Eight Steps to Vet Scientific Information for Policy Fitness

Peer review- necessary but not sufficient.

For some mysterious reason, my previous post here on the GAO report on the Forest Service has received a great many hits on the blog over the past few weeks. I am interpreting that to mean that at least some readers are interested in the science/policy interface, but who knows? Anyway, this has led to writing a series of posts on my observations on the scientific enterprise. Those who are not interested feel free to flip past.

So here is something I wrote recently, thanks to Bob Zybach and the folks at ESIPRI, when asked what I think a good process would look like for vetting research studies to be used in policy.

Eight Steps to Vet Scientific Information for Policy Fitness

Peer review as currently practiced is probably fine for deciding among proposals or which articles should be published in journals. However, I think that when public funding is at stake for major investments, and people’s lives and property are on the line, we should up our game in terms of review to a more professional level. I was recently asked my thoughts on how to do that, and here they are. It takes all eight steps, outlined below.

1. Is the research structured to answer the policy question? Often the policy question is nuanced.. say, “what should we do to protect homes from wildland fires and protect public and firefighter safety?” This is often where research goes off the rails. Say, historic vegetation ecologists study the past and claim that there was plenty of fire in the past.. but that information is actually not particularly relevant to the policy question. To get funding to do the study (and make the claim that it’s relevant), all they need to do is to pass a panel of scientists who basically use the “it sounds plausible to a person unfamiliar with the policy issues” criterion.

It seems obvious, but for scientific information to be policy relevant, policy folks have to be involved in framing the question. Most, if not all, research production systems that I am aware of do not have this step.

2. Did they choose the right disciplines and/or methods to answer the policy question? Clearly a variety of disciplines could have some useful contribution, as well as an inherent conflict of interest, if you rely on them to tell you if they are relevant or not.

3. Statistical review by a statistician. If you use statistics, this needs to be reviewed, not by a peer, but by a statistician. You can’t depend on journals to do this. The Forest Service used to have station statisticians (and still does?) to review proposals so people worked out their differences in thinking and experimental design before the project was too far down the road.

4. The Quality Assurance /Quality Controls (QA/QC) procedures for the equipment used and data need to be documented (and also reviewed by others). For someone who is unfamiliar with QA/QC applications, you might start with the recent paper attached (lockhart_2009_forest-policy-and-economics), it has a number of citations, and also the implications of the Data Quality Act. What is odd is that the NAPAP program led the way for QA/QC, but it’s not clear how that has been carried forward to today. It might be interesting to take the top-cited papers in forest ecology or management or whatever policy-relevant field you choose and review their QA/QC procedures.

5. Traditional within-discipline peer review.

6. Careful review of the logic path from facts found to conclusions drawn. It is natural for universities or other institutions to hype the importance of research findings. Since people will also use the findings to promote their own policy agenda, and because a paper can be misused even if the scientist is careful (e.g. 4 Mile Fire), it is more important to be specific and careful about your interpretation and conclusions. It is also best that if the findings lead to conclusions that are outside those of the current general consensus, that the authors forthrightly discuss different hypotheses for why their findings are different. Don’t just let the press hype “new findings show” as if the previous studies were irrelevant. The authors know more than anyone else about it, so they should be willing to share what they think about the differences an upfront way. That’s how “science” is supposed to progress, by building on previous work.

7. An important part of professional review for studies involving models should be “what background work did you do?“ Did you use sensitivity analysis for your assumptions? Did you compare model projections to the real world? If not, why not? In fact, the relationship of empirical data to your work should be clearly described, since scientific information derives its legitimacy from its predictive value in the real world, not from being a group hug of scientists within a discipline.

8. Post publication requirements: access by the public to data and open online review. This should be absolutely required for use in important policy discussions.

Do you agree? Do you have additions or deletions or other comments? Why do you think the bar is currently so low in terms of review?”

10 thoughts on “Eight Steps to Vet Scientific Information for Policy Fitness”

  1. Great post, Sharon. Some will never subscribe to all your points, because they prefer to fly under the radar. On the same vein, blind peer review should never be accepted. If you are going to review something, you need to put your name (and reputation) on it! One of my pet peeves is when 80’s style logging is the comparison, instead of modern day surgical-style thinning, with modern equipment and modern tree selection.

  2. “For some mysterious reason, my previous post here on the GAO report on the Forest Service has received a great many hits on the blog over the past few weeks. I am interpreting that to mean that at least some readers are interested in the science/policy interface, but who knows?”

    Hello Sharon, As someone who has access to some of the inner workings of this blog, I too have noticed that the most popular search term, by almost triple, that gets people to this blog is the search term “conveyor belt.”

    I took a screen shot of for the top search terms all time for this blog:


    As you can see some of the terms are very specific to this blog, such as words like: Stream, new century forest planning, spotted owl, biomass. However, other terms are very much just general terms that have nothing really to do with national forest planning such as: conveyor belt, cinderella, cone of silence, superflous, weaving, cinderella background.

    As such, I’m not sure anyone in the world who did a google search for “conveyor belt” or “cinderella” actually intended to visit this blog and talk about national forest planning and management. The fact that your Dec 15, 2010 post about the GAO, in which the term “conveyor belt” is in the title, hasn’t had one comment in nearly two years sort of is an indication that sometimes google searchers come to this site by accident. In other words, I believe the 3,444 people who came to this site via the search term “conveyor belt” were actually looking for “conveyor belts,” not for a national forest planning blog. Maybe this helps solve the mystery. Thanks.

  3. I agree, Matthew, in fact I mentioned both the Cinderella and the Conveyor Belt links in response to questions during my talk at the SAF convention.

    The questions was “how many people read the blog?” I said, I don’t know. I would go with the 135 or so who have subscribed. I said many people go to the blog, but I think many of them are just doing searches on unrelated topics. I said , for example, conveyor belt and Cinderella are very popular.

    Now, conveyor belt had always been popular but it got decidedly more popular since we got back from Convention. Does this recent surge have meaning compared to the background levels? I’ll admit, I don’t know.

    The other thing I said about the numbers was that 130 people or so may not seem like a lot, but when I started this blog my other main volunteer gig was planning music for a church where 10-20 people show up on any given Sunday. So it’s all relative. It’s really not about numbers for me, it’s about giving information to, and learning from others on the blog.

  4. Much of the climate science that is being touted for informing policy would not meet your fitness requirements. I believe climategate lot to do with not making the data availible to the public and doing end runs around FOIA. So I applaud your 8 steps. Sometimes it is hard to separate the advocacy and politics from the science.

  5. “vetting research studies to be used in policy” seems backward. Shouldn’t it read “vetting policy through supported research studies?”

    In other words, policy makers tend to partake in the legal or debate approach of “what science is there to support the policy or argument” while scientists should (ethically) take the approach of “what policy [or argument] does the science say is supported.” Small distinction with huge implications.

    • You may think that, but the idea that “science” should drive policy is not widely accepted. Most important polices have many different ways of achieving them (say, low carbon energy) and there is no one “science” or scientific study that gives a clear path. Take cap’n’trade as a policy for example. How could “science” possibly tell us if it would work or not? And the scientific studies that were done, as I recall, mostly disagreed with each other.

      It seems to me that anything really important has a variety of scientific disciplines that are involved. And the weight to be given to each discipline can’t possible be estimated “scientifically.”

      there are many statements of what scientists should do ethically. My favorite is from the American Fisheries Society, Oregon Chapter here. I don’t think I’ve every seen scientific professionals’ roles in management and policy so clearly articulated.

  6. A key consideration – in the absence of information meeting all these conditions, do you allow “business as usual” or adopt a precautionary principle?

    • Hmm. I don’t know exactly what “precautionary” would mean.
      Are we talking about say, engineered new life forms, or thinning a stand of trees? Being precautionary to me would have a great deal to do with the novelty of the technology.

      The other thing is that we know that coal, wind, solar, natural gas and biomass all have environmental impacts. What would be precautionary dealing with the “science” there? Shutting down the grid?

      I know there is a precautionary “principle,” but I don’t think you can really talk about the idea meaningfully unless you have a specific example.

  7. Hi Sharon: Thanks for posting these! As you know, I share many of the concerns expressed by you, Larry, Matt, and several others regarding the quality of scientific information that goes into formulating public policies — particularly those resulting policies that can cost taxpayers billions of dollars and unnecessary widespread unemployment and poverty, and yet often do not even meet the modest management objectives they are intended to achieve.

    US taxpayers pay for scientific research, equal rights under the law, and provide services for the handicapped. Why can’t we personally consider the scientific information that drives policies that directly affect our lives? Why can’t we see what scientists actually think, instead of being fed such misleading summaries as: “scientific consensus,” “science tells us,” “agency scientists agree,” and other nonsense? If this information was required to be presented in Plain English and to be made publicly available (and debatable) via Internet (Step 8), many of the aforementioned problems would likely go away over time.

    Your steps, if adopted, would cut through a lot of the politics parading as “science” these days. It would also cut through the self-protection afforded scientists via their use of “anonymous” pal reviews and politicized “peer reviewed” scientific information. It is fairly predictable that Democrats will believe in Global Warming with apocalyptic results, while Republicans seem to reject several of the assumptions that go into these predictions of catastrophe. I’m guessing similar political divisions for other “sciences” dealing with evolution, wolf reintroductions, and even species’ taxonomy. This approach should reasonably mitigate those divisions to a large degree by placing actual facts and important background information and methods on the table.

    I think we are doing a great disservice to science, to ourselves, to our rural economies, and to the environment by the politicized distribution of scientific research funds and the subsequent adoption of policies resulting from expected “findings” that has been taking place the past 50 years. Modern technology is just learning to deal with the known weaknesses of this arrangement in a positive way, and this posting shows one way of doing so.

  8. Thanks, Bob, I would only add that I have found many excellent (and other embarrassingly poor) discussions between scientists from different agencies. If the US taxpayer wants to get the advantage of the best information they are buying, some situations could use a public exchange between scientists with differing opinions. (Rather than regulatory agency always trumps land management agency scientists by virtue of which one they work for).


Leave a Comment