Why Peer Reviewed Science is So Important

This just in:

Academic publishing
Science’s Sokal moment

It seems dangerously easy to get scientific nonsense published
Oct 5th 2013 |From the print edition

IN 1996 Alan Sokal, a physicist at New York University, submitted a paper to Social Text, a leading scholarly journal of postmodernist cultural studies. The journal’s peer reviewers, whose job it is to ensure that published research is up to snuff, gave it a resounding thumbs-up. But when the editors duly published the paper, Dr Sokal revealed that it had been liberally, and deliberately, “salted with nonsense”. The Sokal hoax, as it came to be known, demonstrated how easy it was for any old drivel to pass academic quality control in highbrow humanities journals, so long as it contained lots of fancy words and pandered to referees’ and editors’ ideological preconceptions. Hard scientists gloated. That could never happen in proper science, they sniffed. Or could it?

Alas, as a report in this week’s Science shows, the answer is yes, it could. John Bohannon, a biologist at Harvard with a side gig as a science journalist, wrote his own Sokalesque paper describing how a chemical extracted from lichen apparently slowed the growth of cancer cells. He then submitted the study, under a made-up name from a fictitious academic institution, to 304 peer-reviewed journals around the world.

Despite bursting with clangers in experimental design, analysis and interpretation of results, the study passed muster at 157 of them. Only 98 rejected it. (The remaining 49 had either not responded or had not reviewed the paper by the time Science went to press.) Just 36 came back with comments implying that they had cottoned on to the paper’s sundry deficiencies, though Dr Bohannon says that 16 of those eventually accepted it anyway.

The publications Dr Bohannon selected for his sting operation were all open-access journals. These make papers available free, and cover their costs by charging authors a fee (typically $1,000-2,000). Policymakers have been keen on such periodicals of late. Since taxpayers already sponsor most academic research, the thinking goes, providing free access to its fruits does not seem unreasonable. But critics of the open-access model have long warned that making authors rather than readers their client risks skewing publishers’ incentives towards tolerating shoddy science.

Dr Bohannon has shown that the risk is real. Researchers can take comfort that the most prestigious open-access journals, such as those published by the Public Library of Science, an American outfit, did not fall for the jape. But plenty of periodicals run by other prominent publishers, such as Elsevier, Wolters Kluwer and Sage, did. With the number of open-access papers forecast to grow from 194,000 in 2011 (out of a total of 1.7m publications) to 352,000 in 2015, the Bohannon hoax ought to focus editors’ minds—and policymakers’, too.

From the print edition: Science and technology

28 Comments

  1. Thanks, Emily! This has been a serious problem for quite a while. I’m curious why you think it is “timely” now? (It seems very timely to me, too, but I’m guessing for reasons possibly different than yours).

  2. Yeah, we’ve been there and done that — Jerry Franklin’s 30 minute review of the Donato/JBK paper, perhaps? I’ve still got that email somewhere.
    Never mind that a couple years after Bosworth criticized the lack of peer review on Beschta 1, he gave Kauffman that job at the Pacific Islands research center. Hmmmph.

      • Well, I had to downsize my paperwork, that included s-canning my copy of the Roadless Initiative DEIS and most of my FOIA packet from OSU concerning the Donato hijacking. I got a three inch stack and there it was, buried in like the last ten pages. I cited it in Evergreen and Jerry didn’t sue.
        Maybe I should have framed that one or put it in a safe-deposit box, eh?

          • Sharon, this is one of those hindsight things. When I was thrown into the Biscuit Donato gutwagon, I really had no idea what I was getting into. I would have had NO clue except I was in the same press-mobile as grant administrator Tom Sensenig, sitting next to him as we motored up to look at another fire site where the Congs were going to pontificate.
            He and I were the only ones actually looking out the windows, all the “journalists” were gossiping and checking their Blackberries, so I asked, “Dude, what’s your gig?” Glad I asked, really, and somewhat irritated that not one single flipping “reporter” ever did. Ever.
            So, would I realize that Jerry’s e-mail might be handy eight years later? And really, might it be worth it to dredge it up?

            • Dave: As you know, Sensenig is a good guy. What Franklin and the Donato group pulled on him was highly unethical and should have resulted in a public outcry — except perhaps for Tom’s good manners and mostly low-key personality. Turning that responsibility over to professors at OSU — instead of immediately going public — was a real mistake. Somehow (“politics”) the tables were turned and Donato became a victim. You know this story better than I.

        • At one point I had a few boxes of papers obtained via FOIA (and Oregon state open records) about the OSU/Hal Salwasser/Chris West dust up over the Donato Paper. Suffice to say, some interesting emails there between Salwasser and the timber industry and some of the pro-timber professors at OSU. Yep, Salwasser, who was Dean of OSU Forestry at the time even referred to enviros as “goons.”

          • Matt: Salwasser was one of the founders of Conservation Biology, if I remember right, and a major government spotted owl biologist — not a forester! OSU had a lot of environmental types on the faculty and on campus during those years. I think by “goons” he was referring to the “radical environmentalists” who were disrupting meetings, damaging equipment, setting fires, spiking trees, and so on. There was quite a “family” of them around Eugene at that time, and “goons” was a polite term for what they were doing. I attended Donato’s original presentation of his findings at OSU, and the subsequent hearing at Salem. In my opinion, it was pedestrian work, highly politicized, and borderline ethical. So they gave him a PhD instead of a Masters and sent him to the tropical islands for a government-paid vacation/career. I lost a lot of respect for him, his work, and his team during that time.

    • Gil: Are you talking about Conservation Biology again? This is precisely why I don’t like wading through most lists of references. Cherry picking isn’t even a sport anymore — the branches are so heavy with fruit they bend down to the ground so you can grab them by the handful. And almost all of it paid by taxpayers (“us”).

    • One of my colleagues did that. It was in my field so it aroused my curiosity because I wanted to know how he came up with it. Sure enough, I tracked it down in the middle of a paper as a statement without citation. And from then on it was cited as if there were something behind it. It would be funny if it weren’t really quite appalling.

  3. I hope the paper doesn’t imply that things are hunky-dory outside of open access journals. I was (very briefly) an editor and had a great deal of trouble finding people who weren’t related academically, weren’t enemies academically, and who were willing to review papers (work for no pay) in areas that are fairly specific. If you have to understand the literature and there are only 5-10 people publishing in that specific area, well, you do that math.

    Then if you go outside that micro-field people aren’t familiar enough to catch everything. If there’s anything more thankless than being an unpaid reviewer, it’s being an unpaid editor. They all deserve thank yous from society, as they do an impossible job with no resources. So they don’t do it perfectly or sometimes even respectably.. that’s not their fault necessarily (although some have an ideological filter). The problem is a system of expecting hours of difficult work with only intangible rewards. IMHO.

    Anyway note that in my previous post “Eight Steps to Policy Fitness” traditional peer review is only one step.

    • Sharon, I agree with you 100% on this. I get a lot of manuscripts to review, and I try to do a reasonable proportion of them, turning down the ones I don’t feel technically competent to handle (for example, I get asked to review a lot of pretty hardcore molecular biology papers, because I previously published some with my former grad students who went on to become proficient molecular biologists, but I’m not up to that level and turn those down). Along with being an unpaid task as you mention, reviewing manuscripts gets almost no credit from university tenure/promotion committees, so the younger folks doing a lot of it are literally sacrificing career progress to do so. Being an editor gets a little more credit, but not much. I think part of the reason is, that people who end up as university administrators are often those who were never that good at research and publishing, and they just don’t get why peer review is that important and time-consuming. Thanks for letting me rant! -Guy

      • I don’t call that a rant. I call it a reasonable critique of the Academic Establishment, which we could actually use more of. That’s one of the reasons why I don’t work in academic science. I figured out that I could spend much time writing proposals that were never funded. If they were funded, the university could take 60% or so indirect costs (not USDA grants).

        The business model was not attractive, especially not for women, who faced a great deal of prejudice at the time. Plus by the time you worked with your students and applied for grants, and reviewed documents, it seemed to be difficult or impossible to achieve life balance, especially in the earlier (AKA child-bearing) years.

        In contrast, I could work regular hours and get paid with good benefits and retirement and only had to do what I was told, more or less. I think when I see the current faculty folks, that universities have become more family -friendly and I genuinely hope that that’s the case.

        Also, I organized competitive grant panels when I worked for CSREES, and I am not so sure that they are an optimal way of dispensing research bucks, and I worked with people who believed in replacing formula funds with competitive grants with an almost evangelical fervor. But that would be another “rant.” ;).

        • all good points (although at least we state folks aren’t furloughed right now 🙂 ) For what it’s worth, I always thought that CSREES had the best and fairest competitive grants review process of all the big players. I sat on many many panels for them, and always came away impressed with the even-handedness of the other panel members and program admins. By contrast, NSF is a very clubby, good-old-boy funding machine, and EPA is just plain dysfunctional. Unfortunately, CSREES became NIFA, and in the Roger Beachy (Mr. Biotech) years the emphasis shifted to funding much fewer grants for much larger amounts, so the rich got richer and the poor got poorer, to the detriment of U.S. ag reseach (my opinion of course, though widely shared).

          • Well, that is really great to hear.. I remember working very hard on the Fund for Rural America with a variety of good folks, and it seemed to get caught in internal USDA power games. So I had to call a bunch of people we had given grants to (1mill per as I recall) and tell them we were not sending the money after all. In the interests of my long-term professional reputation, I called the FS then and asked to come back.

    • Sharon: We need to stop the academic myth of “work with no pay.” Mike Newton can pull that off now that he is “retired,” but the vast amount of college professors and agency scientists who do peer reviews do it on company time. WE, the people, pay them to do it, and pay them very well. It is part of their job as scientists. What you — and they — mean is they are not paid extra, or paid by a publisher. Only the retired professionals — who are usually “getting by” on excellent retirement plans — can legitimately make that claim most of the time: others it’s just self-pity nonsense. Fortunately, retired people have the most experience and insight and typically make the best reviewers. I don’t know why this myth got started or why it keeps going (I’m guessing laziness and not wanting to work factor in), but I do know that I lot of people who keep spreading it aren’t doing their job of peer review.

      I’m in full agreement with everything else you’ve said, though. Got any thoughts on my one gripe?

      • Bob, I’m not Sharon, but I’ll half agree with you. Professors do review papers and grants on “company time” (although as we tell all new assistant professors, if you plan to work only 40 hrs a week here, you ain’t gonna make it). I guess I get paid “very well” by some standards ($72k/yr after 26 years, at the highest professorial rank). My fed colleagues do a lot better of course, and Idaho is pretty near the bottom for faculty salaries. But I think if faculty received some recognition from university administration for effort spent reviewing manuscripts/grants, then there wouldn’t be many complaints. As it is, you might as well take a nap for a couple hours a day, for all the recognition it gets. But in the big scheme of things, you’re right it’s a fairly minor gripe. In a number of fast-moving fields (molecular biology, bioinformatics, others) it’s typical that retired folks (and some later-career profs) aren’t the best reviewers, because they’ve already fallen behind the curve of current knowledge, so a disproportionate amount of the reviewing work falls on the younger faculty who have less time available to do it. No self pity here, though, life is good 🙂

        • I ran the notion that “in a number of fast-moving fields (molecular biology, bioinformatics, others) it’s typical that retired folks (and some later-career profs) aren’t the best reviewers because they’ve already fallen behind the curve of current knowledge,” by my 84-year-old molecular geneticist dad. He demurs with the observation that a good review panel should include both young bucks and wise heads. The young bucks may know the latest tricks, but often lack the wisdom gained through experience and can be ignorant of first principles. Dad enjoys his retirement from teaching and active lab work by, among other things, reviewing papers for publication.

          • Wow, Andy, what a pioneer he is! I am about 25 years or so younger and the field was still fairly primitive when I got to it. For example, there wasn’t “molecular” yet; I learned the stuff that was around at thein a course called “biochemical genetics.” I resented having to memorize about “operons” in gene regulation as I thought it was too simplistic to be true. Just an idea so folks could write later ” bacterial gene regulation is more complex than previously thought”. Like thinking non-coding regions are Junk-DNA so it can be discovered later that surprisingly they are doing something.

            And like others on this blog, I always hated memorizing. Ironically, I forget many things frequently but can remember Jacob and Monod. Well all that is way off topic. Your Dad was in on the ground floor of work that continues to amaze us at the wonder of it and bless us with useful things, and will be doing so for a long time to come.

      • Guy: Just read your 100% response and see your point, too. It is the administrators that seem to be at the root of the problem, not specifically demanding that their employees do peer reviews, and not requiring students to learn how to do them. That is exactly why I have been working as a Board member for ESIPRI the past three years. On the other hand, those same committee-goers that seem to be in charge of everything see ESIPRI as just another 501 c(3) looking for “donations” and needing to get in line if they’re going to get any. Very frustrating and bordering on idiocy and incompetency, in the dictionary sense of those words.

        If you have the time at some point, I would be very interested in your thoughts of the ESIPRI Guidelines, which I coauthored: http://www.esipri.org/Guidelines/

        Background is on page 1 and pages C-2 to C-4 in Appendix C. The heart of the document is in pages 8-12. Those 9 or 10 pages pretty much tell the story. Need and process are covered in more details in the other pages. You’ll recognize some of the names involved.

  4. One final point that was brought up to me by real alphabet-soup people in terms of peer review.
    Some sciences are hard, with clear cause and effect. Molecular biology is probably pretty hard. Stuff like ecology is much, much softer, more observational and less receptive to “control” or direct causal linkages. The softer stuff is WAY more vulnerable to abuse. Unfortunately, there are practitioners who will, and do, take advantage.

    • Dave: You need to get a copy of Botkin’s book — maybe even join Sharon’s online book club with your thoughts. Hard science arrogance has a strong basis in fact, as you state, but the skill sets are not entirely transferable. Mike Newton has demonstrated that there is plenty of room for academic rigor in forestry as well. Botkin will give you some basic, and valuable, insights as to why there is so much crap in the ecological and environmental sciences these days. It’s not because they are “soft” sciences, necessarily, but because there are some fundamental problems that have caused these disciplines to become “retarded,” as he says.

Leave a Reply

Your email address will not be published. Required fields are marked *