Interesting blog post, “The rise and fall of peer review.” Excerpt:
Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?
It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.
I don’t blame the reviewers.. what do you expect when you ask someone to do something for free?
Here are some other thoughts..
Different disciplines look at things differently. Frequently the reviewers are from the same discipline so there’s some degree of unnecessary reinventing the wheel.
Practitioners have no opportunity to review (other than places like TSW)
Papers that make Big Important claims logically would be subjected to a higher degree of scrutiny. There is no mechanism for that.
Finally, the best review I have ever seen was about some FIA work that was being reviewed by a technical committee. The reviewers were all experts and were paid by various interests to review so they knew the technical stuff, they cared, and they were paid. They really dug into the details.
But we can’t afford that degree of scrutiny for your everyday kind of paper.
“Different disciplines look at things differently. Frequently the reviewers are from the same discipline so there’s some degree of unnecessary reinventing the wheel.”
And sometimes, I think, peer reviewers know the author(s) and may or may not be sympathetic to their work and views. Maybe having reviewers with a mix of disciplines would help.