Peer Review – Where You Thought it Ended? That’s Just the Beginning!

“This is a peer-reviewed study!”

In the increasingly heated battles waged lately by crusaders against innovation in agriculture, such assertions are increasingly thrown down like a gauntlet. The intent is to negate findings by regulators and scientists around the world that crops and foods improved through biotechnology are safe. These advocates argue passionately that “paper X”, published in a scientific journal after being reviewed by anonymous scientists and an editor, is sufficient to overturn the findings of hundreds of previously published reports (see http://www.biofortified.org/genera/guide/) to say nothing of the vast experience accumulated through the consumption of trillions of meals derived from biotech improved crops since they first entered the marketplace in the mid 1990s. When these papers are criticized by scientists post-publication, cries of censorship and persecution inevitably arise, and are routinely coupled with claims that the critics are bought and paid for by vested corporate interests (see  http://www.infiniteunknown.net/2012/12/17/smelling-a-corporate-rat-%E2%80%93-the-move-to-suppress-seralini-gmo-study/ and http://www.globalresearch.ca/gmo-researchers-attacked-evidence-denied-and-a-population-at-risk/5305324). But the noisemakers overlook something fundamental about the culture of science:  where they thought peer review ended -is really where it gets going.

Peer Review – What is it?

Like so much of Western thought, the idea of peer review has its roots in Plato’s dialogues. Following Socrates, it’s all about asking questions, challenging and re-examining presuppositions, and following the data. To that end, credible scientific journals follow a general pattern: when an author submits a manuscript for possible publication, editors solicit several of the author’s peers (the canonical number is three, but it can vary) to review the paper. These “peer reviewers” examine the methodology, data, and reasoning of the authors. Their charge is to make sure the experimental design is sound; the data produced actually address the hypothesis being tested; that previous related work is acknowledged, considered, and cited; and that the conclusions the author presents in the paper follow from the data.  Deficiencies against any of these criteria weigh against the proposal to publish, though if deficiencies identified are minor, or can be addressed fairly easily, authors are often asked to revise as a pre-condition for publication.

Papers that make it through this gauntlet and are then published are said to be “peer reviewed.”  But this does not mean they are now graven in stone, immune to challenge. What it means is that they have passed the first hurdle, and are now fit to be shared with the global scientific community, which gets to check the work of the “peer reviewers.” And sometimes they make mistakes. Let’s consider a recent example.

 

A paper by French scientist Gilles-Eric Séralini[i] that appeared in a journal published by Elsevier, a prominent scientific publisher, claimed that the long-term consumption by rats of biotech improved maize and (separately) glyphosate (the popular herbicide Roundup) produced an excess of tumors in rats. The paper is claimed by many biotech opponents to be the first long-term study of the safety of “gmos” (genetically modified organisms).  It was not. The authors claimed it showed long-term consumption by rats of biotech improved maize and (separately) glyphosate (the popular herbicide Roundup) produced an excess of tumors in rats.  They buttressed the claim with lurid photos of rats with gigantic tumors and their findings were quickly picked up by activists as a smoking gun against genetically modified organisms (GMOs). But there were more than a few problems with the paper.  I won’t re-hash them all here, but they’ve been pointed out in an unprecedented avalanche of letters to the editor of the journal,  critical reviews by official regulatory bodies, critical media coverage  and savaged in the blogosphere.  Of course, from the beginning, it didn’t take a sensitive nose to smell a rat, as the embargoed press release   announcing the paper only to journalists who had to promise to abide by unprecedented constraints designed to minimize the potential for negative media coverage.  The authors refused to release the full description of materials and methods supposedly used in the study and falsely declared they had no conflicts of interest (despite the fact that funding was obtained, at least in part, by commercial entities that benefit from anti-GMO fears). Second, the authors violated basic principles of animal care, which call for euthanizing animals to avoid unnecessary suffering once a clinical endpoint is reached, i.e., visible tumors as opposed to the gross deformities featured in the pictures accompanying the paper. Third, the strain of rats used in the study was one created by researchers to use in testing of chemicals for carcinogenicity, and about 80% of these rats would be expected to develop tumors  over their lifespan no matter what they are fed... which happens to be precisely what the authors report in this paper for both control and experimental animals. The authors further failed to apply any appropriate statistical analysis of their results, and somehow missed noticing that slightly more of their control animals developed tumors than did those fed glyphosate or biotech maize.  In other words, based on their own data, it would seem the biotech maize actually reduced the incidence of tumors compared to controls.[ii]

In summary, the global scientific community of peers reviewed the published paper and reached a clear consensus position, as captured in this statement from the European Food Safety Authority[iii]:

The… authors’ conclusions cannot be regarded as scientifically sound because of inadequacies in the design, reporting and analysis of the study as outlined in the paper. Consequently, it is not possible to draw valid conclusions about the occurrence of tumours in the rats tested. Based on the information published by Séralini et al., EFSA finds there is no need to re-examine its previous safety evaluations of NK603 or to consider these findings in the ongoing assessment of glyphosate.

The failings of the paper are, indeed, so numerous and flagrant as to lead some to charge the authors with “gross scientific misconduct and attempted fraud .”

That’s how peer review works.

There are many ways peer review can go awry, and let an unworthy paper slip into print. Peers can fail to understand or appreciate intricacies of the experimental design, or miss that the wrong statistical tools were used to analyze the results, or the right tools used in the wrong way. This is why three peers are most often consulted – it is hoped, and usually the case, that substantial mistakes will be caught by at least one of the three pairs of eyes. But if the peers are not well chosen, the potential for errors increases.

How are the peers selected?  This is perhaps the Achilles heel of peer review. Reviewers must be individuals versed in the subject matter that is the focus of the paper. If they are not familiar with the field, the value of their opinions and their ability to offer critical insights is reduced; the odds are increased that something will escape notice. And they must be willing to work for free – nobody gets paid to peer-review a paper; and they must be willing to toil in anonymity – they get no credit, except perhaps informal good will with the editor. It is the journal editor’s job to find the right peers to review a paper. So we can see there are a number of perverse incentives in play:  if a peer does a poor job, he/she is at little risk of exposure. To decline a request to review carries some stigma, but the anonymity of peer review limits this, and there is little clear penalty for peers who do not give the task their full attention. But this is why three peers are usually tasked with reviewing a paper. Anonymity also provides a measure of reassurance – it allows a scientist to offer pointed comments without fear of incurring the wrath of a powerful author, or embarrassing a colleague if pointing out a basic mistake – all to the good. But despite all these safeguards, sometimes peers just miss the boat; a paper with significant flaws in methodology or reasoning makes it into print even though it is so defective it should never have seen the light of day.

For this reason scientists (as opposed to propagandists) view publication as the beginning of peer review, rather than the end. The success and the value of the process depends on the participation of the global scientific community. Papers that survive this scrutiny help to map new approaches to problem solving in particular fields in the endlessly iterative efforts of scientists to more clearly understand the world. As Winston Churchill said of democracy, so it is with peer review:  “It’s absolutely the worst form of government [or, in this case, quality assurance], except for all the others.”