Peer review

Sometimes, doing science is not too dissimilar to being in a courtroom drama. There will be lawyers on the other side trying to undermine your arguments, while you are trying to defend your case, hoping that the judge will be reasonable. Sometimes, the trial is easily won. Sometimes, the trial will be endless, filled with drama and intrigue. This is the process called peer review.

One of the main goals of scientists in many fields is to publish their findings in journals. Before some science results can become published, it first needs to be written into a relatively short paper and sent to a journal. But before it will get published, the paper first needs to undergo a process called peer review. This means that the paper gets sent to fellow scientists, who will check the paper to see if the paper is scientifically sound and worth publishing. Such a reviewer has to be knowledgeable about the paper you’ve just written, but also distant enough to be able to judge the paper with some objectivity. The reviewers generally stay anonymous, so they are relatively safe to give criticism, without having to fear retaliation from the writers of the paper. Once the reviewers and the journal editor think the paper is acceptable, normally after some changes have been made, the paper gets to be published.

I think peer review is generally a good thing, as it filters out some of the papers whose conclusions are not supported by a fair and scientific way of doing things. I’ve also had many great, considerate reviewers that have helped me in making my papers better. But there are some downsides to peer review. One obvious point is that the process is rather time-consuming, but this might be the price we have to pay for improving the quality of the papers out there. Instead, one of my main reservations against peer review is that it is not as objective as it is made out to be, leading to a large degree of arbitrariness, and sometimes to very unpleasant behaviour.

In principle, peer review is meant to guarantee scientific accuracy, but in practice the reviewer is just looking for anything ‘wrong’ with the paper. The problem is, there is always something wrong with a paper. If you want, you can always think of something else to change. You can also always think of more work that can be done to support the paper’s conclusions. As a reviewer, you can basically demand books worth of analysis, and keep the writers of the paper trapped in an endless loop of peer review. In principle, the editor of the journal is the fair and benevolent judge that stops this from happening, but in practice the editor can also be part of the problem.

For some editors and reviewers the question of whether a paper is scientifically sound seems to translate into the question of whether the writers do things in a way that they would do things themselves, or in a way that things are commonly done. This can leave a very narrow definition of ‘scientifically sound’, with very little space for new ideas. It also leads to a sort of gatekeeping that can exclude certain groups from participating in the scientific debate, for instance if one doesn’t have the resources to do certain work, or if one is simply discriminated against. Gatekeeping is especially high in ‘high-impact’ journals, which want to publish only the most exciting research, but this is another story for later.

Even when a paper is published and given the peer review seal of approval, it is also hardly a guarantee that the paper isn’t flawed. A reviewer has only limited time to look at a paper and often won’t have the opportunity to try and reproduce the work that is reviewed. Nevertheless, a peer-reviewed paper has a lot more stature than a paper without peer review, both in the scientific community and outside, for instance by journalists. It is often treated as a binary thing: either the paper is peer-reviewed, and hence reliable, or it is not. In reality, how well a published paper is checked varies enormously, depending on the thoroughness of the writers, the reviewers, and the editor. And even when all the scientists in the world think a paper is acceptable for publication, it might still not prove to be 100% correct, as new insights become available over the years.

Some of the bad experiences with peer review have made we think about what I think the job of a reviewer should be. I do think the reviewer should check the scientific soundness of the paper, and give suggestions for improvements, but I think there should be space for imperfect papers. Papers that do not solve all the problems in one go. Papers that do not present something new and exciting, but might provide essential data ten years from now. Papers that provide an unlikely-sounding hypothesis, which cannot be discounted straight away but can be tested in the future. As long as it helps in furthering our understanding of the universe, and adds to the debate, I’m happy for a paper to be published somewhere. It is for us as scientists to stay critical about published papers and to try and improve what is there. 

Published papers will walk their evolution-like path into the future: they will eventually be proven correct, and be a little building block for our scientific understanding of the universe, or they might be discounted and forgotten. They might also be forgotten and re-discovered at a later time, but what I would hate to see is perfectly fine papers not even getting a chance to enter the meme-pool because of the peer review process, and hence present a missed opportunity for science, with a lot of misery at the side of the writers included.