Preprints are great, but post publication peer review isn't ready for prime time

26 Feb 2016 By Jeff Leek

The current publication system works something like this:

Coupled review and publication

1. You write a paper
2. You submit it to a journal
3. It is peer reviewed privately
4. The paper is accepted or rejected a. If rejected go back to step 2 and start over b. If accepted it will be published
5. If published then people can read it

This system has several major disadvantages that bother scientists. It means all research appears on a lag (whatever the time in peer review is). It can be a major lag time if the paper is sent to “top tier journals” and rejected then filters down to “lower tier” journals before ultimate publication. Another disadvantage is that there are two options for most people to publish their papers: (a) in closed access journals where it doesn’t cost anything to publish but then the articles are beyind paywalls and (b) in open access journals where anyone can read them but it costs money to publish. Especially for junior scientists or folks without resources, this creates a difficult choice because they might not be able to afford open access fees.

For a number of years some fields like physics (with the arxiv) and economics (with NBER) have solved this problem by decoupling peer review and publication. In these fields the system works like this:

Decoupled review and publication

1. You write a paper
2. You post a preprint a. Everyone can read and comment
3. You submit it to a journal
4. It is peer reviewed privately
5. The paper is accepted or rejected a. If rejected go back to step 2 and start over b. If accepted it will be published

Lately there has been a growing interest in this same system in molecular and computational biology. I think this is a really good thing, because it makes it easier to publish papers more quickly and doesn’t cost researchers to publish. That is why the papers my group publishes all show up on biorxiv or arxiv first.

While I think this decoupling is great, there seems to be a push for this decoupling and at the same time a move to post publication peer review. I used to argue pretty strongly for post-publication peer review but Rafa set me straight and pointed out that at least with peer review every paper that gets submitted gets evaluated by someone even if the paper is ultimately rejected.

One of the risks of post publication peer review is that there is no incentive to peer review in the current system. In a paper a few years ago I actually showed that under an economic model for closed peer review the Nash equilibrium is for no one to peer review at all. We showed in that same paper that under open peer review there is an increase in the amount of time spent reviewing, but the effect was relatively small. Moreover the dangers of open peer review are clear (junior people reviewing senior people and being punished for it) while the benefits (potentially being recognized for insightful reviews) are much hazier. Even the most vocal proponents of post publication peer review don’t do it that often when given the chance.

The reason is that everyone in academics already have a lot of things they are asked to do. Many review papers either out of a sense of obligation or because they want to be in the good graces of a particular journal. Without this system in place there is a strong chance that peer review rates will drop and only a few papers will get reviewed. This will ultimately decrease the accuracy of science. In our (admittedly contrived/simplified) experiment on peer review accuracy went from 39% to 78% after solutions were reviewed. You might argue that only “important” papers should be peer reviewed but then you are back in the camp of glamour. Say waht you want about glamour journals. They are for sure biased by the names of the people submitting the papers there. But it is possible for someone to get a paper in no matter who they are. If we go to a system where there is no curation through a journal-like mechanism then popularity/twitter followers/etc. will drive readers. I’m not sure that is better than where we are now.

So while I think pre-prints are a great idea I’m still waiting to see a system that beats pre-publication review for maintaining scientific quality (even though it may just be an impossible problem)