Peer Review

I have long had a dislike for peer review, as I believe that it is inherently conservative, hampering the spread of new ideas, and preserving the current mainstream of a discipline. One indication for this is that ground-breaking articles are usually published in obscure minor journals, not in the big ones which are more strongly guarded by peer reviewers.

In theory, peer review should be very useful, picking up on methodological issues, and procedural mistakes. And indeed, in a review of a conference submission a while ago the reviewer wondered why my results all had a certain feature in common, which lead me to have a closer look at my software. And indeed, he (or she!) had spotted a bug which I had missed, thus improving the quality of my paper. Positive result.

But in contrast, I had a project application rejected because the reviewer was obviously working in a different paradigm, and didn't accept that the methods used in the project (not invented by myself, but by eminent people working in my field) "could possibly work". They clearly did, as I had a working prototype at the time, but this was perhaps something the reviewer simply didn't like. Negative result (at least for me and my colleague in the project...)

John Sinclair famously said in his valedictory lecture that he never in his life got a peer reviewed paper published. I can only assume that this is because he often challenged established assumptions, and his ideas usually were a step ahead of most other people.

So the main problem is that peer review does (in my experience) not merely look at whether the work has been executed in the right way, but mainly is about whether the reviewer likes the content. This is not really peer review, but gatekeeping. Anything I don't like I reject, as it could threaten my position in my field.

The reason for this rant is an article I've just read, on alternatives, or peer-to-peer review. The main concern of which peer review is seen as the answer is that everybody can just publish anything, and especially so since the advent of the Web. And people do. There are a lot of wacko websites, where all sorts of rubbish is proclaimed as the truth. But who really cares? You can usually spot them easily, and there are also sites like Bad Science who deal explicitly with this issue. The point is, peer review doesn't stop nonsense being published, for example even the BMJ fell victim to the 'cello scrotum' hoax. The BMJ is peer reviewed, but perhaps they don't review 'brief case report[s]'? Does this not provide a false sense of security, even? It's in the BMJ, so it must be true?

If there is no peer review, you cannot rely so much on this (potentially false) sense of security, but with 'peer-to-peer' review, once a paper would be published, it could be more easily be discussed, and critical comments could point to problematic issues. In effect that would be like making peer reviewers' comments public. And if they are no longer anonymous, then it's a lot harder to be unfairly biased.

Of course this might just open the floodgates, and all sorts of nonsense could be published. But would people not be more careful? Would you still push out papers you didn't think were good enough? You might, because the number of papers you publish is an indicator of research quality. I'll save that for another time, one rant a day is enough...

Why not use Web 2.0 to get out of this dilemma? For example blogs (as mentioned on this blog before) for publications, and comments for the discussion. The whole system of publishing needs revision, IMHO, but I can't see anything happening anytime soon. As it said in the article mentioned above,

[p]eer review thus functions as a self-perpetuating disciplinary system, inculcating the objects of discipline into becoming its subjects. After all, those who manage the current system of peer review are of course those who have successfully negotiated it, granting an enormous inertia to the status quo.