• Peer Review: How to Distinguish the Good from the Bad?

    It seems as if we have a stormy autumn this year – not only out there in the countryside, but also in scholarly publishing. There have been a number of interesting publications this month shaking up the community, still, the most agitating paper seems to be the sting from John Bohannon in Science Magazine. Whereas most of the immediate and fury discussions focus on the damage this paper might have done to the idea of Open Access publication, I believe we should not forget what it says about peer review.

    There has always been debate about the concept of peer-review, but with the sting from John Bohannon it becomes apparent, that the application of peer review leaves something to be desired. Bohannon tried to relate the fact of poor or none peer review to the fact that OA publications have a financial incentive to accept – with or without peer review. This approach helped him to make it a big story, which is a good thing since it ignited so much of discussion.

    The question is now how can we distinguish between the good and the bad. It seems that Beall’s list is limited in its judgment: Some ‘predatory’ publishers did peer review, some editors from well-reputed publishers didn’t. The same can be said for journals belonging to certain lists or associations – COPE, DOAJ or OASPA were not free of failure too. So how can a scientist see ‘from the outside’ whether a journal is doing a honorable job when it comes to peer-review?

    The simplest way would be an open peer review process. There, the author can see other reviews and check, who has done reviews and how it has been done. Some open access projects promote this approach (see wikipedia). So far most reviewers are reluctant to show their identity and opinion (nature trial 2006, Sense about science Peer Review survey 2009). Open review shows benefits and probably will grow, but to what extent is unknown. For the time being, I assume that open review will cannot be applied to the majority of journals.

    Let’s do it professionally

    After all, the use or misuse of peer review is a question of quality management and -control. A recent blog post from Peerage of Science makes an interesting suggestion: „Another way is to create an independent certifying entity that only awards certification to journals that regularly demonstrate sufficient peer reviewing standards.“ They have developed a system of ‚reviewing the reviewer‘, which is certainly an important part of what has to be done. Once we adopt the experiences from other fields of industry where certification is a common thing, we are further towards a professional solution open to the whole community.

    Now what means certification? The simplest way would be a first party certification: This is basically like a pledge: the publisher warrants that his journal has a high quality, assured by double peer review. Bohannon showed that this does not work. Second party certification is done by professional organizations that request certain quality standards from publishers and warrant as an independent party that the journals of these publishers actually do what the public believes them to do. There were different results for several associations, but apparently COPE, DOAJ or OASPA measures were all not sufficient and some of their members accepted the sting paper for publication.

    Other industries have long developed and implemented the principle of third party certification. Here, an independent “third” party makes an assessment and issues a certificate to the service provider stating what services have been tested and found compliant with a certain standard. Such a third party is called a “notified body” and accredited by national or international accreditation institutions according to established rules. Such a notified body would assess the use of a clearly defined standard for publishing houses and their workflows. Such a standard could have parts from three different sources: applicable law, technical norms (ISO etc.) and additional rules such as proven ethics documents.

    In case of our industry I strongly advocate to create a stakeholder forum and develop a standard for peer review processes. This will ensure a maximum of acceptance within the community. Such a standard might then be licensed to notified bodies. It should be noted that standard issuer (or holder) and notified body must not be the same organization.

    The certification process usually runs in several steps: It starts with a first assessment and goes on with regular assessments. The result could be a yes or no decision and a respective certificate. If we look at the apparent problems with peer review compliance, another approach might be better: After a first assessment, a process of continuous improvement might be certified by the notified body. This could be shown with a sort of red-yellow-green traffic light system as we know it from the SHERPA/RoMEO OA classification.

    Certainly, it might take a year to develop such a standard. Besides that, certification is an industry itself and established notified bodies could integrate this business into their portfolio. And costs should not eat up too much of the legendary profits from larger publishers. A cost scheme adapted to the publisher’s size should allow smaller publishers to take part on fair conditions as well.

    After all, what would it bring to the community? It would add a new level of transparency in the requirements of scholarly publishing. It could help to harmonize peer review procedures. But, most of all it would give authors and readers a fair chance to distinguish the good from the bad journals, no matter whether they are open access or subscription based. It would be applied equally to small and big publishers. It would not matter whether the publisher is new or established, a learned society or only in it for the money. It would just state that a publisher does what is reasonably expected of him. It would be professional.