Open Access: death knell for peer review?

A frequent criticism of Open Access (OA) is that it will lead to the traditional peer review process being abandoned, with scientific papers simply thrown on to the Web without being subjected to any quality control or independent assessment. Is this likely? If so, would it matter?

The argument that OA threatens peer review is most often made by scientific publishers. They do so, argue OA advocates, not out of any genuine concern, but in the hope that by alarming people they can ward off the growing calls for research funders to introduce mandates requiring that all the research they fund is made freely available on the Internet.

Their real motive, critics add, is simply to protect the substantial profits that they make from scientific publishing.

Whatever the truth, there is no doubt that STM publishers are currently very keen to derail initiatives like the US Federal Research Public Access Act (FRPAA) — legislation that, if introduced, would require all US Government agencies with annual extramural research expenditures of over $100 million to make manuscripts of journal articles stemming from research they have funded publicly available on the Internet.

And a primary reason publishers give for their opposition to such initiatives is that it would "adversely impact the existing peer review system."

Factually inaccurate

What do publishers mean by peer review? Wikipedia describes it as the process of "subjecting an author's scholarly work or ideas to the scrutiny of others who are experts in the field. It is used primarily by publishers, to select and to screen submitted manuscripts, and by funding agencies, to decide the awarding of monies for research. The peer review process is aimed at getting authors to meet the standards of their discipline and of science generally."

In other words, when a researcher submits a paper for publication in a scholarly journal the editor will ask a number of other researchers in the field whether the submitted paper warrants publication.

And Open Access is the process of making scholarly papers freely available on the Internet.

There are two ways of achieving OA: Researchers can publish their paper in an OA journal like PLoS Biology which, rather than charging readers (or their institutions) a subscription to read the contents of the journal, charges authors (or invariably authors' funders) to publish the paper.

Alternatively, researchers can publish in a traditional subscription-based journal like Nature or Science and then make their paper freely available on the Web by self-archiving it in their institutional repository (IR).

Since both methods still require that papers are peer reviewed, OA advocates point out, publisher claims that making research OA necessitates foregoing the peer review process is factually inaccurate.

And while it is true that some researchers also post their preprints in IRs prior to having them peer reviewed, they add, this is done solely in order to make their research available more quickly, not to avoid peer review. As OA advocate Stevan Harnad frequently points out, OA means "free online access to peer-reviewed research (after — and sometimes before — peer review), not to research free of peer review."

There is, however, a second strand to publishers' claims that OA threatens peer review. If OA is forced on them, they say, they will not be able to survive financially, either because they will discover that there is no stable long-term business model for OA publishing, or because the increasing number of papers researchers post in institutional repositories will cause academic institutions to cancel their journal subscriptions. This poses a threat to peer review, they add, since if publishers exit the market there will be no one left to manage the process.

However, these claims are also rejected by OA advocates, who argue that most publishers have already accommodated themselves to self-archiving. Indeed, they add, there is no indication at all that self-archiving negatively impacts journal subscriptions. Nor is there any reason, they say, to believe that a sustainable OA business model cannot be found.

Far from perfect

But supposing publishers are right, and OA does eventually cause peer review to be abandoned? Would it matter?

After all, while researchers and publishers say they place great store on it, peer review is far from perfect. In September 2000, for instance, the UK Parliamentary Office of Science and Technology (OST) pointed out that many view peer review as "an inherently conservative process … [that] … encourages the emergence of self-serving cliques of reviewers, who are more likely to review each others' grant proposals and publications favourably than those submitted by researchers from outside the group."

Publishers have also been known to concede that there are problems with peer review. Writing in 1997, for instance, the then editor of the British Medical Journal (BMJ), Richard Smith, described peer review as "expensive, slow, prone to bias, open to abuse, possibly anti-innovatory, and unable to detect fraud." He added: "We also know that the published papers that emerge from the process are often grossly deficient."

In fact, it seems that the most that can be said of peer review is that we have failed to come up with anything better. Following the decision by Science to retract papers it had published by Dr Hwang Woo-suk — after it was discovered that he had faked claims that he had obtained stem cells from cloned human embryos — for instance, publications consultant Liz Wager said of peer review "it's a lousy system but it's the best one we have."

True, there have been some attempts to improve things. One frequent criticism of peer review, for instance, is that since the scientists who review submitted papers do so anonymously there is little accountability — no doubt assisting "self-serving cliques of reviewers" to arise.

For this reason a few more enlightened publishers have tried to make the process more transparent. In 1999, for instance, the BMJ announced that it would in future "identify to authors the names of those who have reviewed their papers, including the names of our in house editorial and statistical advisers."

And in June 2006, Nature also appeared to have accepted that greater openness would help, announcing an experiment in which some of the papers submitted to the journal would be subjected both to the traditional confidential peer review process, while also being placed on the Web for open, identifiable public comment.

The aim, Nature said, is to "test the quality of unsolicited comments made during an open peer review process, against the traditional process where no unsolicited comments [are made]".

Too little too late?

Nature's experiment is not a sign that the journal is contemplating abandoning peer review — it is simply exploring what is generally called "open review." We should add that open review is not the same things as Open Access, since Nature remains a traditional journal based on a subscription-based publishing model (although it does permit authors to self-archive papers that they have published in the journal six-month's after publication).

But are traditional journals like Nature in danger  of doing too little too late, and being left behind? For in recent months OA journals have also been exploring new ways of evaluating scientific papers. And they are taking a far more radical approach.

In August, for instance, OA publisher Public Library of Science launched a new journal called PLoS ONE. Papers submitted to PLoS ONE still undergo peer review, but a less rigorous form of peer review. Once published, however, the papers are also subjected to post-publication review on the Web.

As the managing editor of PLoS ONE Chris Surridge explained to me when I interviewed him earlier this year, PLoS ONE referees are asked to answer a simpler question than that asked by traditional peer review. That question is: "Has the science in this paper been done well enough to warrant it being entered into the scientific literature as a whole?"

The key point, added Surridge, is that PLoS ONE does not believe that peer review should end on publication. "Every paper will have a discussion thread attached to it. We are also developing ways to allow people to directly annotate the papers themselves."

What PLoS ONE and Nature have in common, of course, is that they are both experimenting with new types of peer review. However, the PLoS ONE model differs from Nature's in a number of significant ways. First, it will utilise a less rigorous form of peer review prior to publication. Second, the open review part of the process will take place not in parallel with the traditional review stage, but after publication. Third, once published the papers will be OA. As such, they will not be locked behind a financial firewall, available only to subscribers.

It is worth making that last point because Nature's ability to experiment with peer review is surely constrained by the limited accessibility of its papers.

For this reason alone, Nature's approach will inevitably have to be more cautious than an OA journal like PLoS ONE can adopt, even though it clearly acknowledges that profound questions need to be asked about traditional peer review — as evidenced by the online debate Nature has been holding on the topic.

Root and branch

As a consequence, while traditional publishers are having to continue playing up the importance of the current peer review system while tinkering at the edges, OA journals are are able to take a root and branch approach to reform.

On the other hand, however, this leaves the OA Movement vulnerable to allegations from those who oppose initiatives like the FRPAA that it is bent on destroying the scientific process.

Yet the more the failings of traditional peer review are examined, the more radical seem to be the solutions that people feel are necessary. In September, for instance, a group of UK academics keen to improve the way in which scientific research is evaluated launched a new OA journal called Philica.

Unlike both Nature and PLoS ONE, Philica has no editors, and papers are published immediately on submission — without even a cursory review process. Instead, the entire evaluation process takes place after publication, with reviews displayed at the end of each paper.

As such, the aim of the review process is not to decide whether or not to publish a paper, but to provide potential readers with guidance on its importance and quality, and so enabling particularly popular or unpopular works to be easily identified.

Importantly, argues Philica, its approach means that reviewers cannot suppress ideas if they disagree with them.

Specifically, the evaluation process consists of anonymous, recursively weighted reviews. As the journal's FAQ explains, "When people review Philica entries, they rate them on three criteria: originality, importance and overall quality. These ratings are on a scale of 1-7, where 1 is very poor, 7 is very good and 4, the mid-point, represents a rating of middling quality."

Moreover, unlike PLoS ONE, Philica does not charge author-side fees. This is possible, the FAQ explains, because the overheads are minimal. "Philica can be made free to everybody, whilst retaining the benefits of peer-review, because of the open, online submission and refereeing processes."

Philica is not the only new initiative to push the envelope that bit further. Another approach similar in spirit is that adopted by Naboj, which utilises what it calls a dynamical peer review system.

Modelled on the review system of Amazon, Naboj allows users to evaluate both the articles themselves, and the reviews of those articles. The theory is that with a sufficient number of users and reviewers, a convergence process will occur in which a better quality review system emerges.

Currently Naboj users can only review preprints that have been posted in the popular physics preprint server arXiv.org, but there are plans to extend this to allow reviews of papers in other open archives, including presumably the burgeoning number of institutional repositories.

As with Philica, the aim is not to assess whether papers should be published, but to guide people on the quality of the growing number of scholarly papers being made available online, regardless of whether they have been peer reviewed in the traditional sense.

Do these new services herald the end of peer review? Not necessarily. Philica, for instance, makes a point of stressing that it only accepts reviews submitted by bona fide academics, excluding even graduate students from the process — on the grounds that it "is not normal practice for students to do this to the work of fully-qualified academics, and we do not consider it desirable to change that here."

So while the review process may have migrated from the pre-publication phase to the post-publication phase, it is still peer review.

We should perhaps also stress that Naboj does not claim to be an OA journal, but a web site to help people find useful research. As such it is more of an adjunct to the growing tide of OA literature than an alternative scholarly journal.

Moreover, the vast majority of OA journals — most of those for instance published by the two main OA publishers BioMed Central and Public Library of Science — still practice traditional peer review before publishing papers.

Nevertheless the implications of new initiatives like PLoS ONE and Philica are surely clear.

Complicating an already confused picture

The problem for OA advocates, of course, is that such developments are complicating an already confused picture, and leading to a great deal of misunderstanding about the relationship between OA and peer review.

The consequences of this were amply demonstrated at the beginning of October, when an Associated Press news story about PLoS ONE and Philica was published. As is usual with AP stories, the article was syndicated to multiple newspapers — and with every republication the headlines became increasingly alarmist.

The Desert Sun, for instance, reprinted the article with the headline and subtitle: "Online journals threaten scientific review system: Internet sites publishing studies with little or no scrutiny by peers"; The Gainesville Sun, published it with the headline, "Online publishing a threat to peer review"; and The Monterey Herald went with: "Academic journals bypass peers, go to Web."

"Is this routine editorial carelessness or spreading paranoia?" asked OA advocate Peter Suber exasperatedly on his blog.

The answer, perhaps, is a bit of both. Certainly, the spreading confusion is a boon to publishers bent on killing the various proposals intended to make OA mandatory — since it is causing many to conclude that OA represents a serious threat to the scientific process.

The paranoia reached a peak when the AP article attracted the attention of Harvard's college newspaper, The Harvard Crimson, which responded by publishing a muddle-headed editorial called "Keep Science in Print".

The launch of journals like PLoS ONE, it warned, threatens to "create a morass from which science might not emerge. Results will be duplicated, communication retarded, and progress slowed to a standstill … [as] … scientists will have no way of knowing which discoveries and experiments merit their time and interest. Instead they will spend inordinate amounts of time wading through the quicksand of junk science to get to truly interesting work."

Unfortunately, pointed out Suber, The Harvard Crimson editorial was seriously flawed, failing on at least two important counts. "First, it confuses open review with non-review. Second, it assumes that all online-only journals (open access and subscription-based) use open review — i.e. that traditional peer review requires print."

Inevitable

For those impatient to see OA prevail, the spreading confusion is very frustrating. What OA advocates therefore need to do, suggests Harnad, is insist on keeping discussions about reforming peer review separate from the debate about OA.

So while agreeing that peer review "can be made more effective and efficient", Harnad insists that any discussion about reforming it "should not be mixed up with — or allowed to get in the way of — OA, which is free access to the peer-reviewed literature we have, such as it is."

Conflation, however, seems inevitable — not just because the public is confused, but because, as Suber recently pointed out, "there are a lot of exciting synergies to explore between OA and different models of peer review." A good example of the way in which these are being explored at the Los Alamos National Laboratory, he added, was mentioned by Herbert Van de Sompel during the Nature debate on peer review.

These synergies flow from the fact that when papers are made freely available on the Web so much more is possible. And for this reason, some believe that peer review reform and OA are joined at the hip.

This was a point made by Andrew Odlyzko, a mathematician who heads the University of Minnesota's Digital Technology Centre, in a recent response to Harnad on the American Scientist Open Access Forum. "I think you go too far by denying that Open Access has anything to do with peer review," he said. "For many (including myself), Open Access is (among other things) a facilitator of the evolution towards an improved peer review system."

In this light it is not accidental that OA publishers are beginning to lead the way in peer review reform.

In any case, as the editor-in-chief of Wired magazine Chris Anderson has pointed out, it seems inevitable that the Internet will change the way in which peer review is conducted, not least because where in a print world scholarly papers have to jostle for limited space (pages), in an online environment any such constraints go away.

So where the decision in classical peer review is whether a paper is worthy enough to earn a share of a limited resource, in the online world no such decision is necessary, and the admission gates can be thrown open.

Filter and rank

In an online world, by contrast, the issue is not whether a piece of research gets to go through the gates, but how you filter and rank the expanding tide of research passing through. (But note that this implies that there are no access restrictions imposed on that research).

After all, the Internet will inevitable fill up with junk science regardless of the peer review system. The priority, therefore, will increasingly be to highlight research worthy of attention, and to flag the junk, not whether a paper is published or not.

The important point to bear in mind, says Odlyzko, is that peer review has never been the final edict on a work. "It guarantees neither correctness, nor importance, nor originality. All it does is provide some partially quantifiable assurance of those. The final verdict comes decades (and sometimes centuries) later, when scholars go back and reassess individual contributions."

But perhaps the most interesting question raised by current developments is not when and how science is evaluated, but who does it. As Anderson reminds us, on the Internet a new kind of peer review is emerging — one in which "peer is coming to mean everyman more than professional of equal rank." When people talk about peer-to-peer services, for instance, they are not usually referring to services designed to enable scientists to talk to one another.

Can we therefore assume that the work of a scientist will always be evaluated exclusively by fellow scientists? And would it matter if it were not?

Anderson points out, for instance, that Wikipedia contributors "don't have to have PhDs or any sort of professional affiliation: their contributions are considered on their merit, regardless of who they are or how they have become knowledgeable." Likewise they can delete or edit the contributions made by experts

Tellingly, a controversial study conducted by Nature in 2005 concluded that the accuracy level of science entries in Wikipedia was only slight lower than that fount of all expert wisdom, the Encyclopaedia Britannica.

And those who believe that the intellectual elite should only ever be evaluated by their coevals will surely have been depressed by the implications of the ill-informed and self-congratulatory editorial penned by The Harvard Crimson. Even the intellectual elite, it seems, can talk nonsense at times, and it doesn't take a scientist to spot it.

Given the obvious inaccuracies parroted by the Harvard scribblers, peer review traditionalists will also doubtless feel uncomfortable about the parallel the editorial drew with the evaluation of scholarly literature. "Getting into Harvard is hard, very hard," it boasted. "Yearly the gatekeepers in Byerly Hall vet thousands of applicants on their merits, rejecting many times the number of students that they accept. But getting a scientific paper published in Science or Nature, today’s pre-eminent scientific journals, is oftentimes harder."

As one of those who left a comment on the web site of The Harvard Crimson put it (evidently sceptical about the merit systems at play here), "Since most at Harvard are well-connected and can get their papers published no matter what their merit, perhaps there is anxiety about a purely merit-based system?"

Goodbye gatekeeper?

So has OA sounded a death knell for traditional peer review? Perhaps. Would it matter if it has? Probably not. In fact, OA journals seem to be in the process of developing superior ways of evaluating papers, not doing away with evaluation.

A more accurate way of describing developments, perhaps, is that peer review as understood and practised by traditional journals is giving way to a number of new models. These models are more appropriate to the online world, and perhaps offer a more effective and efficient way of evaluating science. Importantly, they require that scholarly papers are made freely available on the Web — which is precisely the aim of the OA Movement.

What the debate also points to, of course, is that in an Internet-enabled world traditional gatekeepers cannot assume that their role will remain the same.

Indeed, as New York University journalism professor, Jay Rosen pointed out when I interviewed him earlier this year, many traditional gates are in the process of being dismantled. As a consequence, he added, "All kinds of knowledge monopolies — and positions of authority based on them — are wearing away."

In other words, those professionals and organisations that have gained control of society's institutions and power structures are going to have to justify the rights and privileges they currently enjoy, or lose them.

Undoubtedly this includes science publishers who believe that they should retain the exclusive right to determine how scientific papers are published; peer review cliques who think they should be the sole arbiters of what science is and is not published; and, indeed, journalists who believe that, in a world full of blogs, they can maintain a monopoly on deciding what gets reported .

Who knows, perhaps it is only a matter of time before the gatekeepers at Byerly Hall, along with their Ivy League colleagues, discover that their pre-eminent right to adjudicate on academic merit has also disappeared