What quality controls are utilised by PLOS ONE when selecting reviewers? Who is deemed eligible?

In 2011, I expressed concern about the PLOS ONE business model and its associated review process. My worries were focused on the use of what some have called light or “lite” peer review, and the “pay-to-publish” system used by PLOS ONE (and now by many other publishers). My worries were subsequently recorded on the PLOS ONE Wikipedia page. 

Recent personal experience has increased my concern, and left me wondering about the way in which reviewers are recruited by PLOS ONE. 

On 12^th July, I received an email from a PLOS ONE academic editor inviting me to peer review a paper. I won’t say what the paper was entitled, or who the authors were, but it was on the topic of open access journals.

Since I am a blogger/journalist rather than an academic I was surprised to receive the invitation, and emailed PLOS ONE with the following question: “I have had an invitation to review the above paper. Can you point me to the rules on the eligibility of PLOS ONE reviewers?”

I received the following (I assume boilerplate) reply:

This did not address my question, so I also emailed the academic editor whose name had been at the bottom of the invitation. I am not going to name him, but I will say that he is based at Universidad de Las Palmas de Gran Canaria, “Was it you who sent me an invitation to review the above paper?” I asked. “If so, I am wondering why you invited me. Can you say?”

I received the following response:

Grunt work

Let me be quite clear at the outset: I had and have no interest whatsoever in reviewing this or any other scholarly work, not least because there is absolutely no incentive for me to devote my time to reviewing papers. Moreover, the one time I did agree to review anything for an academic journal (an editorial rather than a paper), my suggestions were all rejected on the grounds that “the author says he is too busy to make the changes you suggested.” Clearly I had not made very good use of my time!

But as I say, my first response on receiving the PLOS ONE invitation was to wonder whether it is inappropriate for non-academics to review scholarly papers.

With these thoughts in mind I tweeted the invitation under the strapline “PLOS ONE invites journalist to review scholarly paper”. Somewhat to my surprise, everyone who responded said that they saw no problem with my reviewing a scholarly paper on open access (although it could not presumably be defined as “peer” review). Their reasoning was that they are confident that I have the necessary expertise. And Roger Schonfeld commented, “I’d like to see expertise welcomed into the scholarly conversation without regard to academic affiliation.”

On reading these responses I recalled that some OA advocates maintain that an important benefit of OA is that it encourages members of the public to take a greater interest in science, and to even take part in the process themselves – by means of “citizen science”.

A few days later I read this in the Times Higher: “the open access movement in academic publishing, while far from perfect, means that more research papers than ever are available to those outside universities. Are we not entitled to expect graduates in the workforce and the general public to show the curiosity and commitment to engage with research findings relevant to them?”

It added: “Academics should not hide in the ivory tower, but neither should non-academics refuse to re-enter it after they graduate. Its doors remain wide open. Surely it is not too much to ask for policymakers, journalists, non-governmental organisations and industry researchers to make a greater effort to access its treasures?”

That all sounds laudable enough, but I wonder if the authors of the Times Higher article (two cultural studies researchers) would want their scholarly papers reviewed by members of the public.

As it happens, I tend these days to turn a somewhat sceptical ear to people when they start extolling the virtues of citizen science. As I put it in an interview last year, when researchers talk about the benefits of citizen science they usually have in mind a scenario in which the public’s role is simply to provide free labour for crowdsourcing projects. Citizens are despatched to count butterflies, bugs or birds, stare at images of galaxies on their computer, or “swab and send”, and then feed the data back to bona fide scientists in the lab.

“This activity is clearly valuable,” I suggested, “but seems to imply that the role of citizens in science is to do the grunt work, while the real science continues to be undertaken by professionals.”

Left wondering

That said, I was still intrigued as to how and why I had been invited to review a paper for PLOS ONE. So I persisted in my attempts to get an explanation, putting to PLOS ONE the following three questions:

·       Does PLOS ONE allow or not allow people to review a paper where they are not a member of a university or other research institution?

·       Does PLOS ONE have a published list of the criteria by which reviewers can be deemed to be eligible to act as a reviewer? If not, can you tell me what they are anyway?

·       On what grounds was I invited to review the paper in question, and why was it later decided that the invitation was a mistake?

It was not easy to get an explanation out of PLOS ONE, despite two tweets from the publisher saying that it was looking into the matter (here and here).

Eventually (on 22^nd July) I received the following email:

Academic Editors at PLOS ONE have great autonomy when selecting reviewers, and the decision as to whether an individual expert is eligible to act as a reviewer for a specific manuscript lies with the Academic Editor. We have been in touch with [the academic editor] and he has noted, as we mentioned, that the reasons why specific individuals are approached as reviewers are numerous and may vary depending on the submission. In relation to the invitation you received [the academic editor] has reiterated that this was issued in error, he apologized to you for this error when you first raised this to his attention.

From this I conclude that there is in fact no policy on reviewer eligibility at PLOS ONE, although it would have been nice if the publisher had directly acknowledged this.

So I am left wondering how PLOS ONE academic editors make their decisions when recruiting reviews. Presumably there will be quite a diversity of views on eligibility. Is such a loose arrangement really satisfactory?

Interestingly, a few days later Chealsye Bowley, a librarian at Florida Gulf Coast University tweeted about a related problem she had experienced with F1000Research. She had been informed that as she does not have a PhD she is not eligible to submit a paper to the publishing platform.

The good news is that F1000Research does have a published authorship policy. The better news is that the company responded to Chealsye’s concerns by proposing to amend the policy to cater for people like her (although some still view the proposed amendment as unsatisfactory).

Like Chealsye I too do not have a PhD. I did spend a year at Moscow State University in 1981 studying the “village prose” writers (e.g. Vasily Belov and Valentin Rasputin), but the interminable queues I had to stand in at the Lenin Library simply to request access to the books I needed – along with the constantly flowing Столичная vodka – proved too great a distraction. I never finished my thesis.

Not very encouraging

However, the fact that I do not have a PhD would not have been known by the PLOS ONE academic editor since I have no CV online. Presumably, therefore, he withdrew the invitation when he realised I was not a researcher. But should he not have established that before sending me the invitation? When I later asked him how the mistake had occurred, and how I had been targeted, he did not reply.

But there are some clues as to how I was selected, and I don’t find them very encouraging. Let me explain: 

The invitation I received (below) was sent to me at 12:08 on 12 July 2016.

When I looked at my blog statistics I noted that four minutes earlier someone from the University of Las Palmas De Gran Canaria had accessed one of the pages on my web site (this one).

This was the only page the visitor from Gran Canaria accessed that day, so I am inferring that the vetting process went something like this:

1.     The academic editor does a Google search on “open access” and sees my name in one of the hits.

2.     He clicks on the link in that list and is sent to a page on my site.

3.     He quickly scrolls down that page, copies the email address at the bottom, and sends off his invitation.

This suggests that the academic editor did not think to check whether I was a researcher. If so, it is a pretty telling mistake. After all, the banner on the top of the page he accessed on my site quite clearly states that I am an “independent journalist”. In addition, the email addresses he copied clearly describes my status – richard.poynder@journalist.co.uk.

If that’s right, then the mistake the academic editor made was to spend far too little time establishing the eligibility of the person he invited to review his open access paper – around four minutes in fact. This is of course speculation on my part as the academic editor stopped responding to my emails.

This leaves me with three further questions:

1.     Do PLOS ONE editors routinely only do four minutes research when looking for, and establishing the eligibility of, a reviewer, or was this editor in an uncharacteristic hurry?

2.     Does PLOS ONE have no quality controls in place to enable it to itself verify the appropriateness of reviewers being recruited in its name? Does it simply leave everything to the academic editor?

3.     If I had not questioned the invitation, but just sent in a review, any old review, and that review had met the needs of the academic editor, would anyone have known that a “mistake” had been made? After all, PLOS ONE does not operate open peer review – its guidelines state: “Reviewers are anonymous by default. Reviewers’ identities are not revealed to authors or to other reviewers unless reviewers specifically request to be identified by signing their names at the end of their comments.”

Serious issues

And if the situation at PLOS ONE is as I infer it to be then I think it raises some serious issues. As Danny Kingsley has pointed out, problems with peer reviewers are now a big, and growing, problem. “In August 2015 Springer was forced to retract 64 articles from 10 journals, ‘after editorial checks spotted fake email addresses, and subsequent internal investigations uncovered fabricated peer review reports’”, she wrote recently. “They concluded the peer review process had been ‘compromised’.”

Kingsley added, “In November 2014, BioMed Central uncovered a scam where they were forced to retract close to 50 papers because of fake peer review issues. This prompted BioMed Central to produce the blog ‘Who reviews the reviewers?’ and Nature writing a story on Publishing: the peer review scam.”

Is not PLOS ONE laying itself open to similar scrapes and scandals? (As it is, PLOS ONE has already featured on Retraction Watch a number of times).

It is also worth noting that in the Nature article that Kingsley cites (published in 2014) the authors point out that the editorial system used by PLOS ONE (Editorial Manager) is insecure, since it “actually sends out a password, without prompting, whenever it asks a user to sign in, for example to review a new manuscript.” Two years later this is still happening, as I can now attest.

All in all, I feel my experience suggests there are two, related, issues here. First, there appears to be inadequate checking of reviewers at PLOS ONE, and apparently no official policy on reviewer eligibility.

Second, in describing my invitation as a mistake, PLOS ONE would seem to assume that the involvement of citizens in science should, at best, be no more than as occasional passive readers of papers, and/or fodder for citizen science grunt work.

As I said, I have absolutely no wish to review papers myself, but I am confident that many citizens would be more than willing to contribute. And so long as they had the necessary expertise they would probably do a better job than many academics (I say that having read quite a few reviews in my time). Either way, I would suggest that establishing the expertise of a reviewer (whoever they might be) requires more than a quick Google search.

Moreover, if, as many maintain, the boundary between citizens and scientists should be more porous, is PLOS ONE sending out the right message to the public with incidents like this? I also suspect that had I been a researcher I would have received a more detailed and respectful response to my enquiries from PLOS ONE.

The Ivory Tower is dead, long live the Ivory Tower!