Sunday, June 12, 2011

UK politicians puzzle over peer review in an open access environment

The UK House of Commons Science & Technology Committee is currently conducting an inquiry into peer review. The third public event of the inquiry was held on Monday 23rd May, when the Committee heard evidence from experts on open access publishing and post-publication review, and from representatives of the research community.

SOME OF THE ISSUES EXPLORED BY THE COMMITTEE ARE HIGHLIGHTED IN THE EDITED QUESTIONS AND ANSWERS LISTED BELOW. THE ASTERISKED HEADINGS ARE MINE.

I COMMENT ON THE SESSION AT THE BOTTOM OF THIS POST. 

UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.

The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for Ellesmere Port and Neston. Other politicians to pose the questions below were Graham Stringer, Labour MP for Blackley and Broughton, Roger Williams, Liberal Democrat MP for Brecon and Radnorshire, and Stephen Metcalfe, Conservative MP for South Basildon and East Thurrock. (A full list of Committee members is available here).

The hearing was split into two sessions.

== FIRST SESSION ==

Those giving evidence in the first session were Dr Rebecca Lawrence, Director, New Product Development at Faculty of 1000 Ltd., Mark Patterson, Director of Publishing at the Public Library of Science, Dr Michaela Torkar, Editorial Director at Biomed Central, and Dr Malcolm Read OBE, Executive Secretary of JISC.






* On splitting traditional peer review into two separate processes: a) assessing a paper’s technical soundness and b) assessing its significance — a model pioneered by open-access publisher PLoS ONE, and now increasingly being adopted by traditional publishers …

Q162 Chair: We have heard that pre-publication peer review in most journals can be split, broadly, into a technical assessment and an impact assessment. Is it important to have both? 

Dr Torkar: … It is fairly straightforward to think about scientific soundness because it should be the fundamental goal of the peer review process that we ensure all the publications are well controlled, that the conclusions are supported and that the study design is appropriate. That is fairly straightforward as a very important aspect which should be addressed as part of the peer review process.

The question of the importance of impact is more difficult. When we think about high impact papers we think about those studies which describe findings that are far reaching and could influence a wide range of scientific communities and inform their next-stage experiments. Therefore, it is quite important to have journals that are selective and reach out to a broad readership, but the assessment of what is important can be quite subjective. That is why it is important, also, to give space to smaller studies that present incremental advances. Collectively, they can actually move fields forward in the long term.

Dr Patterson: … [B]oth these tasks add something to the research communication process. Traditionally, technical assessment and impact assessment are wrapped up in a single process that happens before publication. We think there is an opportunity and, potentially, a lot to be gained from decoupling these two processes into processes best carried out before publication and those better left until after publication.

One way to look at this is as follows. About 1.5 million articles are published every year. Before any of them are published, they are sorted into 25,000 different journals. So the journals are like a massive filtering and sorting process that goes on before publication. The question we have been thinking about is whether that is the right way to organise research. There are benefits to focusing on just the technical assessment before publication and the impact assessment after publication … Online we have the opportunity to rethink, completely, how that works. Both are important, but we think that, potentially, they can be decoupled ...

Dr Lawrence: … [I]t is not known immediately how important something is. In fact, it takes quite a while to understand its impact. Also, what is important to some people may not be to others. A small piece of research may be very important if you are working in that key area. Therefore, the impact side of it is very subjective.

Dr Read: … Separating the two is important because of the time scale over which you get your answer. The impact is much longer. I guess the technical peer review is a shorter-term issue.
===
* On whether in order to deliver faster publishing times it is necessary to cut corners by, for instance, editing papers more lightly, and whether this approach leads to more submissions …

Q167 Roger Williams: Is light copy editing a feature of how you can deliver faster times?

Dr Patterson: [W]e are balancing these two competing interests of speed and quality. In our production process we focus on delivering really well structured files that will be computable, for example. We don’t expend effort in changing the narrative. Scientific articles aren’t works of literature. That is not to say it wouldn’t be nice if, sometimes, a bit more attention was paid to that. It is also true that one of the criteria for PLoS ONE is that the work is in intelligible English. If an editorial reviewer thinks that something is just not good enough and they can’t really see what is happening, it will be returned to the author.

Q169 Roger Williams: Are there any other corners that your journal “cuts” in order to deliver faster times?

Dr Patterson: I wouldn’t frame it that way. What we are doing is trying to identify and take away any unnecessary barrier to publication …

Q170 Roger Williams: Has your approach and the reputation you have built up resulted in a lot more submissions?

Dr Patterson: PLoS ONE was launched in December 2006 and is still quite a new journal. It is only four and a half years’ old. We published about 4,000 articles in 2009 and 6,700 last year, so it became the biggest peer review journal in existence in four years. It has grown steadily over that time …
===
* On the propagation of the PLoS ONE model and whether the journal might become a victim of its own success … 

Q171 Roger Williams: Has your approach and the reputation and impact of the journal itself increased the number of submissions? 

Dr Patterson: It has. We see a lot of positive feedback … the message that if I have a solid piece of work I’m not going to have to grapple with a journal that is basically biased against publication — the goal of PLoS ONE is to publish all rigorous science — is a very positive one which authors like. Coupled with ideas about how, then, you might assess the impact after publication, it is definitely gaining ground.

The other very significant thing that has happened in the last nine to 12 months is that eight or more big publishers have announced PLoS ONE lookalikes, essentially. That is very striking. The American Institute of Physics and the American Physical Society have both launched physical science versions; Sage has launched a social science version; the BMJ group, who were actually the first, last year launched a clinical research version of PLoS ONE; Nature has launched a natural science version of PLoS ONE, and on it goes. The model is getting that level of endorsement from major publishers and I think, again, that is probably helping to make researchers very comfortable with the way in which PLoS ONE works.

Q172 Roger Williams: But will you be a victim of your own success? Will you be overwhelmed by the volume of submissions and then your time to publication suffers as a result? 

Dr Patterson: I certainly hope not. The growth has been pretty spectacular and has definitely surpassed our expectations …
===
* On whether PLoS ONE’s approach and popularity is impacting on its peer review process, and whether the success of its model might trigger a wider change in peer review …

Q173 Roger Williams: Do you believe that this approach has had an effect on the peer review process perhaps in terms of timing, quality and ease of recruiting or having access to reviewers? 

Dr Patterson: It is beginning to. PLoS ONE has grown very rapidly in the space of four years to become a very big journal. There are now another eight to 10 on the scene that are being launched, or are about to be launched. If another 10, 20 or 30 of these are launched over the next one to two years, which I think is quite likely — because a lot of publishers will be looking very hard and thinking that if they don’t get involved they will potentially lose out — that could make some fairly substantial changes in the way the prepublication peer review process works. There is a lot to say about post-publication but not yet. So I think the model could change …
===
* On whether PLoS ONE’s peer review process could be described as “light touch” …

Q176 Stephen Metcalfe: You wouldn’t describe your approach as "light touch"?

Dr Patterson: No, not at all. It is important to consider not just the peer review process but everything that goes on before an article is accepted for publication as being critical steps in quality control. There are several components to that, of which peer review is one. At PLoS ONE staff are involved in the first step. It goes through a series of quality control steps which are focused. Basically, we want to take stuff away from the academics so that they can focus on the science and we can sort out everything else. We focus on things like whether the competing interest statements are properly indicated; financial disclosures; if the work concerns human participants whether there is an ethics statement and appropriate ethical approval — a whole series of things like that. Hardly any manuscripts get through that without some kind of query going back to the author.

Then there is a step where we involve PhD scientists who scan the work. These are people who have some level of subject expertise. Some — not many — of the submissions are rejected at that point because they are completely out of scope or something. They are also looking for any articles on controversial topics or anything that might require special treatment. They flag work like that. The work then goes to the editors whose responsibility it is to take on the peer review process. It is a pretty involved process.

The peer review part then focuses on seven criteria to do with whether the methodology and analysis are appropriate; whether the conclusions are justified; whether the work is ethically sound and properly reported; and whether data is available as appropriate. There is a set of seven criteria …
===
* On cascading peer review, the difficulties in getting publishers to share reviews, and whether sharing is more likely in an OA environment …

Q181 Stephen Metcalfe: How widely used is the system of cascading submissions and reviews from one journal to another? 

Dr Torkar: … We use this quite extensively at BioMed Central and, in particular, with the BMC series which is more or less our equivalent of PLoS ONE and was launched in 2001. It is a group of more than 60 community journals which are subject specific: BMC Immunology, BMC Genetics, etc. As they also have the premise of publishing all scientifically sound studies without putting too much emphasis on the impact and extent of the advance, they will consider manuscripts that were previously peer reviewed or submitted to some of our flagship journals. Sometimes the transfers will happen before the peer review and sometimes with the reviewers’ reports. That does save time for authors and reduces the burden on the peer reviewers who don’t have to re-review manuscripts for multiple journals.

Dr Patterson: Cascading peer review is a phenomenon that exists at PLoS in its two flagship journals PLoS Biology and PLoS Medicine. Articles can be transferred from there to other journals. To give you a sense of the size of that, about 10% to 15% of submissions to PLoS ONE come from other PLoS journals. It is pretty clear that, internally, that works quite well. A lot of publishers think so and quite a lot of the evidence has shown that.

The much more problematic issue is the sharing of reviews from one publisher to another. I know you heard some talk about the Neuroscience Peer Review Consortium experiment which, interestingly, was not terribly popular with authors, but I am not sure how much publishers were really behind it. For example, it was said that some publishers might feel reluctant to share reviews with another journal or publisher because they have built up relationships with these people and there is some commercial value associated with that. When you hear that you have to ask whether that sense of ownership is in the best interests of science. I am not convinced …

… it is quite natural that journals would feel that way in a world of subscriptions because it is about selling a package of content to a group of readers. That is how the model works. Therefore, anything which allows you to improve that package of content is of value to you commercially. In a way, it is completely understandable that journals in that subscription business model would be reluctant to share their reviews.

When you switch round the model, as BMC, PLoS and many others do now, in terms of supporting and publishing through a publication fee, considering yourselves, as publishers, much more as service providers — you are selling a publishing service to a researcher — your attitude towards sharing peer reviews might be changed. I am not sure.
===
* On the ethics of publishing papers sponsored by pharmaceutical companies aiming to bring new products to the attention of doctors, whether pharmaceutical companies might find PLoS ONE’s peer review process more attractive for these purposes, and whether PLoS ONE has a financial interest in publishing pharma-sponsored papers … 

Q198 Graham Stringer: … How much commercial pressure is there from pharmaceutical companies to publish … and how does that commercial pressure interfere with the publication? A journal that publishes a paper which means doctors can prescribe a particular drug stands to make a lot of money, doesn’t it? How is that pressure dealt with ethically? 

Dr Patterson: This is an issue which has certainly been highlighted in the evidence you have already heard. This is something on which, in particular, the editors of our journal PLoS Medicine have taken a very strong position, to reduce what they call the cycle of dependency in some way between the pharmaceutical industry and medical publishing. One of the ways in which that is manifest is with very substantial reprint revenues associated with high profile, hard-hitting clinical trials; for example, sponsored by the pharmaceutical industry.

What PLoS Medicine and PLoS as a whole have done, in order to keep the two things apart and separate any commercial interest from the editorial integrity of the content to be published, is refuse to accept any form of drug or device advertising, even though it could be a significant revenue stream for us. We feel that is a very strong leadership position to take in that area. The business of open access is also very important to this. The articles we publish are open in the sense that there are no barriers to reusing that content. A lot of publishers retain rights to contents so that they can reprint the article. They are the only people who can reprint that article at the levels of thousands and thousands of copies for redistribution, which then earns them an awful lot of money. We can’t do that.

Q199 Graham Stringer: Are you saying that reproducing your articles is free? 

Dr Patterson: Yes.

Q200 Graham Stringer: You are very different from The Lancet or other journals? 

Dr Patterson: Totally different. We feel that is a very important principle. We have no unique right to take those articles and make that kind of money from them. These are some steps that have been taken. They are not the solution to everything, but I think they are important … 

Q201 Graham Stringer: It struck me, when you spoke earlier, that if a pharmaceutical company wanted to get a drug to market very quickly and within the mindset of GPs and other doctors, your route to publication would be quicker. It might be an incentive, then, for them to go via a route which you said yourself — I can’t remember your exact words — was of a different standard; it wouldn’t be sent back. That worried me slightly, that, commercially, it might be easier for drug companies to make more money by going via your route. But you don’t have a financial interest in that? 

Dr Patterson: There is no financial interest, in that sense. To be clear, we consider work that has been sponsored by the pharmaceutical industry but, obviously, it has to conform to the same criteria as everything else. What might make the pharmaceutical industry reluctant, in terms of thinking about the value of that publication commercially, is that to publish in a very high prestige journal would probably be of great value. That is what might put them off coming to, say, PLoS ONE which does not, in and of itself equal high prestige …
===
* On post-publication peer review, whether publishers should incentivise researchers to contribute to such reviewing, what form post-publication review should take, and how research assessment metrics can be improved … 

Q209 Chair: … I want to go finally to post-publication commenting. Should publishers introduce some system of prestige or credit for post-publication commentary? Dr Patterson, why is article-based methodology a good one? I don’t regard it necessarily as a healthy comment if I make a speech and there is an endless number of blogs. No doubt I will disagree with half of them anyway. Is the F1000 model which uses faculty members to carry out that process a better one, or does it become a biased process? To finish off, let me put to all of you this question: what is a good system of post-publication commenting? Should there be some recognition of the people who participate? 

Dr Patterson: Maybe the starting point is to say that at the moment we have a very blunt instrument for research assessment which is basically a number — an impact factor — associated with a journal. We can do much better than that now. The way we are looking at this is to consider all the things you can potentially measure post-publication. It is not just about a blog comment or something like that. There is a whole range of metrics and indicators, including resources like Faculty of 1000, which can be brought to bear on the question of research assessment. Normally, people are looking at the research literature as a whole, they are identifying the papers that are important to them and they are coming to those papers. We want to provide an indication when they come to that paper of how important this is and what impact it has had through usage data, citation information, blogosphere coverage and social bookmarking. There are so many possibilities.

We have moved in that direction by providing those kinds of metrics and indicators on every article that we publish — we are not the only people doing this but we have probably taken it further than most — to try to move people away from thinking about the merits of an article on the basis of the journal it was published in to thinking about the merits of the work in and of itself. Indicators and metrics can help with that. They aren’t the answer to the question but they will help …

Dr Lawrence: We would agree. Faculty of 1000 is a way of using a panel of experts. We have heads of faculty who then suggest the section heads who then suggest the faculty members. It is all very open. All their comments are against their name. On the question of bias, they also have to sign something to say they haven’t been unduly influenced and, obviously, there are issues of conflicts of interest.

Q210 Chair: Isn’t that a more structured approach to Dr Patterson’s X Factor version? 

Dr Lawrence: I don’t think that any of these different metrics, on their own, are that strong. The point is about bringing together all the various metrics. They all have their own problems. To measure the impact of research you need to use different ones in a sensible way. In a way, the more metrics you have the better your chance of really understanding the impact.

Q211 Chair: I am getting from this that your methodology is making sure that the judges aren’t tone deaf, if I may continue to use my rotten analogy, which is a cruel one to you, Dr Patterson. In Dr Patterson’s case, you don’t care. 

Dr Patterson: No. To be clear, I think both approaches will be required. They are complementary. I would like to see — we probably will shortly — F1000 as one of the indicators on a PLoS article. You go to the article and say, "Ooh! It’s been highlighted in F1000 and this is what the person has said", or something like that. There will be a place for expert assessment, evaluation and organisational content post-publication, as well as grabbing as many metrics and indicators as you can from the world at large.

== SECOND SESSION ==

Those giving evidence in the second session were Dr Janet Metcalfe, Chair of Vitae, Professor Ian Walmsley, Pro Vice Chancellor of the University of Oxford, and Professor Teresa Rees CBE, Professor of Social Science and former Pro Vice Chancellor (Research) at Cardiff University.
===
* On why researchers are under pressure to publish in high impact journals … 

Q216 Chair: … [W]hy is it that researchers are put under so much pressure to get work published in the high impact journals? 

Professor Walmsley: Perhaps a simple answer to that from a parochial view of a university person is that that is the way one’s career advances. As you heard from the previous panel, a lot of very good work gets published in journals that do not have such high visibility, and I think that is quite crucial. None the less, having a highly cited paper in a journal that people would regard as high profile is considered important as a way to raise your visibility and develop your career.

Dr Metcalfe: We have drivers in the system, such as the research assessment exercise, that encourage that, so there is very strong emphasis in terms of the impact of the journal. Coming at it from my perspective as Vitae, it is: how do you support early career researchers to enter into that system and even make decisions about what journals they should be targeting? How do they get a sense of the most appropriate place for them to publish?

Q217 Chair: Doesn’t the tie-in between the research excellence framework and high impact journals potentially create a rather subjective judgment? 

Professor Walmsley: I would argue that the reason peer review works well is the expertise of the community on an inherently subjective set of criteria; that is, one can with any piece of work assess various objective elements of it. Is it right? Is it novel — that is, is it new and not been published before? But the subjective element, which I think differentiates a number of different journals — because they have different subjective criteria — is the piece that is very difficult to assess in an objective way. Knowing that a piece of work is going to be important is a very difficult thing to do. In many ways that is something best assessed post facto …

… I absolutely take the point about RAE. Having sat on one of the RAE panels last time, I can say the panel was very clear that the forum in which the paper had been published was not determinative. It was reading the individual outputs and assessing the value of the work itself that ended up being more important. None the less, when a CV comes across the desk of a head of department for a faculty post, as a first pass through it makes a difference where those papers are published.
===
* On whether researchers should be paid for their work as reviewers …

Q222 Roger Williams: Overall, would your judgment be that in the best of all possible worlds researchers should be paid for their work as peer reviewers?  

Professor Rees: I am not sure of the answer to that question. It is strange that if researchers do the research and publish and then do the peer review and the editing — in some cases now they are asked to pay for their articles to be published — one finds oneself responding to a memo saying, "Which journals do you think we should cut from the library because of budget cuts?" I would say there is a bit of a paradox there. 

Professor Walmsley: I would concur with that, having part of the library as my portfolio too. That is an internal and difficult question to address. As to whether reviewers should be paid, I think that may send incentives in the wrong direction. One wants as wide a fraction of the community with appropriate expertise to be involved as possible. The way we might see it internally in departments at Oxford would be that this is a contribution to the community, just as chairing or sitting on committees in the university is considered part of what you need to do in order to make the place and the business function. But the question is about keeping an appropriate lid on that. There are various ways in which one might do that, but in mentoring terms one would often say, "You want to review twice as many papers as you publish and you want to review three times as many grant applications as you submit." That tempers your workload and makes the whole system work.
===
* On whether researchers should be formally trained to do peer review … 

Q226 Stephen Metcalfe: … Dr Metcalfe, I think that the number of people who take up the opportunity for training either in peer review or other publication training is relatively small. Why do you think that is? 

Dr Metcalfe: The tradition is very much an apprenticeship model. You learn the system by doing it in terms of writing papers, submitting them and maybe getting feedback from your principal investigator. Where that works it is absolutely fantastic in terms of somebody taking an early career researcher through the system and giving them feedback before they submit their articles, maybe having several researchers in their group giving feedback, and showing them how the whole process works. But, because we are a collective in terms of the academic community, there is opportunity for that process not to be as well supported throughout the whole of the academic community as it could be.

The challenge is how to help a researcher maximise their opportunities of publication at submission so that they are reducing the amount of rejections and the amount of comments they have to do through that. Formal training in that process is one way in which you can do that. From some of the research Vitae has done, we have evidence of increases in the success rates of grant applications and fellowship applications by having formal training and development in working within the peer review systems for both of those. We could do more in advance of a researcher having to submit their first paper or grant proposal so that they are better informed and therefore more expert about how the whole process works. 

Q227 Stephen Metcalfe: You would be in favour of moving towards a more formal requirement for training. You consider that it should be provided across all higher education institutions. 

Dr Metcalfe: No, I wouldn’t go down that route. I think the opportunities to have training should be there. The process by which a researcher learns to become expert is very much up to their individual circumstances. If they are getting good individual nurturing and mentoring by their PI, that is great. But there should also be the opportunity, for those researchers who respond more to formal training, to have that available as well. 

Q228 Stephen Metcalfe: Who do you think should pay for that training? 

Dr Metcalfe: Collectively, we all have a responsibility for it to work. I think journals have a responsibility to support and provide more information about what is required and to contribute to the training of their reviewers. I think institutions have a responsibility, as signatories to the concordat for the career development of researchers, to ensure that those opportunities are there. I think research and funding councils and Government have an obligation to provide enough funding within the entire system to make available that kind of training for our early career researchers.
===
* On fraud, and what whether Oxford University has ever sacked anyone for fraud …  

Q234 Graham Stringer: Last week the chair of COPE told us that if a university had not fired at least one academic for fraud there was something wrong with the university. Do you agree with that statement? Do you think she was right? If so, have your universities sacked any academics over the last five years for academic fraud? 

Professor Walmsley: The answer to the second question is no. 

Graham Stringer: So you are not firing. 

Professor Walmsley: Yes. I would say that the answer to the first question is probably no, too, but I want to be careful not to suggest that there are no ethical challenges within publication. We have a process within Oxford, which I am certain is the same at other places, to deal with that. Part of the question is: how does it come to your attention? 

Q235 Graham Stringer: Before you go on, is that process published?  

Professor Walmsley: Yes. It is available on the website through the Research Integrity Portal and there is an access through the SkillsPortal to that as well. 

Q236 Graham Stringer: So, that is true for all?  

Professor Walmsley: Yes. How do you identify and find that out? I think that internally, at the pre-publication end, there is great onus on researchers. As more and more papers are published with joint authors there is joint responsibility for doing that. That could lead in two directions: first, increased pressure to get it right because there are more people involved in the discussion; but, secondly, the chance that you will miss a trick or two because there are more people contributing. It is a difficult tension. Once the paper is out there, if an external party notes something that looks challenging I guess we will hear about that either from the external people or from editors themselves. If an editor writes, we will be able to investigate that internally.

As to the sanction of firing someone, I said I have not known that to happen, but there are certainly lower levels of discipline that can happen. However, I don’t know what the statistics are at Oxford. 

Q237 Graham Stringer: If you were to have an investigation, would you publish the results? Would that become a public document? 

Professor Walmsley: I don’t know the answer to that question. 

Q238 Graham Stringer: Would you write and tell us?  

Professor Walmsley: Yes, I will do that. 

The (uncorrected) transcript of the full session is available here.

The video of the event can be accessed here, if it does not open up automatically here.
=====

Subsequent to giving evidence to MPs a slightly puzzled Rebecca Lawrence penned a blog post about her experience. She concluded: “I think we all left still wondering what the purpose of the whole enquiry is and what they are hoping to achieve.”

She added: “This was made even more evident when at the end of the latest session, following 5 long rounds of oral sessions, MP Graham Stringer suggested that he felt that maybe they should have been looking at the commercial pressure on both editors of journals and researchers instead.”

It’s true that the Committee gave no reason for launching the Inquiry. But there have been hints: In this session, for instance, MPs referred both to the so-called Climategate incident, and to Andrew Wakefield.

It is also true that the Committee has at times seemed a little unfocused. But this was probably inevitable: opening up the topic of peer review is not unlike opening Pandora’s Box — all sorts of things fly out. For this reason perhaps the Committee has not always been able to explore in sufficient depth some of issues that have arisen.

The key theme to emerge from this day of the hearing was whether peer review should/would change in an online, open-access environment; and if so, in what way. I list below some of the areas that I feel could have been examined more closely:
  • There appears to have been little discussion of the possible quality implications of authors (or their funders) being asked to pay a fee to publish in open-access journals. It is widely known (within the open-access movement at least) that a growing number of high-volume, low-quality journals are emerging that appear to offer little more than a vanity publishing service. I should stress that I am not referring to PLoS ONE or BioMed Central. I am referring to a number of start-up companies that have adopted the "author pays" model introduced by BioMed Central — and subsequently copied by PLoS ONE — but which often appear to make little or no effort to have the papers they publish reviewed effectively. The witnesses who appeared in the first part of this session will know of this development, and would surely have been able to talk through the issues. In the process the Committee could have explored the potential conflicts of interest (and how to overcome them) that must surely arise when the income of a publisher of peer-reviewed journals is directly related to the number of papers it accepts. 
  • Likewise, the Committee did not appear to address a topic that has become a serious problem in scholarly publishing, one that surely has implications for peer review. That is, although the stated purpose of publishing scholarly papers is to share new ideas and experimental data with other researchers, the emphasis today is as likely (often more likely) to be on furthering an author's career as it is on communicating research findings — which are not necessarily the same things. In other words, researchers are incentivised to publish as many papers as possible in order to maximise their chances of tenure and/or promotion. Amongst other things this leads to salami slicing, and inevitably to the publication of lower-quality, less-worthwhile papers. In a pay-to-publish environment that can easily morph into vanity publishing if there are inadequate guidelines and processes to ensure high quality this is worrying — not just in terms of quality in fact but, since it can cost 1,000s of dollars to publish a paper, in terms of costs too.
  • And while it did explore the costs of peer review, the Committee seemed more interested in the (very much smaller) costs of reviewers’ time, than the much larger, increasingly burdensome, costs of paying publishers to organise that peer review. And open access looks set to increase these costs (certainly in the short term). This issue was hinted at when Teresa Rees said: “It is strange that if researchers do the research and publish and then do the peer review and the editing — in some cases now they are asked to pay for their articles to be published — one finds oneself responding to a memo saying, ‘Which journals do you think we should cut from the library because of budget cuts?’ I would say there is a bit of a paradox there.” The Committee would have benefitted from trying to unpick that paradox. 
  • Finally, the Committee did not follow through on Stephen Metcalfe’s questions about training researchers. As I understood it, the issue was whether and how researchers are taught to peer review papers. In their answers the witnesses focussed almost exclusively on how one trains researchers to get their papers through the review process, not how to review other researchers’ papers effectively. In the context of the inquiry the latter is surely the more important issue. Indeed, the fact that the witnesses focused on the other side of the process serves to underline the extent to which scholarly communication is now viewed essentially as a career-advancement mechanism, rather than a formal way of sharing research findings. As mentioned, this would seem to have implications for the quality of published research. At least it would have helped had the Committee persisted in looking at both sides of the process — after all, the current pressure to publish as many papers as possible leaves researchers vulnerable to the constant stream of email invitations from publishers of pay-to-publish journals whose review processes are obscure, and may be seriously lacking in rigour.
I realise that, given the time available, it is probably asking too much to expect the Committee to explore all the issues that fly out of the peer review box, certainly in depth. But for an inquiry into peer review that has indicated it plans to examine, amongst others things, “the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public”; “the impact of IT and greater use of online resources on the peer review process”; and “the processes by which reviewers with the requisite skills and knowledge are identified,” the above issues would seem to be highly germane.

That said, Graham Stringer’s suggestion that they ought to be looking more at the commercial pressures faced by editors and researchers suggests that the Committee is beginning to understand some of the deeper forces currently influencing how peer review operates.

It is also worth bearing in mind that when the Science & Technology Select Committee was conducting an inquiry into scientific publishing in 2004 some began to wonder if it had lost its way. However, when its report — Scientific Publications: Free for All? — was eventually published the research community quickly concluded that the politicians had actually understood the issues very clearly and, moreover, had made the correct recommendations.

Meanwhile the current inquiry continues. Time will tell how successfully today’s crop of MPs can get to grips with the issues.

No comments:

Post a Comment