The UK House of Commons Science & Technology Select Committee is currently conducting an inquiry into peer review. It held its fourth oral evidence session on June 8th, taking evidence from both funders of scientific research and from Government.
UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.
While the Committee gave no specific reason for launching the current inquiry it seems evident from the questions MPs have been asking that two particular incidents have been exercising their minds: the long-running saga over Andrew Wakefield and the MMR vaccine scare, and the so-called Climategate incident.
In the June 8th session the Committee asked questions not just about the efficacy of peer review, but about scientific fraud, bias, the willingness of universities to investigate allegations of misconduct, and the need to make research data freely available so that others can access, examine and test them.
MPs seemed particularly concerned that universities may be unwilling to investigate claims of misconduct. As MP Graham Stringer put it in one of the questions he asked the witnesses, “There is a certain amount of evidence that very little fraud is detected in universities and major research institutions in this country. Do you think we should be doing more to try and detect that, because in one sense there is an interest within those bodies not to discover or expose the problems they have, to sweep it under the carpet, isn’t there? If you are running a university and you find you have a researcher who just writes down his figures without doing the work, which has happened in one or two cases, the university doesn’t want to say that it has been employing a fraudster for 10 years, does it?”
Politicians also probed the witnesses about the use of journal impact factors as a “proxy measure for research quality” when assessing the performance of academics, and whether “the growth of online repository journals” like PLoS ONE is a “technically sound” development.
Robust defence
For their part the witnesses put up a robust defence of current practices. They denied that universities would cover up fraud; they dismissed suggestions that the impact factor is used as a proxy measure of quality; and they insisted that, while it might not be perfect, there is no practical alternative to traditional peer review.
In support of the latter claim they repeated the oft-made analogy with Winston Churchill’s description of democracy. Churchill famously described democracy as “the worst form of government, except for all those other forms that have been tried from time to time.” Thus it is with peer review, averred the witnesses: no one has come up with anything better.
Those with any experience or knowledge of how peer review works in practice might have been tempted to conclude that analogising peer review with democracy is to obfuscate the issue. At the very least, it appears oxymoronic.
Such a conclusion was all the more likely in light of the opening question and answer. The Chair suggested that it might be helpful to conduct some research into the efficacy of the current system — on the grounds that “evaluation of peer review is poor”. To this Wellcome Trust director Sir Mark Walport replied: “Peer review is no more and no less than review by experts. I am not sure that we would want to do a comparison of a review by experts with a review by ignoramuses.”
Sir Mark’s statement can only have served to remind the audience that peer review is more oligarchic than democratic in effect. Rather than encouraging egalitarianism, it promotes elitism, and all the privileges one might associate with an old boy’s club (appositely perhaps, there was not a single female witness called to give evidence on June 8th).
Of course the Churchillian analogy is not really meant to suggest that peer review is a democratic process. Nevertheless the witnesses’ repeated claims that the current peer review system is “good enough” would surely be challenged by many junior researchers, who frequently complain that scholarly journals tend to be controlled by small elite groups of insiders, invariably senior researchers.
As one researcher pointed out to me recently, this is particularly problematic for those working outside North American and Europe. As he put it, “Peer-reviewed journals with a high impact factor are either dominated by certain gangs, or groups, or the editors rely on the opinion of reviewers too much.”
The upshot, he added, is that “a small guy from Russia, Brazil or Thailand will never get published, even with excellent results, unless he or she has a prominent Western colleague as a co-author.”
In fact, it is not just researchers in less privileged parts of the world who can struggle to get published in scientific journals today. Nor is it only junior researchers who complain about the peer review system. In 2009, for instance, 14 leading stem cell researchers wrote an open letter to journal editors highlighting their disquiet at the way in which the system operates.
Speaking to the BBC about the letter Professor Lovell-Badge commented: "It's turning things into a clique where only papers that satisfy this select group of a few reviewers who think of themselves as very important people in the field is published.”
Responding in a (separate) BBC interview, Sir Mark downplayed the criticism. Scientists, he said, “are always a bit paranoid” about peer review. And to make his point Sir Mark again used the analogy with democracy — peer review is not perfect, but it is the best system that the research community has been able to come up with.
Wednesday, June 22, 2011
Sunday, June 19, 2011
Open Access by Numbers
Few can now doubt that open access (OA) to scholarly research is set to become an important feature of the scholarly communication landscape. What is less certain is how much of the world’s research literature is currently available on an OA basis, how fast OA is growing, and what percentage of the world’s academic and scientific literature will be OA in the long-term.
Trying to crunch the numbers is complicated by the fact that research papers can be made OA in two ways: Researchers can continue to publish in subscription journals and then make them freely available by self-archiving them in an institutional repository (Green OA), or they can pay to publish their work in an OA journal (either a pure Gold journal or a Hybrid OA journal) so that the publisher will make it freely available for them.
OA enthusiasts like librarian Heather Morrison — who publishes a series called “Dramatic Growth of Open Access” — tend to estimate OA occurrence and growth primarily by the simple counting of things.
In March, for instance, Morrison reported that there are now over 6,000 OA journals listed in the directory of open access journals (DOAJ), and implied that the number of OA articles is now growing more quickly than the number of papers being published in subscription journals. As she put it: “Data is presented that strongly suggests that the success rate for open access journals is already higher than that of subscription journals.”
In the same post, Morrison argued that by counting the number of papers flagged as OA on the Mendeley research sharing service we could conclude that self-archiving had grown by 171% in the first quarter of 2011.
Counting in this way presents an upbeat picture, suggests that the world is in the process of being flooded with OA, and that universal OA is just around the corner.
Refining the counts
Critics, however, point out that simple counting is too crude when trying to measure OA. Counting Gold OA journals, for instance, is not helpful since many of them publish just a handful of papers a year, if that.
Likewise, counting items that have been self-archived can be deceptive: Many records in institutional repositories will consist of metadata alone, or non-target items like presentations and other non-reviewed material.
Certainly publishers describe the incidence and growth of OA in a less upbeat manner. When I spoke to Springer’s Derk Haank at the end of last year, for instance, he estimated that only around 2% to 2.5% of the world’s papers are being published in Gold or Hybrid journals today.
And since the total number of research papers is growing at around 6% to 7% a year, he said, OA remains “just a drop in the ocean”.
In fact, predicted Haank, OA publishing will never be more than a niche activity. “I expect it to remain between 5% and 10% at a maximum,” he said.
Haank did not provide an estimate of Green OA, but implied that it was relatively low. Pointing out that he would be anxious if it did become commonplace he added, “But we are such a long way from that situation today that we are very easy going about author archiving.”
A few researchers, meanwhile, have been busy trying to arrive at more precise figures. When I last wrote on this topic in 2010 I spoke to a number of researchers, including Bo-Christer Björk.
Based at the Hanken School of Economics in Helsinki, Björk has undertaken several studies aimed at sizing the growth of OA, primarily Gold OA.
For a variety of reasons, Björk explained, this is not an easy thing to do. Nevertheless, when I spoke to him in January 2010 Björk estimated that Gold OA was probably increasing its share of the market by 0.5% per annum.
He added, however: “I have no evidence to show any acceleration in growth. On the contrary it seems that growth has been relatively stable, after a short expansive period when BioMed Central and PLoS were founded”.
“Tremendous growth of Gold OA”
Since then, Björk has taken a closer look at the many new OA journals that have been launched from 1993 - 2009, as well as the many subscription journals that have been converted into Gold journals.
There has also been the rise of “mega journals” like PLoS ONE, now the largest peer-reviewed journal in the world, and which expects to publish 12,000 papers in 2011 alone. In the wake of PLoS ONE’s success a number of PLoS ONE clones have recently been launched.
On June 13th 2011 Björk and colleagues published a new paper reporting an average annual growth rate since 2000 of 18% for the number of OA journals and 30% for the number of articles.
This, the paper suggests, “can be contrasted to the reported 3.5% yearly volume increase in journal publishing in general. In 2009 the share of articles in OA journals, of all peer reviewed journal articles, reached 7.7%. Overall, the results document a rapid growth in OA journal publishing over the last fifteen years.”
And in a note he posted on the American Scientist Open Access Forum (AmSci) Björk said that the results, “show the tremendous growth of gold OA over the past decade”.
As we said, Björk’s primary focus is on Gold OA. What about Green OA?
This is an area that Yassine Gargouri, a postdoctoral researcher who works with OA advocate Stevan Harnad at the Université du Québec à Montréal (UQAM), has been working on for the past four years.
Gargouri’s numbers suggest that between 2005 and 2010 the percentage of Green OA rose from about 15% per year to about 21%, which amounts to an increase of about 1% per year.
His numbers also suggest that introducing a Green mandate (requiring all an institution’s researchers to self-archive their papers) triples the yearly percentage of OA papers from the mandating institution.
Taken together with Björk’s work, this would seem to suggest that around 30% of the academic and scientific literature published in 2011 worldwide may now be freely available on the Web, two thirds of it as Green OA and one third of it as Gold OA.
Can Gold alone buy OA?
Nevertheless, it remains difficult to be precise about OA numbers, and especially difficult to make accurate predictions about future growth.
Like all attempts to understand and predict the world by means of numbers and statistics, much depends on how one derives them in the first place, how one crunches them, and how one subsequently interprets the results. In the case of OA, a key question that emerges is whether Gold OA is able on its own to accelerate the growth of OA to the degree that the OA movement would wish.
Why is it necessary to fret over such things? It is necessary for a number of reasons, but above all because if OA advocates knew exactly what was happening, and why, they would be able to put their main effort into those activities most likely to achieve their goal.
Vitally, they would be better able to answer a question that has plagued the movement for many years: Should the priority be given to Green or to Gold OA?
In the PDF file attached below I am publishing a Q&A interview with Gargouri. With a PhD in cognitive informatics, Gargouri has also participated in projects dealing with knowledge management, semantic web applications and ontologies. He has also taught in the computer science department at UQAM.
The interview includes contributions from Harnad — a leading OA advocate and self-styled archivangelist.
####
If you wish to read the interview please click on the link below.
I am publishing it under a Creative Commons licence, so you are free to copy and distribute it as you wish, so long as you credit me as the author, do not alter or transform the text, and do not use it for any commercial purpose.
To read the interview (as a PDF file) click here.
See also: John Whitfield Open access comes of age Nature, 21st June 2011
Trying to crunch the numbers is complicated by the fact that research papers can be made OA in two ways: Researchers can continue to publish in subscription journals and then make them freely available by self-archiving them in an institutional repository (Green OA), or they can pay to publish their work in an OA journal (either a pure Gold journal or a Hybrid OA journal) so that the publisher will make it freely available for them.
OA enthusiasts like librarian Heather Morrison — who publishes a series called “Dramatic Growth of Open Access” — tend to estimate OA occurrence and growth primarily by the simple counting of things.
In March, for instance, Morrison reported that there are now over 6,000 OA journals listed in the directory of open access journals (DOAJ), and implied that the number of OA articles is now growing more quickly than the number of papers being published in subscription journals. As she put it: “Data is presented that strongly suggests that the success rate for open access journals is already higher than that of subscription journals.”In the same post, Morrison argued that by counting the number of papers flagged as OA on the Mendeley research sharing service we could conclude that self-archiving had grown by 171% in the first quarter of 2011.
Counting in this way presents an upbeat picture, suggests that the world is in the process of being flooded with OA, and that universal OA is just around the corner.
Refining the counts
Critics, however, point out that simple counting is too crude when trying to measure OA. Counting Gold OA journals, for instance, is not helpful since many of them publish just a handful of papers a year, if that.
Likewise, counting items that have been self-archived can be deceptive: Many records in institutional repositories will consist of metadata alone, or non-target items like presentations and other non-reviewed material.
Certainly publishers describe the incidence and growth of OA in a less upbeat manner. When I spoke to Springer’s Derk Haank at the end of last year, for instance, he estimated that only around 2% to 2.5% of the world’s papers are being published in Gold or Hybrid journals today.
And since the total number of research papers is growing at around 6% to 7% a year, he said, OA remains “just a drop in the ocean”.
In fact, predicted Haank, OA publishing will never be more than a niche activity. “I expect it to remain between 5% and 10% at a maximum,” he said.
Haank did not provide an estimate of Green OA, but implied that it was relatively low. Pointing out that he would be anxious if it did become commonplace he added, “But we are such a long way from that situation today that we are very easy going about author archiving.”
A few researchers, meanwhile, have been busy trying to arrive at more precise figures. When I last wrote on this topic in 2010 I spoke to a number of researchers, including Bo-Christer Björk.
Based at the Hanken School of Economics in Helsinki, Björk has undertaken several studies aimed at sizing the growth of OA, primarily Gold OA.
For a variety of reasons, Björk explained, this is not an easy thing to do. Nevertheless, when I spoke to him in January 2010 Björk estimated that Gold OA was probably increasing its share of the market by 0.5% per annum.
He added, however: “I have no evidence to show any acceleration in growth. On the contrary it seems that growth has been relatively stable, after a short expansive period when BioMed Central and PLoS were founded”.
“Tremendous growth of Gold OA”
Since then, Björk has taken a closer look at the many new OA journals that have been launched from 1993 - 2009, as well as the many subscription journals that have been converted into Gold journals.
There has also been the rise of “mega journals” like PLoS ONE, now the largest peer-reviewed journal in the world, and which expects to publish 12,000 papers in 2011 alone. In the wake of PLoS ONE’s success a number of PLoS ONE clones have recently been launched.
On June 13th 2011 Björk and colleagues published a new paper reporting an average annual growth rate since 2000 of 18% for the number of OA journals and 30% for the number of articles.
This, the paper suggests, “can be contrasted to the reported 3.5% yearly volume increase in journal publishing in general. In 2009 the share of articles in OA journals, of all peer reviewed journal articles, reached 7.7%. Overall, the results document a rapid growth in OA journal publishing over the last fifteen years.”
And in a note he posted on the American Scientist Open Access Forum (AmSci) Björk said that the results, “show the tremendous growth of gold OA over the past decade”.
As we said, Björk’s primary focus is on Gold OA. What about Green OA?
This is an area that Yassine Gargouri, a postdoctoral researcher who works with OA advocate Stevan Harnad at the Université du Québec à Montréal (UQAM), has been working on for the past four years.
Gargouri’s numbers suggest that between 2005 and 2010 the percentage of Green OA rose from about 15% per year to about 21%, which amounts to an increase of about 1% per year.
His numbers also suggest that introducing a Green mandate (requiring all an institution’s researchers to self-archive their papers) triples the yearly percentage of OA papers from the mandating institution.
Taken together with Björk’s work, this would seem to suggest that around 30% of the academic and scientific literature published in 2011 worldwide may now be freely available on the Web, two thirds of it as Green OA and one third of it as Gold OA.
Can Gold alone buy OA?
Nevertheless, it remains difficult to be precise about OA numbers, and especially difficult to make accurate predictions about future growth.
Like all attempts to understand and predict the world by means of numbers and statistics, much depends on how one derives them in the first place, how one crunches them, and how one subsequently interprets the results. In the case of OA, a key question that emerges is whether Gold OA is able on its own to accelerate the growth of OA to the degree that the OA movement would wish.
Why is it necessary to fret over such things? It is necessary for a number of reasons, but above all because if OA advocates knew exactly what was happening, and why, they would be able to put their main effort into those activities most likely to achieve their goal.
Vitally, they would be better able to answer a question that has plagued the movement for many years: Should the priority be given to Green or to Gold OA?
In the PDF file attached below I am publishing a Q&A interview with Gargouri. With a PhD in cognitive informatics, Gargouri has also participated in projects dealing with knowledge management, semantic web applications and ontologies. He has also taught in the computer science department at UQAM.
The interview includes contributions from Harnad — a leading OA advocate and self-styled archivangelist.
####
If you wish to read the interview please click on the link below.
I am publishing it under a Creative Commons licence, so you are free to copy and distribute it as you wish, so long as you credit me as the author, do not alter or transform the text, and do not use it for any commercial purpose.
To read the interview (as a PDF file) click here.
See also: John Whitfield Open access comes of age Nature, 21st June 2011
Tuesday, June 14, 2011
Bernard Rentier Interview in Portuguese
Brazilian Open Access advocate Dr Hélio Kuramoto is currently translating the recent interview I did with Bernard Rentier, Rector of the University of Liège, into Portuguese.
Dr Kuramoto plans to undertake the translation in stages. The first part was posted today, and can be accessed here.
Thank you Hélio!
Dr Kuramoto plans to undertake the translation in stages. The first part was posted today, and can be accessed here.
Thank you Hélio!
Sunday, June 12, 2011
UK politicians puzzle over peer review in an open access environment
The UK House of Commons Science & Technology Committee is currently conducting an inquiry into peer review. The third public event of the inquiry was held on Monday 23rd May, when the Committee heard evidence from experts on open access publishing and post-publication review, and from representatives of the research community.
SOME OF THE ISSUES EXPLORED BY THE COMMITTEE ARE HIGHLIGHTED IN THE EDITED QUESTIONS AND ANSWERS LISTED BELOW. THE ASTERISKED HEADINGS ARE MINE.
I COMMENT ON THE SESSION AT THE BOTTOM OF THIS POST.
UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.
The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for Ellesmere Port and Neston. Other politicians to pose the questions below were Graham Stringer, Labour MP for Blackley and Broughton, Roger Williams, Liberal Democrat MP for Brecon and Radnorshire, and Stephen Metcalfe, Conservative MP for South Basildon and East Thurrock. (A full list of Committee members is available here).
The hearing was split into two sessions.
Those giving evidence in the first session were Dr Rebecca Lawrence, Director, New Product Development at Faculty of 1000 Ltd., Mark Patterson, Director of Publishing at the Public Library of Science, Dr Michaela Torkar, Editorial Director at Biomed Central, and Dr Malcolm Read OBE, Executive Secretary of JISC.
SOME OF THE ISSUES EXPLORED BY THE COMMITTEE ARE HIGHLIGHTED IN THE EDITED QUESTIONS AND ANSWERS LISTED BELOW. THE ASTERISKED HEADINGS ARE MINE.
I COMMENT ON THE SESSION AT THE BOTTOM OF THIS POST.
UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.
The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for Ellesmere Port and Neston. Other politicians to pose the questions below were Graham Stringer, Labour MP for Blackley and Broughton, Roger Williams, Liberal Democrat MP for Brecon and Radnorshire, and Stephen Metcalfe, Conservative MP for South Basildon and East Thurrock. (A full list of Committee members is available here).
The hearing was split into two sessions.
== FIRST SESSION ==
Tuesday, June 07, 2011
The OA Interviews: Bernard Rentier, Rector of the University of Liège
What is striking about Open Access (OA) is that it so obviously the right and rational way for the research community to respond to the networked world. Indeed, one could aptly describe it as a no-brainer — or, as OA advocate Stevan Harnad likes to put it, OA is “raincoat science” (“It's raining, kids, and you're getting wet: Time to put on your raincoats!”).
What is odd about OA is that so few in the research community yet appear to have understood (or at least accepted) its inevitability. Such myopia is doubtless no accident — many have a vested interest in the status quo, while others instinctively fear change, and have an irrational abhorrence of the new.
As a consequence, what is surely “inevitable and optimal” has been delayed now for over a decade and a half: While universal OA could in theory be realised practically overnight, it is estimated that still only around 30% of the world's academic and scientific literature is freely available on the Web.
Fortunately some do “get it” the moment the concept is explained to them — as did Bernard Rentier, a professor of virology and immunology at the University of Liège (ULg), in 1998.
Rentier had been appointed vice-rector in charge of research policy and libraries at ULg the year before — a position that inevitably focused his professorial mind on a number of problems for which OA turns out to be the solution. This became apparent to him during a conversation he had with a science librarian, who introduced him to the concept behind the then incipient OA movement.
“[I]t was immediately obvious to me that this was the logical approach to take,” says Rentier, “especially as the new technology that was then emerging made it entirely possible.”
What is odd about OA is that so few in the research community yet appear to have understood (or at least accepted) its inevitability. Such myopia is doubtless no accident — many have a vested interest in the status quo, while others instinctively fear change, and have an irrational abhorrence of the new.
As a consequence, what is surely “inevitable and optimal” has been delayed now for over a decade and a half: While universal OA could in theory be realised practically overnight, it is estimated that still only around 30% of the world's academic and scientific literature is freely available on the Web.
Fortunately some do “get it” the moment the concept is explained to them — as did Bernard Rentier, a professor of virology and immunology at the University of Liège (ULg), in 1998.
Rentier had been appointed vice-rector in charge of research policy and libraries at ULg the year before — a position that inevitably focused his professorial mind on a number of problems for which OA turns out to be the solution. This became apparent to him during a conversation he had with a science librarian, who introduced him to the concept behind the then incipient OA movement.
“[I]t was immediately obvious to me that this was the logical approach to take,” says Rentier, “especially as the new technology that was then emerging made it entirely possible.”