Peer Review, Impact Factors, and the Decline of Science

[I've had interesting tabs open for weeks waiting for time to report on them. Sorry for the delay. -dww]

The Economist reported on October 19, 2013 (pp. 21-24) that there is "Trouble at the lab". Indeed. And trouble has been brewing for quite some time without a single identifiable culprit or an easy way to solve the problem. This problem is concerned with predatory publishing, irreproducibility of scientific results, and the use of quantitative data as an attempt to judge quality.

University administrations, search and tenure committees, governments, funding associations, and other bodies need some way of judging people they don't know in order to decide whether to offer them jobs or promotions or funding. This has often boiled down to counting the number of publications, or the impact factors of the journals in which their articles are published. Coupled with the crisis in publishing, with the subscription price of subscription journals exploding, an unhealthy mix is brewing.

Predatory publishers promise quick publication in good-sounding "international" journals, using the Open Access "golden road" to extract fees from authors. They promise peer review, but if at all they only seem to look at the formatting. Established publishers trying to keep up their profits have incorporated more and more journals into their portfolios without keeping a watchful eye on quality control.

Enter John Bohannon. In October 2013 Bohannon published an article in Science, Who's Afraid of Peer Review? He details a sting operation that he conducted between January and August 2013, submitting 304 papers with extremely obvious deficiencies to journals that he chose both from Lund University's "Directory of Open Access Journals" as well as from Jeffrey Beall's list of predatory publishers.

Bohannon has put his data online, showing that 82% of the journals chosen from Beale's list accepted the fabricated paper, as well as 45% of the journals on the DOAJ list. Predictably, DOAJ is not amused and accusing Bohannon of, among other things, racism because he chose African-sounding names for the authors (1 - 2).

In August 2013, Nature journalist Richard van Noorden detailed a scheme by publishers called "citation stacking" in which a group of publishers collude to quote extensively from each other's journals in order to avoid being sanctioned for coercive citation. This activity was described in Science in 2012 by Allen W. Wilhite and Eric A. Fong as a process by which authors are instructed to quote from a publisher's own journals in order to increase the so-called impact factor. van Noorden's article focused on a group of Brazilian journals, so he, too, was accused of racism. This is unfortunate, as it detracts from a very serious problem.

We find ourselves today in a rapidly expanding world with scientific research being conducted in many different places and much money being invested in producing results. People need publications, and have little time for doing peer review, a job that is generally not paid for and performed as a service to the community. Universities in countries without a tradition of rigorous scientific practice have researchers who need publications, and there are people out to make money any way they can. Researchers competing for scarce jobs in countries that are trying to spend less on science and education than they have in the past are also sometimes tempted to follow the path of less resistance and publish with such journals. And some are not aware that they have just selected a publication that sounds like one that is well respected, as Beall has noted.

I don't have a solution to offer, other than boycotting the use of quantitative data about publications and getting people to be aware of the scams going on. We need to get serious about peer review, embracing such concepts as open access pre- and post-publication peer review in order to get more rigor into the publication process. I realize that people have been complaining about the decline of science since at least Charles Babbage (Reflections on the Decline of Science in England, And on Some of Its Causes, 1830). But we are in grave danger of letting bad science get the upper hand.

And what happens to those who try and point out some of the dicier parts of science? Nature just published another article by van Noorden, together with Ed Yong and Heidi Ledford, Research ethics:  3 ways to blow the whistle.

Update 2013-12-01: fixed a typo

Nature wins libel trial

Nature won a long-running libel suit in July brought against them in England by Mohamed El Naschie on account of the following article (which is now back online) by Quirin Schiermeier : Self-publishing editor set to retire: Criticism grows over high number of self-penned articles in physics journal. (Published online 26 November 2008 in Nature 456, 432 (2008) | doi:10.1038/456432a)

Nature has both published a report on the outcome of the suit and an article by the author of the original article describing the process. The main focus of the 2008 article had been about how El Naschie had been self-publishing in an Elsevier journal (Chaos, Solitons and Fractals), of which he was editor-in-chief, at a very unusual rate, and that he was retiring from this position. Schiermeier was looking into how exactly this enormous publishing output was happening.

Schiermeier was also curious to know more about this person, as there were a number of blogs that were criticizing him, for example the El-Naschie Watch, but that were being threatened with litigation. Threats of legal action in science are often in inverse proportion to the scientific value of the contested results.  Schiermeier notes:

Still, unlike other science journalists, such as Simon Singh, who had previously had to go through the libel ordeal on their own, I was in the comparatively comfortable situation that my employers had the resources, the stamina and the willpower to take the case on.

Síle Lane, a campaigns manager at Sense About Science in London, is reported by Nature as noting that "[i]ndividuals who lack the financial support of an institution capable of defending against a libel action will probably look at the amount of time and money it took to defend this case and decide that they shouldn’t speak out."

The judgement ([2012] EWHC 1809 (QB)) is available online at Nature and its 91 pages make a fascinating read. I think it should be required reading for anyone dealing with the scientific publishing process, peer review, impact factors, or junk journals and vanity publishing.

The Honorable Mrs. Justice Sharp, DBE, the judge for this suit, should have been made DBE for her thorough analysis of the situation, but she already is. She looked at the norms of scientific publishing, the peer review system, an the problem of journal self-citation by calling on expert witnesses.

In paragraph 90 we see that El Naschie published 58 papers in 2008 (the year he was being forced to step down as editor). That would be more than one paper a week, and the next most frequent published author at the journal that year had only 9 publications. The next paragraphs contain some fascinating statistics comparing the publication record of other editors-in-chief of top-rate journals. 

In paragraph 114 in response to El Naschie's statement that no peer group existed for assessing his theories, so his only recourse was to self-publish, Sharp states: "We are not living in the age of Galileo. Scientists or those with a keen interest in science as informed amateurs are free to publish and thus disseminate their ideas to the outside world: and in the age of the internet, there are many platforms by which they can do so."

Paragraph 120 details the expert witness of Professor Neil Turok, professor for applied mathematics and theoretical physics. His gruesome task was to actually read and analyze all 58 papers published by El Naschie in 2008 in the journal in question. What an enormous amount of effort!

120.
Through Professor Turok, it is said by the Defendants that on analysis, the 58 papers or articles contained the following defects:
i) A failure to define terminology and concepts, including in particular a failure to present the principles and equations of “E-infinity theory” and the predictions which are said to be deduced from it;
ii) Strongly expressed conclusions, unsupported by any, or any intelligible process of logical reasoning; in particular, the repeated unexplained reliance on numerical coincidences in support of the assertion that the Claimant’s “E-infinity theory” is correct;
iii) Statements which are meaningless or obscure, even to a readers with expertise in the field of theoretical physics;
iv) Statements which are simply wrong;
v) Elementary errors of spelling and grammar;
vi) A lack of any, or any substantial, contribution of new knowledge to the field;
vii) An excessive degree of citation of other articles written or co-written by the Claimant, in particular in order to justify assertions which should have been
supported by self-contained argument or references to the work of independent authors (the articles published by the Claimant in CSF in 2008 contained approximately 301 citations of his own articles in CSF, including citations of “in press” articles: i.e. those articles which were due to be, but which had not at the material time, been formally published);
viii) The use of those articles to advertise other articles by the Claimant.

The judgement then goes on to examine each and every one of these defects. In paragraph 122 already Sharp notes: "But to paraphrase Samuel Johnson, you do not have to be a carpenter to see when the legs of a table are uneven, or even to see that it has no legs at all." On questioning about peer review, Turok states in paragraph 169: "Peer review is essentially what separates theoretical physics from chaos."

It goes on, paragraph after paragraph detailing the bizarreness of the case. This includes the witness Anke Boehm who worked for El Naschie insisting that it was not the policy of the journal to keep reviews and papers once they had been published (paragraph 182) and then speaking of "deleting" the reviews when El Naschie insisted that everything was only done on paper and the he does not use email.  Paragraph 290 notes that El Naschie admitted that some of the people who were speaking for him or working at the journal just did not exist.

The judgment also includes material from Elsevier that makes it clear that even they found the journal problematic (paragraph 228). An internal review of the journal is quoted as stating “Although the journal has a growing impact factor, we are concerned that this is too dependent upon self-citation […] The journal appears to have a high acceptance rate with many accepted papers coming from China." I don't normally praise Elsevier, but I am glad to see that they took the initiative here in terminating the editorship of El Naschie.

I wish to thank Nature for standing up for the name of science reporting and against junk journals! It cost them much time and money, I do hope that they can get the money for the lawyers back from El Naschie, which will probably not be an easy task.

Impact Factor

The blog Mathlog comments in German on a publication in SIAM News by Douglas N. Arnold Integrity Under Attack: The State of Scholarly Publishing.

The publication stems from a report on a particularly disturbing plagiarism series involving 5 plagiarisms by the same duo, M. Sreenivas and T. Srinivas. It seems that the first author was a student of the second one and used the professor's name on the paper without his knowledge. However, Srinivas had the papers listed on his publication list at the department.

The publication goes on to list a number of cases, many of which have been discussed here. A new one is the impact factor of the International Journal of Nonlinear Science and Numerical Simulation (IJNSNS). I quote a paragraph here from Arnold's paper:

Let me describe another element that contributes to IJNSNS's high impact factor. The Institute of Physics (IOP) publishes Journal of Physics: Conference Series (JPCS). Conference organizers pay to
have proceedings of their conferences published in JPCS, and, in the words of IOP, “JPCS asks Conference Organisers to handle the peer review of all papers.” Neither the brochure nor the website
for JPCS lists an editorial board, nor does either describe any process for judging the quality of the conferences. Nonetheless, Thomson Reuters counts citations from JPCS in calculating impact factors.
One of the 49 volumes of JPCS in 2008 was the proceedings of a conference organized by IJNSNS editor-in-chief He at his home campus, Shanghai Donghua University. This one volume contained 221 papers, with 366 references to papers in IJNSNS and 353 references to He. To give you an idea of the effect of this, had IJNSNS not received a single citation in 2008 beyond the ones in this conference proceedings, it would still have been assigned a larger impact factor than any SIAM journal except for SIAM Review.

Hmm - I have a number of Chinese conferences on my dodgy list. Perhaps Chinese conferences should be excluded from the calculation of impact factors? And journals need to institute some real peer review (and not just payment for publication). It also shows that the impact factor is broken, if it can be easily gamed like this. Arnold summarizes his criticism of reliance on bibliometrics: "Counting can never replace expert opinion."