It’s the best of times, and the worst of times, in bioscience publishing. More papers are being published than ever before, reflecting enormous progress in biomedical research. But despite this real and exciting progress, all is not well in this world. Why? A major reason is the widespread belief that scientific publications are increasingly irreproducible. High profile retractions garner enormous attention and understandable calls for explanation and remediation. Peer reviewed publications are the coin of the realm in academia, so we must take concerns about their irreproducibility extremely seriously.
Published research, whether basic science or clinical, must be performed rigorously, with meticulous attention to good scientific practice. But scientists are fallible, and errors will always exist. Identifying errors and correcting them is central to the way science works. Honest errors must be distinguished from fraud or other forms of scientific misconduct. While the latter is always unacceptable, it’s almost certainly responsible for a small fraction of irreproducible results in the scientific literature.
Are irreproducible results more prevalent today? Well-controlled studies to assess prevalence don’t exist, but irreproducibility is likely to be identified more frequently now—even if it’s no more common. Online access to published results, sometimes including raw data, allows more widespread scrutiny. And today we have individuals and organizations, such as Retraction Watch, whose main goal is to identify and publish such claims.
One indicator is clear—the number of retractions is increasing. And some bio-pharma scientists have written that a high fraction of basic science findings from prominent academic labs in fields such as cancer cannot be reproduced within their companies. While this claim can’t be taken at face value without published verification of the primary data, which is not commonly provided, the fact that such views are common within industry is a cause for concern. Companies that question published results from academic labs may have competitive reasons to keep these results private, rather than publish them. However, it would strengthen the scientific enterprise if they could be persuaded to do so.
The causes of irreproducible research fall into three main categories. First are deficiencies in how investigators conduct and analyze their studies, including problems in design, statistical analysis and reagents. A second cause relates to incentives that may tempt, or nudge, susceptible investigators to violate good scientific practice against what should be their professional values and judgement. These incentives include career advancement, promotion, grant funding and recognition, as well as possible financial conflicts of interest. These factors fall squarely within the responsibility of academic institutions such as HMS. The institutional response to conscious violations of good scientific practice should be severe in order to protect research ethics and the trust that the scientific enterprise requires.
A less commonly discussed contributor to irreproducibility relates to how papers are reviewed and accepted for publication—the scientific publishing ecosystem. Since their origin during the 17th century, scientific journals have been the major vehicle for disseminating science. And so it’s surprising how little research is done to determine how best to deliver the results of research through publishing.
I believe that scientific journals can do more to enhance the validity of the work that they publish. How? First, we should consider two simple ideas. The vast majority of journals keep reviewers anonymous to the authors, and with a few notable exceptions, do not publish reviewers’ comments or editors and authors responses. The dominant reason for reviewer anonymity is to protect them from retaliation by unhappy authors. But this practice also promotes self-serving or superficial reviews, so reviewer identification should be encouraged, perhaps through an opt-out mechanism. A strong case can be made that journals should also publish communications that occur between reviewer, editor and author. In addition to being instructive to those seeking to understand the work, this information would serve as data for independent research on the quality of the peer review process, which is now impossible with the reviews and editorial communications hidden from view.
There is also increasing interest in reversing the sequence between publication and review. In physics, nearly all manuscripts are posted online prior to review at one or more repositories, and a national conversation has begun on encouraging such a process in the biosciences. F1000Research publishes papers online prior to review and then solicits open reviews from a panel of reviewers and/or the wider scientific community. This site also encourages publication of confirmatory studies. Although critical to scientific progress, confirmations are of vastly less interest to journals, so new mechanisms to facilitate their timely publication are needed.
Papers are now discussed online after publication in new ways. PubPeer creates a mechanism for open dialogue between the public and authors. In this way, concerns about papers may be brought more quickly to the attention of scientists and publishers. Research on the benefits and risks of this approach are needed, since anonymity of comments has sometimes evolved into what may appear to be harassment of authors. Finally, the mechanisms for correcting mistakes in published papers, whether by retraction or simple correction, are made unnecessarily difficult by publishers, which allows errors to be propagated longer than necessary.
Despite these concerns, the great majority of scientists conduct research to a high standard and publish reproducible results that contribute to the forward march of scientific progress. But concerns about reproducibility are troubling. We must certainly improve research training offered to our students, and sometimes to our faculty. We must better anticipate, manage, and where possible, change those incentives that promote unacceptable behaviors by a subset of susceptible scientists. In parallel, we should encourage publishers to be more transparent and accountable, providing data to facilitate research that will enable improvements to the review process. With all of the above efforts, we will publish more durable and reproducible research, maximizing the results of societies’ critically important investment in biomedical research.