PubMed Commons closure: a step back in post-publication peer review

Open discussion is, or should be, a natural part of the scientific process. With so much literature being published annually, and such a large population of biomedical scientists, one would think that there would be so much to discuss. Despite this, very few integrated discussion platforms exist today that allow for academics and members of the public to interact about biomedical research and the published literature. The most common forms by which academics communicate to each other are personal communication, such as via email, where queries or concerns can be relayed directly to authors by other academics, on personal blogs that may have large visibility or be lost in the masses of websites, academic journal clubs at academic institutes where members of a department, including faculty members and students, can have a group discussion about a paper or the published literature on a topic (1).

Why would academics want to discuss a published paper? Given that science is in most cases a slowly incremental process that builds on earlier achievements, there are ample reasons for biomedical researchers to hold a discussion about a paper, including the application or robustness of a methodology prior to investing in it, the feasibility of a method when human, financial and other resources have been budgeted, the importance or value of a published paper’s results, and possible detection of errors or fraud. Most academics, however, wish to shy away from conflict or scandal, and the vast majority of journal clubs tend to focus on the positive merits rather than on the negative demerits, highlighting the latter only in cases where the experiment may have failed to replicate.

One current example that biomedical researchers can relate to are the CRISPR (clustered regularly interspaced short palindromic repeats) bacterial immune systems that allow for gene and genome editing such as CRISPR/Cas9 that allows for RNA-guided DNA-targeting (2). In May of 2016, Han Chunyu of Hebei University of Science and Technology published a paper (3) in Nature Communications on a gene editing tool, Natronobacterium gregoryi Argonaute (NgAgo) that challenged the supremacy of the CRISPR/Cas9 system. However, many keen biomedical geneticists eager to test and apply this new system to their research, and who invested considerable resources into doing so in their laboratories around the world, were unable to replicate those findings, negative failures that were reported to the journal and on social media. By the end of 2016, the journal had issued an expression of concern, and by August of 2017, the paper had been retracted, dealing a negative blow to the integrity of biomedical genetics and also Chinese-based research.

The ability to discuss the published literature and to critically assess it, especially in the light of failed traditional peer review (4), is broadly termed post-publication peer review (PPPR) (5). PPPR has strengthened in the past few years and three main online platforms evolved in the discussion: PubMed Commons (PMC), PubPeer, and Publons (6). Whereas PMC only allows signed comments, in which the commentator’s identity can be identified, PubPeer allows for signed and anonymous comments, but is mostly used by the latter, while Publons also allows for both but is serving mostly as a repository for peer review as a rewards system platform. However, questions are being raised about the ability of Publons to address the quality of peer review and improve it (7).

PMC (https://www.ncbi.nlm.nih.gov/pubmedcommons) was born on October 22, 2013, a project by the National Institutes of Health (NIH) and National Center for Biotechnology Information (NCBI) with the support of Prof. Robert Tibshirani of Stanford University’s METRICS. Although the precise team that made up PMC was never made public, except for the involvement of Hilda Bastian, the initial response to PMC was positive and it had, until February 3, 2018 attracted 30 online journal clubs who were contributing to commenting on published articles on this online PPPR platform. However, a sudden and brief anonymous notice by PMC on February 1, 2018, indicating that PMC would shut down by mid-February, came as somewhat of a sudden shock to PPPR protagonists. The stated reason was that only 6,000 of the 28 million PubMed-indexed papers had attracted comments, i.e., the PMC project had failed to promote and solidify PPPR within the biomedical community.

There are a few possible reasons why PMC may have failed. Since commenting had to be signed, i.e., commentators’ names always had to be identified, commentators most likely would not have felt comfortable commenting on papers by competing researchers or research groups, either for fear of professional retaliation, or ridicule. Perhaps a few comments may have been seen as “acceptable”, but what if an academic had much to say about many papers, or were to point out flaws across a wide swathe of papers? Without a doubt, despite doing something good, i.e., offering a critical analysis of the published literature, there is no doubt that such an academic would face backlash, professional criticism and ridicule. One of the most prominent show-downs that took place at PMC was in 2015 and 2016 between Professor Michael Blatt and PubPeer’s co-founder Boris Barbour, which to some extent defined the negative tone, the complexities of moderation, and the risks involved with named commenting, especially when opinions were passionate, or stronger. That PMC-based battle, historic to the PPPR movement, lies at the heart of the anonymous versus named PPPR ideological clash of this decade, where some academics feel that safety in being critical of a published paper can only take place behind an anonymous mask, such as at PubPeer (8).

It is undeniable that the sudden death of PMC is a set-back for progressive PPPR, but it is not, in any way, the death of PPPR. PMC represented, to some extent, an acknowledgement that PPPR and the public critique of the scientific literature was not only mainstream, but that it was recognized by a leading biomedical indexing agency. To some extent, the death of PMC after less than 5 years of existence and weak functionality, places PPPR back into the underground movement of literature critique, supplementing anti-science and science-critical blogs and websites that have mushroomed everywhere on the internet.

Biomedical science around the globe is being produced at unprecedented levels, with a concomitant level of publications of varying quality, and there are umpteen challenges facing the biomedical publishing industry (9), especially as it attempts to transition to an open access-only state. The literature is in a tumultuous state, as is the PPPR movement, or perhaps it is in a state of dynamic evolution, as evidenced by the death of PMC, an unimaginable fate just a year ago. However, even though PMC is now gone, a literature fraught with errors or fraud still remains to be explored and corrected. Advancing PPPR in a post-PMC world is anything but impossible, but it brings some sadness to know that there is now one less tool available for critical academic discussion.

Acknowledgements

Funding: None.

Footnote

Conflicts of Interest: The author has completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/amj.2018.02.07). The author has no conflicts of interest to declare.

Ethical Statement: The author is accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Topf JM, Sparks MA, Phelan PJ, et al. The evolution of the journal club: from Osler to Twitter. Am J Kidney Dis 2017;69:827-36. [Crossref] [PubMed]
2. Wang H, La Russa M, Qi LS. CRISPR/Cas9 in genome editing and beyond. Ann Rev Biochem 2016;85:227-64. [Crossref] [PubMed]
3. Gao F, Shen XZ, Jiang F, et al. DNA-guided genome editing using the Natronobacterium gregoryi Argonaute. Nat Biotechnol 2016;34:768-73. [Crossref] [PubMed]
4. Teixeira da Silva JA, Dobránszki J. Problems with traditional science publishing and finding a wider niche for post-publication peer review. Account Res 2015;22:22-40. [Crossref] [PubMed]
5. Teixeira da Silva JA. A PPPR road-map for the plant sciences: cementing a road-worthy action plan. J Educ Social Res 2015;5:15-21.
6. Teixeira da Silva JA, Al-Khatib A, Dobránszki J. Fortifying the corrective nature of post-publication peer review: identifying weakness, use of journal clubs, and rewarding conscientious behavior. Sci Eng Ethics 2017;23:1213-26. [Crossref] [PubMed]
7. Teixeira da Silva JA, Al-Khatib A. The ClarivateTM Analytics acquisition of Publons – an evolution or commodification of peer review? Res Ethics. Available online: http://journals.sagepub.com/doi/abs/10.1177/1747016117739941
8. Teixeira da Silva JA, Blatt MR. Does the anonymous voice have a place in scholarly publishing? Plant Physiol 2016;170:1899-902. [Crossref] [PubMed]
9. Teixeira da Silva JA, Shaughnessy MF. An interview with Jaime A. Teixeira da Silva: insight into improving the efficiency of the publication process. North Am J Psychol 2017;19:325-38.