Next steps for preprint review infrastructure

In December, ASAPbio, EMBO and HHMI convened a meeting about preprint review, one of the latest disruptions in scientific publishing. IOI’s Research Data Analyst, Naomi Penfold, led a panel discussion about sustainable business models for preprint review services and spoke with various attendees in-person and virtually to find out how investment in infrastructure could support this open research innovation. Here’s what she learned.

Peer review is seen as a crucial part of the scientific publishing process, helping to ensure the quality and accuracy of published research. Preprint review is the act of peer review on preprints, opening up the process and outputs of peer review that are usually behind doors at journal publishers. While a peer review service that sits outside journals is not a new idea – see Axios, for example – today’s initiatives in the life sciences are poised to build upon the growing traction of preprints in biology. Preprint review initiatives are in their early days, still finding out their niche and value to scholars, and currently, only a very small number of preprints are being reviewed in this way.

This meeting brought together preprint review initiative leads with funders, publishers, and researchers to discuss policies and practices that could encourage the adoption and development of preprint review in biology. There are different views on the future for preprint review: as a replacement for journals, a complement to the existing system, and/or a training exercise to grow and diversify the reviewer pool. Overall, this meeting highlighted the opportunity to use preprints to build a more collegial and constructive culture of peer review (than that typically experienced at journals). While the focus of the meeting was on policies and practices to encourage the adoption of preprint review, including how to incentivize researchers to contribute reviews, we noted some specific needs and gaps to consider in relation to investing in open infrastructure:

• Efforts to encourage adoption need funding. As well as investing in the technical infrastructure enabling preprint review, we heard the call for funders to support initiatives that encourage scholars to try out preprint review, and that nurture the envisaged culture of collegiality. This support could be provided directly by funders through programmes for the scholars they fund and indirectly through investment in adoption-focussed projects by initiatives. In particular, in this nascent phase of preprint review, now is an opportune moment to fund initiatives focussed on improving diversity and inclusion in the scholarly communications process.
• Preprint servers will need to evolve alongside preprint review initiatives to support a seamless experience for scholars. If preprint review is to be seen as a trusted and valuable contribution, and something worthwhile for researchers to read and use, it will be important to communicate its value clearly from the points at which researchers interact with preprints. Major points of interaction today are through two of the largest preprint servers for the life sciences, bioRxiv and medRxiv. We heard several users report how preprint reviews are not easy to find on the current site design, and that the banner on each preprint stating it has not been reviewed can be misleading. The banner text for preprints that have received reviews was updated in August 2022 to read “This is a preprint. It has not been certified by a journal but peer reviews are available”. We also heard the rationale behind current design decisions at bioRxiv – we perceive that, as with any technology platform, there’s a need to balance user feedback with other technical needs and organizational priorities. We think it will be important for preprint server(s) and review services to continue to improve their user experience and design to meet the evolving needs of users in a way that makes sense, both practically and technically. Several technology requirements for preprints as a whole, including review services, have already been noted. Drawing upon the ethos of open source development here, it may be helpful for preprint infrastructure funders to nurture an ecosystem that centers the needs of a diverse research community in design processes, and supports collaboration between preprint servers and preprint review services to enable a more seamless experience for all users.
• It’s too early for preprint review initiatives to have a plan for financial sustainability. At such an early phase in the development of preprint review, the current focus of preprint review initiatives is to provide a functional service for users to try out, and in doing so hone in on their value proposition and potential revenue streams. As in any early-stage venture, non-profit initiatives seek a sufficiently stable funding runway that provides them with this experimentation time, similarly to that afforded to commercial start-ups through seed funding and support. Where possible, preprint review initiatives may benefit from sharing learnings during this early stage, as they did during the business models panel at the meeting.

• Work to understand the costs of journal publishing could help inform business model development for preprint review services. In response to Plan S, publishers have been gathering data on the breakdown of costs in journal publishing, including the cost of peer review. While we heard there have been challenges in collating accurate information, this data on costs could help inform whether and how preprint review initiatives could provide a better value service, and to whom. However, this data is not publicly available.
• Preprint review infrastructure should support appropriate privacy and security of reviewers’ identities. While there is debate as to whether peer review reports should disclose the identity of the reviewer or not, the pragmatic reality is that the preprint review ecosystem must be able to support both named and anonymous reviewing activity, at least in terms of what is publicly disclosed. For some, publicly disclosing their identity is not possible in such a competitive and biased ecosystem. Considering these factors, the infrastructure underpinning the preprint review ecosystem likely needs to support several minimal requirements:
  1. those who coordinate peer review (including journal editors) are able to validate a reviewer’s identity and manage conflicts of interest;
  2. the reader of the preprint review report is able to verify whether the review report has come from a coordinated process (as in (i) or not);
  3. funders and evaluators are able to verify a scholar’s contributions to peer review irrespective of whether they are named in a review report or not;
  4. the reviewer can trust that the information about their identity (in relation to their reviewing activity) is accessible only to those who they have consented disclosure to. Of note, differential privacy is already enabled for ORCiD profiles, and PREreview is one preprint review initiative with this kind of capability.

We also heard how current initiatives include those to develop the necessary metadata and processes to connect preprint reviews with the rest of scholarly communications, and several discussions at the meeting centered on understanding how to define preprint review (what is a review? Who can be a reviewer?). There is plenty more yet to be discussed about how preprint review infrastructure will work: for example, where should preprint review content be hosted? How should reviews be licensed? Should preprint reviews be archived and preserved for the long term? We look forward to continuing discussions.

For more information, please find video recordings and other outputs from the two-day meeting available online from ASAPbio at https://asapbio.org/recognizing-preprint-peer-review.

Acknowledgement

Thanks to Jessica Polka and Iratxe Puebla for reviewing this blog post, to Sam Hindle for comments, and to the meeting attendees for thought-provoking and informative discussions.

2023-03-03: This blog has been updated to improve the accuracy and clarity of the text discussing how ‘Preprint servers will need to evolve alongside preprint review initiatives to support a seamless experience for scholars’.