Science writers should be careful when reporting on preprint studies

Story by Ryan Prior
Photography by David Jen

Science is progressing hour by hour, and the public needs to know. Now.

Such was the attitude among readers and journalists in the early weeks and months of the COVID-19 pandemic. With the fate of civilization seemingly hanging in the balance, the careful, deliberative work of peer-reviewed science seemed at odds with the prospect of lives lost if early findings about the novel coronavirus weren’t sent out rapidly to the public.

So how and when should public information officers or journalists write stories that rely on findings from scientific papers uploaded to “preprint” servers?

That was the question posed throughout the Oct. 22 ScienceWriters2022 panel “Perplexing Preprints: Insights, Common Fears, and Real Strategies” (#SciWriPrePrints). It was organized by Dawn Levy, a science writer and public information officer at the Oak Ridge National Laboratory, alongside Tom Ulrich, associate director of science communications at the Broad Institute of MIT and Harvard.

Ulrich is himself a veteran of the debate, having appeared on NASW panels about preprints in 2018 and 2020. Nearly three years into a pandemic, the preprint paper has become a “new normal” for COVID-19 news, but hasn’t yet taken over health reporting more broadly. Some 29% of COVID-19 preprints received news coverage, while just 1% of non-COVID-19 preprints got picked up, noted Alice Fleerackers, a PhD candidate at Simon Fraser University who is conducting research into preprints.

Within that gap, it’s worth explaining to readers that findings are provisional: during the peer review process, 17% of COVID-19 preprints underwent major changes. Knowing how to interpret the larger swath of data — and make personal and public decisions amid uncertainty — is vital.

“Many journalists mention that preprint sources muddied the conversations about important topics like children and vaccines or reopening schools, and this was something they were keenly aware of when they were balancing the risks and benefits of deciding to cover them,” Fleerackers said. “Just because I have a PhD in this field doesn’t mean I have the expertise to look at a paper and perfectly judge it.”

Session organizers Dawn Levy (left) and Tom Ulrich speak at the SciWriPrePrints session at ScienceWriters2022. (David Jen for NASW)

But while preprints have been gaining market share in the more rarefied world of science writers, the wider public doesn’t have a handle on how to interpret them, or even what they are. Just 25% of audiences could define the word “preprint,” according to Fleerackers’ research. That’s why if journalists are reporting on preprints, it’s particularly important to explain to readers that the study data is preliminary and has not yet cleared the rigorous peer review from other scientists required to publish in a scientific journal.

Laura Helmuth, the editor-in-chief of Scientific American, noted that when framed correctly, science writers can turn a preprint’s study into a feature, rather than a bug. A study’s preprint status can make a story more exciting, giving readers an insight into the cutting edge of science and critical debates within the scientific community.

Helmuth noted that while preprints are new to health reporting, they’ve long been the norm in reporting on space and physics. Prior to the pandemic, health journalists primarily leaned on preprints only in high-stakes stories such as the first gene-edited human embryos. She emphasized that in the case of COVID-19, countless lives saved were likely saved because of preprints. But science can also be distorted for years if incorrect studies are given air time — such as now widely discredited and retracted research linking autism and vaccines.

As a general principle, she cautioned against reporters relying on any single studies to report any story. Like any study, a preprint is a single tile within the broader mosaic of a particular field. They are useful as news pegs, but reporters should vet them with outside sources weighing in. In this way reporters conduct their own form of journalistic peer review.

“We’re not going back. Preprints are here to stay,” Helmuth said. “Peer review is evolving.”

For those closely tracking trends in how institutions and journals are evolving on these topics, Ulrich recommended a few resources, including the ASAPbio “Preprints in the Public Eye” project and TRANSPOSE, a database of journal policies on preprints and peer review.

“One thing that helps us here is that our audiences are much more aware of how science works than they were three years ago,” Helmuth said.

Ryan Prior (@r_prior) is a science writer from Atlanta. He is currently a fellow and journalist-in-residence at The Century Foundation, a progressive think tank in Washington, DC. He was a features writer for CNN during the COVID-19 pandemic, and is the author of the new book The Long Haul: Solving the Puzzle of the Pandemic’s Long Haulers And How They Are Changing Healthcare Forever. He has also written for USA Today, The Daily Beast, Business Insider, STAT News, The Guardian, and WebMD.