The Reviewer Who Never Saw the Evidence: A Case for Verified Publication

Consider what we actually ask of a peer reviewer. Evaluate whether the methods support the conclusions. Assess whether the statistics are sound. Judge whether the study is well designed. Now consider what we give the reviewer: a manuscript, a figures file, and — increasingly — a data availability statement that says nothing verifiable at all.

The reviewer cannot open the dataset. Cannot rerun the model. Cannot check whether the table labeled 'n = 247' matches the data behind it. Peer review, as practiced by most journals, is an audit of the prose without an audit of the books.

The Honor System Has a Track Record

Defenders of the status quo argue that trust, plus post-publication correction, suffices. The record suggests otherwise. In medicine especially, retraction watches and replication projects have repeatedly found that published conclusions rest on data nobody outside the original lab has examined — and that problems surface only when an outsider finally does. The checks that catch errors are almost always checks that involve looking at the evidence.

The problem is structural, not personal. Reviewers are unpaid, time-pressed volunteers. Journals have no mechanism to grant them access, no budget to fund verification, and no incentive structure that rewards catching problems before publication.

A Tiered System, Honestly Labeled

The remedy is not to hold every paper hostage to full reanalysis, which is often impractical. It is honesty about what was checked. Journals should adopt a graduated badge system:

• Data Available: Underlying data is deposited in a recognized repository and accessible to reviewers and readers.
• Data Reviewed: A qualified reviewer examined the dataset and confirmatory materials before publication.
• Analysis Verified: An independent party reran the primary analysis and reproduced the reported results.

Most papers would earn the first tier. That alone would transform the landscape. The second and third tiers would carry premium weight — and premium fees, because verification is real labor.

Pay the Checkers

That is the uncomfortable part for publishers: verification costs money, and the current model spends none of it. Journals that charge article processing charges in the thousands of dollars can fund a data-checking stipend without material strain. Professional data auditors — a role that barely exists today but should — could be staffed the way statistical editors once were.

Open access solved the distribution problem; it did nothing about the verification problem. Until journals are willing to say, on the published record, exactly what was and was not checked, readers will continue to extend trust that the system has done nothing to earn. Verification is not a courtesy. It is the product.