Before You Analyze: A Due Diligence Checklist for Inherited Data

A dataset arrives by email, shared drive, or repository link. It has a README, a codebook, and a DOI. It looks clean. The temptation is to start analyzing immediately. That temptation is the beginning of many retractions.

Skepticism is not an insult to the researcher who shared the data. It is a recognition that datasets are artifacts shaped by decisions, deadlines, and undocumented workarounds. The following ten questions are a practical checklist for graduate students, reviewers, and analysts who inherit data they did not build. Each question is paired with a common failure pattern and a verification step that can usually be completed in an afternoon.

1. What exactly does the DOI resolve to?

A DOI can resolve to a landing page that says data are available upon request, or to a newer version than the one cited. Verification: open the DOI, download the files, and compare checksums with the version named in the paper. If the DOI resolves but the files do not match, the citation is not reproducible.

2. Is there a visible version history?

Silent in-place revision is common. Files are overwritten, numbers are corrected, and the landing page remains the same. Verification: look for release tags, timestamps, and a changelog. If none exist, ask the depositor which version the paper used and record the answer.

3. What does missing mean in this file?

Missing values may appear as blanks, zeros, negative sentinels, or text such as NA. Verification: tabulate every column's unique values and compare them with the codebook. A value of -999 is not a measurement; it is a warning.

4. Which transformations are undocumented?

Winsorization, log transforms, unit changes, and recoding can happen without a note. Verification: reproduce basic summary statistics from the rawest file available. If the means and ranges do not align, document the gap before modeling.

5. Does the codebook match the columns?

Codebooks go stale. Column names change; labels do not. Verification: join the codebook to the data dictionary and compare names, types, and ranges. Investigate every mismatch rather than renaming columns by intuition.

6. Can you rebuild the analytic sample?

The published N may reflect filters that were never written down. Verification: apply the stated inclusion criteria step by step and count rows at each stage. If you cannot arrive at the same N, the analysis sample is not reproducible.

7. Are units, dates, and time zones consistent?

Mixed units and shifted dates are quiet corrupters. Verification: check minimum and maximum values, duplicate timestamps, and time-zone metadata. A daily series with two observations on one date and none on another deserves scrutiny.

8. What is the license, and what does it permit?

A dataset can be technically open but legally restricted. Verification: read the license, the data use agreement, and any provenance note. If the license is absent, do not assume permission.

9. Who touched the file, and when?

Data passed through email attachments, local edits, and renamed folders. Verification: request a provenance log or checksum history. If none exists, treat the file as a convenience copy, not an authoritative release.

10. Can an independent analyst reproduce one table?

The simplest test is to rebuild one published table or figure from the deposited data and code. Verification: run it in a clean environment and record the software versions. If the table cannot be rebuilt in an afternoon, the problem is not your afternoon.

A Negative Answer Is Not a Disqualification

Not every dataset will pass every question. The purpose is not to reject data but to make risk visible. A dataset with an incomplete codebook can still be useful if the analyst documents the gap. A dataset with no version history can still support analysis if the exact file is preserved and cited. What matters is that the reader, reviewer, and future user can see what was known and what was assumed.

At BAM Dataset, verification is not an adversarial act. It is the ordinary labor that makes open data trustworthy. The ten questions above are a small investment compared with the cost of discovering, months later, that the file changed, the sample was rebuilt, or the codebook described a different study.