What is a replication package?

The data, code and instructions that let someone else regenerate a paper's results, and the parts that most often go missing.

Updated

Definition

A replication package is the bundle of data, code and instructions that lets a reader regenerate the results in a paper without asking the authors. The term comes from economics and the social sciences, where journals began requiring one as a condition of publication. The American Economic Association's Data and Code Availability Policy requires authors to "provide the data, code, and other details of the computations sufficient to permit replication", deposited in an openly accessible trusted repository, and says the AEA Data Editor "will assess compliance with this policy", which includes conducting reproducibility checks.

Strictly, what a package supports is reproduction (same data, same code) rather than replication (new data); see Reproducibility vs replicability. The name has stuck, and data editors, journals and repositories all use it.

What goes in one

The AEA policy asks for a README in the top directory of the package and says a master script is "strongly encouraged"; where there is none, authors should give precise step-by-step instructions. The same shape serves any computational field.

  • Data: the files used, or, where they cannot be shared, the exact source, version, access instructions and checksums.
  • Code: everything from raw data to each table and figure, at the version that produced the published numbers.
  • Environment: language and library versions, operating system, hardware, and the time the main runs take.
  • A README: how to run the package, which script produces which table or figure, what is excluded and why, and known differences from the published numbers.
  • A master script, or one documented command per output.
  • A license for the code and, where possible, the data.

What re-running real packages showed

OpenRepro re-runs the authors' own packages with claims and tolerances registered before any result is recorded. Four cases show which parts of a package carry the weight.

Synthetic Difference-in-Differences, published in the American Economic Review, keeps its Table 1 code as a vignette inside the synthdid package, with a session-info block naming the package commits and R version. Run at those commits, the code reproduced every entry of Table 1 at the reported precision; the only deviations were the platform (Linux instead of an M1 Mac) and one non-estimating build dependency. The session-info was the package's most valuable file. Its one weakness: the paper-era commit is no longer on any branch and is reachable only by hash.

ROCKET keeps its experiment code in a repository that pins no library versions and was rewritten in July 2020. Our first re-run ran the repository's 2022 code rather than code from the paper's date, without declaring a code version first. A corrective re-run declared the first code commit, made 21 hours after the paper's first arXiv version, before any result was recorded, and reproduced: mean accuracy 0.8517 against 0.8510. A package whose code keeps changing after the paper should say which version produced the published tables.

The COVID-19 change-point paper bundles a snapshot of its case data. That snapshot, with the end date the paper states, reproduced all four spreading rates; the live series the notebook downloads by default has since been revised and extended to 2023. For time-varying public data, the snapshot is the package.

Dropout as a Bayesian Approximation has result files committed next to its scripts. The files match the paper, but the scripts were rewritten a day after the files were committed, and the rewritten scripts train with batch size 128 where the paper and the earlier code use 32. Our first re-run used the committed scripts and missed on three of five datasets; at batch 32, five of the six re-tested values came back within the archived uncertainty and the sixth was inconclusive. A package in which outputs and code have drifted apart is the hardest kind to assess, because the reader cannot tell which is wrong.

What a data editor is checking

A reproducibility check by a data editor, or by OpenRepro, is mechanical by design: does running the package, as documented, produce the published numbers? It does not judge the research question. The things that fail it are concrete and fixable: a missing data file, an unpinned library, a script that no longer matches its output, a seed that is fixed in the code but averaged in the text, a step described in prose but absent from the code.

The checklist for authors goes through each of these with the ledger case behind it. How an outcome is decided describes how OpenRepro registers claims, declares the code version, records deviations and computes an outcome from the log.