How to make your paper's results reproducible: a checklist

Pin the code

Name the exact version of the code that produced the tables, and make sure that version stays reachable. From Louvain to Leiden did this with a release tag (networkanalysis 1.0.0); building that tag with the authors' own release commands reproduced the Table II modularity values within 0.0005. Synthetic Difference-in-Differences did it with a session-info block naming the synthdid, MCPanel and glmnet commits; the paper reproduced at the reported precision. The synthdid commit is no longer on any branch of the repository and is retrievable only by its hash, which is a near miss: a repository clean-up could have made the paper's own code unreachable.

Tag a release or record a commit hash in the paper and the README.
Never force-push or garbage-collect the commit that produced the published tables.
If results are regenerated after review, tag the new version too and say which one the published tables come from.

Pin the environment

A method implemented in a library is at the mercy of that library's later versions, and of the helpers around it. The Conformalized Quantile Regression paper does not state library versions; the re-run patched one line of the authors' data loader, replacing scikit-learn's Imputer with SimpleImputer, and built scikit-garden from its 2022 git master because the 0.1.3 release on PyPI ships pre-generated C that does not compile on Python 3.7 or later. Three other ledger papers pin nothing: the dropout repository needed a 2016 Keras and Theano environment reconstructed from the README's hints, and its report still marks that environment as inferred; the ROCKET re-run had to pick the newest scikit-learn that still accepted the script's classifier arguments; and the COVID-19 change-point re-run guessed the releases current in May 2020.

Commit a lockfile, an environment export, a session-info, or at minimum the versions of every library whose version could change a number.
Record the interpreter version, the operating system and the BLAS, and whether a GPU was used. The SDID authors recorded their platform and noted that one estimator's cross-validation uses a platform-dependent random generator; that note let the re-run interpret its own tiny differences.
If your package keeps evolving, pin the paper's replication script, and the data generators it calls, to the paper's release, and say so in the README. ROCKET's code was rewritten in July 2020, and our first re-run ran the repository's 2022 code rather than code from the paper's date.

Pin the data

A live data source is not an input; it is a moving target. The COVID-19 change-point repository bundles a snapshot of the Johns Hopkins case data from April 2020. The re-run used that snapshot with the end date the paper states, and all four spreading rates reproduced (0.43, 0.25, 0.15 and 0.09 against posterior medians of 0.428, 0.251, 0.154 and 0.094). The notebook's default, the latest date of the live series, now points at a 2023 series that has been revised backwards. For Leiden, the paper did not say how network node identifiers were mapped or which random seeds were used, and the current SNAP Amazon file has a slightly different edge count from the one in the paper's table; neither changed the outcome, but both had to be noted as deviations.

Archive the exact data files used, or record a checksum and the download date and version.
State preprocessing in the code, not only in the text: id mapping, filtering, train/test splits.
If the data cannot be shared, share the loader and the checksums so a reader with access can confirm they have the same input.

Make the table come out of the script

The most reliable replication package is one where a single command regenerates the published table. The SDID vignette does this for Table 1 and reproduced every entry. The weakest pattern is result files committed next to scripts that no longer match them. In the dropout repository the committed results match Table 1 exactly, but they were committed a day before the scripts were rewritten, and the rewritten scripts train with batch size 128 where the paper uses 32: run as committed, they miss on three of five datasets, while at batch 32 five of the six re-tested values came back within the archived uncertainty and the sixth was inconclusive. One script was not even valid as committed. The ROCKET re-run regenerated only the ROCKET column of Table 1; its best-mean-rank claim compares that column with the other seven classifiers' published accuracies, which were not re-run.

Provide a master script, or a documented command per table and figure.
Regenerate every published table from a clean checkout before submission, and commit the regenerated outputs together with the scripts that made them.
Cover every column of a results table, not only the one for your own method.

Say how randomness was handled

If a number is an average over runs, say how many, how they were seeded and whether any were excluded. The GCN paper reports means over 100 random initializations while the script fixes one seed; our first re-run of 100 seeds found two collapsed runs on Citeseer that moved the mean outside a fixed band, and fresh cohorts of 100 then gave means close to the paper's 70.3 with intervals too wide to settle the gap. The dropout paper's Table 1 reports standard errors over 20 splits, and those errors are what allowed a sensible allowance (twice the reported standard error) to be set before its settling test.

Report run-to-run spread (a standard deviation or standard error) next to every averaged number.
Make the seed a parameter of the script rather than a constant inside it.
State the policy for failed or diverged runs before you look at the results, and apply it.

State the compute and the margin

Speed claims and "best method" claims are the easiest to lose. ROCKET's first re-run could not time training on the largest dataset under comparable conditions, and its reproduced accuracies left the best-mean-rank claim in an exact tie. A corrective re-run on a dedicated machine met all three timing claims (as upper bounds, on newer hardware, with a 50% allowance) and gave ROCKET a strict lead of 8/170 of a mean rank, a margin the record says ordinary run-to-run variation could move. The paper's own remark that the top methods are not significantly different is the kind of sentence that makes a reader trust the rest.

Give hardware, memory and wall-clock time for the main experiments, so a reader can budget a re-run.
Report comparative claims with their margin, and say whether the margin is significant.
Prefer claims that a reader can check with less compute than you had: a per-dataset table is checkable; a single aggregate rank is fragile.

Keep a README that tells the truth

One of the most useful documents in the ledger's repositories is a warning the authors wrote themselves: the dropout README's report of test-set contamination in the original hyperparameter search. It does not settle a result on its own, but it tells a reader what to expect and where to look.

Record known differences between the current code and the published numbers, with the version that does produce them.
List what the repository does not cover (datasets, columns, figures), so that nobody mistakes untested for failed.
Give a contact for corrections, and keep the README current when the code changes.

The checklist

In one place:

Code: commit hash or release tag named in the paper; the commit kept reachable.
Environment: lockfile or session-info; interpreter, OS, BLAS and GPU recorded.
Data: archived snapshot or checksums plus download date; preprocessing in code.
Scripts: one command regenerates each published table and figure; outputs committed with the scripts that made them.
Randomness: run counts, seeding and exclusion policy stated; spread reported.
Compute: hardware, memory and time given; comparative claims reported with their margin.
README: known discrepancies, untested parts and a contact.