Reproducibility vs replicability: what's the difference?

Two words for two different questions

In 2019 the US National Academies of Sciences, Engineering, and Medicine published a consensus report, Reproducibility and Replicability in Science, which fixed definitions meant to work across fields. In its words, reproducibility is "obtaining consistent results using the same data and code as the original study", and the report treats this as synonymous with computational reproducibility. Replicability means "obtaining consistent results across studies aimed at answering the same scientific question", each study using new data or new methods (summary from the Academies).

The shorthand is simple. Reproducibility: same data, same code, same numbers? Replicability: new data, similar methods, same conclusion? Usage still varies between fields, and some communities use the words the other way around, so it is worth saying which definition you mean whenever the distinction matters.

A reproducibility check is a question about the record, not the idea

A reproducibility check takes the paper's own materials and asks whether they still produce the paper's own numbers. It cannot say whether the finding is true in general. It can say whether the published code, data and description are complete and consistent enough for someone else to get the same result.

That is what OpenRepro does. For each paper, the claims to be checked and the tolerance for each are registered before any result is recorded, the authors' code is run at a version declared in advance, and the outcome is computed from the record. Synthetic Difference-in-Differences is a clean example: the authors' Table 1 code, run at the synthdid commit named in their session information with R 4.0.5, gave every entry of the table at the reported precision, with point estimates within 0.05 of the published values. The paper reproduced. Whether synthetic difference-in-differences is a good estimator is a different question, and a reproduction does not answer it.

From Louvain to Leiden reproduced in the same sense: the authors' Java implementation at the release tag matching the paper gave the Table II modularity values on the DBLP and Amazon networks within 0.0005, and Leiden beat Louvain on both, as the paper says.

Why a reproducibility check can fail when nothing is wrong with the science

Because reproducibility is about the record, the usual reasons for failure are bookkeeping, not fraud or error in the idea, and some of the bookkeeping errors are in the re-run itself. The ledger shows the main kinds.

The re-run can depart from the paper. Our first re-run of ROCKET ran the repository's 2022 code rather than code from the paper's date, without declaring a code version first, and our first re-run of Dropout as a Bayesian Approximation trained with batch size 128, the value in the committed scripts, where the paper and the authors' earlier code use 32. Both cases are described below, and both are listed on What we got wrong.

A setting can differ between the paper and the released code. For Dropout as a Bayesian Approximation, the result files in the repository match Table 1 exactly, but they were committed a day before the experiment scripts were rewritten, and the rewritten scripts train with batch size 128 where the paper and the earlier code use 32. Our first re-run used the committed scripts: Boston (3.09 against 2.97) and Wine (0.624 against 0.62) held, and Concrete, Energy and Yacht did not. A paired test sealed before training then ran those three at both batch sizes. At batch 32, five of the six re-tested values fell within twice the archived standard error and the sixth, Yacht's log-likelihood, was inconclusive; at batch 128 the earlier misses came back. Two settings remain unconfirmed: the dropout rate for Concrete and Yacht, and the original software environment.

Randomness can make a gap impossible to settle. For Semi-Supervised Classification with Graph Convolutional Networks, Cora (81.53 against 81.5) and Pubmed (79.16 against 79.0) reproduced in our first re-run, but on Citeseer 2 of 100 seeds collapsed, to 15.9% and 9.3% accuracy, and pulled the mean below the fixed band we had registered around 70.3. A fixed band ignores run-to-run spread, and the gap was smaller than one standard error of the mean. A settling test ran a fresh cohort of 100 seeds, judged by an interval rule with a precision gate sealed before training: the mean came out at 70.37, compatible with 70.3, but the interval was wider than the gate, so the outcome is inconclusive. One run in that cohort collapsed after early stopping too, and it stays included.

A claim can sit on a knife edge. ROCKET claims the best mean rank among eight classifiers on the 85 bake-off datasets. A first re-run used the repository's 2022 code; its accuracies left ROCKET and TS-CHIEF tied, and a tie rule chosen after the tie was seen marked the claim as missed. A corrective re-run declared the 2019 code and the paper's own ranking rule before any result was recorded: mean accuracy came out at 0.8517 against 0.8510, and ROCKET's mean rank was strictly best (571/170 against 579/170 for TS-CHIEF). Outcome: reproduced, with the first attempt kept as a superseded version. The margin is small enough that run-to-run variation in ROCKET's unseeded random kernels can move it, and the paper itself notes that the differences among the top methods are not statistically significant.

Why check reproducibility first

Reproducibility is the cheaper and more decisive test. A replication needs new data and a judgment about what counts as the same question; a reproduction needs only the published materials and a tolerance registered before results are recorded. If a result does not reproduce, the record usually says why: a package version, a rewritten script, an unspecified seed. If it does reproduce, a later replication starts from a known baseline instead of a number nobody has regenerated.

OpenRepro's outcomes are reproducibility outcomes. How an outcome is decided sets out the rules; the ledger lists every paper checked and what came out.

How to use the words precisely

A few habits keep the distinction useful.

• Say "reproduce" when the same data and code are re-run, and "replicate" when new data is collected.
• Name the code version. "Reproduced at commit 2408a92" is a claim; "reproduced" alone is not.
• State the tolerance. A number that comes out within 0.1 of the published value is a different finding from one that matches to six decimals, and both are different from a number that comes out within 10%.
• Say what was not tested. A reproduction covers only the claims that were checked.