# OpenRepro

> OpenRepro re-runs published computational research and publishes tamper-evident evidence of whether the results hold.

This file is generated from the evidence reports behind https://openrepro.org/. 7 papers listed; ledger updated 19 Oct 2026.
Contact and right of reply: hello@openrepro.org. Authors of any paper listed have a right of reply; their notes can be published alongside the report.

## For agents

How to answer "does <paper> reproduce?" from this site:
- Find the paper in the list below (or in ledger.json) by arXiv id, DOI or title, and cite its report URL.
- Report the outcome in OpenRepro's own words, with its qualifiers (for example a broad bound, a stricter check that missed, or results left untested for budget) and its assurance level. Outcomes are computed from each paper's headline claims, judged against tolerances registered before results were recorded (legacy attempts) or sealed before scored execution (only where a report's assurance says so). "Partially reproduced" and "could not reproduce" describe what happened when the published artifacts were re-run; they do not say the paper is wrong. "Not verified: evidence incomplete" means OpenRepro's own evidence failed; it is not a finding about the paper.
- Each re-run is carried out by an AI agent (Claude) in a sandbox and no domain expert has reviewed the reports. Say so if the user is relying on the result.
- If a paper is not in the ledger, say only that OpenRepro has not released a report on it. Do not guess an outcome. Re-runs can be requested at hello@openrepro.org.
- The verifier at https://openrepro.org/verify/ checks a bundle's log and artifact integrity and that the report names the log's head hash; it does not check the report's contents. https://openrepro.org/release.json lists SHA-256 digests of every released report and bundle; it is published on the same site, so it is not an independent anchor.

## Data, feed, MCP and citation

- [Data page](https://openrepro.org/data/): what each file holds, the schema, stable ids, update cadence and license
- [ledger.json](https://openrepro.org/ledger.json) and [ledger.csv](https://openrepro.org/ledger.csv): one record per listed paper (outcome, qualifiers, assurance, headline claims held, attempt lineage, links, log head hash)
- [claims.json](https://openrepro.org/claims.json) and [claims.csv](https://openrepro.org/claims.csv): one record per registered claim (published value, observed value, tolerance, check kind, verdict)
- [release.json](https://openrepro.org/release.json): the release manifest, every released attempt with SHA-256 digests of its report files and bundle
- [feed.xml](https://openrepro.org/feed.xml): Atom feed, one entry per published report
- [CITATION.cff](https://openrepro.org/CITATION.cff): how to cite the ledger; every report page has its own BibTeX and plain-text citation
- MCP server (read-only, no key): https://mcp.openrepro.org/mcp. Tools: check_paper, search_reports, get_report, how_to_verify, request_rerun. In Claude Code: `claude mcp add --transport http openrepro https://mcp.openrepro.org/mcp`
- License: reports, verdicts, evidence logs and these files are CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Cite as "OpenRepro, <report URL>".

## How an attempt works

1. One paper and the authors' own published code are chosen. Which version of the code counts is declared before any result is recorded: normally the code at the date of the paper version being tested. An attempt may also declare, in advance, that an earlier version of the authors' code may be tried if a claim misses; earlier-code results are reported separately and never replace the primary result.
2. Each number to be checked is registered as a claim: the published value, the exact quote from the paper, and a tolerance with the reason for it. Some claims are marked headline; the outcome is computed from those. Claims registered after results are seen are flagged.
3. The experiments are re-run in a resource-capped container sandbox and the software environment is recorded. Differences from the paper's setup (CPU instead of GPU, fewer runs, a subset of datasets) are recorded as deviations with an impact of none, minor or material.
4. Observed values are checked against the output of the logged command they cite. A value that cannot be found there is flagged in the report, not excluded.
5. The outcome is computed by fixed rules from the registered claims (including which are headline, a choice the agent makes when registering them) and the logged results. Qualifiers (bound check, checks missed, untested for budget) and an assurance level are published with every outcome.

Each attempt is carried out by an AI agent, Claude (made by Anthropic), operating OpenRepro's engine in a sandbox (a Claude Code session). Each run is driven by the AI with no person in the loop, and the log's actor field records this. A person oversees the project, chooses the papers and sets the rules. No domain expert has reviewed the reports. Authors get a right of reply, and their notes can be published alongside the report. The agent does not type the outcome: it is computed by fixed rules from the claims, tolerances and headline choices the agent registered and the results it recorded, so those choices shape it.

Every command, output and decision is an event in an append-only, hash-chained log, so the record is tamper-evident relative to a head hash you hold: if your copy of the head hash matches, the log was not edited after that hash was recorded. The head hash is published on this same site, and independent anchoring is pending, so the chain does not by itself show when a log was written. It does not prove that command output is honest; the recorded code, commits and environment are what let results be re-run. Each released attempt's log, artifacts and report are downloadable as an evidence bundle (https://openrepro.org/bundles/). A one-file verifier (https://openrepro.org/verify/) checks log and artifact integrity and that the report names the log's head hash; it does not check the report's contents. Re-deriving the report from the log is the engine's `repro verify`, which is not yet public. The release manifest (https://openrepro.org/release.json) lists SHA-256 digests of every released report and bundle.

## Outcome definitions

- Reproduced: Every headline claim held within tolerance using the primary code version declared before any result was recorded (normally the authors' code at the paper date).
- Reproduced only with earlier code: All headline claims held, but some only with an earlier version of the authors' code, under a fallback declared before any result was recorded.
- Partially reproduced: Some headline claims held (with primary or declared earlier code) and others did not, or were untested.
- Could not reproduce: None of the headline claims held, with the primary code or any declared earlier code.
- Not verified: evidence incomplete: OpenRepro's own evidence for the attempt is incomplete (for example a result without its supporting log evidence, or a source quote that could not be resolved), so no reproduction outcome is given. It is a gap in OpenRepro's record, not a finding about the paper.
- Not attemptable: The claims could not be tested at all, for example because the code or data is unavailable.
- Incomplete: The attempt has not finished, or it tested none of its headline claims. Incomplete attempts are not listed.
- Incomplete: A claim that was sealed into the plan was withdrawn after the seal. The engine keeps it in the scored set as withdrawn and gives no reproduction outcome ("Incomplete: plan amended after the seal"). Amended attempts are not listed.

Claim verdicts: Held (within tolerance with the primary code); Held with earlier code (within tolerance only under the declared earlier-code fallback); Did not hold; Not tested.

## Papers

### Neural Collaborative Filtering

Does Neural Collaborative Filtering (He et al., 2017) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).

- Authors: Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, Tat-Seng Chua
- Published: arXiv, 2017. arXiv 1708.05031 (https://arxiv.org/abs/1708.05031)
- Field: Recommender systems
- OpenRepro page: https://openrepro.org/papers/1708.05031/
- Full evidence report: https://openrepro.org/evidence/arxiv-1708-05031-20261019T135218Z.html
- Checked: 19 Oct 2026
- Outcome: Reproduced (4 of 4 headline claims held)
- Qualifiers: some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: The authors Keras code (commit 312aff0, identical in code to the current default branch) was run on MovieLens-1M with the exact README environment (Python 2.7.15, Keras 1.0.7, Theano 0.8.0) on CPU, using the README commands with 8 predictive factors and 20 epochs. All six pre-registered Table 2/3/4 values at 8 factors landed within 0.005 of the paper: NeuMF without pre-training HR@10 0.6848 vs 0.688 and NDCG@10 0.4088 vs 0.410; NeuMF with pre-training (initialised from the authors bundled GMF/MLP weights rather than retrained ones) 0.6798 vs 0.684 and 0.4054 vs 0.403; MLP-3 0.6707 vs 0.671 and 0.3979 vs 0.399. Each value is one run reported at the best-HR epoch, which is the scripts own convention. Mid-attempt the shared cloud droplet was destroyed by another process; the environment and code were rebuilt with the same recipe and only the in-flight MLP run was repeated.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: hexiangnan/neural_collaborative_filtering at 312aff0
- Work tree when the attempt ended: hexiangnan/neural_collaborative_filtering at 312aff0
- Primary code version declared before any result was recorded: github.com/hexiangnan/neural_collaborative_filtering at commit 312aff0 (2017-08-21, 'Change the name of Data'), the last commit on or before the arXiv v2 date 2017-08-26 (v1 was 2017-08-16; the PDF tested is v2). Event 9 shows HEAD 4aab159 differs from 312aff0 only in README.md, LICENSE and Dockerfile, so the code is identical to the current default branch.
- Log: 67 hash-chained events, head hash 04ac3824830007550ce23545b33c04ec2f8c290dd7107cca03355bd8ce4ddfc6
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches HR@10 = 0.688 on MovieLens-1M. Published 0.688 ratio; observed 0.6848 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 0.995× the published value. (Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' HR@10 column), p. 7)
  - NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches NDCG@10 = 0.410 on MovieLens-1M. Published 0.41 ratio; observed 0.4088 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 0.997× the published value. (Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' NDCG@10 column), p. 7)
  - NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches HR@10 = 0.684 on MovieLens-1M. Published 0.684 ratio; observed 0.6798 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 0.994× the published value. (Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' HR@10 column), p. 7)
  - NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches NDCG@10 = 0.403 on MovieLens-1M. Published 0.403 ratio; observed 0.4054 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 1.01× the published value. (Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' NDCG@10 column), p. 7)

Other claims checked:
  - MLP with three hidden layers (32->16->8, embedding 16; the README's --layers [64,32,16,8]) reaches HR@10 = 0.671 on MovieLens-1M at 8 predictive factors. Published 0.671 ratio; observed 0.6707 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 3 (MovieLens rows, Factors = 8, MLP-3 column), p. 8)
  - MLP with three hidden layers reaches NDCG@10 = 0.399 on MovieLens-1M at 8 predictive factors. Published 0.399 ratio; observed 0.3979 ratio; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 0.997× the published value. (Table 4 (MovieLens rows, Factors = 8, MLP-3 column), p. 9)

Deviations from the paper's setup:
  - [none] All training and evaluation run on CPU (Theano device=cpu, 3.5-CPU/6 GB container on a shared 4-vCPU DigitalOcean droplet, OMP_NUM_THREADS=1), not on whatever hardware the authors used.
  - [none] Python 2.7.15 with Keras 1.0.7 and Theano 0.8.0 exactly as the README specifies, but with numpy 1.16.5, scipy 1.2.1 and h5py 2.10.0 (the last Python 2 releases from conda-forge/PyPI) rather than the unspecified 2017-era versions; Theano compiles with the sandbox system g++ 14.2.
  - [minor] For the NeuMF with pre-training claims, the pre-trained GMF (8 factors) and MLP ([64,32,16,8]) weights are the authors files bundled in the repository (Pretrain/ml-1m_GMF_8_1501651698.h5 and Pretrain/ml-1m_MLP_[64,32,16,8]_1501652038.h5, as named in the README command) rather than GMF and MLP models retrained in this attempt.
  - [minor] The cloud sandbox was destroyed mid-attempt by another process (shared droplet torn down at ~15:17Z) and rebuilt on a fresh droplet: the Python 2.7 / Keras 1.0.7 / Theano 0.8.0 environment and the authors repository were set up a second time with the same recipe and commit, and the MLP-3 run (job mlp3-8) was restarted from scratch. The NeuMF results were complete before the loss and were not re-run.

### DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network

Does DeepSurv (Katzman et al., 2018) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).

- Authors: Jared Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, Yuval Kluger
- Published: BMC Medical Research Methodology, 2018. arXiv 1606.00931 (https://arxiv.org/abs/1606.00931); DOI 10.1186/s12874-018-0482-1 (https://doi.org/10.1186/s12874-018-0482-1)
- Field: Medical statistics
- OpenRepro page: https://openrepro.org/papers/1606.00931/
- Full evidence report: https://openrepro.org/evidence/arxiv-1606-00931-20261018T225505Z.html
- Checked: 19 Oct 2026
- Outcome: Reproduced (4 of 4 headline claims held)
- Qualifiers: some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: jaredleekatzman/DeepSurv at 079dee7
- Work tree when the attempt ended: jaredleekatzman/DeepSurv at 079dee7
- Primary code version declared before any result was recorded: github.com/jaredleekatzman/DeepSurv (first author; the paper links it) at commit 079dee7, the last commit on or before 2017-08-09, the date of arXiv v3 (the version tested), run with the documented command in experiments/deepsurv/Dockerfile.<experiment> for each experiment
- Log: 63 hash-chained events, head hash b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - DeepSurv reaches test C-index 0.618308 on SUPPORT. Published 0.618308 C-index; observed 0.618582 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 5 (DeepSurv column))
  - DeepSurv reaches test C-index 0.643374 on METABRIC. Published 0.643374 C-index; observed 0.643492 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 5 (DeepSurv column))
  - DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG. Published 0.668402 C-index; observed 0.672011 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 1.01× the published value. (Table 1, p. 5 (DeepSurv column))
  - DeepSurv reaches test C-index 0.862620 on WHAS. Published 0.86262 C-index; observed 0.858003 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 0.995× the published value. (Table 1, p. 5 (DeepSurv column))

Other claims checked:
  - DeepSurv reaches test C-index 0.774019 on simulated linear data. Published 0.774019 C-index; observed 0.773885 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 5 (DeepSurv column))
  - DeepSurv reaches test C-index 0.648902 on simulated nonlinear data. Published 0.648902 C-index; observed 0.645797 C-index; criterion within ±0.01 of the published value; verdict: Held. Check: equivalence; observed 0.995× the published value. (Table 1, p. 5 (DeepSurv column))

Deviations from the paper's setup:
  - [minor] Python 2.7.15 from conda-forge with numpy 1.13.3, scipy 0.19.1, pandas 0.20.3, h5py 2.7.1 and matplotlib 2.2.3 (only for plots), plus Theano 0.9.0, Lasagne master (0.2.dev1) and lifelines 0.9.4, logger 1.4, tensorboard-logger 0.0.3 from PyPI, on CPU with float32, instead of the floydhub/dl-docker:cpu image.
  - [none] The documented commands were run with explicit paths to the model, data and results directories instead of the Docker mounts (/models, /shared/data, /shared/results), with /DeepSurv/deepsurv symlinked to the repository, and the WHAS run used the repository model file at 079dee7 even though it differs from the WHAS hyper-parameters in Table 3.

### Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations

Does PINNs (Raissi et al., 2017) reproduce? Yes, within a broad bound: when OpenRepro re-ran the authors' code, 1 of 1 headline claim held within tolerances registered before results were recorded; Burgers error 3.21× the published value; passes a registered 10× ceiling; a stricter 2× check did not hold; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).

- Authors: Maziar Raissi, Paris G. Perdikaris, George Em Karniadakis
- Published: arXiv, 2017. arXiv 1711.10561 (https://arxiv.org/abs/1711.10561)
- Field: Scientific machine learning
- OpenRepro page: https://openrepro.org/papers/1711.10561/
- Full evidence report: https://openrepro.org/evidence/arxiv-1711-10561-20261018T234638Z.html
- Checked: 19 Oct 2026
- Outcome: Reproduced (1 of 1 headline claim held)
- Qualifiers: Burgers error 3.21× the published value; passes a registered 10× ceiling; a stricter 2× check did not hold; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: Running the authors Burgers.py (the earliest version in their repository, from five months after arXiv v1) with its fixed seeds on CPU gives a relative L2 error of 2.15e-3 for the continuous-time Burgers example, against 6.7e-4 reported. That meets the pre-registered order-of-magnitude headline check (at most 6.7e-3) but not the stricter within-a-factor-of-2 check. The paper own Table 1 shows errors varying between 1e-4 and 1e-2 across neighbouring settings, and this run used a CPU rather than the reported GPU. The Table 1 sweep was not run.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: maziarraissi/PINNs at 13ce051
- Work tree when the attempt ended: maziarraissi/PINNs at 13ce051
- Primary code version declared before any result was recorded: github.com/maziarraissi/PINNs at commit 13ce051 (2018-05-01), the earliest commit containing the Burgers code; the repository was created on 2018-01-20, after arXiv v1 (2017-11-28), so no code exists from the paper date
- Log: 26 hash-chained events, head hash 450dad5c4edbfc2ab0c9dce568dcddd793d19978beba85e4a64f26f6724759da
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - A physics-informed network (9 layers x 20 neurons, N_u = 100, N_f = 10,000) solves the Burgers equation with relative L2 error of order 1e-4 (reported 6.7e-4). Published 0.00067 relative L2; observed 0.00214873 relative L2; criterion at most 0.0067; verdict: Held. Check: broad bound; observed 3.21× the published value. (Section 3.1, Fig. 1 caption, p. 8)

Other claims checked:
  - The Burgers relative L2 error is within a factor of 2 of the reported 6.7e-4. Published 0.00067 relative L2; observed 0.00214873 relative L2; criterion at most 0.00134; verdict: Did not hold. Check: broad bound; observed 3.21× the published value. (Section 3.1, Fig. 1 caption, p. 8)

Deviations from the paper's setup:
  - [minor] CPU (shared 4 vCPU droplet, 2 threads) instead of the NVIDIA Titan X GPU used in the paper; Python 3.6.15, TensorFlow 1.8.0, numpy 1.14.5, scipy 1.1.0, matplotlib 2.2.2 (2018-era versions; the repository pins none). Training took 974 s against about 60 s reported.

### Bag of Tricks for Efficient Text Classification

Does fastText (Joulin et al., 2017) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 8 of 8 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

- Authors: Armand Joulin, Édouard Grave, Piotr Bojanowski, Tomáš Mikolov
- Published: Conference of the European Chapter of the Association for Computational Linguistics, 2017. arXiv 1607.01759 (https://arxiv.org/abs/1607.01759); DOI 10.18653/v1/e17-2068 (https://doi.org/10.18653/v1/e17-2068)
- Field: Natural language processing
- OpenRepro page: https://openrepro.org/papers/1607.01759/
- Full evidence report: https://openrepro.org/evidence/arxiv-1607-01759-20261018T223713Z.html
- Checked: 18 Oct 2026
- Outcome: Reproduced (8 of 8 headline claims held)
- Qualifiers: legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: With fastText at the commit of arXiv v3 and its classification-results.sh, test accuracy with 10 hidden units and bigrams reproduces Table 1 on all eight datasets: 92.2, 96.7, 98.6, 95.7, 63.9, 72.4, 60.2 and 94.6 against 92.5, 96.8, 98.6, 95.7, 63.9, 72.3, 60.2 and 94.6. The unigram row also reproduces, within 0.2 points everywhere. The script own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs, preprocessed with the script own function. Training speed (Table 2) was not tested.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: facebookresearch/fastText at 206179d
- Work tree when the attempt ended: facebookresearch/fastText at 206179d
- Primary code version declared before any result was recorded: github.com/facebookresearch/fastText (linked in the paper) at the last commit on or before 2016-08-09 (arXiv v3), building with make and running classification-results.sh, whose header says it produces Table 1
- Log: 56 hash-chained events, head hash 87b5ac6197ec5f210cb74ba6aaa5412fe8e2237bdf845f1afee95be9043de61d
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News. Published 92.5 percent; observed 92.2 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.997× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News. Published 96.8 percent; observed 96.7 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.999× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia. Published 98.6 percent; observed 98.6 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity. Published 95.7 percent; observed 95.7 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full. Published 63.9 percent; observed 63.9 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers. Published 72.3 percent; observed 72.4 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full. Published 60.2 percent; observed 60.2 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity. Published 94.6 percent; observed 94.6 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)

Other claims checked:
  - fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News. Published 91.5 percent; observed 91.4 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.999× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News. Published 93.9 percent; observed 93.7 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.998× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia. Published 98.1 percent; observed 98.3 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity. Published 93.8 percent; observed 93.9 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full. Published 60.4 percent; observed 60.3 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.998× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers. Published 72 percent; observed 71.8 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.997× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full. Published 55.8 percent; observed 55.6 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.996× the published value. (Table 1, p. 3)
  - fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity. Published 91.2 percent; observed 91.1 percent; criterion within ±0.3 of the published value; verdict: Held. Check: equivalence; observed 0.999× the published value. (Table 1, p. 3)

Deviations from the paper's setup:
  - [minor] Datasets came from the public s3.amazonaws.com/fast-ai-nlp mirror (tarball hashes in the job output) instead of the script own googledrive.com/host download, then preprocessed with the script own normalize_text function.
  - [none] The unigram row was produced with the same command as the script but -wordNgrams 1; the script itself only runs the bigram configuration.
  - [minor] Built with g++ 14 on Debian 13 and trained with 4 threads on a 4 vCPU shared droplet.

### Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions

Does COVID-19 change points (Dehning et al., 2020) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

- Authors: Jonas Dehning, Johannes Zierenberg, F. Paul Spitzner, Michael Wibral, João Pinheiro Neto, Michael Wilczek, Viola Priesemann
- Published: Science, 2020. arXiv 2004.01105 (https://arxiv.org/abs/2004.01105); DOI 10.1126/science.abb9789 (https://doi.org/10.1126/science.abb9789)
- Field: Epidemiology
- OpenRepro page: https://openrepro.org/papers/2004.01105/
- Full evidence report: https://openrepro.org/evidence/arxiv-2004-01105-20261018T033025Z.html
- Checked: 18 Oct 2026
- Outcome: Reproduced (4 of 4 headline claims held)
- Qualifiers: legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: Fitting the authors three-change-point SIR model with weekly reporting modulation (repository state on the date of arXiv v3) to German case data until 21 April 2020, using the JHU snapshot bundled with the repository, reproduces all registered values: spreading rates 0.43, 0.25, 0.15 and 0.09 (posterior medians 0.428, 0.251, 0.154, 0.094) and change points around March 6, 15 and 23 (posterior medians March 5.8, 15.2 and 22.6). The credible intervals also match Fig. 3 closely. Only the main three-change-point model was refitted; the 0-2 change-point comparisons and forecasts were not.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: Priesemann-Group/covid19_inference_forecast at 2960f28
- Work tree when the attempt ended: Priesemann-Group/covid19_inference_forecast at 2960f28
- Primary code version declared before any result was recorded: github.com/Priesemann-Group/covid19_inference_forecast (linked in the paper) at the last commit on or before 2020-05-04 (arXiv v3), using its Germany three-change-point model with the data end date pinned to 2020-04-21 and the earliest JHU snapshot available in the repository at that commit, rather than the current, back-revised JHU series
- Log: 44 hash-chained events, head hash 90e4d597839451c42440893ea4dea7451ef11eaedc3887da05aa3c80dcc11e78
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - Before the first change point the spreading rate in Germany was 0.43 per day. Published 0.43 per day; observed 0.4275 per day; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 0.994× the published value. (Abstract, p. 1)
  - After the first change point the spreading rate was 0.25 per day. Published 0.25 per day; observed 0.2509 per day; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Abstract, p. 1)
  - After the second change point the spreading rate was 0.15 per day. Published 0.15 per day; observed 0.1544 per day; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 1.03× the published value. (Abstract, p. 1)
  - After the third change point the spreading rate was 0.09 per day. Published 0.09 per day; observed 0.0937 per day; criterion within ±0.02 of the published value; verdict: Held. Check: equivalence; observed 1.04× the published value. (Abstract, p. 1)

Other claims checked:
  - The first change point occurred around March 6, 2020. Published 6 day of March; observed 5.79 day of March; criterion within ±2 of the published value; verdict: Held. Check: equivalence; observed 0.965× the published value. (Abstract, p. 1)
  - The second change point occurred around March 15, 2020. Published 15 day of March; observed 15.22 day of March; criterion within ±2 of the published value; verdict: Held. Check: equivalence; observed 1.01× the published value. (Abstract, p. 1)
  - The third change point occurred around March 23, 2020. Published 23 day of March; observed 22.63 day of March; criterion within ±2 of the published value; verdict: Held. Check: equivalence; observed 0.984× the published value. (Abstract, p. 1)

Deviations from the paper's setup:
  - [minor] Data end pinned to 2020-04-21 and data taken from the repository bundled JHU snapshot (confirmed_global_fallback_2020-04-28.csv) instead of the live JHU download; only the three-change-point model is fitted; pm.sample uses a fixed random_seed.
  - [minor] Python 3.7 with PyMC3 3.8, Theano 1.0.4, numpy 1.18.5, pandas 1.0.5, scipy 1.4.1 and arviz 0.7.0 from conda-forge, CPU only.

### From Louvain to Leiden: guaranteeing well-connected communities

Does From Louvain to Leiden (Traag et al., 2019) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

- Authors: V. A. Traag, L. Waltman, Nees Jan van Eck
- Published: Scientific Reports, 2019. arXiv 1810.08473 (https://arxiv.org/abs/1810.08473); DOI 10.1038/s41598-019-41695-z (https://doi.org/10.1038/s41598-019-41695-z)
- Field: Network science
- OpenRepro page: https://openrepro.org/papers/1810.08473/
- Full evidence report: https://openrepro.org/evidence/arxiv-1810-08473-20261018T013946Z.html
- Checked: 18 Oct 2026
- Outcome: Reproduced (4 of 4 headline claims held)
- Qualifiers: DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: Using the authors Java implementation at the release tag matching the paper (networkanalysis 1.0.0) on the SNAP DBLP and Amazon networks, with the paper protocol of 10 random starts of 10 iterations each, all tested Table II values reproduced within 0.0005: DBLP Louvain 0.8267 vs 0.8262 and Leiden 0.8384 vs 0.8387; Amazon Louvain 0.9298 vs 0.9301 and Leiden 0.9340 vs 0.9341. Leiden found higher modularity than Louvain on both networks, as claimed. The other four networks in Table II were out of scope (download blocked, proprietary, or too large for the sandbox), and the badly connected community percentages in Fig. 4 were not tested.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: CWTSLeiden/networkanalysis at 2d38563
- Work tree when the attempt ended: CWTSLeiden/networkanalysis at 2d38563
- Log: 42 hash-chained events, head hash 578432e9062b72e8e2576db5eeed0bf5ac96e6f5b649204a2e77964f446edc6c
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - On DBLP, the maximal modularity found by the Leiden algorithm over 10 replications of 10 iterations is 0.8387. Published 0.8387 modularity; observed 0.838377 modularity; criterion within ±0.002 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table II, p. 6 (text p. 9))
  - On Amazon, the maximal modularity found by the Leiden algorithm is 0.9341. Published 0.9341 modularity; observed 0.934022 modularity; criterion within ±0.002 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table II, p. 6 (text p. 9))
  - On DBLP, Leiden finds higher maximal modularity than Louvain (paper: +0.0125). Published 0.0125 modularity; observed 0.011724 modularity; criterion at least 0; verdict: Held. Check: broad bound; observed 0.938× the published value. (Table II, p. 6 (text p. 9))
  - On Amazon, Leiden finds higher maximal modularity than Louvain (paper: +0.0040). Published 0.004 modularity; observed 0.004176 modularity; criterion at least 0; verdict: Held. Check: broad bound; observed 1.04× the published value. (Table II, p. 6 (text p. 9))

Other claims checked:
  - On DBLP, the maximal modularity found by the Louvain algorithm is 0.8262. Published 0.8262 modularity; observed 0.826653 modularity; criterion within ±0.002 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table II, p. 6 (text p. 9))
  - On Amazon, the maximal modularity found by the Louvain algorithm is 0.9301. Published 0.9301 modularity; observed 0.929846 modularity; criterion within ±0.002 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table II, p. 6 (text p. 9))

Deviations from the paper's setup:
  - [none] Built and run with OpenJDK 21 instead of a 2018-era JDK.
  - [minor] SNAP node ids were renumbered to 0..n-1 in sorted order, and runs used a fixed seed (20261018) rather than an unspecified random one.

### Synthetic Difference-in-Differences

Does Synthetic Difference-in-Differences (Arkhangelsky et al., 2021) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

- Authors: Dmitry Arkhangelsky, Susan Carleton Athey, David A. Hirshberg, Guido W. Imbens, Stefan Wager
- Published: American Economic Review, 2021. arXiv 1812.09970 (https://arxiv.org/abs/1812.09970); DOI 10.1257/aer.20190159 (https://doi.org/10.1257/aer.20190159)
- Field: Economics
- OpenRepro page: https://openrepro.org/papers/1812.09970/
- Full evidence report: https://openrepro.org/evidence/arxiv-1812-09970-20261018T010328Z.html
- Checked: 18 Oct 2026
- Outcome: Reproduced (4 of 4 headline claims held)
- Qualifiers: legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
- Coverage: see the summary
- Summary: Running the authors own Table 1 code (vignettes/paper-results.Rmd) at the synthdid commit named in their session-info, with R 4.0.5 and the pinned MCPanel and glmnet commits, reproduced every entry of Table 1 at the reported precision: SDID -15.6 (8.4), SC -19.6 (9.9), DID -27.3 (17.7), MC -20.2 (11.5), DIFP -11.1 (9.5). Point estimates differ from the paper by at most 0.05, and placebo standard errors by at most 0.05. Deviations were limited to platform (x86_64 Linux instead of an M1 Mac) and one non-estimating build dependency. The paper-era synthdid commit is no longer on any branch of the repository but is still retrievable from GitHub by its hash.
- Assurance: Legacy. Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen. Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites. Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed). Report re-derived from the log: Passed: the engine re-derived the report from the log. Engine version: 0.2.1. Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Commit the recorded results name: 2408a92
- Work tree when the attempt ended: synth-inference/synthdid at 70c1ce3; susanathey/MCPanel at 6b2706f; cran/glmnet at f4fc95a; synth-inference/synthdid at 2408a92
- Log: 59 hash-chained events, head hash 8b90233f1aba9384bcb8d580e05cbaaf9a5947ebf64d2d62c4f0f4b157f1f7cb
- Run by: Claude, an AI model made by Anthropic, operating OpenRepro's engine in a sandbox

Headline claims (these decide the outcome):
  - The SDID estimate of the average effect of Proposition 99 on California per-capita cigarette sales over the 12 post-treatment years is -15.6 packs. Published -15.6 packs per capita; observed -15.6038 packs per capita; criterion within ±0.1 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The placebo-method standard error of the SDID estimate is 8.4. Published 8.4 packs per capita; observed 8.36799 packs per capita; criterion within ±0.84 of the published value; verdict: Held. Check: equivalence; observed 0.996× the published value. (Table 1, p. 8 (placebo method, Section 5, Algorithm 4))
  - The synthetic control estimate is -19.6. Published -19.6 packs per capita; observed -19.6197 packs per capita; criterion within ±0.1 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The difference-in-differences estimate is -27.3. Published -27.3 packs per capita; observed -27.3491 packs per capita; criterion within ±0.1 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)

Other claims checked:
  - The matrix completion estimate is -20.2. Published -20.2 packs per capita; observed -20.2181 packs per capita; criterion within ±0.5 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The synthetic control with intercept (DIFP) estimate is -11.1. Published -11.1 packs per capita; observed -11.1046 packs per capita; criterion within ±0.1 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The placebo-method standard error of the SC estimate is 9.9. Published 9.9 packs per capita; observed 9.91743 packs per capita; criterion within ±0.99 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The placebo-method standard error of the DID estimate is 17.7. Published 17.7 packs per capita; observed 17.7403 packs per capita; criterion within ±1.77 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The placebo-method standard error of the MC estimate is 11.5. Published 11.5 packs per capita; observed 11.5099 packs per capita; criterion within ±1.15 of the published value; verdict: Held. Check: equivalence; observed 1× the published value. (Table 1, p. 8)
  - The placebo-method standard error of the DIFP estimate is 9.5. Published 9.5 packs per capita; observed 9.45566 packs per capita; criterion within ±0.95 of the published value; verdict: Held. Check: equivalence; observed 0.995× the published value. (Table 1, p. 8)

Deviations from the paper's setup:
  - [none] isoband 0.2.5 (from the 2021-08-01 CRAN snapshot) instead of the 0.2.4 in the 2021-05-01 snapshot used for all other CRAN packages.
  - [minor] Platform differs from the authors Table 1 run: x86_64 Debian 13 with R reference BLAS/LAPACK on a shared-CPU DigitalOcean droplet, versus an M1 MacBook Air with OpenBLAS 0.3.13. R 4.0.5 and the pinned synthdid, MCPanel and glmnet commits match their session-info.
  - [none] future.batchtools is not installed or loaded; the vignette imports it alongside rngtools, future and doFuture.