OpenRepro

Every paper re-run, and its outcome

Outcomes are computed from each paper's headline claims, each judged against a tolerance registered before results were recorded. Qualifiers (a broad bound, a check that missed) sit under each outcome. What the outcomes mean.

PaperOutcomeClaimsHeadline heldCheckedEvidence
Neural Collaborative Filtering He et al. · arXiv, 2017 · arXiv 1708.05031 Recommender systems Reproducedsome claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy Neural Collaborative Filtering: every claim checked6 claims: 6 held. 4 of 4 Report
DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network Katzman et al. · BMC Medical Research Methodology, 2018 · arXiv 1606.00931 Medical statistics Reproducedsome claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy DeepSurv: every claim checked6 claims: 6 held. 4 of 4 Report
Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations Raissi et al. · arXiv, 2017 · arXiv 1711.10561 Scientific machine learning ReproducedBurgers error 3.21× the published value; passes a registered 10× ceiling; a stricter 2× check did not hold; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy PINNs: every claim checked2 claims: 1 held, 1 did not hold. 1 of 1 Report
Bag of Tricks for Efficient Text Classification Joulin et al. · Conference of the European Chapter of the Association for Computational Linguistics, 2017 · arXiv 1607.01759 Natural language processing Reproducedlegacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy fastText: every claim checked16 claims: 16 held. 8 of 8 Report
Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions Dehning et al. · Science, 2020 · arXiv 2004.01105 Epidemiology Reproducedlegacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy COVID-19 change points: every claim checked7 claims: 7 held. 4 of 4 Report
From Louvain to Leiden: guaranteeing well-connected communities Traag et al. · Scientific Reports, 2019 · arXiv 1810.08473 Network science ReproducedDBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy From Louvain to Leiden: every claim checked6 claims: 6 held. 4 of 4 Report
Synthetic Difference-in-Differences Arkhangelsky et al. · American Economic Review, 2021 · arXiv 1812.09970 Economics Reproducedlegacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before itAssurance: Legacy Synthetic Difference-in-Differences: every claim checked10 claims: 10 held. 4 of 4 Report
4 attempts in progress, listed when they finish.

11 more reports are not yet released (author right-of-reply, or held for a re-run) and will appear when their release records allow. Right of reply.

The claims column is the paper's strip from the chart: every claim by how far the re-run landed from the published number, in tolerances; headline claims are the larger points. The field under each title is a label assigned by OpenRepro for browsing; everything else on this page comes from the evidence reports. Machine-readable: ledger.json, ledger.csv, claims.csv (one row per claim), claims.json, Atom feed.