What does 'reproduced' mean on OpenRepro?

Every headline claim came out within its registered tolerance when the authors' own published code was re-run at the code version declared before any result was recorded. It is a statement about the published materials under the recorded conditions, not an endorsement of the paper. When a headline check is a broad bound, or another check missed, the answer says so next to the outcome.

Does 'could not reproduce' mean the paper is wrong?

No. It means none of the headline claims came out within tolerance when the authors' code was re-run under the recorded conditions, with the primary code or any declared earlier code. The record often shows a likely cause, such as a missing seed, a package version or a dataset. The authors have a right of reply, and their notes can be published alongside the report.

What does 'Not verified: evidence incomplete' mean?

That OpenRepro's own evidence for the attempt failed a check, for example a result without its supporting log evidence or a source quote that could not be resolved, so no reproduction outcome is given. It is a gap in OpenRepro's record, not a finding about the paper.

What do 'bound check' and 'check missed' mean next to an outcome?

They are qualifiers. A bound check accepts any value under a ceiling or over a floor rather than a match to the published number, and a broad bound admits values more than twice (or less than half) the published one; the page gives the observed value as a multiple of the published one and the ceiling. 'Check missed' means a non-headline check, often a stricter version of a headline check, did not hold. 'Partly untested' means some results were left untested for budget.

Are the reports peer reviewed or checked by an expert?

No. No domain expert has reviewed the reports, and nothing on this site has been peer reviewed. The authors' right of reply, the published evidence log and the verifier are the checks.

How are the tolerances set?

Each claim is registered with the published value, the exact sentence or table cell it comes from, and a tolerance with the reason for it. In current (legacy) attempts this happens before results are recorded, which does not show that no output had been seen first; attempts whose plan is sealed before scored execution say so in their Assurance block. The log event at which each claim was registered, and the event of its first result, are printed on the paper's page. A claim registered after a result was recorded is flagged.

What does the hash chain prove?

That a log matches the head hash you hold: if it does, the log was not edited after that hash was recorded. The head hash is published on this same site and independent anchoring is pending, so it does not stop an operator rewriting a whole log before publishing, and it does not prove that command output is honest; the recorded code, commits and environment are what let results be re-run.

What does the verifier check?

The standalone verifier checks log and artifact integrity (every event's hash, the chain, every artifact's SHA-256) and that the report names the log's head hash. It does not check the report's contents against the log; that re-derivation is the engine's repro verify, which is not yet public. The release manifest at /release.json lists SHA-256 digests of every released report and bundle.

What happens before a report is published?

Every outcome other than 'reproduced' gets a heads-up to the authors and 14 days before publication. Each attempt is released only by an explicit release record that names the notices sent and the release date; without one, the paper does not appear on the site at all. 'Reproduced' gets a courtesy note at publication. Authors can reply at any time, and their notes can be published alongside the report.

How can I ask an AI assistant whether a paper reproduces?

The ledger is a read-only MCP server at https://mcp.openrepro.org/mcp, which answers with the outcome, its qualifiers and a link to the report; it serves only what this site publishes. The same facts are in https://openrepro.org/llms.txt and https://openrepro.org/ledger.json. A paper that is not in the ledger has no released report; that is not an outcome.

Can I reuse the data?

Yes. Reports, verdicts and evidence logs are CC BY 4.0. The ledger and claims are at /ledger.json, /ledger.csv, /claims.json and /claims.csv, described on the data page, with a CITATION.cff for citing the ledger and a citation block on every report page.

How an outcome is decided

Every outcome is computed by the same fixed rules from the record of the attempt. Which claims are registered, their tolerances and which count as headline are choices the agent makes when it registers them, and they shape the label; each paper page shows them, with the outcome's qualifiers and the report's assurance level.

What an attempt checks

An attempt takes one paper and the authors' own published code, and re-runs the experiments behind specific numbers in the paper. Before any result is recorded, each number to be checked is registered as a claim: the published value, the exact sentence or table row it comes from (matched against the paper's text), and a tolerance with the reason for it. A tolerance can be a band around the published value (within ±0.005) or a one-sided limit (at most, at least).

Registering a claim before its result is recorded is not the same as registering it before anything is observed. In current attempts, which this site labels legacy, the agent may have run commands and seen output before registering; the log shows the order of registration and result entry, not what was seen. Attempts whose whole plan is sealed before scored execution say so in their Assurance block, and only those are described that way.

Some claims are marked headline: the results the paper is chiefly known for. The outcome is computed from the headline claims; other claims are reported alongside them. If no claim is marked headline, all of them count. Which claims are headline is chosen when they are registered, and that choice is editorial: the qualifiers below make a narrow or loose headline visible.

Which version of the authors' code counts is also declared before any result is recorded, as the primary code: normally the authors' code at the date of the paper version being tested, but sometimes a replication script the authors published later. An attempt may declare, before any result is recorded, that an earlier version of the authors' code may be tried if a claim misses. Earlier-code results are always reported separately and never replace the primary result.

The outcomes

Reproduced: Every headline claim came out within its tolerance using the primary code version declared before any result was recorded (normally the authors' code at the paper date). Read it with its qualifiers: a reproduction within a broad bound is weaker than a match to the published number.
Reproduced only with earlier code: Every headline claim came out within its tolerance, but at least one only with an earlier version of the authors' code, under a fallback declared before any result was recorded. The published numbers can be obtained from the authors' code, but not from the primary version, and the record shows which version does produce them.
Partially reproduced: Some headline claims held and others did not, or could not be tested. A claim that held only with earlier code counts as held here. This is a finding in its own right, and often the most useful one: it separates the results that stand up from the ones that depend on something the paper does not pin down, such as a random seed, a package version or a dataset. Read the claims table to see which is which.
Could not reproduce: None of the headline claims came out within tolerance, with the primary code or any declared earlier code. This is a statement about the published materials under the recorded conditions, not about the authors' honesty or the idea's merit. The record often shows a likely cause; authors are invited to reply, and their notes can be published alongside the report.
Not verified: evidence incomplete: OpenRepro's own evidence for the attempt failed a check, for example a result recorded without its supporting log evidence, or a source quote that could not be resolved. No reproduction outcome is given, the label is grey, and nothing is counted as held or missed. It is a gap in OpenRepro's record, not a finding about the paper.

Three further labels exist: Not attemptable, when the claims could not be tested at all (for example, the code or data is unavailable); Incomplete, for an attempt that has not finished, or that tested none of its headline claims; and Incomplete: plan amended after the seal, when a claim sealed into the plan was withdrawn afterwards (the engine keeps it in the scored set as withdrawn and gives no outcome). Incomplete and amended attempts are not listed.

Qualifiers

An outcome is never shown alone when the evidence behind it is narrower than the label suggests. The qualifier travels with the outcome everywhere: the answer under the paper's title, the ledger row, the badge, the data files, the feed, the llms files and the MCP server.

Bound check. Every headline check is one-sided (at most, at least), so it accepts any value under a ceiling or over a floor rather than a match to the published number.
Broad bound. A headline bound admits values more than twice (or less than half) the published value. The page states the observed value as a multiple of the published one and the bound, for example Burgers error 3.21× the published value; passes a registered 10× ceiling.
Check missed. A non-headline check did not hold, often a stricter version of a headline check (a stricter 2× check did not hold).
Partly untested. Some of the paper's results were left untested to stay within the compute budget. Reports made before the engine recorded this say so instead of guessing.
Retry after a miss. A claim's final result passed on a rerun after an earlier result missed, and the plan's rerun rule was not one the engine checks mechanically.
Late registration. Some claims were registered after command output had already been observed.
Tree changed in run. The code tree changed while a cited run was in progress (a file patched, or patched and restored).
Unregistered extraction. At least one value was read from an excerpt the agent chose after seeing the output, rather than by a rule fixed before the seal. Such an outcome is never shown as an unqualified reproduction.
Legacy. The attempt predates the sealed plan: claims were frozen only by their first result, and values were matched in the logged output rather than bound to it. Every currently listed report is legacy.

Reports carry these qualifiers from the engine, and the short forms above travel with the outcome on the badge, in page titles and on share cards; a qualifier the site does not recognise is still carried verbatim rather than dropped. For reports made before the engine recorded qualifiers, the site derives the first four from the registered claims by the same definitions, and says that it did.

Claim results

Each claim is marked Held, Held with earlier code, Did not hold or Not tested. In legacy attempts each observed value is matched against the text of the logged output it cites: that shows the number appears in that output, not that it is the registered measurement, and a value that cannot be found is flagged in the report without changing the verdict. Newer attempts bind each value to the output of a scored run started after the seal, and extract it by a rule (a metric key, a pattern or a table cell) registered before the seal when the claim has one; the value is judged unrounded. A value read from an excerpt the agent chose after seeing the output is labelled as such and qualifies the outcome. An attempt whose evidence is incomplete gets no outcome. Checks decided after results were recorded are notes and diagnostics: published in full, never a change to a verdict.

Deviations and scope

Re-running rarely matches the original setup exactly: a CPU instead of a GPU, a newer operating system, a subset of datasets. Every such difference is recorded as a deviation with an impact of none, minor or material, and shown on the paper's page. Parts of a paper that were not tested are named in the summary. An outcome covers only the claims that were checked.

The record

Attempts run in a resource-capped container sandbox, and the software environment is recorded. Every command, its output, and every decision is an event in an append-only log; each event includes the hash of the previous one, and command outputs are stored as hashed files. The engine generates the report, including the outcome, from that log.

Each released attempt's log, artifacts and report are downloadable as an evidence bundle. A single Python file checks every event's hash, the chain linking them, every artifact's SHA-256, and that the report names the log's head hash. release.json, the release manifest, lists every released attempt with the SHA-256 of its report.json, report.md, report.html and bundle.

What the evidence does and does not prove

It does show:

That a log matching the head hash you hold was not edited after that hash was recorded, and that its artifacts are the ones it names.
That a report file matches the digest in the release manifest, so the copy you have is the one OpenRepro released.
The order of events: when each claim was registered relative to when its first result was recorded.

It does not show:

That the report's contents follow from the log. The standalone verifier checks the log, the artifacts and the report's head reference only. Re-deriving the report from the log is the engine's repro verify, which is not yet public.
When anything happened. The head hashes and the manifest are published on this same site; independent anchoring is pending, so the chain does not stop an operator rewriting a whole log before publishing.
That claims were registered before any output was seen, in legacy attempts (see above).
That command output is honest, or that a number in it is the registered measurement in legacy attempts. That is why the scripts, commits and environment are recorded: so the results can be re-run.
That an attempt is the only one made. Released attempts and their superseded versions are listed; attempts held for a re-run are counted, not named.

Assurance levels

Each paper page has an Assurance block. The level is the strongest row the report meets; a legacy report is never described as meeting a rule introduced after it was made.

Level	Plan	Evidence	Execution manifest	Report re-derived
Legacy	Claims registered before results were recorded; no sealed plan. The report says so itself (`legacy`), whichever engine version regenerated it	Number matched in the logged output text	No	As recorded; the attempt predates the checks it lists under predates
Partial	Recorded by the engine, but not sealed before scored execution	As recorded	As recorded	As recorded
Sealed plan	Claims, tolerances, extraction rules, headline choice, stopping and rerun rules sealed before scored execution; every result cites a scored run started after the seal	As recorded	As recorded	As recorded
Sealed and bound	Sealed before scored execution	Each value bound to an exact excerpt of the cited run output (by a sealed rule, or by an agent-selected excerpt, which the page says)	Yes: the command, the Git checkout state and the hashes of the named driver files recorded at the start and end of each scored run. Data files and the environment are not bound	As recorded
Sealed, bound and re-derived	Sealed before scored execution	Bound, and every value extracted by a rule sealed before the run	Yes, as above	Passed: the engine re-derived the report from the log, rebuilt the sealed plan and recomputed every flag

Who runs the attempts, and who reviews them

Re-runs are carried out by an AI agent, Claude (made by Anthropic), operating OpenRepro's engine in a sandbox, in a Claude Code session.
Each run is driven by the AI with no person in the loop, and the log's actor field records this. Each paper page names the model that ran it.
A person oversees the project: choosing the papers and setting the rules the agent works under.
No domain expert has reviewed the reports. They are not peer reviewed. The authors' right of reply, below, is the check.
Authors get a right of reply, and their notes can be published alongside the report.

The agent does not type the outcome: it is computed by fixed rules from the claims, tolerances and headline choices the agent registered and the results it recorded, so those choices shape it.

How papers are chosen

Papers are chosen for open code and data, low compute cost and citation impact across a mix of fields. Finished attempts are released after their right-of-reply period, including failures. An attempt held for a re-run is counted but not named, and an attempt that a re-run replaced stays public as a superseded version.

Right of reply and release

Every outcome other than “reproduced” gets a heads-up to the authors and 14 days before publication. “Reproduced” gets a courtesy note to the authors at publication. Authors can reply at any time, before or after publication, and their notes can be published alongside the report.

Release is fail-closed. Each attempt appears only when it has an explicit release record naming the notices sent (date, kind, number of recipients), the release date and the authors' reply status, and the build checks that a non-“reproduced” release comes at least 14 days after the latest notice. Without such a record the paper does not appear anywhere: not on the site, in the data files, the feed, the badges, the evidence bundles, the MCP server or the dataset mirrors. The ledger says only how many reports are not yet released, and site-wide totals say they are not counted yet. Every released attempt is listed in release.json.

Versions and corrections

Published reports are immutable and are never edited in place. A correction is a new attempt; the earlier one stays public as a superseded version, linked from the paper's page with the reason it was replaced, and both appear in the data files' attempt lineage.

Badge

Every listed paper has an embeddable badge at /badge/<arxiv-id>.svg that states the outcome, how many headline claims held and any qualifier (a bound check, a check that missed). It is generated from the report at every build, so it always reflects the current report and a correction updates it; a paper without a due release record has no badge. Authors and maintainers may embed it, and each paper page gives copy-ready Markdown and HTML that link the badge to the report.

Use it from an AI assistant

The ledger is also available as a read-only MCP server at https://mcp.openrepro.org/mcp, so an assistant such as Claude can answer “does this paper reproduce?” with the outcome, its qualifiers, the assurance level, the attempt history and a link to the report. In Claude Code: claude mcp add --transport http openrepro https://mcp.openrepro.org/mcp. It serves only what this site publishes, stores nothing and needs no key.

Citing OpenRepro

Each paper page ends with a plain-text citation and BibTeX for its report. Cite the report, not the paper's ledger row: the citation carries the attempt id and the log head hash, which identify exactly the version read, because published reports are never edited in place. The ledger as a whole is a dataset; CITATION.cff gives its citation, versioned by build date.

License

Reports, verdicts and evidence logs: CC BY 4.0.

Questions and answers

What does 'reproduced' mean on OpenRepro?: Every headline claim came out within its registered tolerance when the authors' own published code was re-run at the code version declared before any result was recorded. It is a statement about the published materials under the recorded conditions, not an endorsement of the paper. When a headline check is a broad bound, or another check missed, the answer says so next to the outcome.
Does 'could not reproduce' mean the paper is wrong?: No. It means none of the headline claims came out within tolerance when the authors' code was re-run under the recorded conditions, with the primary code or any declared earlier code. The record often shows a likely cause, such as a missing seed, a package version or a dataset. The authors have a right of reply, and their notes can be published alongside the report.
What does 'Not verified: evidence incomplete' mean?: That OpenRepro's own evidence for the attempt failed a check, for example a result without its supporting log evidence or a source quote that could not be resolved, so no reproduction outcome is given. It is a gap in OpenRepro's record, not a finding about the paper.
What do 'bound check' and 'check missed' mean next to an outcome?: They are qualifiers. A bound check accepts any value under a ceiling or over a floor rather than a match to the published number, and a broad bound admits values more than twice (or less than half) the published one; the page gives the observed value as a multiple of the published one and the ceiling. 'Check missed' means a non-headline check, often a stricter version of a headline check, did not hold. 'Partly untested' means some results were left untested for budget.
Who runs the attempts?: An AI agent, Claude (made by Anthropic), operating OpenRepro's engine in a sandbox. Each run is driven by the AI with no person in the loop, and the log's actor field records this. A person oversees the project, chooses the papers and sets the rules. The agent does not type the outcome: it is computed by fixed rules from the claims, tolerances and headline choices the agent registered and the results it recorded, so those choices shape it.
Are the reports peer reviewed or checked by an expert?: No. No domain expert has reviewed the reports, and nothing on this site has been peer reviewed. The authors' right of reply, the published evidence log and the verifier are the checks.
How are the tolerances set?: Each claim is registered with the published value, the exact sentence or table cell it comes from, and a tolerance with the reason for it. In current (legacy) attempts this happens before results are recorded, which does not show that no output had been seen first; attempts whose plan is sealed before scored execution say so in their Assurance block. The log event at which each claim was registered, and the event of its first result, are printed on the paper's page. A claim registered after a result was recorded is flagged.
What does the hash chain prove?: That a log matches the head hash you hold: if it does, the log was not edited after that hash was recorded. The head hash is published on this same site and independent anchoring is pending, so it does not stop an operator rewriting a whole log before publishing, and it does not prove that command output is honest; the recorded code, commits and environment are what let results be re-run.
What does the verifier check?: The standalone verifier checks log and artifact integrity (every event's hash, the chain, every artifact's SHA-256) and that the report names the log's head hash. It does not check the report's contents against the log; that re-derivation is the engine's repro verify, which is not yet public. The release manifest at /release.json lists SHA-256 digests of every released report and bundle.
How are papers chosen?: For open code and data, low compute cost and citation impact across a mix of fields. Finished attempts are released after their right-of-reply period, including failures; an attempt held for a re-run is counted but not named, and an attempt that a re-run replaced stays visible as a superseded version.
What happens before a report is published?: Every outcome other than 'reproduced' gets a heads-up to the authors and 14 days before publication. Each attempt is released only by an explicit release record that names the notices sent and the release date; without one, the paper does not appear on the site at all. 'Reproduced' gets a courtesy note at publication. Authors can reply at any time, and their notes can be published alongside the report.
How can I ask an AI assistant whether a paper reproduces?: The ledger is a read-only MCP server at https://mcp.openrepro.org/mcp, which answers with the outcome, its qualifiers and a link to the report; it serves only what this site publishes. The same facts are in https://openrepro.org/llms.txt and https://openrepro.org/ledger.json. A paper that is not in the ledger has no released report; that is not an outcome.
Can I reuse the data?: Yes. Reports, verdicts and evidence logs are CC BY 4.0. The ledger and claims are at /ledger.json, /ledger.csv, /claims.json and /claims.csv, described on the data page, with a CITATION.cff for citing the ledger and a citation block on every report page.

Right of reply, and reporting an error

Authors of any paper on this site may submit a response, and their notes can be published alongside the report, unedited. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.