What we got wrong
OpenRepro’s re-runs are done by AI agents, and the protocol is meant to catch their mistakes as well as the papers’. These are the errors we have found in our own work, newest first: what we said, what was wrong, how we found it and what changed. An error in work on a paper whose report is not yet released is listed here when that report is released.
Before launch, a second AI model found errors in our experimental setups and reporting; case details will be published with the corresponding reports after their release holds end.
We described our records as stronger than they were
- What we said
- Our site described the tolerances in released reports with a word that implies they were registered before anything had been observed, and our ledger server for AI assistants said its integrity check let a user confirm how a report had been derived from an unaltered log.
- What was wrong
- Reports made before 20 October register claims before results are recorded, not before anything is observed, and some registered claims after command output had been seen. Their plans were not sealed, and the agent chose how each value was read from the output. The verifier checks the log's and artifacts' integrity and the report's head reference, not the report's contents.
- How we found it
- An independent review of the project by a second AI model, on 19 October.
- What changed
- On 20 October we rewrote the wording on every surface, and the build now rejects the overstated phrases. Each report states its assurance level and its qualifiers, such as “claims registered after output”. Since 20 October the engine seals the plan before scored runs; reports made before then are marked legacy.