Statistics · 2019
Reproducibility report: Conformalized Quantile Regression
5 of 5 headline claims held
legacyunregistered extractionThe full answer and its qualifiers
Does Conformalized Quantile Regression (Romano et al., 2019) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 5 of 5 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 20 Oct 2026).
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
Does Conformalized Quantile Regression (Romano et al., 2019) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 5 of 5 headline claims held within tolerances registered before results were recorded (checked 20 Oct 2026).
- Checked
- Released
- Assurance
- Legacy
- Evidence
- Full report · report.json
- Data
- JSON record · How to cite
- Assistants
- Ask via MCP
Does Conformalized Quantile Regression reproduce?
Yes, on its headline claims. 5 of 5 headline claims held within tolerances registered before results were recorded.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
- Held (large: headline claim; small: other claim)
Reading the marks
- Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| Conformal methods, including both CQR variants, attain the nominal 90% coverage; observed value is the LOWEST split-averaged test coverage among all CQR (method, dataset) cells run in this attempt. (headline) | 90 | 89.65 | at least 88 | -0.17 | Held |
| CQR with a quantile neural network gives average 90% interval length 0.57 on the bike sharing dataset (20 random 80/20 splits). (headline) | 0.57 | 0.5704 | within ±0.057 | +0.01 | Held |
| CQR with a quantile neural network gives average 90% interval length 1.76 on the community dataset (20 random 80/20 splits). (headline) | 1.76 | 1.8561 | within ±0.176 | +0.55 | Held |
| CQR with quantile random forests gives average 90% interval length 1.55 on the community dataset (20 random 80/20 splits). (headline) | 1.55 | 1.507 | within ±0.155 | -0.28 | Held |
| CQR with quantile random forests gives average 90% interval length 0.56 on the concrete dataset (20 random 80/20 splits). (headline) | 0.56 | 0.5584 | within ±0.056 | -0.03 | Held |
| Standard split conformal prediction with a conditional-mean random forest gives average 90% interval length 1.94 on the community dataset, longer than CQR random forests (1.55). | 1.94 | 1.8578 | within ±0.194 | -0.42 | Held |
What the attempt found
Clean re-run of attempt arxiv-1905-03222-20261019T184203Z with the same protocol and the same six pre-registered claims, and no foreign or withdrawn claims. Ran the authors' reproducible_experiments/run_experiment.py (commit 2b9b5c7, the last before arXiv v1) unchanged except for the same one-line sklearn Imputer patch, on community, concrete and bike with the same rebuilt Python 3.8 stack, 5 splits (2 for bike). All six claims landed within tolerance: CQR quantile-forest length 1.5070 vs 1.55 (community) and 0.5584 vs 0.56 (concrete), CQR quantile-net 1.8561 vs 1.76 (community) and 0.5704 vs 0.57 (bike), split-conformal RF 1.8578 vs 1.94 (community, longer than CQR as reported), and minimum CQR coverage 89.65% against the 90% nominal level. Compared with the earlier attempt, five values are identical to 4 decimals; only the bike quantile-net cell moved (0.5704 vs 0.5695, coverage 89.65 vs 89.26), likely from a different torch thread count. The paper's Table 1 averages over all 11 datasets were not tested.
Assurance
- Level
- Legacy (what the levels mean)
- Plan
- Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
- Evidence
- Legacy matching: each observed value was matched against the text of the logged output it cites
- Execution manifest
- No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
- Predates
- plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Conformal methods, including both CQR variants, attain the nominal 90% coverage; observed value is the LOWEST split-averaged test coverage among all CQR (method, dataset) cells run in this attempt.¶Section 6.2, p. 10 · registered at event 7 · quote verified on p. 10 · bound check, observed 0.996× the published value | 90% | 89.65 | at least 88 | Held |
| CQR with a quantile neural network gives average 90% interval length 0.57 on the bike sharing dataset (20 random 80/20 splits).¶Figure 4 (bike panel), p. 15 · registered at event 8 · quote verified on p. 15 · equivalence check | 0.57rescaled response units | 0.5704 | within ±0.057 | Held |
| CQR with a quantile neural network gives average 90% interval length 1.76 on the community dataset (20 random 80/20 splits).¶Figure 5 (community panel), p. 16 · registered at event 9 · quote verified on p. 16 · equivalence check | 1.76rescaled response units | 1.8561 | within ±0.176 | Held |
| CQR with quantile random forests gives average 90% interval length 1.55 on the community dataset (20 random 80/20 splits).¶Figure 5 (community panel), p. 16 · registered at event 10 · quote verified on p. 16 · equivalence check | 1.55rescaled response units | 1.507 | within ±0.155 | Held |
| CQR with quantile random forests gives average 90% interval length 0.56 on the concrete dataset (20 random 80/20 splits).¶Figure 5 (concrete panel), p. 16 · registered at event 11 · quote verified on p. 16 · equivalence check | 0.56rescaled response units | 0.5584 | within ±0.056 | Held |
Other claims checked
1 claim: 1 held
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Standard split conformal prediction with a conditional-mean random forest gives average 90% interval length 1.94 on the community dataset, longer than CQR random forests (1.55).¶Figure 5 (community panel), p. 16 · registered at event 12 · quote verified on p. 16 · equivalence check | 1.94rescaled response units | 1.8578 | within ±0.194 | Held |
Deviations from the paper's setup
4 recorded: 3 minor, 1 none
- noneOne-line patch to the authors' datasets/datasets.py (community loader): `from sklearn.preprocessing import Imputer` / `Imputer(missing_values='NaN', strategy='mean', axis=0)` replaced by `from sklearn.impute import SimpleImputer as Imputer` / `Imputer(missing_values=np.nan, strategy='mean')`. The full git diff is in the output of event 16; it is byte-identical to the patch in the earlier attempt arxiv-1905-03222-20261019T184203Z.
- minorSoftware environment rebuilt in 2026 (same stack as the earlier attempt): Python 3.8.20, numpy 1.18.5, scipy 1.4.1, scikit-learn 0.22.2.post1, pandas 1.0.5, torch 1.4.0+cpu, and scikit-garden built from its git master (fbd9a5b, 2022) because the 0.1.3 PyPI sdist ships pre-generated C that does not compile on Python >= 3.7. The quantile-forest Python code in master differs from 0.1.3 only in import paths and removal of the `presort` argument (diff in event 15). The paper does not state library versions; the 2019 originals were likely Python 3.6/3.7, scikit-learn 0.20/0.21 and torch 1.0/1.1. GCC needed `-Wno-error=incompatible-pointer-types -Wno-error=int-conversion` to compile the Cython output (build flags only).
- minorOnly 3 of the paper's 11 datasets (community, concrete, bike) and 3 of its 8 interval methods (CQR quantile random forest, CQR quantile neural net, split-conformal random forest; the RF-local variant runs alongside split RF because the authors' script fits both) are run. The paper's Table 1 numbers are averages over all 11 datasets and are not tested here; per-dataset claims are taken from the numbers printed in Figures 4-5.
- minorFewer random training/test splits than the paper's 20: splits (seeds) 0-4 for split-conformal RF on community, CQR quantile RF on community and concrete, and CQR quantile net on community; only splits 0-1 for CQR quantile net on bike. The authors' run_experiment() is called unchanged per seed, so each seed reproduces the paper's split for that seed. The split counts are the same as the earlier attempt and were fixed before any job started (events 19-22).
The run
- Wall time
- 40 min 38 s, 19 October 2026 to 20 October 2026
- Job compute
- 1 h 4 min across 4 jobs
- Work tree at the end
- yromano/cqr at 2b9b5c7
- Container image
- repro-sandbox:0.2 sha256:319dfea6bd8d…
- Sandbox
- 3.5 CPUs, 6 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-1905-03222-20261019T233532Z
- Log
- 55 events, head 1a032d5f
Evidence
log seal, 55 hash-chained events, 1 earlier version
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 55 hash-chained events
1a032d5f5c4c462b52fd0bd3b2c0405dabe9029151ad90ac318c6df4d3518877
Earlier versions of this report
How to cite this report
OpenRepro (2026). Reproducibility report: Conformalized Quantile Regression. Reproduced; 5 of 5 headline claims held (legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1905-03222-20261019T233532Z, log head 1a032d5f5c4c462b, checked 20 Oct 2026. https://openrepro.org/papers/1905.03222/
APA style
OpenRepro. (2026). Reproducibility report: Conformalized Quantile Regression (Attempt arxiv-1905-03222-20261019T233532Z; log head 1a032d5f5c4c462b). https://openrepro.org/papers/1905.03222/
BibTeX
@techreport{openrepro-1905.03222,
author = {{OpenRepro}},
title = {Reproducibility report: Conformalized Quantile Regression},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/1905.03222/},
note = {Attempt arxiv-1905-03222-20261019T233532Z; log head 1a032d5f5c4c462b; checked 20 Oct 2026}
}
References
1 paper
- Yaniv Romano, Evan Patterson, Emmanuel J. Candès (2019). Conformalized Quantile Regression. arXiv 1905.03222
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
Copy the badge as Markdown or HTML
[](https://openrepro.org/papers/1905.03222/)<a href="https://openrepro.org/papers/1905.03222/"><img src="https://openrepro.org/badge/1905.03222.svg" alt="OpenRepro: Reproduced. 5 of 5 headline claims held; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="416" height="20"></a>Right of reply, and reporting an error
Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.