Replication record · arXiv:1905.03222

Conformalized Quantile Regression

Yaniv Romano, Evan Patterson, Emmanuel J. Candès · 2019 · arXiv

Reproduced

Ran the authors' reproducible_experiments/run_experiment.py (commit 2b9b5c7, the last before arXiv v1) unchanged except for a one-line sklearn Imputer compatibility patch, on three bundled datasets with a rebuilt 2019-era Python 3.8 stack. All six registered per-dataset claims taken from Figures 4-5 landed within tolerance: CQR quantile-forest interval length 1.507 vs 1.55 (community) and 0.558 vs 0.56 (concrete), CQR quantile-net 1.856 vs 1.76 (community) and 0.570 vs 0.57 (bike), split-conformal RF 1.858 vs 1.94 (community, longer than CQR as reported), and every CQR cell averaged at least 89.26% coverage against the 90% nominal level. Splits were reduced from 20 to 5 (2 for bike), and the paper's headline Table 1 averages over all 11 datasets were not tested because MEPS and the larger UCI datasets were outside this attempt's data and CPU budget. Seven claims from an unrelated paper were registered by operator error and withdrawn as non-headline placeholders before any result (see note at event 43).

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Some claims were registered after command output had already been observed. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 5 of 5 consistent
Other checks: 1 of 1 consistent
Deviations: 4 (0 material)
Integrity flags: 0
Logged compute: 2.78 h over 24 commands and 3 jobs
Primary code (declared before the first recorded result): github.com/yromano/cqr at 2b9b5c7d62de7f63af49e86fc570db5c2c77d637 (2019-05-03), the last commit on or before the arXiv v1 date 2019-05-08 (v1 is the only arXiv version). Experiments run through the authors' reproducible_experiments/run_experiment.py with its hard-coded settings.
Work tree when the attempt ended: yromano/cqr@2b9b5c7d62de in /work/cqr, 1 file(s) modified locally (see code-diff.patch)

Protocol

Declared before the first recorded result: primary code is github.com/yromano/cqr at 2b9b5c7d62de7f63af49e86fc570db5c2c77d637 (2019-05-03), the last commit on or before the arXiv v1 date 2019-05-08 (v1 is the only arXiv version). Experiments run through the authors' reproducible_experiments/run_experiment.py with its hard-coded settings.; earlier versions of the authors' code are not used. arXiv 1905.03222 has only v1 (8 May 2019); 2b9b5c7 is the last commit before it and already contains reproducible_experiments/ with the paper's hyper-parameters. Later commits (2019-06 bug fixes, 2022 MEPS cleaning fix, 2026 numpy fix) postdate the paper and are not used.

Claims

Conformal methods, including both CQR variants, attain the nominal 90% coverage; observed value is the LOWEST split-averaged test coverage among all CQR (method, dataset) cells run in this attempt.headline

Consistent

paper 90observed 89.26

Paper 90 %Observed 89.26 %Accepted if at least 88

all methods based on conformal prediction successfully construct prediction bands at the nominal coverage rate of 90%

Section 6.2, p. 10 · quote verified on p. 10 · bound check · observed/published 0.9918 · registered at event 25, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. minimum over tested CQR (method, dataset) cells of mean test coverage (%) over random splits

Why this tolerance. Coverage >= 1-alpha is a finite-sample guarantee of the method, so the right test is one-sided (at_least). Split-averaged empirical coverage still fluctuates: on concrete (206 test points, ~412 calibration points) the per-split coverage SD is ~2.6 points, so a 10-20 split mean has SD ~0.6-0.8 points, and taking the minimum over up to 5 cells pushes it lower. A floor of 88% (2 points below nominal) is ~2.5-3 SD and would still catch a real failure of the guarantee such as the 88.51% undercoverage the paper reports for non-conformalized quantile nets.

Event 92. Observed 89.26, delta -0.74, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Minimum over the five CQR (method, dataset) cells run (CQR RF community 90.53, CQR RF concrete 90.19, CQR Net community 89.87, CQR Net bike 89.26) of the split-averaged test coverage in percent. Split-conformal RF on community: 89.97; RF-local 90.13. Lowest single split for any CQR cell: 86.89 (CQR RF, concrete, split 4).

CQR with a quantile neural network gives average 90% interval length 0.57 on the bike sharing dataset (20 random 80/20 splits).headline

Consistent

paper 0.57observed 0.5695

Paper 0.57 rescaled response unitsObserved 0.5695 rescaled response unitsAccepted if within ±0.057

0.57 0.62 0.71 0.76 0.72 0.70 2.26 2.20 Avg. Length Avg. Coverage 0.5 1.0 1.5 2.0 80% 85% 90% 95% 100% CQR Neural Net CQR Random Forests Neural Net Local Neural Net Random Forests Local Random Forests Ridge Local Ridge bike

Figure 4 (bike panel), p. 15 · quote verified on p. 15 · equivalence check · observed/published 0.9991 · registered at event 26, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. average interval length on test set, mean over random splits, response rescaled by mean |y| of training set

Why this tolerance. The paper reports only the 20-split mean (rounded to 2 decimals) and shows split-to-split spread only as boxplots, so no numeric std is available. Tolerance is +/-10% of the paper value: it covers the standard error of a 10-20 split mean on these small datasets (split-level length std is typically 5-10% of the mean, so SE ~1-3%), unseeded/version-dependent NN and forest randomness from a 2026 rebuild of a 2019 stack, and 2-decimal rounding, while staying well below the 15-25% gaps between CQR and split conformal that the paper's comparison rests on. Figures 3-6 print 8 average lengths per dataset; the extracted text lists them in the fixed order CQR Neural Net, CQR Random Forests, Neural Net, Neural Net Local, Random Forests, Random Forests Local, Ridge, Ridge Local. This mapping was fixed BEFORE any run by averaging each position over the 11 datasets: the means (1.397, 1.407, 2.165, 1.805, 2.241, 1.823, 3.062, 2.944) reproduce Table 1 (1.40, 1.41, 2.16, 1.81, 2.24, 1.82, 3.06, 2.94); swapping any pair breaks the match.

Event 91. Observed 0.5695, delta -0.0005, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Mean over splits 0-1 of the 'CQR Net' average test-interval length on bike from run_experiment(..., 'cqr_quantile_net', seed), averaged by summarize.py. Only 2 of 20 splits (0.580, 0.559): each bike split takes ~36-40 min on CPU.

CQR with a quantile neural network gives average 90% interval length 1.76 on the community dataset (20 random 80/20 splits).headline

Consistent

paper 1.76observed 1.8561

Paper 1.76 rescaled response unitsObserved 1.8561 rescaled response unitsAccepted if within ±0.176

1.76 1.55 2.06 1.98 1.94 1.83 1.90 1.85 Avg. Length Avg. Coverage 1.0 1.5 2.0 2.5 80% 85% 90% 95% 100% CQR Neural Net CQR Random Forests Neural Net Local Neural Net Random Forests Local Random Forests Ridge Local Ridge community

Figure 5 (community panel), p. 16 · quote verified on p. 16 · equivalence check · observed/published 1.055 · registered at event 27, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. average interval length on test set, mean over random splits, response rescaled by mean |y| of training set

Event 88. Observed 1.8561, delta +0.0961, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Mean over splits 0-4 of the 'CQR Net' average test-interval length on community from the authors' run_experiment(..., 'cqr_quantile_net', seed) (splits 0-1 in job nn-s0-5, 2-4 in job nn-comm-s2-5), averaged by summarize.py. Per-split lengths 1.829, 1.629, 2.301, 1.666, 1.856 (split SD 0.27, so a 5-split mean has SE ~0.12); 5 of 20 splits.

CQR with quantile random forests gives average 90% interval length 1.55 on the community dataset (20 random 80/20 splits).headline

Consistent

paper 1.55observed 1.507

Paper 1.55 rescaled response unitsObserved 1.507 rescaled response unitsAccepted if within ±0.155

1.76 1.55 2.06 1.98 1.94 1.83 1.90 1.85 Avg. Length Avg. Coverage 1.0 1.5 2.0 2.5 80% 85% 90% 95% 100% CQR Neural Net CQR Random Forests Neural Net Local Neural Net Random Forests Local Random Forests Ridge Local Ridge community

Figure 5 (community panel), p. 16 · quote verified on p. 16 · equivalence check · observed/published 0.9723 · registered at event 28, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. average interval length on test set, mean over random splits, response rescaled by mean |y| of training set

Event 87. Observed 1.507, delta -0.043, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Mean over splits 0-4 of the 'CQR RF' average test-interval length on community, from the authors' run_experiment(dataset='community', test_method='cqr_quantile_forest', seed) rows in results.csv (job events 68), averaged by summarize.py. Per-split lengths 1.571, 1.344, 1.536, 1.517, 1.566; 5 of 20 splits.

CQR with quantile random forests gives average 90% interval length 0.56 on the concrete dataset (20 random 80/20 splits).headline

Consistent

paper 0.56observed 0.5584

Paper 0.56 rescaled response unitsObserved 0.5584 rescaled response unitsAccepted if within ±0.056

0.48 0.56 0.50 0.49 0.54 0.53 0.99 0.92 Avg. Length Avg. Coverage 0.4 0.6 0.8 1.0 80% 85% 90% 95% 100% CQR Neural Net CQR Random Forests Neural Net Local Neural Net Random Forests Local Random Forests Ridge Local Ridge concrete

Figure 5 (concrete panel), p. 16 · quote verified on p. 16 · equivalence check · observed/published 0.9971 · registered at event 29, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. average interval length on test set, mean over random splits, response rescaled by mean |y| of training set

Event 90. Observed 0.5584, delta -0.0016, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Mean over splits 0-4 of the 'CQR RF' average test-interval length on concrete from run_experiment(..., 'cqr_quantile_forest', seed), averaged by summarize.py. 5 of 20 splits.

Standard split conformal prediction with a conditional-mean random forest gives average 90% interval length 1.94 on the community dataset, longer than CQR random forests (1.55).

Consistent

paper 1.94observed 1.8578

Paper 1.94 rescaled response unitsObserved 1.8578 rescaled response unitsAccepted if within ±0.194

1.76 1.55 2.06 1.98 1.94 1.83 1.90 1.85 Avg. Length Avg. Coverage 1.0 1.5 2.0 2.5 80% 85% 90% 95% 100% CQR Neural Net CQR Random Forests Neural Net Local Neural Net Random Forests Local Random Forests Ridge Local Ridge community

Figure 5 (community panel), p. 16 · quote verified on p. 16 · equivalence check · observed/published 0.9576 · registered at event 31, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. average interval length on test set, mean over random splits, response rescaled by mean |y| of training set

Event 89. Observed 1.8578, delta -0.0822, consistent. Evidence events [86] (value matched by the legacy rule; not bound to an excerpt). Mean over splits 0-4 of the 'RF' (standard split conformal, conditional-mean random forest) average test-interval length on community, from run_experiment(..., 'random_forest', seed), averaged by summarize.py. Per-split 1.795, 1.859, 1.834, 1.885, 1.915; CQR RF on the same splits is 1.507, i.e. shorter, as the paper reports. RF-local (not a registered claim) gave 1.7613 vs paper 1.83.

Withdrawn registrations

Registered, then withdrawn before the claim had any result. Withdrawn before the seal: not scored, not counted and not listed above. Withdrawn after the seal: still scored and counted above, as withdrawn.

withdrawn
cond-dr-simple-att, cond-dr-simple-se, data-treated-pop-mean, uncond-att-2004-2007, uncond-event-study-overall, uncond-simple-att, uncond-simple-se (marked WITHDRAWN by re-registration, no result, events 36, 37, 38, 39, 40, 41, 42): not a claim of this paper. Registered at event 23 by operator error: a stale claim file from an unrelated attempt (arXiv 1803.09015, Callaway & Sant'Anna) sat in the operator's shared scratch directory and was picked up by a glob. Re-registered here as a non-headline placeholder; it is never tested and does not count toward the outcome. (reason quoted from the first; the others differ only in event numbers)

Deviations from the paper's protocol

none
One-line patch to the authors' datasets/datasets.py (community loader): `from sklearn.preprocessing import Imputer` / `Imputer(missing_values='NaN', strategy='mean', axis=0)` replaced by `from sklearn.impute import SimpleImputer as Imputer` / `Imputer(missing_values=np.nan, strategy='mean')`. The full git diff is in the output of event 49.
sklearn.preprocessing.Imputer was removed in scikit-learn 0.22, the oldest release with Python 3.8 wheels available in the sandbox. SimpleImputer performs the same column-mean imputation of the single column with missing values (OtherPerCap). event 52
minor
Software environment rebuilt in 2026: Python 3.8.20, numpy 1.18.5, scipy 1.4.1, scikit-learn 0.22.2.post1, pandas 1.0.5, torch 1.4.0+cpu, and scikit-garden built from its git master (fbd9a5b, 2022) because the 0.1.3 PyPI sdist ships pre-generated C that does not compile on Python >= 3.7. The quantile-forest Python code in master differs from 0.1.3 only in import paths and removal of the `presort` argument (diff in event 21). The paper does not state library versions; the 2019 originals were likely Python 3.6/3.7, scikit-learn 0.20/0.21 and torch 1.0/1.1. GCC 14 needed `-Wno-error=incompatible-pointer-types` to compile the Cython output (build flag only).
The exact 2019 environment is not recorded and Python 3.7 is not available from the sandbox's package manager; this is the closest buildable stack. event 53
minor
Only 3 of the paper's 11 datasets (community, concrete, bike) and 3 of its 8 interval methods (CQR quantile random forest, CQR quantile neural net, split-conformal random forest; the RF-local variant runs alongside split RF because the authors' script fits both) are run. The paper's Table 1 numbers are averages over all 11 datasets and are not tested here; per-dataset claims are taken from the numbers printed in Figures 4-5.
Budget of ~2 hours on 3.5 shared CPUs. MEPS 19/20/21 require separately downloading and processing AHRQ survey files (not bundled), blog and facebook_1/2 are separate UCI downloads, and the larger datasets (bio, blog, facebook) with 1000-tree quantile forests and up-to-1000-epoch networks exceed the CPU budget. event 54
minor
Fewer random training/test splits than the paper's 20: splits (seeds) 0-4 for split-conformal RF on community, CQR quantile RF on community and concrete, and CQR quantile net on community; only splits 0-1 for CQR quantile net on bike (split 1 only if it finishes in time). The authors' run_experiment() is called unchanged per seed, so each seed reproduces the paper's split for that seed.
Budget of ~2 hours on 3.5 shared CPUs. One bike split of the CQR quantile net (1000 full epochs plus a refit, as the authors' learner does) took about 40 minutes on CPU; a community split takes 4-5 minutes per method. The split counts were fixed from these timings, before any per-dataset averages were computed. event 72

Notes from the attempt

event 43
Operator error: events 23, 24, 30, 32, 33, 34 and 35 registered seven claims (cond-dr-*, data-treated-pop-mean, uncond-*) that belong to an unrelated paper (arXiv 1803.09015). They came from stale claim files left in the operator scratch directory by an earlier session and were picked up by a glob. No result had been recorded. Each has been re-registered as a non-headline WITHDRAWN placeholder so it cannot affect the outcome. The six CQR claims (events 25-29, 31) are the real pre-registered claims of this attempt.
event 73
Observation on the authors code (not changed): run_experiment.py fits the feature StandardScaler and the response scale (mean |y|) on the full 80% training split, i.e. proper-training plus calibration, whereas footnote 5 of the paper says these statistics are computed only on the proper training set. This attempt keeps the authors code as is.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-19T18:42:03.732Z → 2026-10-19T20:14:58.900Z
Event log: 97 events, hash-chained; head 9be22707fb1af66d77eac9d6701115985d4a21d70fbc92346c6d32d8699240a2
Verify: python3 openrepro_verify.py arxiv-1905-03222-20261019T184203Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.