Statistics · 2019
Reproducibility report: Generalized random forests
3 of 3 headline claims held
bound check1 other claim missedscored run not a direct sourceThe full answer and its qualifiers
Do Generalized Random Forests (Athey et al., 2019) reproduce? Yes, on bound checks: when OpenRepro re-ran the authors' code, 3 of 3 headline claims held within tolerances sealed before scored execution; every headline check is a one-sided bound, not a match to the published value; 1 other check did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one (checked 20 Oct 2026).
- Every headline check is a one-sided bound, not a match to the published value.
- 1 other check did not hold.
- A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.
Do Generalized Random Forests (Athey et al., 2019) reproduce? Yes, on bound checks: when OpenRepro re-ran the authors' code, 3 of 3 headline claims held within tolerances sealed before scored execution (checked 20 Oct 2026).
- Checked
- Released
- Assurance
- Sealed, bound and re-derived
- Evidence
- Full report · report.json
- Data
- JSON record · How to cite
- Assistants
- Ask via MCP
Do Generalized Random Forests reproduce?
Yes, on bound checks. 3 of 3 headline claims held within tolerances sealed before scored execution.
- Every headline check is a one-sided bound, not a match to the published value.
- 1 other check did not hold.
- A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.
- Held (large: headline claim; small: other claim)
- Did not hold
Reading the marks
- Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly
- Chevron at the edge: beyond ±3 tolerances, with the actual multiple printed beside it (∞ when the tolerance was zero)
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| Plan compatibility rule, part 1: the lower end of the literal-paper-DGP (P) interval for C. GRF MSE x 10 at p=20, n=800 is at most 0.935, the top of the rounding interval of the published 0.93. (headline) | 0.93 | 0.904628 | at most 0.935 | -5.07 | Held |
| Plan compatibility rule, part 2: the upper end of the literal-paper-DGP (P) interval for C. GRF MSE x 10 at p=20, n=800 is at least 0.925, the bottom of the rounding interval of the published 0.93. (headline) | 0.93 | 1.12361 | at least 0.925 | +38.72 | Held |
| Plan precision gate: the half-width q*SE_P of the P interval is at most 0.11858505, half the previously observed 1.1671701-0.93 gap. paper_value is the plan's gate threshold, not a published number; the quote is the caption's 60-replication design that the gate is about. (headline) | 0.118585 | 0.109491 | at most 0.118585 | beyond limit | Held |
| Plan diagnostic: the paired helper excess H-P (helper DGP minus literal DGP, same seeds) has an interval whose lower end is above 0. paper_value 0 is the plan's sign threshold, not a published number. | 0 | -0.0895492 | at least 0 | beyond limit | Did not hold |
| Reporting only: the primary number, the mean of the 60 P-arm MSE x 10 values (literal paper DGP), for comparison with the published 0.93. Its compatibility is judged only by the two interval-end claims. | 0.93 | 1.01412 | within ±1000 | +0.00 | Held |
| Reporting only: mean of the 60 H-arm MSE x 10 values (2020 aw3 helper DGP, same seeds and grf 0.9.5), the control; the predecessor's helper value was 1.1671701. | 0.93 | 1.12473 | within ±1000 | +0.00 | Held |
| Reporting only: mean paired difference H-P of MSE x 10 over the 60 replications. paper_value 0 is a reference, not a published number. | 0 | 0.110612 | within ±1000 | +0.00 | Held |
What the attempt found
Sealed settling test of the disputed Table 1 cell (C. GRF, confounding and heterogeneity, p=20, n=800, published 0.93 MSE x 10), pairing the paper's literal DGP (P) with the 2020 aw3 helper (H) under grf 0.9.5, 60 replications with fixed seeds. P mean 1.014, interval [0.905, 1.124] (q*SE 0.109, inside the 0.1186 precision gate), intersects the rounding interval [0.925, 0.935]: compatible with the published point at the stated precision. H mean 1.125, interval [0.980, 1.270]; paired excess H-P 0.111, interval [-0.090, 0.311], not resolved as positive. Plan branch: the literal-paper cell is compatible with the published point, but this experiment does not establish that the helper caused our earlier gap.
Assurance
- Level
- Sealed, bound and re-derived (what the levels mean)
- Plan
- Sealed before scored execution; 6 outputs observed before the seal (all of it in the log)
- Evidence
- Bound: each value was extracted from the cited run output by a rule fixed before the seal
- Execution manifest
- Yes: for each scored run the command, the Git checkout state (HEAD, status, diff hash) and the hashes of the driver files the command names were recorded when it started and when it ended. Data files and the environment are not bound by this
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Plan compatibility rule, part 1: the lower end of the literal-paper-DGP (P) interval for C. GRF MSE x 10 at p=20, n=800 is at most 0.935, the top of the rounding interval of the published 0.93.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 5 · quote verified on p. 23 · bound check, observed 0.973× the published value | 0.93MSE x 10 | 0.904628 | at most 0.935 | Held |
| Plan compatibility rule, part 2: the upper end of the literal-paper-DGP (P) interval for C. GRF MSE x 10 at p=20, n=800 is at least 0.925, the bottom of the rounding interval of the published 0.93.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 6 · quote verified on p. 23 · bound check, observed 1.21× the published value | 0.93MSE x 10 | 1.12361 | at least 0.925 | Held |
| Plan precision gate: the half-width q*SE_P of the P interval is at most 0.11858505, half the previously observed 1.1671701-0.93 gap. paper_value is the plan's gate threshold, not a published number; the quote is the caption's 60-replication design that the gate is about.¶Table 1 caption, p. 23 · registered at event 7 · quote verified on p. 23 · bound check, observed 0.923× the published value | 0.118585MSE x 10 | 0.109491 | at most 0.118585 | Held |
Other claims checked
4 claims: 3 held, 1 did not hold
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Plan diagnostic: the paired helper excess H-P (helper DGP minus literal DGP, same seeds) has an interval whose lower end is above 0. paper_value 0 is the plan's sign threshold, not a published number.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 8 · quote verified on p. 23 · bound check | 0MSE x 10 | -0.0895492 | at least 0 | Did not hold |
| Reporting only: the primary number, the mean of the 60 P-arm MSE x 10 values (literal paper DGP), for comparison with the published 0.93. Its compatibility is judged only by the two interval-end claims.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 9 · quote verified on p. 23 · broad bound | 0.93MSE x 10 | 1.01412 | within ±1000 | Held |
| Reporting only: mean of the 60 H-arm MSE x 10 values (2020 aw3 helper DGP, same seeds and grf 0.9.5), the control; the predecessor's helper value was 1.1671701.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 10 · quote verified on p. 23 · broad bound | 0.93MSE x 10 | 1.12473 | within ±1000 | Held |
| Reporting only: mean paired difference H-P of MSE x 10 over the 60 replications. paper_value 0 is a reference, not a published number.¶Table 1, p. 23 (row conf.=yes, heterog.=yes, p=20, n=800; column C. GRF) · registered at event 11 · quote verified on p. 23 · broad bound | 0MSE x 10 | 0.110612 | within ±1000 | Held |
Deviations from the paper's setup
7 recorded: 3 minor, 4 none
- minorThe sandbox base image was rebuilt on a new box (image id sha256:35f8fd46..., Debian 13, gcc 14.2.0-19), not the archived job's image sha256:36b1e94b..., which no longer exists. R 4.0.2 (deb sha256 7786359e..., verified), the dependency set from the 2020-09-15 snapshot (Rcpp 1.0.5, RcppEigen 0.3.3.7.0, DiceKriging 1.5.6, lmtest, sandwich, zoo; Matrix 1.2-18 shipped with R), the grf 0.9.5 tarball and the build flags are identical by hash; the environment is pinned by content digest in /work/settle/env-digest.txt (sha256 78e92889...) before the seal.
- nonegrf 0.9.5 was built from the CRAN archive tarball grf_0.9.5.tar.gz (sha256 46ead518..., packaged 2018-01-09), as in the archived job, not from a checkout of commit a1b8df1b (tag v0.9.5, 2018-01-04). Inspection (event 14): causal_forest.R and input_utilities.R differ only in roxygen comments; the mtry default changes affect only standalone regression/quantile/instrumental forests (causal_forest passes mtry to its nuisance forests explicitly); the C++ differs only by trailing newlines, plus the generated RcppExports.cpp.
- noneBuild-only patch: grf 0.9.5 compiled with -include cstddef -include cstdint (Makevars sha256 b243d82c..., the same flags as archived job 00056). No source change.
- minorThe plan's compatibility tolerance 0.005 + q*SE (SE-dependent) is registered as two fixed bounds on the interval ends (P_ci_lower <= 0.935, P_ci_upper >= 0.925), plus the precision gate as a third headline bound; the H-P sign test is registered as >= 0 (plan: > 0). The harness extracts and judges these values but does not itself rederive SDs, SEs or q from the 120 rows; that computation is the sealed analysis script's (sha256 3400b615...), whose 120 recomputed rows are printed in the cited artifact so anyone can recompute it. Per the plan this is a qualified statistical analysis, not a harness-rederived verdict on the formula. Three report-only claims carry deliberately vacuous bands and their verdicts mean nothing.
- noneForest seeding: set.seed(163000+r) then forest seed = runif(1, 0, .Machine$integer.max), i.e. grf 0.9.5's own rule for seed = NULL applied to the reset stream (as in the authors' script, which leaves seed NULL), passed explicitly and logged; not causal_forest(seed = 163000 + r) literally.
- minorThe sealed plan is bound in this attempt's hash-chained log (seal event 25, plan hash 74d6a77e...; plan document sha256 eec2efa4... written at event 17) but was not anchored outside the log before execution; external anchoring happens only when the attempt is committed/released.
- noneDedicated box in DigitalOcean tor1 (s-4vcpu-8gb-intel, the archived job's size) instead of nyc3.
The run
- Wall time
- 21 min 1 s, 20 October 2026 to 20 October 2026
- Job compute
- 7 min 3 s across 2 jobs
- Code the results name
- grf-labs/grf at a1b8df1
- Work tree at the end
- grf-labs/grf at a1b8df1
- Container image
- repro-sandbox:0.2 sha256:35f8fd46ec06…
- Sandbox
- 3.5 CPUs, 6 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-1610-01271-20261020T030230Z
- Log
- 54 events, head 6252e100
Evidence
log seal, 54 hash-chained events, 2 earlier versions
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 54 hash-chained events
6252e1001c1a681990a613fbd32a2e02f351160eff41659fdcafdf705bf1c3b1
Earlier versions of this report
How to cite this report
OpenRepro (2026). Reproducibility report: Generalized random forests. Reproduced; 3 of 3 headline claims held (every headline check is a one-sided bound, not a match to the published value; 1 other check did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one). Attempt arxiv-1610-01271-20261020T030230Z, log head 6252e1001c1a6819, checked 20 Oct 2026. https://openrepro.org/papers/1610.01271/
APA style
OpenRepro. (2026). Reproducibility report: Generalized random forests (Attempt arxiv-1610-01271-20261020T030230Z; log head 6252e1001c1a6819). https://openrepro.org/papers/1610.01271/
BibTeX
@techreport{openrepro-1610.01271,
author = {{OpenRepro}},
title = {Reproducibility report: Generalized random forests},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/1610.01271/},
note = {Attempt arxiv-1610-01271-20261020T030230Z; log head 6252e1001c1a6819; checked 20 Oct 2026}
}
References
1 paper
- Susan Carleton Athey, Julie Tibshirani, Stefan Wager (2019). Generalized random forests. The Annals of Statistics. arXiv 1610.01271 · doi:10.1214/18-aos1709
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
Copy the badge as Markdown or HTML
[](https://openrepro.org/papers/1610.01271/)<a href="https://openrepro.org/papers/1610.01271/"><img src="https://openrepro.org/badge/1610.01271.svg" alt="OpenRepro: Reproduced. 3 of 3 headline claims held; every headline check is a one-sided bound, not a match to the published value; 1 other check did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one." width="537" height="20"></a>Right of reply, and reporting an error
1 author reply
Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.