Replication record · arXiv:1610.01271

Generalized random forests

Susan Carleton Athey, Julie Tibshirani, Stefan Wager · 2019 · doi:10.1214/18-aos1709 · arXiv

Partially reproduced

The authors Table 1 replication script, run with the grf package from the same 2020 commit, matches only 3 of the 12 C. GRF values: with confounding only, the 2020 package gives much lower MSE than the paper (e.g. 0.11 vs 0.27), and in the heterogeneity settings higher MSE (e.g. 1.41 vs 0.93). Running the same script with grf 0.9.5, the release current at the paper date (a declared fallback), reproduces 11 of the 12 values, including all four confounding-only cells to two decimals; the exception is confounding plus heterogeneity with p=20, n=800 (1.17 vs 0.93). The README for these replication files warns that the package has changed since the paper. The WA-1, WA-2 and uncentered GRF columns were not tested.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 1 of 4 consistent (2 more only with earlier code)
Other checks: 8 of 8 consistent
Deviations: 1 (0 material)
Integrity flags: 0
Logged compute: 2.99 h over 6 commands and 4 jobs
Primary code (declared before the first recorded result): github.com/grf-labs/grf (the grf package named in the paper as available from CRAN): the authors replication script experiments/aos19/causal_mse/causal_mse.R, with the grf R package built from the same commit at which that script was last changed
Commits the recorded results name: e93a79fe: 12 results, named in the result's method text; grf v0.9.5 (2018-01-04) (earlier code): 9 results, the result's code_ref
Work tree when the attempt ended: grf-labs/grf@e93a79fe5e8c in /work/grf

Protocol

Declared before the first recorded result: primary code is github.com/grf-labs/grf (the grf package named in the paper as available from CRAN): the authors replication script experiments/aos19/causal_mse/causal_mse.R, with the grf R package built from the same commit at which that script was last changed; earlier versions of the authors' code may be tried as a fallback. The paper (arXiv v4, 2018-04-05) links no repository; the grf repository contains a replication script dedicated to this paper (aos19 = Annals of Statistics 2019), written after the paper. That script with its contemporaneous package is the most faithful authors code. If claims miss, the grf release current at the paper date may be tried as a declared fallback.

Claims

C. GRF (causal forest with local centering) has MSE x 10 of 0.87 with confounding=no, heterogeneity=yes, p=10, n=800.

Consistent

paper 0.87observed 0.980271

Paper 0.87 MSE x 10Observed 0.980271 MSE x 10Accepted if within ±0.131

no yes 10 800 1.37 6.48 0.85 0.87

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.127 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Why this tolerance. Each value is a Monte Carlo average over 60 simulated datasets, which leaves several percent of sampling noise, and the replication script runs a later package version; 15% (at least 0.03) covers both.

Event 38. Observed 0.980271, delta +0.1103, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

C. GRF (causal forest with local centering) has MSE x 10 of 0.59 with confounding=no, heterogeneity=yes, p=10, n=1600.

Consistent

paper 0.59observed 0.557989

Paper 0.59 MSE x 10Observed 0.557989 MSE x 10Accepted if within ±0.088

no yes 10 1600 0.63 6.23 0.58 0.59

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 0.9457 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 39. Observed 0.557989, delta -0.03201, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

C. GRF (causal forest with local centering) has MSE x 10 of 0.93 with confounding=no, heterogeneity=yes, p=20, n=800.

Consistent with earlier code

paper 0.93observed 1.16781

Paper 0.93 MSE x 10Observed 1.16781 MSE x 10Earlier code 0.958962 MSE x 10Accepted if within ±0.14

no yes 20 800 2.05 8.02 0.92 0.93

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.256 · registered at event 13, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 40. Observed 1.16781, delta +0.2378, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 61 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.958962, delta +0.02896, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.52 with confounding=no, heterogeneity=yes, p=20, n=1600.

Consistent with earlier code

paper 0.52observed 0.653395

Paper 0.52 MSE x 10Observed 0.653395 MSE x 10Earlier code 0.550854 MSE x 10Accepted if within ±0.078

no yes 20 1600 0.71 7.61 0.52 0.52

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.257 · registered at event 14, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 41. Observed 0.653395, delta +0.1334, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 62 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.550854, delta +0.03085, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.27 with confounding=yes, heterogeneity=no, p=10, n=800.

Consistent with earlier code

paper 0.27observed 0.108426

Paper 0.27 MSE x 10Observed 0.108426 MSE x 10Earlier code 0.276805 MSE x 10Accepted if within ±0.041

yes no 10 800 0.81 0.16 1.12 0.27

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 0.4016 · registered at event 15, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 42. Observed 0.108426, delta -0.1616, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 63 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.276805, delta +0.006804, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.2 with confounding=yes, heterogeneity=no, p=10, n=1600.

Consistent with earlier code

paper 0.2observed 0.0657384

Paper 0.2 MSE x 10Observed 0.0657384 MSE x 10Earlier code 0.200031 MSE x 10Accepted if within ±0.03

yes no 10 1600 0.68 0.10 0.80 0.20

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 0.3287 · registered at event 16, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 43. Observed 0.0657384, delta -0.1343, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 64 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.200031, delta +3.14e-05, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.17 with confounding=yes, heterogeneity=no, p=20, n=800.

Consistent with earlier code

paper 0.17observed 0.0858192

Paper 0.17 MSE x 10Observed 0.0858192 MSE x 10Earlier code 0.167058 MSE x 10Accepted if within ±0.03

yes no 20 800 0.90 0.13 1.17 0.17

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 0.5048 · registered at event 17, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 44. Observed 0.0858192, delta -0.08418, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 65 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.167058, delta -0.002942, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.11 with confounding=yes, heterogeneity=no, p=20, n=1600.

Consistent with earlier code

paper 0.11observed 0.0477852

Paper 0.11 MSE x 10Observed 0.0477852 MSE x 10Earlier code 0.110018 MSE x 10Accepted if within ±0.03

yes no 20 1600 0.77 0.09 0.95 0.11

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 0.4344 · registered at event 18, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 45. Observed 0.0477852, delta -0.06221, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 66 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.110018, delta +1.78e-05, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.91 with confounding=yes, heterogeneity=yes, p=10, n=800.headline

Consistent with earlier code

paper 0.91observed 1.15953

Paper 0.91 MSE x 10Observed 1.15953 MSE x 10Earlier code 1.03662 MSE x 10Accepted if within ±0.137

yes yes 10 800 4.51 7.67 1.92 0.91

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.274 · registered at event 19, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 46. Observed 1.15953, delta +0.2495, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 67 [earlier code grf v0.9.5 (2018-01-04)]. Observed 1.03662, delta +0.1266, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.62 with confounding=yes, heterogeneity=yes, p=10, n=1600.headline

Consistent

paper 0.62observed 0.66785

Paper 0.62 MSE x 10Observed 0.66785 MSE x 10Accepted if within ±0.093

yes yes 10 1600 2.45 7.94 1.51 0.62

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.077 · registered at event 20, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 47. Observed 0.66785, delta +0.04785, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

C. GRF (causal forest with local centering) has MSE x 10 of 0.93 with confounding=yes, heterogeneity=yes, p=20, n=800.headline

Inconsistent

paper 0.93observed 1.40629

Paper 0.93 MSE x 10Observed 1.40629 MSE x 10Earlier code 1.16717 MSE x 10Accepted if within ±0.14

yes yes 20 800 5.93 8.68 1.92 0.93

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.512 · registered at event 21, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 48. Observed 1.40629, delta +0.4763, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 68 [earlier code grf v0.9.5 (2018-01-04)]. Observed 1.16717, delta +0.2372, inconsistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

C. GRF (causal forest with local centering) has MSE x 10 of 0.57 with confounding=yes, heterogeneity=yes, p=20, n=1600.headline

Consistent with earlier code

paper 0.57observed 0.735746

Paper 0.57 MSE x 10Observed 0.735746 MSE x 10Earlier code 0.613453 MSE x 10Accepted if within ±0.085

yes yes 20 1600 3.54 8.61 1.55 0.57

Table 1, p. 23 (column C. GRF) · quote verified on p. 23 · equivalence check · observed/published 1.291 · registered at event 22, before the first recorded result

Method, tolerance rationale and evidence

Metric. test-set mean squared error of the CATE estimate x 10, averaged over 60 replications with 1,000 test points, B = 2,000 trees

Event 49. Observed 0.735746, delta +0.1657, inconsistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Authors replication script experiments/aos19/causal_mse/causal_mse.R, unmodified, with grf built from the same commit e93a79fe (R 4.0.2, CRAN snapshot 2020-09-15 for dependencies); 60 replications per cell with 1,000 test points, default causal_forest (2,000 trees); value printed as C.GRF (MSE x 10).

Event 69 [earlier code grf v0.9.5 (2018-01-04)]. Observed 0.613453, delta +0.04345, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Declared fallback (protocol event 10): the same replication script with grf v0.9.5 (CRAN release of 2018-01-04, the last before arXiv v4) built from source with -include cstddef/cstdint, and the authors DGP functions loaded from dgps.R at e93a79fe (one added line, diff in the job output).

Deviations from the paper's protocol

none
grf v0.9.5 was compiled with -include cstddef -include cstdint (build flags only) and run with R 4.0.2 and 2020 dependency versions.
Its C++ sources rely on headers that older compilers included implicitly; GCC 14 requires them explicitly. The source code is unchanged. event 71

Notes from the attempt

event 23
Scope, declared before any run: the 12 C. GRF values in Table 1. The WA-1, WA-2 and uncentered GRF columns are not tested because the replication script produces only the C. GRF column.
event 30
The first run (event 27) installed the CRAN grf 1.2.0 release instead of building the package from commit e93a79fe as the protocol at event 10 specifies, and failed because generate_causal_data was added to the package at that commit (dgps.R moved into R/), after the 1.2.0 release. The next run builds grf from source at e93a79fe as declared; the CRAN release is not used for any result.
event 37
The job at event 35 exited 1 only because the authors script begins with rm(list = ls()), which removed the wrapper timer variable t0, so the final elapsed-time print failed after the full results table had been printed. The wall time from the job record is 4,863 s (it shared the droplet with another attempt for part of that).
event 70
For completeness, the fallback run (event 59) also gives the three cells whose primary result was already consistent: no confounding p=10 n=800 0.938, n=1600 0.582; both p=10 n=1600 0.645. Those are not recorded as results because earlier code is only a fallback for inconsistent primary results.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T02:36:19.448Z → 2026-10-18T06:17:37.196Z
Event log: 76 events, hash-chained; head 51d9021ae87e4225ae417c985d3db4c095731b53a8ec8e69c29b669625dc4686
Verify: python3 openrepro_verify.py arxiv-1610-01271-20261018T023619Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.