Generalized random forests

Protocol

Sealed with the plan: primary code is grf R package 0.9.5 (github.com/grf-labs/grf, tag v0.9.5 = commit a1b8df1be078078f58ed9ccc50a0b40c245b2cca; built from the CRAN archive tarball grf_0.9.5.tar.gz as in the archived job, with the hashed build-only -include cstddef/cstdint flags) in BOTH arms. Arm H data: generate_causal_data(dgp='aw3', sigma.m=1, sigma.tau=1, sigma.noise=1) from r-package/grf/R/dgps.R at commit e93a79fe5e8ca3ac31f5d8af6d6bd63e78684588 (sha256 8be0a4e28f1b28355616ef6f63e7cf8707c5a0c1de2f92854ab54a0d60ed2ec8), sourced alone; the 2020 forest package is not installed. Arm P data: the paper's Sec. 6.2 DGP written literally. R 4.0.2 (Posit build, deb sha256 7786359e2c471666872d2686a487243b6b4d11bdda3eb8fd3cd629959cd06d3c) with the archived job's dependency set from the CRAN snapshot of 2020-09-15.; earlier versions of the authors' code are not used. Settling test of the disputed Table 1 cell from the predecessor attempt arxiv-1610-01271-20261018T023619Z (held as partially reproduced; its 0.9.5 fallback gave 1.1671701 vs 0.93). An independent desk check (office/audits/codex-negative-findings-2026-10-19.md, sha256 5b6e553e...) found that run used the 2020 helper DGP (X1 confounding, rescaling, separate train/test normalization), not the paper's literal DGP. This attempt executes the settling plan ~/Desktop/office/plans/1610.01271-settling-plan.md (sha256 eec2efa48ac27ed5bbaae691b696dded8733d6ad6677479dd83cd7ebf0b73363): a paired comparison of the literal DGP (P) and the helper (H) under the same 0.9.5 release, 60 replications, fixed seeds. 0.9.5 is the already-disclosed historical-release hypothesis, not a proven original binary; no version search is allowed, so earlier code is not allowed.

Run policy (sealed at event 25, plan hash 74d6a77e47225f4eef33fc116c07980ad7735070094fe0800e7a391d0d3c9fd5): Exactly 60 paired replications r=1..60 of the cell conf=yes, heterog=yes, p=20, n=800 (n_test=1000), arm P then arm H for each r, one forest fit at a time (job matrix /work/settle/job-matrix.csv sha256 12e2d0cd...). RNGkind Mersenne-Twister / Inversion / Rejection. Per arm: set.seed(161000+r) then generate the training set; set.seed(162000+r) then generate the test set; set.seed(163000+r) then forest seed = runif(1, 0, .Machine$integer.max) (grf 0.9.5's own seed=NULL rule), passed to causal_forest; identical seeds in both arms, so X and N(0,1) noise are matched and W may differ (checked mechanically, not required for W). Both arms: grf 0.9.5 causal_forest(sample.fraction=0.5, mtry=20, num.trees=2000, num.threads=4, min.node.size=NULL, honesty=TRUE, ci.group.size=2, precompute.nuisance=TRUE, alpha=0.05, lambda=0, downweight.penalty=FALSE), two 500-tree OOB nuisance regression forests, predict without variance estimation. Exactly two scored jobs, with the commands in /work/settle/commands.txt (sha256 934a79d6...): grf-settle-sim (Rscript /work/settle/sim.R, sha256 a2c5344d...) then grf-settle-analysis (Rscript /work/settle/analysis.R, sha256 3400b615...). Environment digest /work/settle/env-digest.txt sha256 78e92889...; plan document /work/settle/plan/1610.01271-settling-plan.md sha256 eec2efa4.... No pilots and no unscored simulation or analysis. Aggregation: Per replication and arm: mse = mean((prediction - tau)^2) over the 1,000 test points against that arm's own test tau, mse_x10 = 10*mse (the only unit conversion), recomputed by the analysis job from the saved prediction files and checked against replicates.csv; 120 rows with unique (arm, replicate), finite values and 1,000 ordered rows per prediction file, else the analysis fails (evidence incomplete). Estimates: P = mean of the 60 P values (the primary number); H = mean of the 60 H values; H-P = mean of the 60 paired differences. Each: SE = sample SD (n-1) / sqrt(60) over its 60 values; q = qt(1 - 0.05/(2*3), 59) (Bonferroni over the three estimates); interval = estimate +/- q*SE. Decisions, in this order: precision gate q*SE_P <= 0.11858505 (else inconclusive even if the interval contains 0.93); compatible iff the P interval intersects [0.925, 0.935] (registered as P_ci_lower <= 0.935 and P_ci_upper >= 0.925); paired excess positive iff the H-P interval lower end > 0. The applicable plan branch is reported verbatim. Display rounding only after judgement. Stopping rule: One pass: the simulation job runs once over all 120 fits under a 3600 s wall cap (timeout), then the analysis job runs once. No replications are added after seeing SEs; no seed changes; never average only survivors; exceeding the cap or the $2 cloud cap makes the comparison evidence-incomplete. Low performance, early stopping, OOM, NaN or optimizer failure are not reasons to rerun. Reruns: allowed, At most one retry per scored job, only after a documented external interruption (host loss or transient I/O) of that job, with identical inputs, seeds and command (only the job name and its fresh output directory change); both attempts are retained and the first complete invocation is scored..

Claims

Deviations from the paper's protocol

• minor

  The sandbox base image was rebuilt on a new box (image id sha256:35f8fd46..., Debian 13, gcc 14.2.0-19), not the archived job's image sha256:36b1e94b..., which no longer exists. R 4.0.2 (deb sha256 7786359e..., verified), the dependency set from the 2020-09-15 snapshot (Rcpp 1.0.5, RcppEigen 0.3.3.7.0, DiceKriging 1.5.6, lmtest, sandwich, zoo; Matrix 1.2-18 shipped with R), the grf 0.9.5 tarball and the build flags are identical by hash; the environment is pinned by content digest in /work/settle/env-digest.txt (sha256 78e92889...) before the seal.

  The archived image was built from a floating base tag on a destroyed droplet and cannot be recovered byte for byte; the plan's environment identity (R build, dependency/build inventory) was recovered by hash. event 40

• none

  grf 0.9.5 was built from the CRAN archive tarball grf_0.9.5.tar.gz (sha256 46ead518..., packaged 2018-01-09), as in the archived job, not from a checkout of commit a1b8df1b (tag v0.9.5, 2018-01-04). Inspection (event 14): causal_forest.R and input_utilities.R differ only in roxygen comments; the mtry default changes affect only standalone regression/quantile/instrumental forests (causal_forest passes mtry to its nuisance forests explicitly); the C++ differs only by trailing newlines, plus the generated RcppExports.cpp.

  Same binary as the archived 0.9.5 fallback; functionally equal to the pinned commit on the causal_forest path. event 41

• none

  Build-only patch: grf 0.9.5 compiled with -include cstddef -include cstdint (Makevars sha256 b243d82c..., the same flags as archived job 00056). No source change.

  GCC 14 no longer includes these headers transitively; the plan allows separately hashed header/include repairs. event 42

• minor

  The plan's compatibility tolerance 0.005 + q*SE (SE-dependent) is registered as two fixed bounds on the interval ends (P_ci_lower <= 0.935, P_ci_upper >= 0.925), plus the precision gate as a third headline bound; the H-P sign test is registered as >= 0 (plan: > 0). The harness extracts and judges these values but does not itself rederive SDs, SEs or q from the 120 rows; that computation is the sealed analysis script's (sha256 3400b615...), whose 120 recomputed rows are printed in the cited artifact so anyone can recompute it. Per the plan this is a qualified statistical analysis, not a harness-rederived verdict on the formula. Three report-only claims carry deliberately vacuous bands and their verdicts mean nothing.

  The engine supports only fixed tolerances; the bound pair is exactly equivalent to the plan's intersection rule. event 43

• none

  Forest seeding: set.seed(163000+r) then forest seed = runif(1, 0, .Machine$integer.max), i.e. grf 0.9.5's own rule for seed = NULL applied to the reset stream (as in the authors' script, which leaves seed NULL), passed explicitly and logged; not causal_forest(seed = 163000 + r) literally.

  The plan specifies R RNG streams reset per role; this reading was fixed before the seal and is identical in both arms. event 44

• minor

  The sealed plan is bound in this attempt's hash-chained log (seal event 25, plan hash 74d6a77e...; plan document sha256 eec2efa4... written at event 17) but was not anchored outside the log before execution; external anchoring happens only when the attempt is committed/released.

  The engine has no pre-execution external anchoring step. event 45

• none

  Dedicated box in DigitalOcean tor1 (s-4vcpu-8gb-intel, the archived job's size) instead of nyc3.

  No 4 vCPU / 8 GB size had capacity in nyc3 at run time. event 46

Notes from the attempt

• event 47

  Correction of the predecessor attempt arxiv-1610-01271-20261018T023619Z, required by the settling plan regardless of the result: “Our earlier GRF comparison used a later helper whose confounding variable, outcome construction and sample normalization differ from the paper. This also affected our 0.9.5 fallback. We withdraw its interpretation as a test of Table 1 as written. Our statement that all four confounding-only cells matched to two decimals was also incorrect; they were within our selected tolerance.” Applicable branch (precision gate passed: half-width 0.109 <= 0.11858505; P interval [0.905, 1.124] intersects [0.925, 0.935]; H-P interval [-0.090, 0.311] includes 0): “The literal-paper cell is compatible with the published point, but this experiment does not establish that the helper caused our earlier gap.” H's actual interval, published as the plan requires: H mean 1.125, interval [0.980, 1.270] (the predecessor's helper value 1.1671701 lies inside it). P mean 1.014 (interval [0.905, 1.124]). Compatibility means interval intersection with the rounding interval, not proof of equivalence; the paper's Monte Carlo SE is unavailable. This one-cell result is not a 12/12 claim; the historical 3/12 and 11/12 counts describe the old helper comparisons only. 0.9.5 is the disclosed historical-release hypothesis, not the authors' proven build.

• event 48

  Process notes: (1) A setup-only predecessor attempt arxiv-1610-01271-20261020T025027Z was abandoned before any claim, seal or scored job, because its claims would have been registered after setup output; this attempt registered the protocol and all claims (events 4-11) before any command output. (2) Mechanical pairing check (analysis job): X identical in both arms in all 60 replications (train and test), outcome noise matched to max |diff| 1.8e-15, W agreement about 50% (expected: the arms' propensities depend on different coordinates). (3) Simulation wall time 423 s, well inside the 3600 s cap; no retries, no failed invocations. (4) Values are displayed rounded here only after judgement; the cited artifact holds 17 significant digits.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed

yes

Sealed before scored execution

yes

Evidence bound to artifact excerpts

yes

Values extracted by a sealed rule

yes

Execution manifest for cited jobs

yes

Semantic re-derivation passed

yes

Observed before the seal

6 shell; provenance captured

Engine

report built by repro 0.2.1; attempt created by repro 0.2.1

Redaction before release

Before release, the operator's local home-directory path was replaced with ~ in 4 places. Every event hash from event 17 on changed with it. The seal event's bytes changed (its chain hash); its plan hash did not. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now.

Rule: operator home directory prefix → ~. Reason: Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~ Recorded at event 53; the head the operator states for the log before the redaction is fbcb86abd9ba16348600f4b2887cd874fe7b3bad216c20bbf2d5fbaec138cd89 (not checkable from this log; a copy anchored elsewhere or the operator's kept original can check it).

Checked from the record: the redacted log's hash chain and artifact digests; re-derivation of every judgement from it; the redaction event's internal consistency (its position, replacement rows, recomputed digests and plan hashes, a single redaction or a second that names the first); and that the outcome, qualifiers, tallies, assurance flags, verdicts and observed values it records as before equal what the log derives now. Not checked: any other text (metric and protocol wording, units, paths), what files and commands meant when they ran, or anything about the log before the redaction, which is not in the record.

Provenance

Driven by

operator:claude-opus-5-5 (Claude Code subagent, AI-driven)

Sandbox

3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2

Period

2026-10-20T03:02:30.482Z → 2026-10-20T03:23:31.892Z

Event log

54 events, hash-chained; head 6252e1001c1a681990a613fbd32a2e02f351160eff41659fdcafdf705bf1c3b1

Verify

python3 openrepro_verify.py arxiv-1610-01271-20261020T030230Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).

Right of reply

Authors may submit a response, which is published alongside this report unedited.