Replication record · arXiv:1506.02142

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani · 2015 · arXiv

Partially reproduced

Sealed settling test of MC dropout Table 1 (Concrete, Energy, Yacht; RMSE and test LL): 360 paired fits of the authors a6259f1 net.py at batch 32 (paper) and 128 (committed code) on the 20 committed splits with three seeds each, in a pinned Keras 1.1.0/Theano 0.8.2 environment. Five of six batch-32 cells fall within the predeclared allowance of 2 archive SEs (18-way Bonferroni t intervals); Yacht LL is inconclusive because its interval is wider than the allowance, though its point estimate is 0.002 nats from the archive. Batch 128 is outside the allowance in the worse direction for Energy and Yacht and inconclusive for Concrete, and every paired contrast favours batch 32, so batch-size drift explains the earlier Energy and Yacht RMSE gaps. Dropout 0.01 for Concrete/Yacht and the original environment remain unresolved.

Plan. Sealed before scored execution; 12 outputs observed before sealing (all of it in the log).

Qualifiers. Non-headline claims (stricter checks or registered controls) did not hold; the claims list names each. A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the details name each one. 2 scored jobs started after the seal; not the direct source of any recorded value: fits (event 70).

Scored (headline): 10 of 12 consistent
Other checks: 8 of 18 consistent
Deviations: 7 (1 material)
Integrity flags: 1
Logged compute: 0.28 h over 13 commands and 2 jobs
Primary code (declared before the first recorded result): github.com/yaringal/DropoutUncertaintyExps at a6259f1db8f5d3e2d743f88ecbde425a07b12445 (latest repository commit before arXiv v6; not proven to be the table-generating code). Both arms use its <dataset>/net/net/net.py for Concrete, Energy and Yacht, its committed data/ splits and experiment.py's loading logic. Arm B128 = net.py as committed (batch_size = 128). Arm B32 = the same file with only batch_size = 32 (the paper's stated mini-batch size), plus common seed/output instrumentation identical in both arms.
Commits the recorded results name: yaringal/DropoutUncertaintyExps@a6259f1db8f5: 30 results, work-tree HEAD logged when the result was recorded
Work tree when the attempt ended: yaringal/DropoutUncertaintyExps@a6259f1db8f5 in /work/due

Protocol

Sealed with the plan: primary code is github.com/yaringal/DropoutUncertaintyExps at a6259f1db8f5d3e2d743f88ecbde425a07b12445 (latest repository commit before arXiv v6; not proven to be the table-generating code). Both arms use its <dataset>/net/net/net.py for Concrete, Energy and Yacht, its committed data/ splits and experiment.py's loading logic. Arm B128 = net.py as committed (batch_size = 128). Arm B32 = the same file with only batch_size = 32 (the paper's stated mini-batch size), plus common seed/output instrumentation identical in both arms.; earlier versions of the authors' code are not used. Settling plan ~/Desktop/office/plans/1506.02142-settling-plan.md, section 2: pre-rewrite commit 0eeeb7f11ce3a2e945608f3b51b57462b194ebd8 (batch 32) establishes the batch hypothesis only; it is not a post-hoc rescue arm, and neither it nor the later nested-validation/4,000-epoch experiment is run if B32 misses. Any historical-API experiment needs its own prior plan.

Run policy (sealed at event 69, plan hash d1f5148f2e4eaa3b7a09f257b5115c07decaa852540511d746e1cba4b9151c5f): Settling plan ~/Desktop/office/plans/1506.02142-settling-plan.md (sha256 134dba2b4a8a538cd6968118f1c4a6159b27e898ec4367d07d162b16d147745a; copy at /work/settle/plan/) executed exactly. Exactly 360 fits = 3 datasets (d=1 Concrete, 2 Energy, 3 Yacht) x 20 committed splits (i=0..19) x 3 initialization seeds (r=0..2) x 2 batch sizes (B32, B128); seed = 10000*d + 100*i + r, set for Python random and NumPy immediately before model construction in a fresh Python 2.7 process with PYTHONHASHSEED=0 (Keras 1.1.0 draws weights, dropout-stream seeds and shuffles from the NumPy global RNG). Order: split-major, repetition-major, B32 then B128; both arms always run. Job matrix digest (sha256 of lines 'dataset<TAB>i<TAB>r<TAB>arm<TAB>seed<LF>' in that order) 5877d7f5d5e6e00a950a8945a09b38711faa875ed3e615a87fc3c60641bebf96. Code: authors' a6259f1 net.py per dataset; B128 as committed, B32 differs only in batch_size (arm files and instrumentation in /work/settle/arms). 400 epochs, 10,000 MC passes, dropout/tau/lengthscale as committed (Concrete .01/.0557319157756/.01, Energy .05/.427114830213/.01, Yacht .01/.541869562034/.1). Environment: Python 2.7.15, NumPy 1.11.3, SciPy 0.18.1, scikit-learn 0.18.2, Keras 1.1.0, Theano 0.8.2 CPU float32 (conda-forge via micromamba 1.5.10 + pip), Theano compiling with system g++ 14.2; locked in /work/settle/env/legacy-env.lock. All inputs (scripts, arms, reference files, env lock, plan copy) are bound by /work/settle/inputs.lock.json sha256 3e83385a524931a7fe0b9896dcc3b04d17801faac025dfaa3321201e4c95c031; paper text pages.json sha256 c1357a059e3eaa9865b0fbe3a4b41756395c7a94bd23c84b1882fcd5ec5bb884; data manifests (47 files each) Concrete 6481427cc5ac86c628449a84d733eb0de53ff97415027365985e3a60b8b85a67, Energy acacec3a06fee05e98a90c8bd7e5e8de77dc95511148dcf6e0a9600892110702, Yacht cae3750fe36ff4e8463e9a2faad324391cfc95dd8e4eccd3bc3d847c3516f079, re-checked by the fits job before any fit. Jobs: (1) scored job 'fits' = bash /work/settle/run_fits.sh (input checks, then three dataset workers in parallel, one math-library thread each, then collect.py prints mc/results.csv); (2) scored job 'analysis' = /env/py27/bin/python /work/settle/analysis.py /work/out/fits /work/settle/reference /work/out/analysis. No pilots, no unscored training. Aggregation: Per fit: rmse and ll columns of the fresh mc/results.csv = values returned by the authors' net.predict (source arithmetic), each re-derived by analysis.py from the retained MC array (|diff| <= 1e-9 or the dataset is invalid). Split mean = mean of the three repetitions; dataset metric = mean of the 20 split means (never pooled test rows, never 60 independent values). D_i = split mean - archived A_i (A_i = line i+1 of a6259f1 <dataset>/net/results/test_MC_rmse.txt or test_ll.txt, copied read-only to /work/settle/reference, 20 finite rows, unrounded mean must round to the paper cell). E_i = split mean(B32) - split mean(B128). SE = sample SD of the 20 differences / sqrt(20). 18 intervals (12 batch/reference + 6 contrasts): mean difference +/- q*SE with q = t_19(1 - .05/(2*18)) = 3.4346619924588992. delta = 2 * archive SE per cell (Concrete RMSE .12109881527976792, LL .020875732440461817; Energy .04438037725415345, .020281023086837466; Yacht .08721140058311183, .02761054750297169). Classification: interval wholly inside [-delta, +delta] = within; wholly below -delta or above +delta = outside; otherwise inconclusive. Claims b32-*/b128-* report the interval ends on the metric scale (archive mean + D-interval end) and pass only when within delta of the archive mean, i.e. both ends inside. Contrast claims pass only when the E interval excludes zero in the improving direction (RMSE upper end < 0; LL lower end > 0). Headline (outcome) = the 12 B32 interval-end claims; B128 and contrast claims are diagnostic. Batch size supports an explanation for a cell only when B32 is within, B128 is outside in the worse direction, and the contrast excludes zero in the improving direction. Section 5 correction text is printed mechanically by analysis.py. Stopping rule: One pass over the sealed matrix, then one analysis. Cap: 120 wall-minutes total execution, three dataset workers, one math-library thread each, <= 6 GB aggregate RAM; the fits job stops launching fits 105 minutes after it starts. No extension based on apparent closeness; stop at the cap, not when a result passes. A failed, NaN, OOM or capped fit is recorded and never replaced; a dataset with any missing, invalid, unpaired (initial weight/RNG digests differ within a batch pair) or non-recomputable fit is evidence-incomplete for all its comparisons, and survivors are never averaged. One retry of a job is allowed only for a documented external interruption (host loss or transient I/O), with identical inputs, seeds and command; both attempts are retained and the first complete invocation is scored. Reruns: not allowed.

Claims

Batch-32 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).headline

Consistent

paper 5.2299observed 5.06134

Paper 5.2299 RMSE, target unitsObserved 5.06134 RMSE, target unitsAccepted if within ±0.242198

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.9678 · registered at event 9, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.12109881527976792 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 79. Observed 5.06134, delta -0.1686, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_concrete_rmse_lower 5.061344861682844'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).headline

Consistent

paper 5.2299observed 5.34285

Paper 5.2299 RMSE, target unitsObserved 5.34285 RMSE, target unitsAccepted if within ±0.242198

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.022 · registered at event 10, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 80. Observed 5.34285, delta +0.113, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_concrete_rmse_upper 5.342854364077325'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).

Consistent

paper 5.2299observed 5.363

Paper 5.2299 RMSE, target unitsObserved 5.363 RMSE, target unitsAccepted if within ±0.242198

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.025 · registered at event 11, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 81. Observed 5.363, delta +0.1331, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_concrete_rmse_lower 5.362996005986037'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).

Inconsistent

paper 5.2299observed 5.68132

Paper 5.2299 RMSE, target unitsObserved 5.68132 RMSE, target unitsAccepted if within ±0.242198

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.086 · registered at event 12, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 82. Observed 5.68132, delta +0.4514, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_concrete_rmse_upper 5.681315115746546'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).

Consistent

paper 0observed -0.209382

Paper 0 RMSE, target unitsObserved -0.209382 RMSE, target unitsAccepted if at most 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 13, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units

Why this tolerance. Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.

Event 83. Observed -0.209382, delta -0.2094, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_concrete_rmse_upper -0.2093820114881758'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).headline

Consistent

paper -3.03891observed -3.06444

Paper -3.03891 nats per test observationObserved -3.06444 nats per test observationAccepted if within ±0.0417515

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.008 · registered at event 14, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020875732440461817 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 84. Observed -3.06444, delta -0.02554, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_concrete_ll_lower -3.0644440025785884'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).headline

Consistent

paper -3.03891observed -3.00724

Paper -3.03891 nats per test observationObserved -3.00724 nats per test observationAccepted if within ±0.0417515

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.9896 · registered at event 15, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 85. Observed -3.00724, delta +0.03166, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_concrete_ll_upper -3.0072436764796526'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).

Inconsistent

paper -3.03891observed -3.14992

Paper -3.03891 nats per test observationObserved -3.14992 nats per test observationAccepted if within ±0.0417515

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.037 · registered at event 16, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 86. Observed -3.14992, delta -0.111, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_concrete_ll_lower -3.1499155608217624'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).

Consistent

paper -3.03891observed -3.07235

Paper -3.03891 nats per test observationObserved -3.07235 nats per test observationAccepted if within ±0.0417515

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.011 · registered at event 17, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 87. Observed -3.07235, delta -0.03344, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_concrete_ll_upper -3.07234898391703'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).

Consistent

paper 0observed 0.0532861

Paper 0 nats per test observationObserved 0.0532861 nats per test observationAccepted if at least 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 18, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation

Event 88. Observed 0.0532861, delta +0.05329, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_concrete_ll_lower 0.053286060278996586'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).headline

Consistent

paper 1.66341observed 1.6082

Paper 1.66341 RMSE, target unitsObserved 1.6082 RMSE, target unitsAccepted if within ±0.0887608

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.9668 · registered at event 19, sealed before scored execution

Method, tolerance rationale and evidence

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.04438037725415345 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 89. Observed 1.6082, delta -0.05521, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_energy_rmse_lower 1.6081951689222833'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).headline

Consistent

paper 1.66341observed 1.70967

Paper 1.66341 RMSE, target unitsObserved 1.70967 RMSE, target unitsAccepted if within ±0.0887608

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.028 · registered at event 20, sealed before scored execution

Method, tolerance rationale and evidence

Event 90. Observed 1.70967, delta +0.04626, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_energy_rmse_upper 1.7096686138436987'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).

Inconsistent

paper 1.66341observed 1.9639

Paper 1.66341 RMSE, target unitsObserved 1.9639 RMSE, target unitsAccepted if within ±0.0887608

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.181 · registered at event 21, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 91. Observed 1.9639, delta +0.3005, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_energy_rmse_lower 1.9639005883030252'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).

Inconsistent

paper 1.66341observed 2.1157

Paper 1.66341 RMSE, target unitsObserved 2.1157 RMSE, target unitsAccepted if within ±0.0887608

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.272 · registered at event 22, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 92. Observed 2.1157, delta +0.4523, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_energy_rmse_upper 2.115701136179798'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).

Consistent

paper 0observed -0.312057

Paper 0 RMSE, target unitsObserved -0.312057 RMSE, target unitsAccepted if at most 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 23, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units

Event 93. Observed -0.312057, delta -0.3121, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_energy_rmse_upper -0.3120570105299714'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).headline

Consistent

paper -1.98527observed -2.00112

Paper -1.98527 nats per test observationObserved -2.00112 nats per test observationAccepted if within ±0.040562

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.008 · registered at event 24, sealed before scored execution

Method, tolerance rationale and evidence

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020281023086837466 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 94. Observed -2.00112, delta -0.01585, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_energy_ll_lower -2.001121564376307'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).headline

Consistent

paper -1.98527observed -1.95489

Paper -1.98527 nats per test observationObserved -1.95489 nats per test observationAccepted if within ±0.040562

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.9847 · registered at event 25, sealed before scored execution

Method, tolerance rationale and evidence

Event 95. Observed -1.95489, delta +0.03038, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_energy_ll_upper -1.954894546782792'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).

Inconsistent

paper -1.98527observed -2.19885

Paper -1.98527 nats per test observationObserved -2.19885 nats per test observationAccepted if within ±0.040562

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.108 · registered at event 26, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 96. Observed -2.19885, delta -0.2136, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_energy_ll_lower -2.1988543438067762'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).

Inconsistent

paper -1.98527observed -2.1271

Paper -1.98527 nats per test observationObserved -2.1271 nats per test observationAccepted if within ±0.040562

Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.071 · registered at event 27, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 97. Observed -2.1271, delta -0.1418, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_energy_ll_upper -2.1271023810418503'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).

Consistent

paper 0observed 0.152

Paper 0 nats per test observationObserved 0.152 nats per test observationAccepted if at least 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 28, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation

Event 98. Observed 0.152, delta +0.152, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_energy_ll_lower 0.15199974766449087'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).headline

Consistent

paper 1.10925observed 0.96519

Paper 1.10925 RMSE, target unitsObserved 0.96519 RMSE, target unitsAccepted if within ±0.174423

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.8701 · registered at event 29, sealed before scored execution

Method, tolerance rationale and evidence

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.08721140058311183 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 99. Observed 0.96519, delta -0.1441, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_yacht_rmse_lower 0.9651899983004549'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).headline

Consistent

paper 1.10925observed 1.26781

Paper 1.10925 RMSE, target unitsObserved 1.26781 RMSE, target unitsAccepted if within ±0.174423

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.143 · registered at event 30, sealed before scored execution

Method, tolerance rationale and evidence

Event 100. Observed 1.26781, delta +0.1586, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_yacht_rmse_upper 1.2678071673635172'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).

Inconsistent

paper 1.10925observed 1.99836

Paper 1.10925 RMSE, target unitsObserved 1.99836 RMSE, target unitsAccepted if within ±0.174423

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.802 · registered at event 31, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 101. Observed 1.99836, delta +0.8891, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_yacht_rmse_lower 1.9983550081679469'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).

Inconsistent

paper 1.10925observed 2.76932

Paper 1.10925 RMSE, target unitsObserved 2.76932 RMSE, target unitsAccepted if within ±0.174423

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 2.497 · registered at event 32, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 102. Observed 2.76932, delta +1.66, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_yacht_rmse_upper 2.7693175213950565'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).

Consistent

paper 0observed -0.906752

Paper 0 RMSE, target unitsObserved -0.906752 RMSE, target unitsAccepted if at most 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 33, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units

Event 103. Observed -0.906752, delta -0.9068, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_yacht_rmse_upper -0.9067515886921714'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).headline

Inconsistent

paper -1.55406observed -1.61949

Paper -1.55406 nats per test observationObserved -1.61949 nats per test observationAccepted if within ±0.0552211

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.042 · registered at event 34, sealed before scored execution

Method, tolerance rationale and evidence

Why this tolerance. Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.02761054750297169 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.

Event 104. Observed -1.61949, delta -0.06543, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_yacht_ll_lower -1.6194904845928275'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-32 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).headline

Inconsistent

paper -1.55406observed -1.49281

Paper -1.55406 nats per test observationObserved -1.49281 nats per test observationAccepted if within ±0.0552211

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 0.9606 · registered at event 35, sealed before scored execution

Method, tolerance rationale and evidence

Event 105. Observed -1.49281, delta +0.06125, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b32_yacht_ll_upper -1.4928128048110798'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).

Inconsistent

paper -1.55406observed -2.54495

Paper -1.55406 nats per test observationObserved -2.54495 nats per test observationAccepted if within ±0.0552211

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.638 · registered at event 36, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 106. Observed -2.54495, delta -0.9909, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_yacht_ll_lower -2.544950474590947'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Batch-128 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).

Inconsistent

paper -1.55406observed -1.99479

Paper -1.55406 nats per test observationObserved -1.99479 nats per test observationAccepted if within ±0.0552211

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.284 · registered at event 37, sealed before scored execution

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.

Event 107. Observed -1.99479, delta -0.4407, inconsistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'b128_yacht_ll_upper -1.9947939232545668'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).

Consistent

paper 0observed 0.441131

Paper 0 nats per test observationObserved 0.441131 nats per test observationAccepted if at least 0

mini-batches of size 32 and the Adam optimiser

Section 5 (UCI regression experiments), p. 7 · quote verified on p. 7 · bound check · registered at event 38, sealed before scored execution

Method, tolerance rationale and evidence

Metric. mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation

Event 108. Observed 0.441131, delta +0.4411, consistent. Evidence: 00077-job-analysis-stdout.txt (event 77), excerpt 'contrast_yacht_ll_lower 0.4411305830252684'; verified. Job start event 76. Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.

Deviations from the paper's protocol

material
Concrete and Yacht use the released scripts' dropout 0.01 (Energy 0.05), whereas the paper text lists dropout 0.05 and 0.005 without a dataset mapping.
Settling plan section 2: source-code values are held fixed; this is a disclosed, unresolved mismatch. No search over .005/.05 is made. event 62
minor
Environment is an inferred legacy stack (Python 2.7.15, NumPy 1.11.3, SciPy 0.18.1, scikit-learn 0.18.2, matplotlib 2.2.3 imported only, Keras 1.1.0, Theano 0.8.2 CPU float32; conda-forge with netlib libblas for Theano and OpenBLAS-built SciPy; Theano compiles with Debian g++ 14.2), not the authors' recovered original environment.
The original environment is not recorded by the authors; the plan pins this stack (same as the earlier attempt) and both arms run in it. event 63
minor
Each fit runs in a fresh process with seed 10000*d+100*i+r set immediately before model construction, instead of experiment.py's single np.random.seed(1) stream over 20 splits; three seeds per split instead of one; experiment.py's loop is replaced by a runner (fit_one.py) that reproduces its loading code for one split and writes results outside the checkout.
Settling plan section 3: an explicitly new paired ablation, so that B32 and B128 share initial weights, dropout-stream seeds and RNG state per pair. event 64
none
Common instrumentation added to both arms' net.py: digests of the initial weights and NumPy RNG state after compile, digest of trained weights after fit, and the denormalized MC array kept on the object for lossless saving. B32 additionally changes batch_size 128 -> 32. No other line differs from a6259f1.
Required by the plan's pairing check and MC-array retention; the arithmetic of training, prediction and metrics is unchanged. event 65
none
Yacht's encoding-only patch (coding line in experiment.py) is not applied: experiment.py is not executed, so its UTF-8 comment never reaches the Python 2 parser; net.py for all three datasets is ASCII and used unpatched apart from the instrumentation above.
Not needed with the runner; the plan allows but does not require the patch. event 66
minor
The 360 MC prediction arrays are retained (float32 as produced by net.py, savez_compressed, one file per fit) on the attempt's box volume during the attempt only; their SHA-256 digests are in the logged job output, but the arrays themselves are destroyed with the dedicated box at finalize and are not part of the published attempt.
The engine publishes /work only through the log; several GB of arrays are not committed. The analysis job re-derives every RMSE/LL from the arrays before scoring. event 67
minor
The plan asks for one externally anchored plan binding document, code, environment and matrix. Here the binding is the engine's seal (plan hash in the hash-chained log) whose text cites the sha256 of /work/settle/inputs.lock.json, the plan copy, paper text, data manifests and job matrix; it is anchored externally only when the attempt's log head is committed/anchored, not before execution.
The engine has no pre-execution external anchoring step; anchors/ is managed at release time and is out of scope for this run. event 68

Integrity flags

flag
At least one deviation from the paper's protocol is marked material.

Notes from the attempt

event 109
Three-way classification under the sealed rule (analysis event 77; delta = 2 x archive SE; 18-way Bonferroni t intervals, q = 3.43466). B32 vs archive: Concrete RMSE within, Concrete LL within, Energy RMSE within, Energy LL within, Yacht RMSE within, Yacht LL inconclusive (point difference -0.0021 nats, interval -0.0654 to +0.0612 vs delta 0.0552: wider than the allowance, so it cannot pass). B128 vs archive: Concrete RMSE inconclusive, Concrete LL inconclusive, Energy RMSE outside (worse), Energy LL outside (worse), Yacht RMSE outside (worse), Yacht LL outside (worse). All six paired contrasts exclude zero in the improving direction (B32 better). Batch size supports the explanation for Energy RMSE, Energy LL and Yacht RMSE; Concrete RMSE/LL are within at B32 but B128 is only inconclusive there, so this experiment does not isolate batch size for Concrete; Yacht LL is inconclusive at B32. Five of six B32 cells are within the allowance, so the six-cell setup is not declared a match. The engine verdicts on the interval-end claims are the same rule: a cell passes only when both of its ends are within delta.
event 110
Section 5 correction, printed mechanically by analysis.py (event 77): preamble as in the plan; Concrete RMSE and Concrete test LL: 'Batch 32 is within the historical uncertainty allowance, but this experiment does not isolate batch size as the cause of the old gap.' Energy RMSE, Energy test LL, Yacht RMSE: 'The batch-32 rerun is within the predeclared historical uncertainty allowance, while the paired batch comparison supports batch-size drift as an explanation for our earlier gap.' Yacht test LL: 'The comparison is inconclusive at the predeclared precision; we will not turn uncertainty into a pass or enlarge the allowance.' Always appended: Concrete and Yacht still use the released scripts' dropout .01 rather than a confirmed mapping of the paper's .05/.005, and the original Keras environment has not been established. The earlier attempt arxiv-1506-02142-20261018T043211Z is not modified; this attempt supersedes its interpretation only.
event 111
Execution: all 360 fits completed (status ok 360) in 907.5 s wall in job fits (event 74), well inside the 120-minute cap; no fit failed, none was retried. Every RMSE/LL re-derived from its retained MC array matched the value net.py returned exactly (max abs difference 0.0). Initial-weight and RNG-state digests were equal within every B32/B128 pair. The B128 arm reproduces the earlier attempt's batch-128 misses (Energy RMSE 2.040 vs 2.018 then, Yacht RMSE 2.384 vs 2.365, Concrete RMSE 5.522 vs 5.474), now with three seeds per split. Scope: a qualified setup reproduction test, not proof that the historical hyperparameter selection was sound.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: yes
Sealed before scored execution: yes
Evidence bound to artifact excerpts: yes
Values extracted by a sealed rule: yes
Execution manifest for cited jobs: yes
Semantic re-derivation passed: yes
Observed before the seal: 12 shell; provenance captured
Engine: report built by repro 0.2.1; attempt created by repro 0.2.1

Redaction before release

Before release, the operator's local home-directory path was replaced with ~ in 8 places. Every event hash from event 39 on changed with it. The sealed plan's text and its plan hash (event 69) changed because of the substitution; the seal now pins the substituted text. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now.

Rule: operator home directory prefix → ~. Reason: Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~ Recorded at event 116; the head the operator states for the log before the redaction is 050491ce90f47cb00cf5ec5a60eea67a1abfbe1acff274b5bea6dd6f073414ac (not checkable from this log; a copy anchored elsewhere or the operator's kept original can check it).

Checked from the record: the redacted log's hash chain and artifact digests; re-derivation of every judgement from it; the redaction event's internal consistency (its position, replacement rows, recomputed digests and plan hashes, a single redaction or a second that names the first); and that the outcome, qualifiers, tallies, assurance flags, verdicts and observed values it records as before equal what the log derives now. Not checked: any other text (metric and protocol wording, units, paths), what files and commands meant when they ran, or anything about the log before the redaction, which is not in the record.

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-20T02:49:12.310Z → 2026-10-20T03:21:10.246Z
Event log: 117 events, hash-chained; head 0e96b30ad495309e93a1a1d8b07fe06f7f0b1e24e4f4938a272874a1bb4646db
Verify: python3 openrepro_verify.py arxiv-1506-02142-20261020T024912Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.