{ "schema": "repro.report/v3", "engine_version": "0.2.1", "attempt_id": "arxiv-1506-02142-20261020T024912Z", "legacy": false, "paper": { "arxiv_id": "1506.02142", "title": "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning", "authors": [ "Yarin Gal", "Zoubin Ghahramani" ], "year": 2015, "doi": null, "venue": null, "openalex_id": "https://openalex.org/W582134693" }, "code": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps", "matches_primary": true } ], "code_used": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "repo": "yaringal/DropoutUncertaintyExps", "code_version": "primary", "source": "harness", "results": [ 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108 ] } ], "primary_code_source": "protocol", "outcome": "partially_reproduced", "outcome_label": "Partially reproduced", "qualifiers": [ "other_checks_failed", "extra_scored_runs" ], "qualifier_details": [ "2 scored jobs started after the seal; not the direct source of any recorded value: fits (event 70)" ], "evidence_gate_failed": [], "assurance": { "plan_sealed": true, "sealed_before_scored_execution": true, "evidence_bound": true, "extraction_sealed": true, "execution_manifest": true, "semantic_rederivation_passed": true, "predates": [], "engine_version": "0.2.1", "attempt_engine_version": "0.2.1", "setup_outputs_before_seal": 12, "observations_before_seal": { "shell": 12, "job_finished": 0, "job_status": 0, "read_file": 0 }, "observation_provenance": "captured", "plan_hash": "d1f5148f2e4eaa3b7a09f257b5115c07decaa852540511d746e1cba4b9151c5f", "paper_text_sha256_ok": true }, "plan": { "event": 69, "setup_outputs_observed": 12, "observations_before_seal": { "shell": 12, "job_finished": 0, "job_status": 0, "read_file": 0 }, "observation_provenance": "captured", "reruns": "not_allowed", "rerun_policy": "", "rerun_rule": "none", "run_policy": "Settling plan ~/Desktop/office/plans/1506.02142-settling-plan.md (sha256 134dba2b4a8a538cd6968118f1c4a6159b27e898ec4367d07d162b16d147745a; copy at /work/settle/plan/) executed exactly. Exactly 360 fits = 3 datasets (d=1 Concrete, 2 Energy, 3 Yacht) x 20 committed splits (i=0..19) x 3 initialization seeds (r=0..2) x 2 batch sizes (B32, B128); seed = 10000*d + 100*i + r, set for Python random and NumPy immediately before model construction in a fresh Python 2.7 process with PYTHONHASHSEED=0 (Keras 1.1.0 draws weights, dropout-stream seeds and shuffles from the NumPy global RNG). Order: split-major, repetition-major, B32 then B128; both arms always run. Job matrix digest (sha256 of lines 'datasetirarmseed' in that order) 5877d7f5d5e6e00a950a8945a09b38711faa875ed3e615a87fc3c60641bebf96. Code: authors' a6259f1 net.py per dataset; B128 as committed, B32 differs only in batch_size (arm files and instrumentation in /work/settle/arms). 400 epochs, 10,000 MC passes, dropout/tau/lengthscale as committed (Concrete .01/.0557319157756/.01, Energy .05/.427114830213/.01, Yacht .01/.541869562034/.1). Environment: Python 2.7.15, NumPy 1.11.3, SciPy 0.18.1, scikit-learn 0.18.2, Keras 1.1.0, Theano 0.8.2 CPU float32 (conda-forge via micromamba 1.5.10 + pip), Theano compiling with system g++ 14.2; locked in /work/settle/env/legacy-env.lock. All inputs (scripts, arms, reference files, env lock, plan copy) are bound by /work/settle/inputs.lock.json sha256 3e83385a524931a7fe0b9896dcc3b04d17801faac025dfaa3321201e4c95c031; paper text pages.json sha256 c1357a059e3eaa9865b0fbe3a4b41756395c7a94bd23c84b1882fcd5ec5bb884; data manifests (47 files each) Concrete 6481427cc5ac86c628449a84d733eb0de53ff97415027365985e3a60b8b85a67, Energy acacec3a06fee05e98a90c8bd7e5e8de77dc95511148dcf6e0a9600892110702, Yacht cae3750fe36ff4e8463e9a2faad324391cfc95dd8e4eccd3bc3d847c3516f079, re-checked by the fits job before any fit. Jobs: (1) scored job 'fits' = bash /work/settle/run_fits.sh (input checks, then three dataset workers in parallel, one math-library thread each, then collect.py prints mc/results.csv); (2) scored job 'analysis' = /env/py27/bin/python /work/settle/analysis.py /work/out/fits /work/settle/reference /work/out/analysis. No pilots, no unscored training.", "aggregation": "Per fit: rmse and ll columns of the fresh mc/results.csv = values returned by the authors' net.predict (source arithmetic), each re-derived by analysis.py from the retained MC array (|diff| <= 1e-9 or the dataset is invalid). Split mean = mean of the three repetitions; dataset metric = mean of the 20 split means (never pooled test rows, never 60 independent values). D_i = split mean - archived A_i (A_i = line i+1 of a6259f1 /net/results/test_MC_rmse.txt or test_ll.txt, copied read-only to /work/settle/reference, 20 finite rows, unrounded mean must round to the paper cell). E_i = split mean(B32) - split mean(B128). SE = sample SD of the 20 differences / sqrt(20). 18 intervals (12 batch/reference + 6 contrasts): mean difference +/- q*SE with q = t_19(1 - .05/(2*18)) = 3.4346619924588992. delta = 2 * archive SE per cell (Concrete RMSE .12109881527976792, LL .020875732440461817; Energy .04438037725415345, .020281023086837466; Yacht .08721140058311183, .02761054750297169). Classification: interval wholly inside [-delta, +delta] = within; wholly below -delta or above +delta = outside; otherwise inconclusive. Claims b32-*/b128-* report the interval ends on the metric scale (archive mean + D-interval end) and pass only when within delta of the archive mean, i.e. both ends inside. Contrast claims pass only when the E interval excludes zero in the improving direction (RMSE upper end < 0; LL lower end > 0). Headline (outcome) = the 12 B32 interval-end claims; B128 and contrast claims are diagnostic. Batch size supports an explanation for a cell only when B32 is within, B128 is outside in the worse direction, and the contrast excludes zero in the improving direction. Section 5 correction text is printed mechanically by analysis.py.", "stopping_rule": "One pass over the sealed matrix, then one analysis. Cap: 120 wall-minutes total execution, three dataset workers, one math-library thread each, <= 6 GB aggregate RAM; the fits job stops launching fits 105 minutes after it starts. No extension based on apparent closeness; stop at the cap, not when a result passes. A failed, NaN, OOM or capped fit is recorded and never replaced; a dataset with any missing, invalid, unpaired (initial weight/RNG digests differ within a batch pair) or non-recomputable fit is evidence-incomplete for all its comparisons, and survivors are never averaged. One retry of a job is allowed only for a documented external interruption (host loss or transient I/O), with identical inputs, seeds and command; both attempts are retained and the first complete invocation is scored.", "plan_hash": "d1f5148f2e4eaa3b7a09f257b5115c07decaa852540511d746e1cba4b9151c5f", "claims": [ "b32-concrete-rmse-lower", "b32-concrete-rmse-upper", "b128-concrete-rmse-lower", "b128-concrete-rmse-upper", "contrast-concrete-rmse-upper", "b32-concrete-ll-lower", "b32-concrete-ll-upper", "b128-concrete-ll-lower", "b128-concrete-ll-upper", "contrast-concrete-ll-lower", "b32-energy-rmse-lower", "b32-energy-rmse-upper", "b128-energy-rmse-lower", "b128-energy-rmse-upper", "contrast-energy-rmse-upper", "b32-energy-ll-lower", "b32-energy-ll-upper", "b128-energy-ll-lower", "b128-energy-ll-upper", "contrast-energy-ll-lower", "b32-yacht-rmse-lower", "b32-yacht-rmse-upper", "b128-yacht-rmse-lower", "b128-yacht-rmse-upper", "contrast-yacht-rmse-upper", "b32-yacht-ll-lower", "b32-yacht-ll-upper", "b128-yacht-ll-lower", "b128-yacht-ll-upper", "contrast-yacht-ll-lower" ], "claims_registered_after_output": [] }, "not_attemptable_reason": null, "summary": "Sealed settling test of MC dropout Table 1 (Concrete, Energy, Yacht; RMSE and test LL): 360 paired fits of the authors a6259f1 net.py at batch 32 (paper) and 128 (committed code) on the 20 committed splits with three seeds each, in a pinned Keras 1.1.0/Theano 0.8.2 environment. Five of six batch-32 cells fall within the predeclared allowance of 2 archive SEs (18-way Bonferroni t intervals); Yacht LL is inconclusive because its interval is wider than the allowance, though its point estimate is 0.002 nats from the archive. Batch 128 is outside the allowance in the worse direction for Energy and Yacht and inconclusive for Concrete, and every paired contrast favours batch 32, so batch-size drift explains the earlier Energy and Yacht RMSE gaps. Dropout 0.01 for Concrete/Yacht and the original environment remain unresolved.", "claims": [ { "claim_id": "b32-concrete-rmse-lower", "comparison": "within", "extraction": { "key": "b32_concrete_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 5.229903994142711, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-32 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).", "tolerance": 0.24219763055953583, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.12109881527976792 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 9, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.04631014849044796, "extraction_registered": true, "results": [ { "event": 79, "observed_value": 5.061344861682844, "delta": -0.1685591324598663, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_rmse_lower 5.061344861682844", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 79, "observed_value": 5.061344861682844, "delta": -0.1685591324598663, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_rmse_lower 5.061344861682844", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9677701287349355, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-concrete-rmse-upper", "comparison": "within", "extraction": { "key": "b32_concrete_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 5.229903994142711, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-32 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).", "tolerance": 0.24219763055953583, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.12109881527976792 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 10, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.04631014849044796, "extraction_registered": true, "results": [ { "event": 80, "observed_value": 5.342854364077325, "delta": 0.11295036993461416, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_rmse_upper 5.342854364077325", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 80, "observed_value": 5.342854364077325, "delta": 0.11295036993461416, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_rmse_upper 5.342854364077325", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0215970255020195, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-concrete-rmse-lower", "comparison": "within", "extraction": { "key": "b128_concrete_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 5.229903994142711, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-128 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).", "tolerance": 0.24219763055953583, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.12109881527976792 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 11, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.04631014849044796, "extraction_registered": true, "results": [ { "event": 81, "observed_value": 5.362996005986037, "delta": 0.13309201184332675, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_rmse_lower 5.362996005986037", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 81, "observed_value": 5.362996005986037, "delta": 0.13309201184332675, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_rmse_lower 5.362996005986037", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0254482705595331, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-concrete-rmse-upper", "comparison": "within", "extraction": { "key": "b128_concrete_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 5.229903994142711, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-128 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).", "tolerance": 0.24219763055953583, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.12109881527976792 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 12, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.04631014849044796, "extraction_registered": true, "results": [ { "event": 82, "observed_value": 5.681315115746546, "delta": 0.4514111216038357, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_rmse_upper 5.681315115746546", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 82, "observed_value": 5.681315115746546, "delta": 0.4514111216038357, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_rmse_upper 5.681315115746546", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.0863134623712785, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-concrete-rmse-upper", "comparison": "at_most", "extraction": { "key": "contrast_concrete_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "RMSE, target units", "registered_event": 13, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 83, "observed_value": -0.2093820114881758, "delta": -0.2093820114881758, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_concrete_rmse_upper -0.2093820114881758", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 83, "observed_value": -0.2093820114881758, "delta": -0.2093820114881758, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_concrete_rmse_upper -0.2093820114881758", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-concrete-ll-lower", "comparison": "within", "extraction": { "key": "b32_concrete_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -3.0389055163159577, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-32 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).", "tolerance": 0.041751464880923635, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020875732440461817 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 14, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.013738980911633811, "extraction_registered": true, "results": [ { "event": 84, "observed_value": -3.0644440025785884, "delta": -0.025538486262630666, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_ll_lower -3.0644440025785884", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 84, "observed_value": -3.0644440025785884, "delta": -0.025538486262630666, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_ll_lower -3.0644440025785884", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0084038434645348, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-concrete-ll-upper", "comparison": "within", "extraction": { "key": "b32_concrete_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -3.0389055163159577, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-32 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).", "tolerance": 0.041751464880923635, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020875732440461817 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 15, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.013738980911633811, "extraction_registered": true, "results": [ { "event": 85, "observed_value": -3.0072436764796526, "delta": 0.031661839836305106, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_ll_upper -3.0072436764796526", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 85, "observed_value": -3.0072436764796526, "delta": 0.031661839836305106, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_concrete_ll_upper -3.0072436764796526", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9895811700408874, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-concrete-ll-lower", "comparison": "within", "extraction": { "key": "b128_concrete_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -3.0389055163159577, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-128 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).", "tolerance": 0.041751464880923635, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020875732440461817 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 16, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.013738980911633811, "extraction_registered": true, "results": [ { "event": 86, "observed_value": -3.1499155608217624, "delta": -0.11101004450580465, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_ll_lower -3.1499155608217624", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 86, "observed_value": -3.1499155608217624, "delta": -0.11101004450580465, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_ll_lower -3.1499155608217624", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.0365296136749855, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-concrete-ll-upper", "comparison": "within", "extraction": { "key": "b128_concrete_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -3.0389055163159577, "quote": "Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02", "statement": "Batch-128 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).", "tolerance": 0.041751464880923635, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020875732440461817 (sample SD of the 20 archived split values in concrete/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 17, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.013738980911633811, "extraction_registered": true, "results": [ { "event": 87, "observed_value": -3.07234898391703, "delta": -0.033443467601072197, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_ll_upper -3.07234898391703", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 87, "observed_value": -3.07234898391703, "delta": -0.033443467601072197, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_concrete_ll_upper -3.07234898391703", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0110051027981994, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-concrete-ll-lower", "comparison": "at_least", "extraction": { "key": "contrast_concrete_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "nats per test observation", "registered_event": 18, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 88, "observed_value": 0.053286060278996586, "delta": 0.053286060278996586, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_concrete_ll_lower 0.053286060278996586", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 88, "observed_value": 0.053286060278996586, "delta": 0.053286060278996586, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_concrete_ll_lower 0.053286060278996586", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-energy-rmse-lower", "comparison": "within", "extraction": { "key": "b32_energy_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.663405433671937, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-32 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).", "tolerance": 0.0887607545083069, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.04438037725415345 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 19, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.05336086603514884, "extraction_registered": true, "results": [ { "event": 89, "observed_value": 1.6081951689222833, "delta": -0.05521026474965374, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_rmse_lower 1.6081951689222833", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 89, "observed_value": 1.6081951689222833, "delta": -0.05521026474965374, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_rmse_lower 1.6081951689222833", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9668088947937497, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-energy-rmse-upper", "comparison": "within", "extraction": { "key": "b32_energy_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.663405433671937, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-32 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).", "tolerance": 0.0887607545083069, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.04438037725415345 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 20, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.05336086603514884, "extraction_registered": true, "results": [ { "event": 90, "observed_value": 1.7096686138436987, "delta": 0.046263180171761675, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_rmse_upper 1.7096686138436987", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 90, "observed_value": 1.7096686138436987, "delta": 0.046263180171761675, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_rmse_upper 1.7096686138436987", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0278123295952188, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-energy-rmse-lower", "comparison": "within", "extraction": { "key": "b128_energy_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.663405433671937, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-128 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).", "tolerance": 0.0887607545083069, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.04438037725415345 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 21, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.05336086603514884, "extraction_registered": true, "results": [ { "event": 91, "observed_value": 1.9639005883030252, "delta": 0.3004951546310881, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_rmse_lower 1.9639005883030252", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 91, "observed_value": 1.9639005883030252, "delta": 0.3004951546310881, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_rmse_lower 1.9639005883030252", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.1806505789558175, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-energy-rmse-upper", "comparison": "within", "extraction": { "key": "b128_energy_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.663405433671937, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-128 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).", "tolerance": 0.0887607545083069, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.04438037725415345 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 22, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.05336086603514884, "extraction_registered": true, "results": [ { "event": 92, "observed_value": 2.115701136179798, "delta": 0.4522957025078611, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_rmse_upper 2.115701136179798", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 92, "observed_value": 2.115701136179798, "delta": 0.4522957025078611, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_rmse_upper 2.115701136179798", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.2719094776006752, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-energy-rmse-upper", "comparison": "at_most", "extraction": { "key": "contrast_energy_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "RMSE, target units", "registered_event": 23, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 93, "observed_value": -0.3120570105299714, "delta": -0.3120570105299714, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_energy_rmse_upper -0.3120570105299714", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 93, "observed_value": -0.3120570105299714, "delta": -0.3120570105299714, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_energy_rmse_upper -0.3120570105299714", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-energy-ll-lower", "comparison": "within", "extraction": { "key": "b32_energy_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.9852732326905922, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-32 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).", "tolerance": 0.04056204617367493, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020281023086837466 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 24, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.020431467823047302, "extraction_registered": true, "results": [ { "event": 94, "observed_value": -2.001121564376307, "delta": -0.015848331685714756, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_ll_lower -2.001121564376307", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 94, "observed_value": -2.001121564376307, "delta": -0.015848331685714756, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_ll_lower -2.001121564376307", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0079829473468678, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-energy-ll-upper", "comparison": "within", "extraction": { "key": "b32_energy_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.9852732326905922, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-32 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).", "tolerance": 0.04056204617367493, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020281023086837466 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 25, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.020431467823047302, "extraction_registered": true, "results": [ { "event": 95, "observed_value": -1.954894546782792, "delta": 0.03037868590780013, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_ll_upper -1.954894546782792", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 95, "observed_value": -1.954894546782792, "delta": 0.03037868590780013, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_energy_ll_upper -1.954894546782792", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9846979824199671, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-energy-ll-lower", "comparison": "within", "extraction": { "key": "b128_energy_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.9852732326905922, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-128 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).", "tolerance": 0.04056204617367493, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020281023086837466 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 26, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.020431467823047302, "extraction_registered": true, "results": [ { "event": 96, "observed_value": -2.1988543438067762, "delta": -0.21358111111618405, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_ll_lower -2.1988543438067762", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 96, "observed_value": -2.1988543438067762, "delta": -0.21358111111618405, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_ll_lower -2.1988543438067762", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.1075827284623803, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-energy-ll-upper", "comparison": "within", "extraction": { "key": "b128_energy_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.9852732326905922, "quote": "Energy Efficiency 768 8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02", "statement": "Batch-128 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).", "tolerance": 0.04056204617367493, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.020281023086837466 (sample SD of the 20 archived split values in energy/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 27, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.020431467823047302, "extraction_registered": true, "results": [ { "event": 97, "observed_value": -2.1271023810418503, "delta": -0.14182914835125815, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_ll_upper -2.1271023810418503", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 97, "observed_value": -2.1271023810418503, "delta": -0.14182914835125815, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_energy_ll_upper -2.1271023810418503", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.0714406188608308, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-energy-ll-lower", "comparison": "at_least", "extraction": { "key": "contrast_energy_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "nats per test observation", "registered_event": 28, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 98, "observed_value": 0.15199974766449087, "delta": 0.15199974766449087, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_energy_ll_lower 0.15199974766449087", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 98, "observed_value": 0.15199974766449087, "delta": 0.15199974766449087, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_energy_ll_lower 0.15199974766449087", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-yacht-rmse-lower", "comparison": "within", "extraction": { "key": "b32_yacht_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.1092497489252557, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-32 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).", "tolerance": 0.17442280116622366, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.08721140058311183 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 29, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.15724394018138899, "extraction_registered": true, "results": [ { "event": 99, "observed_value": 0.9651899983004549, "delta": -0.1440597506248008, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_rmse_lower 0.9651899983004549", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 99, "observed_value": 0.9651899983004549, "delta": -0.1440597506248008, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_rmse_lower 0.9651899983004549", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.8701286605975082, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-yacht-rmse-upper", "comparison": "within", "extraction": { "key": "b32_yacht_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.1092497489252557, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-32 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).", "tolerance": 0.17442280116622366, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.08721140058311183 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 30, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.15724394018138899, "extraction_registered": true, "results": [ { "event": 100, "observed_value": 1.2678071673635172, "delta": 0.15855741843826143, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_rmse_upper 1.2678071673635172", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 100, "observed_value": 1.2678071673635172, "delta": 0.15855741843826143, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_rmse_upper 1.2678071673635172", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.1429411352960743, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-yacht-rmse-lower", "comparison": "within", "extraction": { "key": "b128_yacht_rmse_lower", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.1092497489252557, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-128 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).", "tolerance": 0.17442280116622366, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.08721140058311183 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 31, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.15724394018138899, "extraction_registered": true, "results": [ { "event": 101, "observed_value": 1.9983550081679469, "delta": 0.8891052592426911, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_rmse_lower 1.9983550081679469", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 101, "observed_value": 1.9983550081679469, "delta": 0.8891052592426911, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_rmse_lower 1.9983550081679469", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.8015374897349663, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-yacht-rmse-upper", "comparison": "within", "extraction": { "key": "b128_yacht_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test RMSE (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": 1.1092497489252557, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-128 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).", "tolerance": 0.17442280116622366, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.08721140058311183 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "RMSE, target units", "registered_event": 32, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.15724394018138899, "extraction_registered": true, "results": [ { "event": 102, "observed_value": 2.7693175213950565, "delta": 1.6600677724698008, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_rmse_upper 2.7693175213950565", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 102, "observed_value": 2.7693175213950565, "delta": 1.6600677724698008, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_rmse_upper 2.7693175213950565", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 2.4965680849405003, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-yacht-rmse-upper", "comparison": "at_most", "extraction": { "key": "contrast_yacht_rmse_upper", "occurrence": "only", "transform": "none", "units": "RMSE, target units" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i + q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); RMSE, target units", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "RMSE, target units", "registered_event": 33, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 103, "observed_value": -0.9067515886921714, "delta": -0.9067515886921714, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_yacht_rmse_upper -0.9067515886921714", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 103, "observed_value": -0.9067515886921714, "delta": -0.9067515886921714, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_yacht_rmse_upper -0.9067515886921714", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-yacht-ll-lower", "comparison": "within", "extraction": { "key": "b32_yacht_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.5540594381496233, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-32 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).", "tolerance": 0.05522109500594338, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.02761054750297169 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 34, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.03553345107037453, "extraction_registered": true, "results": [ { "event": 104, "observed_value": -1.6194904845928275, "delta": -0.06543104644320419, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_ll_lower -1.6194904845928275", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 104, "observed_value": -1.6194904845928275, "delta": -0.06543104644320419, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_ll_lower -1.6194904845928275", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.042103310103191, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b32-yacht-ll-upper", "comparison": "within", "extraction": { "key": "b32_yacht_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": true, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 32, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.5540594381496233, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-32 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).", "tolerance": 0.05522109500594338, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.02761054750297169 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 35, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.03553345107037453, "extraction_registered": true, "results": [ { "event": 105, "observed_value": -1.4928128048110798, "delta": 0.06124663333854352, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_ll_upper -1.4928128048110798", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 105, "observed_value": -1.4928128048110798, "delta": 0.06124663333854352, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b32_yacht_ll_upper -1.4928128048110798", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 0.9605892594356183, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-yacht-ll-lower", "comparison": "within", "extraction": { "key": "b128_yacht_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) - q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D - q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.5540594381496233, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-128 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).", "tolerance": 0.05522109500594338, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.02761054750297169 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 36, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.03553345107037453, "extraction_registered": true, "results": [ { "event": 106, "observed_value": -2.544950474590947, "delta": -0.9908910364413237, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_ll_lower -2.544950474590947", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 106, "observed_value": -2.544950474590947, "delta": -0.9908910364413237, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_ll_lower -2.544950474590947", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.6376146317937177, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "b128-yacht-ll-upper", "comparison": "within", "extraction": { "key": "b128_yacht_ll_upper", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Table 1, p. 7 (Dropout columns)", "metric": "average test predictive log-likelihood (MC dropout, 10,000 passes); value = mean over 20 splits of (three-seed split mean) + q*SE, where SE = sample SD of the 20 paired differences D_i = split mean - archived A_i over sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18). Equivalently archive mean + (mean D + q*SE). Fresh process per fit, seed = 10000*d+100*i+r, batch size 128, 400 epochs, author net.py at a6259f1 otherwise unchanged.", "paper_value": -1.5540594381496233, "quote": "Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03", "statement": "Batch-128 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).", "tolerance": 0.05522109500594338, "tolerance_rationale": "Settling plan section 4: fixed historical uncertainty allowance delta = 2 x archive SE = 2 x 0.02761054750297169 (sample SD of the 20 archived split values in yacht/net/results at a6259f1, over sqrt(20)), centred on the unrounded archive mean. Both interval ends within delta <=> the interval lies wholly inside [-delta, +delta]. Registered before any output of this attempt.", "unit": "nats per test observation", "registered_event": 37, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.03553345107037453, "extraction_registered": true, "results": [ { "event": 107, "observed_value": -1.9947939232545668, "delta": -0.4407344851049435, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_ll_upper -1.9947939232545668", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 107, "observed_value": -1.9947939232545668, "delta": -0.4407344851049435, "consistent": false, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "b128_yacht_ll_upper -1.9947939232545668", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "inconsistent", "result_revisions": 0, "ratio_to_published": 1.2836020774274337, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "contrast-yacht-ll-lower", "comparison": "at_least", "extraction": { "key": "contrast_yacht_ll_lower", "occurrence": "only", "transform": "none", "units": "nats per test observation" }, "headline": false, "location": "Section 5 (UCI regression experiments), p. 7", "metric": "mean of E_i - q*SE_E over the 20 splits, SE_E = sample SD of E_i / sqrt(20), q = 3.4346619924588992 = t_19 quantile at 1 - 0.05/(2*18); nats per test observation", "paper_value": 0.0, "quote": "mini-batches of size 32 and the Adam optimiser", "statement": "Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).", "tolerance": 0.0, "tolerance_rationale": "Settling plan section 4: batch size supports the explanation for a cell only if this paired contrast excludes zero in the improving direction (RMSE negative, LL positive). Diagnostic, not a paper claim; non-headline, so it never enters the outcome.", "unit": "nats per test observation", "registered_event": 38, "quote_verified": true, "quote_page": 7, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": null, "extraction_registered": true, "results": [ { "event": 108, "observed_value": 0.4411305830252684, "delta": 0.4411305830252684, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_yacht_ll_lower 0.4411305830252684", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] } ], "withdrawn_after_seal": false, "final": { "event": 108, "observed_value": 0.4411305830252684, "delta": 0.4411305830252684, "consistent": true, "evidence_events": [ 77, 74 ], "evidence": { "artifact": "00077-job-analysis-stdout.txt", "event": 77, "excerpt": "contrast_yacht_ll_lower 0.4411305830252684", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 76, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Sealed analysis job (analysis.py) over the 360 fits of scored job fits (event 74): split means of three seeds, paired differences against the archived a6259f1 split values (or B32-B128 contrasts), Bonferroni-t interval with q=3.4346619924588992; value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "a6259f1db8f5d3e2d743f88ecbde425a07b12445", "full_name": "yaringal/DropoutUncertaintyExps", "modified_files": 0, "path": "/work/due", "url": "https://github.com/yaringal/DropoutUncertaintyExps" } ] }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true } ], "counts": { "claims_scored": 12, "consistent": 10, "consistent_earlier_code": 0, "inconsistent": 2, "untested": 0, "withdrawn_after_seal": 0, "other": { "claims": 18, "consistent": 8, "inconsistent": 10, "untested": 0 } }, "withdrawn": [], "protocol": { "earlier_code": "not_allowed", "primary_code": "github.com/yaringal/DropoutUncertaintyExps at a6259f1db8f5d3e2d743f88ecbde425a07b12445 (latest repository commit before arXiv v6; not proven to be the table-generating code). Both arms use its /net/net/net.py for Concrete, Energy and Yacht, its committed data/ splits and experiment.py's loading logic. Arm B128 = net.py as committed (batch_size = 128). Arm B32 = the same file with only batch_size = 32 (the paper's stated mini-batch size), plus common seed/output instrumentation identical in both arms.", "rationale": "Settling plan ~/Desktop/office/plans/1506.02142-settling-plan.md, section 2: pre-rewrite commit 0eeeb7f11ce3a2e945608f3b51b57462b194ebd8 (batch 32) establishes the batch hypothesis only; it is not a post-hoc rescue arm, and neither it nor the later nested-validation/4,000-epoch experiment is run if B32 misses. Any historical-API experiment needs its own prior plan." }, "deviations": [ { "event": 62, "description": "Concrete and Yacht use the released scripts' dropout 0.01 (Energy 0.05), whereas the paper text lists dropout 0.05 and 0.005 without a dataset mapping.", "impact": "material", "reason": "Settling plan section 2: source-code values are held fixed; this is a disclosed, unresolved mismatch. No search over .005/.05 is made." }, { "event": 63, "description": "Environment is an inferred legacy stack (Python 2.7.15, NumPy 1.11.3, SciPy 0.18.1, scikit-learn 0.18.2, matplotlib 2.2.3 imported only, Keras 1.1.0, Theano 0.8.2 CPU float32; conda-forge with netlib libblas for Theano and OpenBLAS-built SciPy; Theano compiles with Debian g++ 14.2), not the authors' recovered original environment.", "impact": "minor", "reason": "The original environment is not recorded by the authors; the plan pins this stack (same as the earlier attempt) and both arms run in it." }, { "event": 64, "description": "Each fit runs in a fresh process with seed 10000*d+100*i+r set immediately before model construction, instead of experiment.py's single np.random.seed(1) stream over 20 splits; three seeds per split instead of one; experiment.py's loop is replaced by a runner (fit_one.py) that reproduces its loading code for one split and writes results outside the checkout.", "impact": "minor", "reason": "Settling plan section 3: an explicitly new paired ablation, so that B32 and B128 share initial weights, dropout-stream seeds and RNG state per pair." }, { "event": 65, "description": "Common instrumentation added to both arms' net.py: digests of the initial weights and NumPy RNG state after compile, digest of trained weights after fit, and the denormalized MC array kept on the object for lossless saving. B32 additionally changes batch_size 128 -> 32. No other line differs from a6259f1.", "impact": "none", "reason": "Required by the plan's pairing check and MC-array retention; the arithmetic of training, prediction and metrics is unchanged." }, { "event": 66, "description": "Yacht's encoding-only patch (coding line in experiment.py) is not applied: experiment.py is not executed, so its UTF-8 comment never reaches the Python 2 parser; net.py for all three datasets is ASCII and used unpatched apart from the instrumentation above.", "impact": "none", "reason": "Not needed with the runner; the plan allows but does not require the patch." }, { "event": 67, "description": "The 360 MC prediction arrays are retained (float32 as produced by net.py, savez_compressed, one file per fit) on the attempt's box volume during the attempt only; their SHA-256 digests are in the logged job output, but the arrays themselves are destroyed with the dedicated box at finalize and are not part of the published attempt.", "impact": "minor", "reason": "The engine publishes /work only through the log; several GB of arrays are not committed. The analysis job re-derives every RMSE/LL from the arrays before scoring." }, { "event": 68, "description": "The plan asks for one externally anchored plan binding document, code, environment and matrix. Here the binding is the engine's seal (plan hash in the hash-chained log) whose text cites the sha256 of /work/settle/inputs.lock.json, the plan copy, paper text, data manifests and job matrix; it is anchored externally only when the attempt's log head is committed/anchored, not before execution.", "impact": "minor", "reason": "The engine has no pre-execution external anchoring step; anchors/ is managed at release time and is out of scope for this run." } ], "notes": [ { "event": 109, "text": "Three-way classification under the sealed rule (analysis event 77; delta = 2 x archive SE; 18-way Bonferroni t intervals, q = 3.43466). B32 vs archive: Concrete RMSE within, Concrete LL within, Energy RMSE within, Energy LL within, Yacht RMSE within, Yacht LL inconclusive (point difference -0.0021 nats, interval -0.0654 to +0.0612 vs delta 0.0552: wider than the allowance, so it cannot pass). B128 vs archive: Concrete RMSE inconclusive, Concrete LL inconclusive, Energy RMSE outside (worse), Energy LL outside (worse), Yacht RMSE outside (worse), Yacht LL outside (worse). All six paired contrasts exclude zero in the improving direction (B32 better). Batch size supports the explanation for Energy RMSE, Energy LL and Yacht RMSE; Concrete RMSE/LL are within at B32 but B128 is only inconclusive there, so this experiment does not isolate batch size for Concrete; Yacht LL is inconclusive at B32. Five of six B32 cells are within the allowance, so the six-cell setup is not declared a match. The engine verdicts on the interval-end claims are the same rule: a cell passes only when both of its ends are within delta." }, { "event": 110, "text": "Section 5 correction, printed mechanically by analysis.py (event 77): preamble as in the plan; Concrete RMSE and Concrete test LL: 'Batch 32 is within the historical uncertainty allowance, but this experiment does not isolate batch size as the cause of the old gap.' Energy RMSE, Energy test LL, Yacht RMSE: 'The batch-32 rerun is within the predeclared historical uncertainty allowance, while the paired batch comparison supports batch-size drift as an explanation for our earlier gap.' Yacht test LL: 'The comparison is inconclusive at the predeclared precision; we will not turn uncertainty into a pass or enlarge the allowance.' Always appended: Concrete and Yacht still use the released scripts' dropout .01 rather than a confirmed mapping of the paper's .05/.005, and the original Keras environment has not been established. The earlier attempt arxiv-1506-02142-20261018T043211Z is not modified; this attempt supersedes its interpretation only." }, { "event": 111, "text": "Execution: all 360 fits completed (status ok 360) in 907.5 s wall in job fits (event 74), well inside the 120-minute cap; no fit failed, none was retried. Every RMSE/LL re-derived from its retained MC array matched the value net.py returned exactly (max abs difference 0.0). Initial-weight and RNG-state digests were equal within every B32/B128 pair. The B128 arm reproduces the earlier attempt's batch-128 misses (Energy RMSE 2.040 vs 2.018 then, Yacht RMSE 2.384 vs 2.365, Concrete RMSE 5.522 vs 5.474), now with three seeds per split. Scope: a qualified setup reproduction test, not proof that the historical hyperparameter selection was sound." } ], "flags": [ "At least one deviation from the paper's protocol is marked material." ], "compute": { "shell_commands": 13, "sandbox_seconds": 81.2, "jobs": 2, "job_seconds": 921.7, "sandbox": { "container": "repro-arxiv-1506-02142-20261020t024912z", "docker_daemon": { "mem_bytes": 8326955008, "ncpu": 4 }, "effective": { "cpus": 3.5, "memory_bytes": 6442450944 }, "image": "repro-sandbox:0.2", "image_id": "sha256:18eae942523ffa32c03a8060875b4fba8f769a6aacdfb12f5aa07f29eaec3b79", "limits": { "cpus": 3.5, "gpu": null, "memory": "6g", "network": true, "shm": "2g" }, "target": "cloud", "target_detail": { "box": "arxiv-1506-02142-20261020t024912z", "droplet": "repro-arxiv-1506-02142-20261020-0252", "image": "ubuntu-24-04-x64", "price_hourly_usd": 0.08333, "provider": "digitalocean", "region": "tor1", "size": "s-4vcpu-8gb-intel" }, "volume": "repro-arxiv-1506-02142-20261020t024912z-work", "warnings": [] }, "environment_event": 113 }, "drivers": [ "operator:claude-opus-5-5 (Claude Code subagent, AI-driven)" ], "model_usage": {}, "model_cost_usd": null, "started_at": "2026-10-20T02:49:12.310Z", "ended_at": "2026-10-20T03:21:10.246Z", "log": { "events": 117, "head_hash": "0e96b30ad495309e93a1a1d8b07fe06f7f0b1e24e4f4938a272874a1bb4646db" }, "right_of_reply": "Authors may submit a response, which is published alongside this report unedited.", "redactions": [ { "event": 116, "rule": "operator home directory prefix → ~", "reason": "Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~", "replacements": 8, "pre_redaction_head": "050491ce90f47cb00cf5ec5a60eea67a1abfbe1acff274b5bea6dd6f073414ac", "pre_redaction_events": 116, "chain_rehashed_from_event": 39, "plan_hash_events": [ 69 ], "seal_bytes_changed": true, "results_before_match": true, "results_before_status": "operator assertion, checked for internal consistency against the current log", "disclosure": "Before release, the operator's local home-directory path was replaced with ~ in 8 places. Every event hash from event 39 on changed with it. The sealed plan's text and its plan hash (event 69) changed because of the substitution; the seal now pins the substituted text. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now." } ] }