Machine learning · 2016

Reproducibility report: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Report by OpenRepro, released

Paper by Yarin Gal, Zoubin Ghahramani. International Conference on Machine Learning (ICML), 2016. arXiv 1506.02142

Also known as Monte Carlo dropout.

Partially reproduced

10 of 12 headline claims held

10 other claims missedscored run not a direct source
The full answer and its qualifiers

Does MC dropout (Gal and Ghahramani, 2016) reproduce? Partly: when OpenRepro re-ran the authors' code, 10 of 12 headline claims held within tolerances sealed before scored execution; 10 other checks did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one (checked 20 Oct 2026).

  • 10 other checks did not hold.
  • A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.

Does MC dropout (Gal and Ghahramani, 2016) reproduce? Partly: when OpenRepro re-ran the authors' code, 10 of 12 headline claims held within tolerances sealed before scored execution (checked 20 Oct 2026).

Checked
Released
Assistants
Ask via MCP

Does MC dropout reproduce?

Partly. 10 of 12 headline claims held within tolerances sealed before scored execution; 2 did not.

Read the answer with its qualifiers:

  • 10 other checks did not hold.
  • A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.

Looking for a replication of MC dropout? This report is a computational reproduction: the authors' own published code was re-run and its results compared with the numbers in the paper. It does not set out to test the findings on new data, and it is not peer reviewed. What an attempt checks.

How far the re-run landed from the published number, in tolerances30 claims re-run, 12 headline
  1. MC dropout: every claim checked30 claims: 18 held, 12 did not hold.∞∞3.4×5.1×∞5.3×3.5×∞5.1×9.5×∞17.9×8.0×∞
  • Held (large: headline claim; small: other claim)
  • Did not hold
Reading the marks
  • Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly
  • Chevron at the edge: beyond ±3 tolerances, with the actual multiple printed beside it (∞ when the tolerance was zero)
The data behind this chart
ClaimPublishedObservedAccepted ifDeviation ÷ toleranceResult
Batch-32 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23). (headline)5.22995.06134within ±0.242198-0.70Held
Batch-32 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23). (headline)5.22995.34285within ±0.242198+0.47Held
Batch-128 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).5.22995.363within ±0.242198+0.55Held
Batch-128 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).5.22995.68132within ±0.242198+1.86Did not hold
Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).0-0.209382at most 0beyond limitHeld
Batch-32 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04). (headline)-3.03891-3.06444within ±0.0417515-0.61Held
Batch-32 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04). (headline)-3.03891-3.00724within ±0.0417515+0.76Held
Batch-128 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).-3.03891-3.14992within ±0.0417515-2.66Did not hold
Batch-128 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).-3.03891-3.07235within ±0.0417515-0.80Held
Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).00.0532861at least 0beyond limitHeld
Batch-32 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66). (headline)1.663411.6082within ±0.0887608-0.62Held
Batch-32 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66). (headline)1.663411.70967within ±0.0887608+0.52Held
Batch-128 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).1.663411.9639within ±0.0887608+3.39Did not hold
Batch-128 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).1.663412.1157within ±0.0887608+5.10Did not hold
Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).0-0.312057at most 0beyond limitHeld
Batch-32 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99). (headline)-1.98527-2.00112within ±0.040562-0.39Held
Batch-32 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99). (headline)-1.98527-1.95489within ±0.040562+0.75Held
Batch-128 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).-1.98527-2.19885within ±0.040562-5.27Did not hold
Batch-128 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).-1.98527-2.1271within ±0.040562-3.50Did not hold
Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).00.152at least 0beyond limitHeld
Batch-32 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11). (headline)1.109250.96519within ±0.174423-0.83Held
Batch-32 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11). (headline)1.109251.26781within ±0.174423+0.91Held
Batch-128 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).1.109251.99836within ±0.174423+5.10Did not hold
Batch-128 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).1.109252.76932within ±0.174423+9.52Did not hold
Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).0-0.906752at most 0beyond limitHeld
Batch-32 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55). (headline)-1.55406-1.61949within ±0.0552211-1.18Did not hold
Batch-32 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55). (headline)-1.55406-1.49281within ±0.0552211+1.11Did not hold
Batch-128 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).-1.55406-2.54495within ±0.0552211-17.94Did not hold
Batch-128 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).-1.55406-1.99479within ±0.0552211-7.98Did not hold
Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).00.441131at least 0beyond limitHeld

What the attempt found

Sealed settling test of MC dropout Table 1 (Concrete, Energy, Yacht; RMSE and test LL): 360 paired fits of the authors a6259f1 net.py at batch 32 (paper) and 128 (committed code) on the 20 committed splits with three seeds each, in a pinned Keras 1.1.0/Theano 0.8.2 environment. Five of six batch-32 cells fall within the predeclared allowance of 2 archive SEs (18-way Bonferroni t intervals); Yacht LL is inconclusive because its interval is wider than the allowance, though its point estimate is 0.002 nats from the archive. Batch 128 is outside the allowance in the worse direction for Energy and Yacht and inconclusive for Concrete, and every paired contrast favours batch 32, so batch-size drift explains the earlier Energy and Yacht RMSE gaps. Dropout 0.01 for Concrete/Yacht and the original environment remain unresolved.

Assurance

Level
Sealed, bound and re-derived (what the levels mean)
Plan
Sealed before scored execution; 12 outputs observed before the seal (all of it in the log)
Evidence
Bound: each value was extracted from the cited run output by a rule fixed before the seal
Execution manifest
Yes: for each scored run the command, the Git checkout state (HEAD, status, diff hash) and the hashes of the driver files the command names were recorded when it started and when it ended. Data files and the environment are not bound by this
Report re-derived from the log
Passed: the engine re-derived the report from the log
Engine version
0.2.1

Headline claims

These decide the outcome. Each row gives the published value, the value from the re-run and the accepted-if rule registered before results were recorded. Each row has its own ¶ link.

ClaimPublishedObservedAccepted ifResult
Batch-32 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).Table 1, p. 7 (Dropout columns) · registered at event 9 · quote verified on p. 7 · equivalence check5.2299RMSE, target units5.06134within ±0.242198Held
Batch-32 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).Table 1, p. 7 (Dropout columns) · registered at event 10 · quote verified on p. 7 · equivalence check5.2299RMSE, target units5.34285within ±0.242198Held
Batch-32 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).Table 1, p. 7 (Dropout columns) · registered at event 14 · quote verified on p. 7 · equivalence check-3.03891nats per test observation-3.06444within ±0.0417515Held
Batch-32 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).Table 1, p. 7 (Dropout columns) · registered at event 15 · quote verified on p. 7 · equivalence check-3.03891nats per test observation-3.00724within ±0.0417515Held
Batch-32 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).Table 1, p. 7 (Dropout columns) · registered at event 19 · quote verified on p. 7 · equivalence check1.66341RMSE, target units1.6082within ±0.0887608Held
Batch-32 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).Table 1, p. 7 (Dropout columns) · registered at event 20 · quote verified on p. 7 · equivalence check1.66341RMSE, target units1.70967within ±0.0887608Held
Batch-32 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).Table 1, p. 7 (Dropout columns) · registered at event 24 · quote verified on p. 7 · equivalence check-1.98527nats per test observation-2.00112within ±0.040562Held
Batch-32 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).Table 1, p. 7 (Dropout columns) · registered at event 25 · quote verified on p. 7 · equivalence check-1.98527nats per test observation-1.95489within ±0.040562Held
Batch-32 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).Table 1, p. 7 (Dropout columns) · registered at event 29 · quote verified on p. 7 · equivalence check1.10925RMSE, target units0.96519within ±0.174423Held
Batch-32 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).Table 1, p. 7 (Dropout columns) · registered at event 30 · quote verified on p. 7 · equivalence check1.10925RMSE, target units1.26781within ±0.174423Held
Batch-32 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).Table 1, p. 7 (Dropout columns) · registered at event 34 · quote verified on p. 7 · equivalence check-1.55406nats per test observation-1.61949within ±0.0552211Did not hold
Batch-32 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).Table 1, p. 7 (Dropout columns) · registered at event 35 · quote verified on p. 7 · equivalence check-1.55406nats per test observation-1.49281within ±0.0552211Did not hold

Other claims checked

18 claims: 8 held, 10 did not hold

Reported alongside the headline claims; they do not change the outcome.

ClaimPublishedObservedAccepted ifResult
Batch-128 arm, Concrete Strength RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).Table 1, p. 7 (Dropout columns) · registered at event 11 · quote verified on p. 7 · equivalence check5.2299RMSE, target units5.363within ±0.242198Held
Batch-128 arm, Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 5.229903994142711 (paper cell 5.23).Table 1, p. 7 (Dropout columns) · registered at event 12 · quote verified on p. 7 · equivalence check5.2299RMSE, target units5.68132within ±0.242198Did not hold
Concrete Strength RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 13 · quote verified on p. 7 · bound check0RMSE, target units-0.209382at most 0Held
Batch-128 arm, Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).Table 1, p. 7 (Dropout columns) · registered at event 16 · quote verified on p. 7 · equivalence check-3.03891nats per test observation-3.14992within ±0.0417515Did not hold
Batch-128 arm, Concrete Strength LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -3.0389055163159577 (paper cell -3.04).Table 1, p. 7 (Dropout columns) · registered at event 17 · quote verified on p. 7 · equivalence check-3.03891nats per test observation-3.07235within ±0.0417515Held
Concrete Strength LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 18 · quote verified on p. 7 · bound check0nats per test observation0.0532861at least 0Held
Batch-128 arm, Energy Efficiency RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).Table 1, p. 7 (Dropout columns) · registered at event 21 · quote verified on p. 7 · equivalence check1.66341RMSE, target units1.9639within ±0.0887608Did not hold
Batch-128 arm, Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.663405433671937 (paper cell 1.66).Table 1, p. 7 (Dropout columns) · registered at event 22 · quote verified on p. 7 · equivalence check1.66341RMSE, target units2.1157within ±0.0887608Did not hold
Energy Efficiency RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 23 · quote verified on p. 7 · bound check0RMSE, target units-0.312057at most 0Held
Batch-128 arm, Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).Table 1, p. 7 (Dropout columns) · registered at event 26 · quote verified on p. 7 · equivalence check-1.98527nats per test observation-2.19885within ±0.040562Did not hold
Batch-128 arm, Energy Efficiency LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.9852732326905922 (paper cell -1.99).Table 1, p. 7 (Dropout columns) · registered at event 27 · quote verified on p. 7 · equivalence check-1.98527nats per test observation-2.1271within ±0.040562Did not hold
Energy Efficiency LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 28 · quote verified on p. 7 · bound check0nats per test observation0.152at least 0Held
Batch-128 arm, Yacht Hydrodynamics RMSE: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).Table 1, p. 7 (Dropout columns) · registered at event 31 · quote verified on p. 7 · equivalence check1.10925RMSE, target units1.99836within ±0.174423Did not hold
Batch-128 arm, Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean 1.1092497489252557 (paper cell 1.11).Table 1, p. 7 (Dropout columns) · registered at event 32 · quote verified on p. 7 · equivalence check1.10925RMSE, target units2.76932within ±0.174423Did not hold
Yacht Hydrodynamics RMSE: the upper end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (RMSE lower with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 33 · quote verified on p. 7 · bound check0RMSE, target units-0.906752at most 0Held
Batch-128 arm, Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).Table 1, p. 7 (Dropout columns) · registered at event 36 · quote verified on p. 7 · equivalence check-1.55406nats per test observation-2.54495within ±0.0552211Did not hold
Batch-128 arm, Yacht Hydrodynamics LL: the upper end of the predeclared Bonferroni-t interval for the paired split difference (three-seed split mean minus archived split value), on the metric scale, lies within delta = 2 x archive SE of the unrounded archive mean -1.5540594381496233 (paper cell -1.55).Table 1, p. 7 (Dropout columns) · registered at event 37 · quote verified on p. 7 · equivalence check-1.55406nats per test observation-1.99479within ±0.0552211Did not hold
Yacht Hydrodynamics LL: the lower end of the predeclared Bonferroni-t interval for the paired batch contrast E_i = mean(B32_i) - mean(B128_i) over the 20 splits excludes zero in the improving direction (LL higher with batch 32).Section 5 (UCI regression experiments), p. 7 · registered at event 38 · quote verified on p. 7 · bound check0nats per test observation0.441131at least 0Held

Deviations from the paper's setup

7 recorded: 1 material, 4 minor, 2 none
  • materialConcrete and Yacht use the released scripts' dropout 0.01 (Energy 0.05), whereas the paper text lists dropout 0.05 and 0.005 without a dataset mapping.
  • minorEnvironment is an inferred legacy stack (Python 2.7.15, NumPy 1.11.3, SciPy 0.18.1, scikit-learn 0.18.2, matplotlib 2.2.3 imported only, Keras 1.1.0, Theano 0.8.2 CPU float32; conda-forge with netlib libblas for Theano and OpenBLAS-built SciPy; Theano compiles with Debian g++ 14.2), not the authors' recovered original environment.
  • minorEach fit runs in a fresh process with seed 10000*d+100*i+r set immediately before model construction, instead of experiment.py's single np.random.seed(1) stream over 20 splits; three seeds per split instead of one; experiment.py's loop is replaced by a runner (fit_one.py) that reproduces its loading code for one split and writes results outside the checkout.
  • noneCommon instrumentation added to both arms' net.py: digests of the initial weights and NumPy RNG state after compile, digest of trained weights after fit, and the denormalized MC array kept on the object for lossless saving. B32 additionally changes batch_size 128 -> 32. No other line differs from a6259f1.
  • noneYacht's encoding-only patch (coding line in experiment.py) is not applied: experiment.py is not executed, so its UTF-8 comment never reaches the Python 2 parser; net.py for all three datasets is ASCII and used unpatched apart from the instrumentation above.
  • minorThe 360 MC prediction arrays are retained (float32 as produced by net.py, savez_compressed, one file per fit) on the attempt's box volume during the attempt only; their SHA-256 digests are in the logged job output, but the arrays themselves are destroyed with the dedicated box at finalize and are not part of the published attempt.
  • minorThe plan asks for one externally anchored plan binding document, code, environment and matrix. Here the binding is the engine's seal (plan hash in the hash-chained log) whose text cites the sha256 of /work/settle/inputs.lock.json, the plan copy, paper text, data manifests and job matrix; it is anchored externally only when the attempt's log head is committed/anchored, not before execution.

The run
Wall time
31 min 58 s, 20 October 2026 to 20 October 2026
Job compute
15 min 22 s across 2 jobs
Container image
repro-sandbox:0.2 sha256:18eae942523f…
Sandbox
3.5 CPUs, 6 GB, no GPU
Run by
Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt
arxiv-1506-02142-20261020T024912Z
Log
117 events, head 0e96b30a

Evidence

log seal, 117 hash-chained events, 1 earlier version

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 117 hash-chained events

0e96b30ad495309e93a1a1d8b07fe06f7f0b1e24e4f4938a272874a1bb4646db

The head hash of the attempt's append-only log. The engine generated this report from that log. The verifier checks the log and its artifacts against this hash, and that the report names it. It does not check the report's contents. The evidence bundle for this attempt is not published yet.

Released 4 November 2026 · Settling update sent 20 October 2026 to the first author, with a request to share it with the co-author. The report was regenerated after the update, adding qualifiers; the outcome did not change. No reply received.

Earlier versions of this report

The report above is the current version. Each earlier attempt stays public, unedited, with the reason it was replaced.

  • arxiv-1506-02142-20261018T043211Z, 18 October 2026: Partially reproduced. Superseded by arxiv-1506-02142-20261020T024912Z: It trained with batch size 128 from the committed scripts, while the paper and the authors' earlier code use 32; the sealed settling test re-ran the three missed datasets at both batch sizes.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Partially reproduced; 10 of 12 headline claims held (10 other checks did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one). Attempt arxiv-1506-02142-20261020T024912Z, log head 0e96b30ad495309e, checked 20 Oct 2026. https://openrepro.org/papers/1506.02142/

APA style

OpenRepro. (2026). Reproducibility report: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (Attempt arxiv-1506-02142-20261020T024912Z; log head 0e96b30ad495309e). https://openrepro.org/papers/1506.02142/

BibTeX
@techreport{openrepro-1506.02142,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/1506.02142/},
  note        = {Attempt arxiv-1506-02142-20261020T024912Z; log head 0e96b30ad495309e; checked 20 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

References

1 paper
  1. Yarin Gal, Zoubin Ghahramani (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. International Conference on Machine Learning (ICML). arXiv 1506.02142

Badge

OpenRepro: Partially reproduced. 10 of 12 headline claims held; 10 other checks did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Copy the badge as Markdown or HTML

Markdown

[![OpenRepro: Partially reproduced. 10 of 12 headline claims held; 10 other checks did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one.](https://openrepro.org/badge/1506.02142.svg)](https://openrepro.org/papers/1506.02142/)

HTML

<a href="https://openrepro.org/papers/1506.02142/"><img src="https://openrepro.org/badge/1506.02142.svg" alt="OpenRepro: Partially reproduced. 10 of 12 headline claims held; 10 other checks did not hold; a scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the report names each one." width="611" height="20"></a>

Right of reply, and reporting an error

Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.