Replication record · arXiv:1506.02142

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani · 2015 · arXiv

Partially reproduced

Rerunning the first authors experiment scripts at the commit matching arXiv v6 (Python 2.7, Keras 1.1.0, Theano 0.8.2, CPU) on five UCI datasets reproduces MC dropout RMSE and log-likelihood on Boston (3.09 vs 2.97) and Wine (0.624 vs 0.62), but not on Concrete (5.47 vs 5.23, just outside the band), Energy (2.02 vs 1.66) or Yacht (2.37 vs 1.11). The result files the authors committed match Table 1 exactly, but they were committed a day before the scripts were rewritten, and the committed scripts do not regenerate them; the Yacht script as committed is not even valid Python 2 without an encoding line. The authors README separately reports test-set contamination in the original hyperparameter search. Five larger datasets were out of scope.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Non-headline claims (stricter checks or registered controls) did not hold; the claims list names each. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 2 of 5 consistent
Other checks: 2 of 5 consistent
Deviations: 2 (0 material)
Integrity flags: 0
Logged compute: 0.14 h over 16 commands and 4 jobs
Primary code (declared before the first recorded result): github.com/yaringal/DropoutUncertaintyExps (the first authors experiments repository; its README names this paper; the paper itself links no code) at the last commit on or before 2016-10-04, the date of arXiv v6, run with its documented command for each dataset
Commits the recorded results name: a6259f1: 10 results, named in the result's method text
Work tree when the attempt ended: yaringal/DropoutUncertaintyExps@a6259f1db8f5 in /work/due, 26 file(s) modified locally (see code-diff.patch)

Protocol

Declared before the first recorded result: primary code is github.com/yaringal/DropoutUncertaintyExps (the first authors experiments repository; its README names this paper; the paper itself links no code) at the last commit on or before 2016-10-04, the date of arXiv v6, run with its documented command for each dataset; earlier versions of the authors' code may be tried as a fallback. arXiv v6 is the version tested and includes the October 2016 standard-error erratum. If a claim misses with the v6-date code, earlier commits may be tried as a declared fallback and are reported separately.

Claims

MC dropout reaches average test RMSE 2.97 on Boston Housing.headline

Consistent

paper 2.97observed 3.0889

Paper 2.97 RMSEObserved 3.0889 RMSEAccepted if within ±0.38

Boston Housing 506 13 4.32 ±0.29 3.01 ±0.18 2.97 ±0.19 -2.90 ±0.07 -2.57 ±0.09 -2.46 ±0.06

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.04 · registered at event 9, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test RMSE over the 20 UCI splits, MC dropout

Why this tolerance. Each value is a mean over 20 random train/test splits with the reported standard error; a run with new random initialisations can move by about one standard error, and environment differences add a little, so the band is twice the reported standard error (at least 0.02).

Event 55. Observed 3.0889, delta +0.1189, consistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test log-likelihood -2.46 on Boston Housing.

Consistent

paper -2.46observed -2.5096

Paper -2.46 natsObserved -2.5096 natsAccepted if within ±0.12

Boston Housing 506 13 4.32 ±0.29 3.01 ±0.18 2.97 ±0.19 -2.90 ±0.07 -2.57 ±0.09 -2.46 ±0.06

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.02 · registered at event 10, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood over the 20 UCI splits, MC dropout

Event 56. Observed -2.5096, delta -0.0496, consistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test RMSE 5.23 on Concrete Strength.headline

Inconsistent

paper 5.23observed 5.4741

Paper 5.23 RMSEObserved 5.4741 RMSEAccepted if within ±0.24

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.047 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test RMSE over the 20 UCI splits, MC dropout

Event 57. Observed 5.4741, delta +0.2441, inconsistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test log-likelihood -3.04 on Concrete Strength.

Inconsistent

paper -3.04observed -3.0983

Paper -3.04 natsObserved -3.0983 natsAccepted if within ±0.04

Concrete Strength 1,030 8 7.19 ±0.12 5.67 ±0.09 5.23 ±0.12 -3.39 ±0.02 -3.16 ±0.02 -3.04 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.019 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood over the 20 UCI splits, MC dropout

Event 58. Observed -3.0983, delta -0.0583, inconsistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test RMSE 1.66 on Energy Efficiency.headline

Inconsistent

paper 1.66observed 2.0182

Paper 1.66 RMSEObserved 2.0182 RMSEAccepted if within ±0.08

8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.216 · registered at event 13, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test RMSE over the 20 UCI splits, MC dropout

Event 59. Observed 2.0182, delta +0.3582, inconsistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test log-likelihood -1.99 on Energy Efficiency.

Inconsistent

paper -1.99observed -2.1425

Paper -1.99 natsObserved -2.1425 natsAccepted if within ±0.04

8 2.65 ±0.08 1.80 ±0.05 1.66 ±0.04 -2.39 ±0.03 -2.04 ±0.02 -1.99 ±0.02

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.077 · registered at event 14, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood over the 20 UCI splits, MC dropout

Event 60. Observed -2.1425, delta -0.1525, inconsistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test RMSE 0.62 on Wine Quality Red.headline

Consistent

paper 0.62observed 0.6243

Paper 0.62 RMSEObserved 0.6243 RMSEAccepted if within ±0.02

Wine Quality Red 1,599 11 0.65 ±0.01 0.64 ±0.01 0.62 ±0.01 -0.98 ±0.01 -0.97 ±0.01 -0.93 ±0.01

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.007 · registered at event 15, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test RMSE over the 20 UCI splits, MC dropout

Event 61. Observed 0.6243, delta +0.0043, consistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test log-likelihood -0.93 on Wine Quality Red.

Consistent

paper -0.93observed -0.9396

Paper -0.93 natsObserved -0.9396 natsAccepted if within ±0.02

Wine Quality Red 1,599 11 0.65 ±0.01 0.64 ±0.01 0.62 ±0.01 -0.98 ±0.01 -0.97 ±0.01 -0.93 ±0.01

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.01 · registered at event 16, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood over the 20 UCI splits, MC dropout

Event 62. Observed -0.9396, delta -0.0096, consistent. Evidence events [43] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test RMSE 1.11 on Yacht Hydrodynamics.headline

Inconsistent

paper 1.11observed 2.365

Paper 1.11 RMSEObserved 2.365 RMSEAccepted if within ±0.18

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 2.131 · registered at event 17, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test RMSE over the 20 UCI splits, MC dropout

Event 63. Observed 2.365, delta +1.255, inconsistent. Evidence events [50] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

MC dropout reaches average test log-likelihood -1.55 on Yacht Hydrodynamics.

Inconsistent

paper -1.55observed -2.26485

Paper -1.55 natsObserved -2.26485 natsAccepted if within ±0.06

Yacht Hydrodynamics 308 6 6.89 ±0.67 1.02 ±0.05 1.11 ±0.09 -3.43 ±0.16 -1.63 ±0.02 -1.55 ±0.03

Table 1, p. 7 (Dropout columns) · quote verified on p. 7 · equivalence check · observed/published 1.461 · registered at event 18, before the first recorded result

Method, tolerance rationale and evidence

Metric. average test predictive log-likelihood over the 20 UCI splits, MC dropout

Event 64. Observed -2.26485, delta -0.7149, inconsistent. Evidence events [50] (value matched by the legacy rule; not bound to an excerpt). Authors experiment.py (paper protocol: 40 x 10 epochs, tuned dropout and tau hard-coded per dataset) at commit a6259f1, Python 2.7, Keras 1.1.0, Theano 0.8.2 on CPU; mean over the 20 UCI splits of the MC-dropout test RMSE / test log-likelihood written by the script.

Deviations from the paper's protocol

none
Added the line # -*- coding: utf-8 -*- at the top of yacht/net/experiment.py.
As committed it is not valid Python 2 source (non-ASCII character in a comment, no encoding declaration). The added line is a declaration only and changes no code. event 48
minor
Python 2.7.15 from conda-forge with numpy 1.11.3, scipy 0.18.1, scikit-learn 0.18.2, matplotlib 2.2 (only imported), Keras 1.1.0 and Theano 0.8.2 on CPU with float32; Theano compiled with the system g++ 14 plus conda libgfortran 3 and libxcrypt headers.
Reconstruction of the September 2016 environment (the README targets the then-latest Keras; Theano with GPU). No versions are pinned in the repository. event 65

Notes from the attempt

event 19
Scope, declared before any run: RMSE and test log-likelihood of MC dropout on Boston, Concrete, Energy, Wine and Yacht (Table 1). Kin8nm, Naval, Power, Protein and Year are not tested because of the compute budget. The VI and PBP baseline columns come from other papers and are not tested.
event 47
Yacht did not run in job mcdropout-uci-c (event 43): yacht/net/experiment.py at commit a6259f1 contains a UTF-8 character (the e-acute in Jose in the comment on line 2) without an encoding declaration, which Python 2 rejects with a SyntaxError before any code runs. The Yacht figures printed by that job came from the result files committed in the repository, not from this run, and are not recorded as results (git status at event 46 shows only the other four datasets results were rewritten). The other four experiment.py files spell the name in ASCII.
event 54
The result files committed in the repository (commit 3e9e154, 2016-09-14) match Table 1 exactly (MC RMSE Boston 2.9735, Concrete 5.2299, Energy 1.6634, Wine 0.6202, Yacht 1.1093; event 52). The experiment scripts were rewritten the next day (889f56d, 2016-09-15: net.py updated for the then-current Keras, experiment.py added for Energy and Yacht), and rerunning the committed scripts at the paper-version commit does not reproduce the committed results for Concrete, Energy and Yacht. The declared earlier-code fallback was not run: no earlier runnable script exists for Energy and Yacht, and the earlier Boston/Concrete code targets a pre-1.0 Keras API, so the published numbers cannot be traced to a runnable earlier version in this repository. The README later documents test-set contamination in the original hyperparameter search and reports a re-run of the paper code giving Boston 2.83 +- 0.17.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T04:32:11.804Z → 2026-10-18T04:59:00.922Z
Event log: 70 events, hash-chained; head 00288e424a438806ab2c4319dce4024be4abe8b19b23e43cd22ae4c785383e34
Verify: python3 openrepro_verify.py arxiv-1506-02142-20261018T043211Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.