Why do machine-learning results fail to reproduce?
When a published number does not come out again, the cause is usually in the record, and the record usually names it.
It is rarely the idea
When a machine-learning paper's headline number does not come out of the authors' own code, the first suspicion is that the method does not work. In the cases on the OpenRepro ledger, that is not what the record shows. The methods ran and produced numbers close to the paper's. What failed was something around the method: how randomness was handled, which version of the code ran, a setting that differs between the paper and the released code, or how much margin a claim had in the first place. In some of these cases the departure was in our own first re-run, and the record shows that too.
Each case below is a paper whose code was re-run with claims and tolerances registered before any result was recorded. The numbers are from the evidence reports linked on each paper's page.
Random seeds, and the runs that collapse
The GCN paper, Semi-Supervised Classification with Graph Convolutional Networks, reports accuracies as the mean of 100 runs with different random weight initializations. The released training script fixes a single seed. To follow the paper, our first re-run changed that one line so the seed could be set from the environment, then ran seeds 1 to 100.
Cora came out at 81.53 against 81.5 reported, and Pubmed at 79.16 against 79.0. Citeseer came out below the 0.5-point band registered around 70.3, because two of the 100 runs collapsed to 15.9% and 9.3%, below chance for six classes. But a fixed band does not see run-to-run spread, and the gap was smaller than one standard error of the mean. A settling test then ran a fresh cohort of 100 seeds with the released code and settings, judged by an interval rule with a precision gate sealed before training. The mean came out at 70.37 against 70.3; one run collapsed again, to 26.0% after early stopping, and stays included; and the interval, though it overlaps 70.3, extends further from the mean than the gate allows. The outcome is inconclusive: the test was not precise enough to settle the gap. The paper does not say whether any runs were excluded.
The lesson for authors is not to hide the collapsed runs. It is to say how many runs there were, how they were seeded, and what was done with outliers, so that a reader can tell a rare failure of optimization from a failure of the method. The lesson for anyone re-running a paper is to judge a mean by its uncertainty, not by a fixed band.
When the re-run, not the paper, departs
Two of the cases below began with a departure of our own. Our first re-run of the dropout paper trained with batch size 128, the value in the committed scripts, where the paper and the authors' earlier code use 32. Our first re-run of ROCKET ran the repository's 2022 code rather than code from the paper's date, without declaring a code version first, and the tie rule that decided its label was chosen after the tie was seen. Checks of our own work found both, and both are listed on What we got wrong with what changed.
A setting that differs between the paper and the code
Dropout as a Bayesian Approximation has an experiments repository with result files committed alongside the scripts. The committed results match Table 1 exactly. They were committed on 14 September 2016; the scripts were rewritten the next day. Our first re-run used the scripts as committed, at the commit matching the tested paper version, with Python 2.7, Keras 1.1.0 and Theano 0.8.2. It reproduced the Boston (3.09 against 2.97) and Wine (0.624 against 0.62) values but not Concrete (5.47 against 5.23), Energy (2.02 against 1.66) or Yacht (2.37 against 1.11).
A desk check of our work then found the main cause on our side: the rewritten scripts train with batch size 128, while the paper and the code from before the rewrite use 32. A paired test sealed before training re-ran Concrete, Energy and Yacht at both batch sizes, 360 fits over the 20 committed splits with three seeds each, judged against twice the archived standard error. At batch 32, five of the six re-tested values fell within that allowance; Yacht's log-likelihood was inconclusive, its point estimate 0.002 nats from the archive but its interval wider than the allowance. At batch 128 the earlier misses came back, and every paired contrast favored batch 32.
The outcome is still partially reproduced, because the Yacht log-likelihood check did not pass, and two things remain open: Concrete and Yacht use the scripts' dropout of 0.01, which the paper does not map to datasets, and the original software environment is inferred. One hyperparameter that differs between the paper and the released code is enough to make a published table look unreproducible.
A narrow lead at the top
Some claims turn on a margin that is about the size of the noise. ROCKET reports the best mean rank among eight classifiers on the 85 bake-off datasets. A corrective re-run fixed the code version (the authors' first code commit, from 2019) and the ranking rule before any result was recorded, taking the rule from the paper: ranks within a dataset are averaged when tied, and a shared best mean rank would count as not reproduced. ROCKET's accuracies reproduced closely: mean accuracy 0.8517 against 0.8510 and a mean absolute per-dataset difference of 0.20 percentage points. With those accuracies substituted into the ranking, ROCKET's mean rank is strictly best, 571/170 against 579/170 for TS-CHIEF. Outcome: reproduced, 3 of 3 headline claims held.
The first attempt, kept as a superseded version, gave a different answer. It ran the repository's 2022 code, its accuracies left ROCKET and TS-CHIEF tied at exactly 288/85, and the tie rule that decided its 'partially reproduced' label was chosen after the tie was seen. The two attempts sampled different random kernels as well as different code, so the gap between a tie and a lead of 8/170 is within ordinary run-to-run variation. The paper says the differences among the top methods are not statistically significant, and the record agrees: a reader who wants to know whether ROCKET is fast and accurate gets a yes; a reader who wants to know whether it is first gets a narrow lead that another run could erase.
Hardware, memory and the budget to re-run
Re-runs rarely have the authors' hardware. The GCN runs used a CPU instead of a GPU, and because a Pubmed run took about six minutes on CPU, Pubmed was run 20 times instead of 100, a deviation declared before any accuracy was seen. The first ROCKET re-run hit the memory limit on 19 of 85 datasets in a sandbox with about 2 GB free and ran those datasets with a low-memory driver; its total of 111 minutes was inflated by memory pressure and that workaround. The corrective re-run used a dedicated machine with enough memory for the authors' in-memory path on every dataset, one single-threaded process per dataset: 40.5 minutes in total against the paper's 1 hour 50 minutes. Because that CPU is newer than the paper's, the timings confirm the paper's upper bounds, not its exact figures.
Every such difference is recorded as a deviation with an impact of none, minor or material, and shown on the paper's page. The point is not that the deviation is harmless but that it is visible.
Nothing was pinned
Three of the papers above pin no library versions at all. The ROCKET repository pins none, so the re-run chose scikit-learn 1.1.3, the newest release that still supports the exact classifier call in the authors' script. The dropout repository pins none, so a September 2016 environment had to be reconstructed. The repository behind the COVID-19 change-point paper pins none either; the re-run used the releases current in May 2020 and reproduced all four spreading rates, but only because the repository also bundles a snapshot of the case data. The live data series now runs to 2023 and has been revised backwards, so the notebook's default of using the latest date would not have reproduced the paper.
Unpinned environments do not always break a result. They always make it harder to tell why a result broke.
What a reader can check before trusting a number
None of this requires re-running anything.
- Does the paper say how many runs a reported number averages over, and does the released code run that many?
- Is there a pinned environment: a lockfile, a session-info, a tagged release, or at least named versions?
- Does the repository state which commit or release produced the tables, and does that commit still exist?
- Are the result files in the repository regenerated by the scripts in the repository, or merely stored next to them?
- How large is the margin behind a comparative claim such as "best" or "state of the art", and does the paper say whether it is significant?
- Is the data a pinned snapshot, or a live download that may have changed since?