Findings so far
Short notes on what individual re-runs turned up: what was run, what came out, and what the test does not settle. Each note is about one listed paper and links to its report.
A settling test of GCN's Citeseer result ran a fresh cohort of 100 training seeds with the authors' released code and settings: the mean was 70.37% against the published 70.3%. Its interval overlaps the published value but extends further from the mean than the precision gate set before training, so the outcome is “Inconclusive: a registered validity/precision condition was not satisfied”, and the test does not settle the earlier shortfall. One of the 100 runs collapsed to 26.0% after early stopping and stays included.
A first ROCKET re-run used the repository's 2022 code and a tie rule chosen after its ranks tied. A corrective re-run declared the 2019 code and the paper's ranking rule before any result was recorded: mean accuracy over the 85 bake-off datasets came out at 0.8517 against 0.8510, and ROCKET's mean rank was strictly best (571/170 against 579/170 for TS-CHIEF), a margin that run-to-run variation can move.
A first re-run trained the MC dropout networks with batch size 128, the value in the committed scripts, while the paper and the authors' earlier code use 32. In a paired test sealed before training (360 fits on the 20 committed splits), batch 32 brought five of the six re-tested Concrete, Energy and Yacht values within twice the archived standard error; Yacht's log-likelihood was inconclusive at that precision, and batch 128 reproduced the earlier misses.
Our first re-run of Table 1 simulated data with a helper function from the grf package's 2020 code, not the design the paper describes. A sealed paired test of the one disputed cell (C. GRF, p=20, n=800) gave 1.014, interval [0.905, 1.124], under the paper's own design, compatible with the published 0.93; the helper gave 1.125, and the test does not show that the helper alone caused the earlier gap.
fastText at the commit of arXiv v3 reproduces Table 1 on all eight datasets, every accuracy within 0.3 points of the paper. The script's own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs.