Machine learning · 2020
Reproducibility report: ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels
3 of 3 headline claims held
late registrationlegacyunregistered extractionThe full answer and its qualifiers
Does ROCKET (Dempster et al., 2020) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 3 of 3 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).
- Some claims were registered after command output had been observed.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
Does ROCKET (Dempster et al., 2020) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 3 of 3 headline claims held within tolerances registered before results were recorded (checked 19 Oct 2026).
- Checked
- Released
- Assurance
- Legacy
- Evidence
- Full report · report.json
- Data
- JSON record · How to cite
- Assistants
- Ask via MCP
Does ROCKET reproduce?
Yes, on its headline claims. 3 of 3 headline claims held within tolerances registered before results were recorded.
- Some claims were registered after command output had been observed.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
- Held (large: headline claim; small: other claim)
Reading the marks
- Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| Averaged over the 85 bake-off datasets, ROCKET (10,000 kernels, ridge classifier, 10 runs) attains the per-dataset test accuracies reported in Table 1, whose mean is 0.8510. (headline) | 0.851002 | 0.851669 | within ±0.005 | +0.13 | Held |
| ROCKET has the strictly best (lowest) mean rank of the eight classifiers in Table 1 over the 85 bake-off datasets. A tie for the best mean rank does NOT count as reproducing this claim. (headline) | 0 | 0 | at most 0 | +0.00 | Held |
| Per-dataset ROCKET accuracies in Table 1 are reproducible: the mean absolute difference between reproduced and reported accuracy across the 85 datasets is small. | 0 | 0.001975 | at most 0.01 | +0.20 | Held |
| Total compute time (training and test, single CPU core per dataset per run, averaged over runs) for ROCKET over all 85 bake-off datasets is 1 hour 50 minutes. (headline) | 110 | 40.54 | at most 165 | -1.26 | Held |
| Training ROCKET on ElectricDevices (8,926 training examples) takes about 6 minutes on a single core. | 6 | 2.35 | at most 9 | -1.22 | Held |
| Training ROCKET on HandOutlines (series length 2,709) takes 4 minutes 52 seconds on a single core. | 4.8667 | 1.81 | at most 7.3 | -1.26 | Held |
What the attempt found
Corrective re-run that supersedes arxiv-1910-13051-20261017T030206Z. The primary code version was declared first: angus924/rocket 4c24610, the first code commit, 21 hours after arXiv v1; the earlier attempt ran 2022 HEAD with the July 2020 rewrite. All claims were then registered, including the best-mean-rank tie rule taken from the paper before any result. Running the authors' 2019 bake-off code on all 85 UCR datasets (10 runs, single-core processes, dedicated box), mean accuracy is 0.8517 against 0.8510 in Table 1, with a mean absolute per-dataset difference of 0.20 points. Using the paper's fractional-rank method, ROCKET has the strictly best mean rank (571/170 vs 579/170 for TS-CHIEF), so the tie rule (a tie would count as not reproduced) was not triggered. The earlier attempt's tie appears to reflect run-to-run variation of unseeded random kernels on a margin the paper calls not significant. Single-core compute times (40.5 min total, 2.35 min ElectricDevices training, 1.81 min HandOutlines) are within the paper's figures on newer hardware.
Assurance
- Level
- Legacy (what the levels mean)
- Plan
- Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
- Evidence
- Legacy matching: each observed value was matched against the text of the logged output it cites
- Execution manifest
- No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
- Predates
- plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Averaged over the 85 bake-off datasets, ROCKET (10,000 kernels, ridge classifier, 10 runs) attains the per-dataset test accuracies reported in Table 1, whose mean is 0.8510.¶Table 1, pp. 25-26 (mean of the Rocket column, computed from Table 1 at event 19; the paper does not print the mean itself) · registered at event 25 · quote verified on p. 25 · equivalence check | 0.851002accuracy | 0.851669 | within ±0.005 | Held |
| ROCKET has the strictly best (lowest) mean rank of the eight classifiers in Table 1 over the 85 bake-off datasets. A tie for the best mean rank does NOT count as reproducing this claim.¶Section 4, p. 11 (also p. 11: 'ranks slightly ahead of HIVE-COTE, TS-CHIEF, and InceptionTime'); Figure 1, p. 1; ranking method p. 11 (Demsar 2006) and p. 18 (rank figures produced with code from Ismail Fawaz et al. 2019a) · registered at event 26 · quote verified on p. 11 · bound check | 0classifiers | 0 | at most 0 | Held |
| Total compute time (training and test, single CPU core per dataset per run, averaged over runs) for ROCKET over all 85 bake-off datasets is 1 hour 50 minutes.¶Section 1, p. 2 · registered at event 28 · quote verified on p. 2 · bound check, observed 0.369× the published value | 110minutes | 40.54 | at most 165 | Held |
Other claims checked
3 claims: 3 held
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Per-dataset ROCKET accuracies in Table 1 are reproducible: the mean absolute difference between reproduced and reported accuracy across the 85 datasets is small.¶Section 4.1.1, p. 11; Table 1 · registered at event 27 · quote verified on p. 11 · broad bound | 0accuracy | 0.001975 | at most 0.01 | Held |
| Training ROCKET on ElectricDevices (8,926 training examples) takes about 6 minutes on a single core.¶Section 1, p. 2 · registered at event 29 · quote verified on p. 2 · bound check, observed 0.392× the published value | 6minutes | 2.35 | at most 9 | Held |
| Training ROCKET on HandOutlines (series length 2,709) takes 4 minutes 52 seconds on a single core.¶Section 1, p. 2 · registered at event 30 · quote verified on p. 2 · bound check, observed 0.372× the published value | 4.8667minutes | 1.81 | at most 7.3 | Held |
Deviations from the paper's setup
5 recorded: 3 minor, 2 none
- minorLibrary versions differ from the paper era: Python 3.10, numpy 1.23.5, scipy 1.9.3, scikit-learn 1.1.3, pandas 1.5.3, numba 0.56.4 (event 11), identical to the earlier attempt. The repository pins no versions.
- minorData come from the UCR archive's own 2018 release (UCRArchive_2018.zip from cs.ucr.edu, sha256 a7163f6e..., event 14), tab-separated files renamed to the _TRAIN.txt/_TEST.txt names the 2019 script loads, instead of the txt files from timeseriesclassification.com that the script's comments name.
- noneEach dataset ran as a separate process of analysis/bakeoff_one.py, the authors' reproduce_experiments_bakeoff.py with only the dataset-name tuple replaced by one read from an environment variable (complete diff at event 33); kernels, transform, classifier, 10 runs, timers and output are the authors' text. The 85 per-dataset CSVs were merged by analysis/calc.py.
- minorTimings were measured on a 4-vCPU AMD EPYC 9555P droplet (2024-era CPU) with 3 single-threaded processes running concurrently under a 3.5-CPU quota, rather than on the paper's Xeon E5-2680 v3 / Gold 6150 cluster cores.
- noneThe best-mean-rank claim compares the reproduced ROCKET column with the other seven classifiers' published Table 1 accuracies (not re-run), with reproduced ROCKET accuracies rounded half-up to 4 decimals before ranking.
The run
- Wall time
- 2 h 29 min, 19 October 2026 to 19 October 2026
- Job compute
- 2 h 16 min across 1 job
- Work tree at the end
- angus924/rocket at 4c24610
- Container image
- repro-sandbox:0.2 sha256:0f6b339a04af…
- Sandbox
- 3.5 CPUs, 10 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-1910-13051-20261019T203424Z
- Log
- 63 events, head 68c626f8
Evidence
log seal, 63 hash-chained events, 1 earlier version
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 63 hash-chained events
68c626f8d6f80233c3b267a4dde2358204dbe838f3ad78eb1893343a1be27175
Earlier versions of this report
How to cite this report
OpenRepro (2026). Reproducibility report: ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Reproduced; 3 of 3 headline claims held (some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1910-13051-20261019T203424Z, log head 68c626f8d6f80233, checked 19 Oct 2026. https://openrepro.org/papers/1910.13051/
APA style
OpenRepro. (2026). Reproducibility report: ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels (Attempt arxiv-1910-13051-20261019T203424Z; log head 68c626f8d6f80233). https://openrepro.org/papers/1910.13051/
BibTeX
@techreport{openrepro-1910.13051,
author = {{OpenRepro}},
title = {Reproducibility report: ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/1910.13051/},
note = {Attempt arxiv-1910-13051-20261019T203424Z; log head 68c626f8d6f80233; checked 19 Oct 2026}
}
References
1 paper
- Angus Dempster, François Petitjean, Geoffrey I. Webb (2020). ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery. arXiv 1910.13051 · doi:10.1007/s10618-020-00701-z
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
Copy the badge as Markdown or HTML
[](https://openrepro.org/papers/1910.13051/)<a href="https://openrepro.org/papers/1910.13051/"><img src="https://openrepro.org/badge/1910.13051.svg" alt="OpenRepro: Reproduced. 3 of 3 headline claims held; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="517" height="20"></a>Right of reply, and reporting an error
Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.