Machine learning · 2021
Reproducibility report: Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
2 of 3 headline claims held
bound checkpartly untestedlegacyunregistered extractionThe full answer and its qualifiers
Does Informer (Zhou et al., 2021) reproduce? Partly: when OpenRepro re-ran the authors' code, 2 of 3 headline claims held within tolerances registered before results were recorded; every headline check is a one-sided bound, not a match to the published value; some results were left untested for budget; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 20 Oct 2026).
- Every headline check is a one-sided bound, not a match to the published value.
- Some results were left untested for budget.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
Does Informer (Zhou et al., 2021) reproduce? Partly: when OpenRepro re-ran the authors' code, 2 of 3 headline claims held within tolerances registered before results were recorded (checked 20 Oct 2026).
- Checked
- Released
- Assurance
- Legacy
- Evidence
- Full report · report.json
- Data
- JSON record · How to cite
- Assistants
- Ask via MCP
Does Informer reproduce?
Partly. 2 of 3 headline claims held within tolerances registered before results were recorded; 1 was not tested.
- Every headline check is a one-sided bound, not a match to the published value.
- Some results were left untested for budget.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
- Held (large: headline claim; small: other claim)
Reading the marks
- Dashed circle in the right-hand lane: the claim was not tested
- Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| Informer reaches test MSE 0.098 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1). (headline) | 0.098 | — | at most 0.118 | — | Not tested |
| Informer reaches test MAE 0.247 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1). | 0.247 | — | at most 0.297 | — | Not tested |
| Informer reaches test MSE 0.577 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2). (headline) | 0.577 | 0.5046 | at most 0.692 | -0.63 | Held |
| Informer reaches test MAE 0.549 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2). | 0.549 | 0.5151 | at most 0.659 | -0.31 | Held |
| Informer reaches test MSE 0.685 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2). (headline) | 0.685 | 0.6021 | at most 0.822 | -0.61 | Held |
| Informer reaches test MAE 0.625 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2). | 0.625 | 0.57 | at most 0.75 | -0.44 | Held |
What the attempt found
Re-run of attempt arxiv-2012-07436-20261019T132134Z on a dedicated 4-vCPU CPU droplet, with the same pre-registered claims, protocol (authors code at c648cc6) and pinned environment (Python 3.7, torch 1.8.0+cpu, numpy 1.19.4, pandas 0.25.1, scikit-learn 0.21.3), running the authors scripts/ETTh1.sh commands with --itr 1 per iteration. ETTh1 multivariate horizon 24 (Table 2), mean of 4 unseeded runs: MSE 0.5046 and MAE 0.5151 vs paper 0.577 and 0.549 (earlier attempt, 1 run: 0.4941 / 0.5111), consistent and better than reported. ETTh1 multivariate horizon 48 (Table 2), untested in the earlier attempt, single run: MSE 0.6021 and MAE 0.5700 vs paper 0.685 and 0.625, consistent and better than reported. The univariate horizon-24 claims (Table 1, seq_len 720) remain untested: at 27.6 s per training step on this box one iteration needs at least ~7.6 h, beyond the budget. No code was modified.
Assurance
- Level
- Legacy (what the levels mean)
- Plan
- Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
- Evidence
- Legacy matching: each observed value was matched against the text of the logged output it cites
- Execution manifest
- No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
- Predates
- plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Coverage
- Some of the paper's results were left untested to stay within the compute budget; the summary names them.
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Informer reaches test MSE 0.098 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).¶Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · registered at event 7 · quote verified on p. 5 · bound check | 0.098MSE | — | at most 0.118 | Not tested |
| Informer reaches test MSE 0.577 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).¶Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · registered at event 9 · quote verified on p. 6 · bound check, observed 0.875× the published value | 0.577MSE | 0.5046 | at most 0.692 | Held |
| Informer reaches test MSE 0.685 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).¶Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · registered at event 11 · quote verified on p. 6 · bound check, observed 0.879× the published value | 0.685MSE | 0.6021 | at most 0.822 | Held |
Other claims checked
3 claims: 2 held, 1 not tested
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Informer reaches test MAE 0.247 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).¶Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · registered at event 8 · quote verified on p. 5 · bound check | 0.247MAE | — | at most 0.297 | Not tested |
| Informer reaches test MAE 0.549 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).¶Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · registered at event 10 · quote verified on p. 6 · bound check, observed 0.938× the published value | 0.549MAE | 0.5151 | at most 0.659 | Held |
| Informer reaches test MAE 0.625 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).¶Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · registered at event 12 · quote verified on p. 6 · bound check, observed 0.912× the published value | 0.625MAE | 0.57 | at most 0.75 | Held |
Deviations from the paper's setup
4 recorded: 1 material, 2 minor, 1 none
- minorAll training and evaluation on CPU (dedicated 4 vCPU DigitalOcean droplet s-4vcpu-8gb-intel, 3.5-CPU container quota, torch 1.8.0+cpu) instead of the single Nvidia V100 32GB GPU the paper states. The authors code falls back to CPU automatically when CUDA is unavailable (main_informer.py: args.use_gpu = torch.cuda.is_available() and args.use_gpu); no code was changed.
- noneOMP_NUM_THREADS=1 for each training run; the horizon-24 and horizon-48 multivariate runs execute concurrently in two separate clean copies of the code at c648cc6.
- materialClaims etth1-uni-24-mse/mae (univariate, features S, seq_len 720, label_len 168, pred_len 24) were registered but not run on this dedicated box either: they remain untested.
- minorFewer training iterations than the --itr 5 in the authors script: ETTh1 multivariate horizon 24 was run as 4 independent --itr 1 iterations (observed value = their mean), and multivariate horizon 48 as a single --itr 1 iteration (n=1). --itr is the scripts own option; no other hyperparameter was changed.
The run
- Wall time
- 5 h 26 min, 19 October 2026 to 20 October 2026
- Job compute
- 9 h 18 min across 6 jobs
- Work tree at the end
- zhouhaoyi/Informer2020 at c648cc6
zhouhaoyi/Informer2020 at c648cc6
zhouhaoyi/Informer2020 at c648cc6
zhouhaoyi/Informer2020 at c648cc6
zhouhaoyi/Informer2020 at c648cc6 - Container image
- repro-sandbox:0.2 sha256:0f62397ca0ed…
- Sandbox
- 3.5 CPUs, 6 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-2012-07436-20261019T202050Z
- Log
- 69 events, head cf28ac49
Evidence
log seal, 69 hash-chained events, 1 earlier version
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 69 hash-chained events
cf28ac49bd513cc6b2d4438a5e99cbbfd7f7210680dec7afd136f609a40513f6
Earlier versions of this report
How to cite this report
OpenRepro (2026). Reproducibility report: Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Partially reproduced; 2 of 3 headline claims held (every headline check is a one-sided bound, not a match to the published value; some results were left untested for budget; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-2012-07436-20261019T202050Z, log head cf28ac49bd513cc6, checked 20 Oct 2026. https://openrepro.org/papers/2012.07436/
APA style
OpenRepro. (2026). Reproducibility report: Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (Attempt arxiv-2012-07436-20261019T202050Z; log head cf28ac49bd513cc6). https://openrepro.org/papers/2012.07436/
BibTeX
@techreport{openrepro-2012.07436,
author = {{OpenRepro}},
title = {Reproducibility report: Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/2012.07436/},
note = {Attempt arxiv-2012-07436-20261019T202050Z; log head cf28ac49bd513cc6; checked 20 Oct 2026}
}
References
1 paper
- Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang (2021). Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence. arXiv 2012.07436 · doi:10.1609/aaai.v35i12.17325
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
Copy the badge as Markdown or HTML
[](https://openrepro.org/papers/2012.07436/)<a href="https://openrepro.org/papers/2012.07436/"><img src="https://openrepro.org/badge/2012.07436.svg" alt="OpenRepro: Partially reproduced. 2 of 3 headline claims held; every headline check is a one-sided bound, not a match to the published value; some results were left untested for budget; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="643" height="20"></a>Right of reply, and reporting an error
Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.