Reproducibility report: Are Transformers Effective for Time Series Forecasting?

OpenRepro

Machine learning · 2023

Reproducibility report: Are Transformers Effective for Time Series Forecasting?

Report by OpenRepro, released 12 Nov 2026

Paper by Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu. Proceedings of the AAAI Conference on Artificial Intelligence, 2023. arXiv 2205.13504 · doi:10.1609/aaai.v37i9.26317

Also known as LTSF-Linear, cure-lab/LTSF-Linear.

Reproduced only with earlier code

8 of 8 (8 with earlier code) headline claims held

Does DLinear (Zeng et al., 2023) reproduce? Yes, with the authors' earlier code: when OpenRepro re-ran the authors' code, all 8 of 8 headline claims held within tolerances sealed before scored execution, 8 of them only with a declared earlier version of the code (checked 20 Oct 2026).

Checked: 20 Oct 2026
Released: 12 Nov 2026
Assurance: Sealed, bound and re-derived
Evidence: Full report · report.json
Data: JSON record · How to cite
Assistants: Ask via MCP

What was tested

This report is a settling test of the eight DLinear values in Table 2 for the ETTh1 dataset: test MSE and MAE at forecast horizons 96, 192, 336 and 720. Each value was produced by one fit with seed 2021, on two versions of the authors' code: the paper-date snapshot 6f6cce8 (15 August 2022) and the earlier commit 721ab58 (6 August 2022, before the authors' 15 August update of scripts and models). The earlier commit was declared as the only allowed earlier version before any training, and each value was judged unrounded within 0.0005, half the last printed digit.

What reproduces

With the 6 August 2022 code, all eight values match Table 2 at the printed precision; the largest gap is 0.000487. With the paper-date code, none match: the gaps run from +0.005 to +0.038. The outcome is therefore “Reproduced only with earlier code”. Both sets of values are bit-identical to the first attempt's, which ran them on a different machine.

In the first attempt, which this report supersedes, Linear and NLinear, the paper's two other linear models, reproduced Table 2 on ETTh1 at every horizon tested, each value within 0.0005 of the published number.

What the report does not say

The two versions run with different settings from their own scripts (batch size, learning rate and weight initialization), so the comparison is between those bundles. It does not identify which code change causes the drift. Only ETTh1 was tested, on CPU rather than the GPU the paper used.

Does DLinear reproduce?

Yes, with the authors' earlier code. All 8 headline claims held within tolerances sealed before scored execution, 8 of them only with a declared earlier version of the authors' code.

Looking for a replication of DLinear? This report is a computational reproduction: the authors' own published code was re-run and its results compared with the numbers in the paper. It does not set out to test the findings on new data, and it is not peer reviewed. What an attempt checks.

How far the re-run landed from the published number, in tolerances8 claims re-run, 8 headline

Held with earlier code

Reading the marks

Ring: where the earlier-code result landed, joined to the primary result
Chevron at the edge: beyond ±3 tolerances, with the actual multiple printed beside it (∞ when the tolerance was zero)

The data behind this chart

Claim	Published	Observed	Accepted if	Deviation ÷ tolerance	Result
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 96 is 0.375 (Table 2). (headline)	0.375	0.384095	within ±0.0005	+18.19	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 96 is 0.399 (Table 2). (headline)	0.399	0.4047	within ±0.0005	+11.40	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 192 is 0.405 (Table 2). (headline)	0.405	0.443401	within ±0.0005	+76.80	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 192 is 0.416 (Table 2). (headline)	0.416	0.449908	within ±0.0005	+67.82	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 336 is 0.439 (Table 2). (headline)	0.439	0.446904	within ±0.0005	+15.81	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 336 is 0.443 (Table 2). (headline)	0.443	0.44811	within ±0.0005	+10.22	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 720 is 0.472 (Table 2). (headline)	0.472	0.503159	within ±0.0005	+62.32	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 720 is 0.490 (Table 2). (headline)	0.49	0.51379	within ±0.0005	+47.58	Held with earlier code

What the attempt found

Prospective settling test of eight ETTh1 DLinear cells of Table 2 (MSE and MAE at horizons 96/192/336/720), sealed before any training: two author code versions x four horizons, seed 2021, one fit each, judged unrounded within 0.0005. The paper-date code 6f6cce8 (primary) misses all eight cells by 0.005 to 0.038; the earlier author code 721ab58 (2022-08-06, fixed in advance as the only allowed earlier version) matches all eight. Both sets of values are bit-identical to the earlier attempt arxiv-2205-13504-20261018T015559Z. This confirms reproduction with pre-update code and version-dependent numerical drift; it does not identify which code change caused the drift.

Assurance

Level: Sealed, bound and re-derived (what the levels mean)
Plan: Sealed before scored execution; 3 outputs observed before the seal (all of it in the log)
Evidence: Bound: each value was extracted from the cited run output by a rule fixed before the seal
Execution manifest: Yes: for each scored run the command, the Git checkout state (HEAD, status, diff hash) and the hashes of the driver files the command names were recorded when it started and when it ended. Data files and the environment are not bound by this
Report re-derived from the log: Passed: the engine re-derived the report from the log
Engine version: 0.2.1
Qualifiers: None

Headline claims

These decide the outcome. Each row gives the published value, the value from the re-run and the accepted-if rule registered before results were recorded. Earlier code: where a claim missed with the primary code, and the attempt had declared before any result was recorded that an earlier version of the authors' code could be tried, that result has its own column and is marked “held with earlier code”. Each row has its own ¶ link.

Claim	Published	Observed	Earlier code	Accepted if	Result
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 96 is 0.375 (Table 2).¶Table 2, p. 5, ETTh1 row T=96, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 5 · quote verified on p. 5 · equivalence check	0.375MSE (standardized)	0.384095	0.374825	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 96 is 0.399 (Table 2).¶Table 2, p. 5, ETTh1 row T=96, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 6 · quote verified on p. 5 · equivalence check	0.399MAE (standardized)	0.4047	0.399474	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 192 is 0.405 (Table 2).¶Table 2, p. 5, ETTh1 row T=192, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 7 · quote verified on p. 5 · equivalence check	0.405MSE (standardized)	0.443401	0.405482	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 192 is 0.416 (Table 2).¶Table 2, p. 5, ETTh1 row T=192, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 8 · quote verified on p. 5 · equivalence check	0.416MAE (standardized)	0.449908	0.415857	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 336 is 0.439 (Table 2).¶Table 2, p. 5, ETTh1 row T=336, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 9 · quote verified on p. 5 · equivalence check	0.439MSE (standardized)	0.446904	0.438513	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 336 is 0.443 (Table 2).¶Table 2, p. 5, ETTh1 row T=336, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 10 · quote verified on p. 5 · equivalence check	0.443MAE (standardized)	0.44811	0.442729	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 720 is 0.472 (Table 2).¶Table 2, p. 5, ETTh1 row T=720, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 11 · quote verified on p. 5 · equivalence check	0.472MSE (standardized)	0.503159	0.471906	within ±0.0005	Held with earlier code
On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 720 is 0.490 (Table 2).¶Table 2, p. 5, ETTh1 row T=720, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · registered at event 12 · quote verified on p. 5 · equivalence check	0.49MAE (standardized)	0.51379	0.49024	within ±0.0005	Held with earlier code

Deviations from the paper's setup

6 recorded: 3 minor, 3 none

minorFull-precision pred.npy/true.npy arrays and selected checkpoints were written on the box but are not retained in the attempt after the box is destroyed; the log keeps their SHA-256 digests, the metrics.csv row, resolved args, train/validation history and the test-index manifest (all printed in each scored job's stdout).
minorThe plan asks for one externally anchored plan before execution. The sealed plan (protocol, claims, run/aggregation/stopping/rerun policy, with the SHA-256 of the plan document, drivers, input manifest, environment freeze, data file, source trees and image) is hash-chained in this log at the seal event, but was not anchored outside OpenRepro before the scored jobs ran.
minorCPU execution (4-vCPU droplet, 4 OMP/MKL threads) with Python 3.9.25, torch 1.9.0+cpu, NumPy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2; the authors' README sets up Python 3.6.9 and the paper ran on GPU.
noneLoader override --num_workers 0 for both versions (the runner default is 10); ./dataset/ is a symlink to /work/data in each fit's fresh output directory.
noneEngine roles: the plan calls 721ab58 the primary historical-paper hypothesis and 6f6cce8 the version-drift control; this attempt records 6f6cce8 as code_version primary and 721ab58 as earlier (code_ref 721ab58...), as the operator was instructed. The jobs ran in the sealed order (721ab58 first); results were recorded primary first because the engine records an earlier-code result only after the claim's primary result.
noneThe plan's tie rule (an exact |gap| = 0.0005 is inconclusive) cannot be expressed in the harness, whose 'within' rule counts a tie as consistent; it was checked by hand and no tie occurred.

The run

Wall time: 11 min 33 s, 20 October 2026 to 20 October 2026
Job compute: 3 min 20 s across 8 jobs
Code the results name: cure-lab/LTSF-Linear at 721ab58
cure-lab/LTSF-Linear at 6f6cce8
Work tree at the end: cure-lab/LTSF-Linear at 721ab58 (differs from the declared primary code)
cure-lab/LTSF-Linear at 6f6cce8
Container image: repro-sandbox:0.2 sha256:f5d3ab8203a8…
Sandbox: 3.5 CPUs, 6 GB, no GPU
Run by: Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt: arxiv-2205-13504-20261020T025623Z
Log: 81 events, head 6d8fe8bb

Evidence

log seal, 81 hash-chained events, 2 earlier versions

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 81 hash-chained events

6d8fe8bb8cfec5b3ae7018267b9386ac4915947961ec36fda06755c64b5de76b

The head hash of the attempt's append-only log. The engine generated this report from that log. The verifier checks the log and its artifacts against this hash, and that the report names it. It does not check the report's contents. The evidence bundle for this attempt is not published yet.

Released 12 November 2026 · Settling update sent 20 October 2026 to three of the four authors (one address bounced). An author replied on 24 October 2026. We have requested permission to publish the reply; it remains private unless permission is granted.

Earlier versions of this report

The report above is the current version. Each earlier attempt stays public, unedited, with the reason it was replaced.

arxiv-2205-13504-20261020T024800Z, 20 October 2026: Abandoned before any result. Superseded by arxiv-2205-13504-20261020T025623Z: A setup-only attempt, abandoned before any claim was registered or any result recorded because its setup output preceded the claims; the settling test was restarted in the attempt that supersedes it, which registered every claim before any command output.
arxiv-2205-13504-20261018T015559Z, 18 October 2026: Could not reproduce. Superseded by arxiv-2205-13504-20261020T025623Z: Its earlier-code match was a diagnostic decided after the main results, so it could not count; the sealed settling re-run declared the authors' 6 August 2022 code as the only allowed earlier version before any training, and all eight DLinear values reproduced with it.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: Are Transformers Effective for Time Series Forecasting?. Reproduced only with earlier code; 8 of 8 headline claims held. Attempt arxiv-2205-13504-20261020T025623Z, log head 6d8fe8bb8cfec5b3, checked 20 Oct 2026. https://openrepro.org/papers/2205.13504/

APA style

OpenRepro. (2026). Reproducibility report: Are Transformers Effective for Time Series Forecasting? (Attempt arxiv-2205-13504-20261020T025623Z; log head 6d8fe8bb8cfec5b3). https://openrepro.org/papers/2205.13504/

BibTeX

@techreport{openrepro-2205.13504,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: Are Transformers Effective for Time Series Forecasting?},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/2205.13504/},
  note        = {Attempt arxiv-2205-13504-20261020T025623Z; log head 6d8fe8bb8cfec5b3; checked 20 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

References

1 paper

Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu (2023). Are Transformers Effective for Time Series Forecasting?. Proceedings of the AAAI Conference on Artificial Intelligence. arXiv 2205.13504 · doi:10.1609/aaai.v37i9.26317

Badge

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Copy the badge as Markdown or HTML

Markdown

[![OpenRepro: Reproduced only with earlier code. 8 of 8 (8 with earlier code) headline claims held.](https://openrepro.org/badge/2205.13504.svg)](https://openrepro.org/papers/2205.13504/)

HTML

<a href="https://openrepro.org/papers/2205.13504/"><img src="https://openrepro.org/badge/2205.13504.svg" alt="OpenRepro: Reproduced only with earlier code. 8 of 8 (8 with earlier code) headline claims held." width="245" height="20"></a>

Right of reply, and reporting an error

Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.

[1] Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu (2023). Are Transformers Effective for Time Series Forecasting?. Proceedings of the AAAI Conference on Artificial Intelligence. arXiv 2205.13504 · doi:10.1609/aaai.v37i9.26317