Replication record · arXiv:2205.13504

Are Transformers Effective for Time Series Forecasting?

Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu · 2023 · doi:10.1609/aaai.v37i9.26317 · arXiv

Could not reproduce

With the authors code at the last commit before arXiv v3 (2022-08-17) and their ETTh1 script, Linear and NLinear reproduce Table 2 to within 0.0005 at every horizon, but DLinear does not: its test MSE is 0.384/0.443/0.447/0.503 against 0.375/0.405/0.439/0.472 reported for horizons 96 to 720. A post-hoc diagnostic shows the published DLinear numbers do reproduce, to the reported precision, with the authors earlier DLinear code and scripts from before their 2022-08-15 update, which the README at the paper date describes as giving slightly different DLinear results. The later change to how test batches are dropped moves ETTh1 results by at most 0.0012 and does not explain the gap. Only ETTh1 was tested.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Non-headline claims (stricter checks or registered controls) did not hold; the claims list names each. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 0 of 4 consistent
Other checks: 8 of 12 consistent
Deviations: 2 (0 material)
Integrity flags: 0
Logged compute: 0.34 h over 5 commands and 3 jobs
Commits the recorded results name: 6f6cce8: 16 results, named in the result's method text
Work tree when the attempt ended: cure-lab/LTSF-Linear@721ab5879ba7 in /work/LTSF-Linear (differs from the commit the results name)

Claims

On ETTh1 (multivariate, look-back 336), DLinear's test MSE at horizon 96 is 0.375.headline

Inconsistent

paper 0.375observed 0.384095

Paper 0.375 MSEObserved 0.384095 MSEAccepted if within ±0.005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96 · quote verified on p. 5 · equivalence check · observed/published 1.024 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, DLinear, ETTh1, horizon 96

Why this tolerance. Training uses the repository's fixed seed (2021), but the paper ran on GPU and this run uses CPU and a different PyTorch build, so small numerical differences are expected; values are reported to 3 decimals, and 0.005 is about 1% of the error level.

Event 46. Observed 0.384095, delta +0.009095, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, DLinear's test MAE at horizon 96 is 0.399.

Inconsistent

paper 0.399observed 0.4047

Paper 0.399 MAEObserved 0.4047 MAEAccepted if within ±0.005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96 · quote verified on p. 5 · equivalence check · observed/published 1.014 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE, DLinear, ETTh1, horizon 96

Event 47. Observed 0.4047, delta +0.0057, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, NLinear's test MSE at horizon 96 is 0.374.

Consistent

paper 0.374observed 0.373666

Paper 0.374 MSEObserved 0.373666 MSEAccepted if within ±0.005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96 · quote verified on p. 5 · equivalence check · observed/published 0.9991 · registered at event 13, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, NLinear, ETTh1, horizon 96

Event 54. Observed 0.373666, delta -0.0003336, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, Linear's test MSE at horizon 96 is 0.375.

Consistent

paper 0.375observed 0.374709

Paper 0.375 MSEObserved 0.374709 MSEAccepted if within ±0.005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96 · quote verified on p. 5 · equivalence check · observed/published 0.9992 · registered at event 14, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, Linear, ETTh1, horizon 96

Event 58. Observed 0.374709, delta -0.0002908, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1 (multivariate, look-back 336), DLinear's test MSE at horizon 192 is 0.405.headline

Inconsistent

paper 0.405observed 0.443401

Paper 0.405 MSEObserved 0.443401 MSEAccepted if within ±0.005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192 · quote verified on p. 5 · equivalence check · observed/published 1.095 · registered at event 15, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, DLinear, ETTh1, horizon 192

Event 48. Observed 0.443401, delta +0.0384, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, DLinear's test MAE at horizon 192 is 0.416.

Inconsistent

paper 0.416observed 0.449908

Paper 0.416 MAEObserved 0.449908 MAEAccepted if within ±0.005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192 · quote verified on p. 5 · equivalence check · observed/published 1.082 · registered at event 16, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE, DLinear, ETTh1, horizon 192

Event 49. Observed 0.449908, delta +0.03391, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, NLinear's test MSE at horizon 192 is 0.408.

Consistent

paper 0.408observed 0.407889

Paper 0.408 MSEObserved 0.407889 MSEAccepted if within ±0.005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192 · quote verified on p. 5 · equivalence check · observed/published 0.9997 · registered at event 17, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, NLinear, ETTh1, horizon 192

Event 55. Observed 0.407889, delta -0.000111, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, Linear's test MSE at horizon 192 is 0.418.

Consistent

paper 0.418observed 0.418304

Paper 0.418 MSEObserved 0.418304 MSEAccepted if within ±0.005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192 · quote verified on p. 5 · equivalence check · observed/published 1.001 · registered at event 18, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, Linear, ETTh1, horizon 192

Event 59. Observed 0.418304, delta +0.0003043, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1 (multivariate, look-back 336), DLinear's test MSE at horizon 336 is 0.439.headline

Inconsistent

paper 0.439observed 0.446904

Paper 0.439 MSEObserved 0.446904 MSEAccepted if within ±0.005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336 · quote verified on p. 5 · equivalence check · observed/published 1.018 · registered at event 19, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, DLinear, ETTh1, horizon 336

Event 50. Observed 0.446904, delta +0.007904, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, DLinear's test MAE at horizon 336 is 0.443.

Inconsistent

paper 0.443observed 0.44811

Paper 0.443 MAEObserved 0.44811 MAEAccepted if within ±0.005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336 · quote verified on p. 5 · equivalence check · observed/published 1.012 · registered at event 20, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE, DLinear, ETTh1, horizon 336

Event 51. Observed 0.44811, delta +0.00511, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, NLinear's test MSE at horizon 336 is 0.429.

Consistent

paper 0.429observed 0.428545

Paper 0.429 MSEObserved 0.428545 MSEAccepted if within ±0.005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336 · quote verified on p. 5 · equivalence check · observed/published 0.9989 · registered at event 21, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, NLinear, ETTh1, horizon 336

Event 56. Observed 0.428545, delta -0.0004546, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, Linear's test MSE at horizon 336 is 0.479.

Consistent

paper 0.479observed 0.479137

Paper 0.479 MSEObserved 0.479137 MSEAccepted if within ±0.005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336 · quote verified on p. 5 · equivalence check · observed/published 1 · registered at event 22, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, Linear, ETTh1, horizon 336

Event 60. Observed 0.479137, delta +0.0001375, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1 (multivariate, look-back 336), DLinear's test MSE at horizon 720 is 0.472.headline

Inconsistent

paper 0.472observed 0.503159

Paper 0.472 MSEObserved 0.503159 MSEAccepted if within ±0.005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720 · quote verified on p. 5 · equivalence check · observed/published 1.066 · registered at event 23, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, DLinear, ETTh1, horizon 720

Event 52. Observed 0.503159, delta +0.03116, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, DLinear's test MAE at horizon 720 is 0.490.

Inconsistent

paper 0.49observed 0.51379

Paper 0.49 MAEObserved 0.51379 MAEAccepted if within ±0.005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720 · quote verified on p. 5 · equivalence check · observed/published 1.049 · registered at event 24, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE, DLinear, ETTh1, horizon 720

Event 53. Observed 0.51379, delta +0.02379, inconsistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, NLinear's test MSE at horizon 720 is 0.440.

Consistent

paper 0.44observed 0.439957

Paper 0.44 MSEObserved 0.439957 MSEAccepted if within ±0.005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720 · quote verified on p. 5 · equivalence check · observed/published 0.9999 · registered at event 25, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, NLinear, ETTh1, horizon 720

Event 57. Observed 0.439957, delta -4.331e-05, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

On ETTh1, Linear's test MSE at horizon 720 is 0.624.

Consistent

paper 0.624observed 0.62391

Paper 0.624 MSEObserved 0.62391 MSEAccepted if within ±0.005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720 · quote verified on p. 5 · equivalence check · observed/published 0.9999 · registered at event 26, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE, Linear, ETTh1, horizon 720

Event 61. Observed 0.62391, delta -9.005e-05, consistent. Evidence events [38] (value matched by the legacy rule; not bound to an excerpt). Authors ETTh1 script (scripts/EXP-LongForecasting/Linear/etth1.sh) at commit 6f6cce8, the last commit on or before arXiv v3 (2022-08-17), model_name set per model; fixed seed 2021 from run_longExp.py; CPU, torch 1.9.0; test MSE/MAE as printed by the script.

Deviations from the paper's protocol

minor
Runs used CPU (4 shared vCPU) with Python 3.9, torch 1.9.0+cpu, numpy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2; the README sets up Python 3.6.9 and the paper ran on GPU.
CPU sandbox. torch 1.9.0 matches requirements.txt. Linear and NLinear reproduce to within 0.0005 in this environment, so the environment does not explain the DLinear gap. event 64
none
The sandbox was recreated with 2 GB of shared memory after the first run (event 31) died with a DataLoader bus error; the job was rerun unchanged (event 38).
Docker gives containers 64 MB of /dev/shm by default, too little for the DataLoader worker processes the code starts. event 65

Notes from the attempt

event 27
Declared before cloning: the PDF tested is arXiv v3 (17 Aug 2022). Primary results use the authors repository (github.com/cure-lab/LTSF-Linear, linked on p. 1) at the last commit dated on or before 2022-08-17, running the authors ETTh1 script settings for each model. As a sensitivity check only (not used for any recorded result), the same runs are repeated at the current HEAD, which includes the March 2023 change of the test loader from drop_last=True to drop_last=False. The automated repository finder missed this repository because the URL is split across a line break (LTSF- Linear); that is a harness defect, noted for fixing.
event 41
Checked at the paper-era commit 6f6cce8: README says Linear, NLinear and DLinear now use the same scripts and that some DLinear results are slightly different now. The README results image at that commit (pics/Mul-results.png, sha256 6584ed90e5f8...) shows ETTh1 DLinear MSE 0.375/0.405/0.439/0.472, identical to arXiv v3 Table 2, so the authors presented these numbers alongside the updated scripts. Next, a post-hoc diagnostic only (decided after seeing the results, so it cannot change any verdict): DLinear on ETTh1 with the scripts as they were before the 2022-08-15 script update.
event 62
Post-hoc diagnostic result (event 44, not used for any verdict): with the authors DLinear script and model as of commit 721ab58 (2022-08-06, before the 2022-08-15 update that set batch size 32 and learning rate 0.005 and changed models/DLinear.py), DLinear on ETTh1 gives MSE 0.3748/0.4055/0.4385/0.4719 and MAE 0.3995/0.4159/0.4427/0.4902 for horizons 96/192/336/720, matching arXiv v3 Table 2 (0.375/0.405/0.439/0.472 and 0.399/0.416/0.443/0.490) to the reported precision. The published DLinear ETTh1 numbers therefore reproduce from the authors earlier code, but not from the code at the paper version date, whose README says DLinear results are slightly different now. With the paper-date code, DLinear at horizon 192 (0.443) would be worse than FEDformer (0.420) in Table 2.
event 63
Sensitivity check at current HEAD 0c11366 (event 38), which includes the March 2023 change of the test loader to drop_last=False: all twelve ETTh1 results move by at most 0.0012 MSE (e.g. DLinear 0.3841/0.4435/0.4469/0.5042, NLinear 0.3737/0.4079/0.4291/0.4412, Linear 0.3747/0.4183/0.4792/0.6250). On ETTh1 the drop_last change does not explain the DLinear gap.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T01:55:59.648Z → 2026-10-18T02:22:09.550Z
Event log: 70 events, hash-chained; head 59d4e20753c01cf2601986473f8f3a82496fa2cea248822cb59ff48d65cfe4ec
Verify: python3 openrepro_verify.py arxiv-2205-13504-20261018T015559Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.