Replication record · arXiv:2205.13504

Are Transformers Effective for Time Series Forecasting?

Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu · 2023 · doi:10.1609/aaai.v37i9.26317 · arXiv

Reproduced only with earlier code

Prospective settling test of eight ETTh1 DLinear cells of Table 2 (MSE and MAE at horizons 96/192/336/720), sealed before any training: two author code versions x four horizons, seed 2021, one fit each, judged unrounded within 0.0005. The paper-date code 6f6cce8 (primary) misses all eight cells by 0.005 to 0.038; the earlier author code 721ab58 (2022-08-06, fixed in advance as the only allowed earlier version) matches all eight. Both sets of values are bit-identical to the earlier attempt arxiv-2205-13504-20261018T015559Z. This confirms reproduction with pre-update code and version-dependent numerical drift; it does not identify which code change caused the drift.

Plan. Sealed before scored execution; 3 outputs observed before sealing (all of it in the log).

Scored (headline): 0 of 8 consistent (8 more only with earlier code)
Deviations: 6 (0 material)
Integrity flags: 0
Logged compute: 0.06 h over 3 commands and 8 jobs
Primary code (declared before the first recorded result): github.com/cure-lab/LTSF-Linear at commit 6f6cce8bc8fa52704059a663181f100d94fc75bf (2022-08-15, the paper-date snapshot: latest code commit before arXiv v3 of 2022-08-17), DLinear with the settings of scripts/EXP-LongForecasting/Linear/etth1.sh (batch 32, learning rate 0.005, released random initialization).
Commits the recorded results name: cure-lab/LTSF-Linear@721ab5879ba7: 8 results, work-tree HEAD logged when the result was recorded; cure-lab/LTSF-Linear@6f6cce8bc8fa: 8 results, work-tree HEAD logged when the result was recorded; cure-lab/LTSF-Linear@721ab5879ba7 (earlier code): 8 results, work-tree HEAD logged when the result was recorded; cure-lab/LTSF-Linear@6f6cce8bc8fa (earlier code): 8 results, work-tree HEAD logged when the result was recorded
Work tree when the attempt ended: cure-lab/LTSF-Linear@721ab5879ba7 in /work/src-721ab58 (differs from the declared primary code)
Work tree when the attempt ended: cure-lab/LTSF-Linear@6f6cce8bc8fa in /work/src-6f6cce8

Protocol

Sealed with the plan: primary code is github.com/cure-lab/LTSF-Linear at commit 6f6cce8bc8fa52704059a663181f100d94fc75bf (2022-08-15, the paper-date snapshot: latest code commit before arXiv v3 of 2022-08-17), DLinear with the settings of scripts/EXP-LongForecasting/Linear/etth1.sh (batch 32, learning rate 0.005, released random initialization).; earlier versions of the authors' code may be tried as a fallback. Settling test per office/plans/2205.13504-settling-plan.md. The earlier code is fixed in advance to exactly one author-produced version, 721ab5879ba72fe67fab0846ba2933c3074a0c53 (2022-08-06, before the 2022-08-15 'Update scripts and models' commit), with the settings of scripts/EXP-LongForecasting/DLinear/etth1.sh (batch 8, runner-default learning rate 0.0001, 1/seq_len weight initialization). It was selected on existing evidence: a post-hoc diagnostic (event 44 of attempt arxiv-2205-13504-20261018T015559Z) matched all eight cells with it, while the paper-date code missed DLinear (that attempt's verdict stands as recorded). This attempt is a prospective confirmation; the plan calls 721ab58 the historical-paper hypothesis and 6f6cce8 the version-drift control. Both versions are run for all eight cells regardless of the primary results, and both are reported equally; no other commit may be used. The comparison is between version bundles and does not isolate learning rate from initialization.

Run policy (sealed at event 22, plan hash 59e7a2da58ab424b26f27eaccc311fbc35d71d1e778efa4df0b7911d6958de23): Exactly eight fits: commit {721ab5879ba72fe67fab0846ba2933c3074a0c53 (earlier code), 6f6cce8bc8fa52704059a663181f100d94fc75bf (primary)} x horizon {96,192,336,720}; seed 2021 set by the authors' runner (Python random, NumPy, torch), itr=1, one repetition per fit, one scored job per fit, each a fresh Python process, one at a time, in this order: fit-721ab58-h96, -h192, -h336, -h720, then fit-6f6cce8-h96, -h192, -h336, -h720; all eight run regardless of scores. Job command: bash /work/settle/run_fit.sh /work/settle/fit.py <commit7> <h>. Arguments = the commit's own ETTh1 script (721ab58: scripts/EXP-LongForecasting/DLinear/etth1.sh, batch 8, runner-default lr 0.0001; 6f6cce8: scripts/EXP-LongForecasting/Linear/etth1.sh with model DLinear, batch 32, lr 0.005), features M, seq_len 336, label_len 48, enc_in 7, individual False, kernel 25, 10 max epochs, patience 3, Adam, lradj type1, authors' early stopping/best checkpoint, test loader drop_last True; plus the single declared loader override --num_workers 0 for both (torch 1.9 draws the iterator base seed and the RandomSampler seed from the global generator identically for 0 and >0 workers; inspected, event 16). Cap: 45 wall-minutes from the start of the first fit, enforced by run_fit.sh with timeout; no pilots, extra epochs, learning-rate sweeps or other seeds. Bound inputs (SHA-256): plan 1ca775a18cee300e242d5a35f97e7857048c4f04eae61a87ed5c02917d709ec8; fit.py 0c9db45ec1dac400afd290f4dbdbd33436ec81968325cb34caae8edea3132cf1; run_fit.sh 455f5970b0141d1b4626569177dff040a58e9b6dc1ca3e4629ffacff98eb211a; manifest_data.py 52119b7395bbb4e3219d7c5c59681be793b7def4dd51176f135fe1968336c5a3; data-manifest.json 3b8d3bf8a099360c6c8df623b38b6e6a7c3730b86cf146eb68d1ee71e30ca58b; ETTh1.csv f18de3ad269cef59bb07b5438d79bb3042d3be49bdeecf01c1cd6d29695ee066; requirements.lock.txt 4839181cb5b75de9cdf676eba44fee2ee7164f730d20872e2171d173f12f9705; env-freeze.txt 1ff64db86b473d6769c9772f50598809f3641ba22f64d4daf3fc5a9003b46f94; env-records.sha256 c9e89a411df7f93cb95873d58b427194c75857895ecc1206901cd21c9a983f57; git trees 721ab58 27aebe2f91c036087007a296c5173bac6e7cff58, 6f6cce8 2583889e5111f2a1a8c59495084eb7e1b8050477 (clean); sandbox image sha256:f5d3ab8203a84c807ae92399a5a0d7fbf2f4c83f32838bb0e52739f80bbc9a05. Environment: Python 3.9.25, torch 1.9.0+cpu, NumPy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2 (full freeze bound), CPU only, OMP/MKL 4 threads, 4-vCPU droplet. Aggregation: No aggregation across runs. Each claim's observed value is the single fit's unrounded test MSE or MAE returned by the authors' utils/metrics.py over all retained test windows x horizon x 7 standardized variables (no inverse transform, no percent conversion, no per-batch averaging), re-scored by fit.py from the saved pred/true arrays and required to equal the runtime value exactly (otherwise the job prints no key lines and exits 3). Primary results come from the 6f6cce8 jobs; earlier-code results (code_ref 721ab5879ba72fe67fab0846ba2933c3074a0c53) from the 721ab58 jobs; both are recorded for all eight claims. Judgement per cell: |observed - paper| <= 0.0005 absolute, unrounded (half the last printed decimal). An exact |gap| = 0.0005 is reported as rounding-boundary/inconclusive in a note, whatever the harness's <= rule says. Per version, 'all selected cells match printed precision' is the conjunction of its eight comparisons; every mismatch is listed with its signed gap. No multiplicity adjustment and no standard error: this is a fixed-seed numerical reconstruction, not a sampling study. The plan's section-4 branch is selected mechanically from the two conjunctions (earlier all-eight / primary all-eight). Stopping rule: One pass over the eight-fit matrix in the fixed order; stop at completion or at the 45-minute cap, whichever comes first; never stop early because a result passes; no further runs after seeing results. A required fit that does not complete leaves its two cells evidence-incomplete; no averaging over survivors. Reruns: allowed, At most one retry per fit, only after a documented external interruption (host loss or transient I/O), with identical inputs, seed and command under a new job name; both invocations are retained and the first complete invocation is scored. Low performance, early stopping, OOM, NaNs and optimizer failure are not grounds for a rerun..

Claims

On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 96 is 0.375 (Table 2).headline

Consistent with earlier code

paper 0.375observed 0.384095

Paper 0.375 MSE (standardized)Observed 0.384095 MSE (standardized)Earlier code 0.374825 MSE (standardized)Accepted if within ±0.0005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.024 · registered at event 5, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 96, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 96 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Why this tolerance. Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.

Event 52. Observed 0.384095, delta +0.009095, inconsistent. Evidence: 00041-job-fit-6f6cce8-h96-stdout.txt (event 41), excerpt 'dlinear_etth1_h96_mse 0.3840945363044739'; verified. Job start event 40. Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 60 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.374825, delta -0.0001753, consistent. Evidence: 00024-job-fit-721ab58-h96-stdout.txt (event 24), excerpt 'dlinear_etth1_h96_mse 0.3748247027397156'; verified. Job start event 23. Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 96 is 0.399 (Table 2).headline

Consistent with earlier code

paper 0.399observed 0.4047

Paper 0.399 MAE (standardized)Observed 0.4047 MAE (standardized)Earlier code 0.399474 MAE (standardized)Accepted if within ±0.0005

96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399

Table 2, p. 5, ETTh1 row T=96, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.014 · registered at event 6, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 96, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 96 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 53. Observed 0.4047, delta +0.0057, inconsistent. Evidence: 00041-job-fit-6f6cce8-h96-stdout.txt (event 41), excerpt 'dlinear_etth1_h96_mae 0.4046996533870697'; verified. Job start event 40. Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 61 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.399474, delta +0.0004735, consistent. Evidence: 00024-job-fit-721ab58-h96-stdout.txt (event 24), excerpt 'dlinear_etth1_h96_mae 0.39947351813316345'; verified. Job start event 23. Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 192 is 0.405 (Table 2).headline

Consistent with earlier code

paper 0.405observed 0.443401

Paper 0.405 MSE (standardized)Observed 0.443401 MSE (standardized)Earlier code 0.405482 MSE (standardized)Accepted if within ±0.0005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.095 · registered at event 7, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 192, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 192 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 54. Observed 0.443401, delta +0.0384, inconsistent. Evidence: 00044-job-fit-6f6cce8-h192-stdout.txt (event 44), excerpt 'dlinear_etth1_h192_mse 0.44340065121650696'; verified. Job start event 43. Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 62 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.405482, delta +0.0004821, consistent. Evidence: 00027-job-fit-721ab58-h192-stdout.txt (event 27), excerpt 'dlinear_etth1_h192_mse 0.40548211336135864'; verified. Job start event 26. Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 192 is 0.416 (Table 2).headline

Consistent with earlier code

paper 0.416observed 0.449908

Paper 0.416 MAE (standardized)Observed 0.449908 MAE (standardized)Earlier code 0.415857 MAE (standardized)Accepted if within ±0.0005

192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416

Table 2, p. 5, ETTh1 row T=192, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.082 · registered at event 8, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 192, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 192 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 55. Observed 0.449908, delta +0.03391, inconsistent. Evidence: 00044-job-fit-6f6cce8-h192-stdout.txt (event 44), excerpt 'dlinear_etth1_h192_mae 0.44990822672843933'; verified. Job start event 43. Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 63 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.415857, delta -0.0001427, consistent. Evidence: 00027-job-fit-721ab58-h192-stdout.txt (event 27), excerpt 'dlinear_etth1_h192_mae 0.4158572554588318'; verified. Job start event 26. Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 336 is 0.439 (Table 2).headline

Consistent with earlier code

paper 0.439observed 0.446904

Paper 0.439 MSE (standardized)Observed 0.446904 MSE (standardized)Earlier code 0.438513 MSE (standardized)Accepted if within ±0.0005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.018 · registered at event 9, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 336, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 336 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 56. Observed 0.446904, delta +0.007904, inconsistent. Evidence: 00047-job-fit-6f6cce8-h336-stdout.txt (event 47), excerpt 'dlinear_etth1_h336_mse 0.44690412282943726'; verified. Job start event 46. Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 64 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.438513, delta -0.0004872, consistent. Evidence: 00030-job-fit-721ab58-h336-stdout.txt (event 30), excerpt 'dlinear_etth1_h336_mse 0.43851277232170105'; verified. Job start event 29. Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 336 is 0.443 (Table 2).headline

Consistent with earlier code

paper 0.443observed 0.44811

Paper 0.443 MAE (standardized)Observed 0.44811 MAE (standardized)Earlier code 0.442729 MAE (standardized)Accepted if within ±0.0005

336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443

Table 2, p. 5, ETTh1 row T=336, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.012 · registered at event 10, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 336, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 336 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 57. Observed 0.44811, delta +0.00511, inconsistent. Evidence: 00047-job-fit-6f6cce8-h336-stdout.txt (event 47), excerpt 'dlinear_etth1_h336_mae 0.4481101632118225'; verified. Job start event 46. Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 65 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.442729, delta -0.0002706, consistent. Evidence: 00030-job-fit-721ab58-h336-stdout.txt (event 30), excerpt 'dlinear_etth1_h336_mae 0.4427293539047241'; verified. Job start event 29. Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 720 is 0.472 (Table 2).headline

Consistent with earlier code

paper 0.472observed 0.503159

Paper 0.472 MSE (standardized)Observed 0.503159 MSE (standardized)Earlier code 0.471906 MSE (standardized)Accepted if within ±0.0005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.066 · registered at event 11, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 720, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 720 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 58. Observed 0.503159, delta +0.03116, inconsistent. Evidence: 00050-job-fit-6f6cce8-h720-stdout.txt (event 50), excerpt 'dlinear_etth1_h720_mse 0.5031593441963196'; verified. Job start event 49. Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 66 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.471906, delta -9.432e-05, consistent. Evidence: 00038-job-fit-721ab58-h720-stdout.txt (event 38), excerpt 'dlinear_etth1_h720_mse 0.4719056785106659'; verified. Job start event 37. Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 720 is 0.490 (Table 2).headline

Consistent with earlier code

paper 0.49observed 0.51379

Paper 0.49 MAE (standardized)Observed 0.51379 MAE (standardized)Earlier code 0.49024 MAE (standardized)Accepted if within ±0.0005

720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490

Table 2, p. 5, ETTh1 row T=720, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9 · quote verified on p. 5 · equivalence check · observed/published 1.049 · registered at event 12, sealed before scored execution

Method, tolerance rationale and evidence

Metric. Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 720, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 720 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.

Event 59. Observed 0.51379, delta +0.02379, inconsistent. Evidence: 00050-job-fit-6f6cce8-h720-stdout.txt (event 50), excerpt 'dlinear_etth1_h720_mae 0.5137903690338135'; verified. Job start event 49. Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.

Event 67 [earlier code 721ab5879ba72fe67fab0846ba2933c3074a0c53]. Observed 0.49024, delta +0.0002403, consistent. Evidence: 00038-job-fit-721ab58-h720-stdout.txt (event 38), excerpt 'dlinear_etth1_h720_mae 0.49024033546447754'; verified. Job start event 37. Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.

Deviations from the paper's protocol

minor
Full-precision pred.npy/true.npy arrays and selected checkpoints were written on the box but are not retained in the attempt after the box is destroyed; the log keeps their SHA-256 digests, the metrics.csv row, resolved args, train/validation history and the test-index manifest (all printed in each scored job's stdout).
Arrays are up to ~43 MB per fit and the engine stores only command output as artifacts; the plan asks for these artifacts per fit. event 70
minor
The plan asks for one externally anchored plan before execution. The sealed plan (protocol, claims, run/aggregation/stopping/rerun policy, with the SHA-256 of the plan document, drivers, input manifest, environment freeze, data file, source trees and image) is hash-chained in this log at the seal event, but was not anchored outside OpenRepro before the scored jobs ran.
No external anchoring step is available before execution in engine 0.2.1; anchors are written only for released attempts. event 71
minor
CPU execution (4-vCPU droplet, 4 OMP/MKL threads) with Python 3.9.25, torch 1.9.0+cpu, NumPy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2; the authors' README sets up Python 3.6.9 and the paper ran on GPU.
Environment pinned by the settling plan; CPU-only sandbox. torch 1.9.0 is the version in the repository's requirements.txt. event 72
none
Loader override --num_workers 0 for both versions (the runner default is 10); ./dataset/ is a symlink to /work/data in each fit's fresh output directory.
Declared resource override in the plan; torch 1.9 draws the iterator base seed and sampler seed from the global generator identically for 0 and >0 workers (inspected before the seal), and the values reproduce the earlier attempt's 10-worker values bit for bit. event 73
none
Engine roles: the plan calls 721ab58 the primary historical-paper hypothesis and 6f6cce8 the version-drift control; this attempt records 6f6cce8 as code_version primary and 721ab58 as earlier (code_ref 721ab58...), as the operator was instructed. The jobs ran in the sealed order (721ab58 first); results were recorded primary first because the engine records an earlier-code result only after the claim's primary result.
Engine semantics of primary/earlier code. event 74
none
The plan's tie rule (an exact |gap| = 0.0005 is inconclusive) cannot be expressed in the harness, whose 'within' rule counts a tie as consistent; it was checked by hand and no tie occurred.
Harness comparison vocabulary. event 75

Notes from the attempt

event 13
Disclosure: a setup-only attempt, arxiv-2205-13504-20261020T024800Z, was abandoned just before this one (no claims, no seal, no training, no inference, no results). It cloned the two commits, built the pinned environment, fetched ETTh1.csv and drafted the collector; it was abandoned because its setup ran before the claims were registered. The same setup is repeated here after registration. The driver files written below were drafted there; every one is logged here before the seal.
event 68
Sealed judgement, applied after all eight fits: 721ab58 (earlier code) matches all eight cells within 0.0005 (signed gaps: h96 MSE -0.000175, MAE +0.000474; h192 MSE +0.000482, MAE -0.000143; h336 MSE -0.000487, MAE -0.000271; h720 MSE -0.000094, MAE +0.000240). 6f6cce8 (paper-date, primary) misses all eight (h96 MSE +0.009095, MAE +0.005700; h192 MSE +0.038401, MAE +0.033908; h336 MSE +0.007904, MAE +0.005110; h720 MSE +0.031159, MAE +0.023790). No |gap| equals 0.0005 exactly, so no cell is a rounding-boundary tie. Plan section 4 branch, selected mechanically: 'Earlier all-eight match; newer has misses.'
event 69
All sixteen values are bit-identical to the earlier attempt arxiv-2205-13504-20261018T015559Z: the 721ab58 values equal its post-hoc diagnostic (event 44) and the 6f6cce8 values equal its paper-era run (event 38), on a different droplet, with --num_workers 0 instead of the default 10. Each fit's re-score from the saved arrays equalled the runtime metric exactly, and every retained test window matched the dataset's own windows (fit.py checks). All eight jobs exited 0; total fit wall time about 6 minutes, well inside the 45-minute cap; no retry was needed or used.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: yes
Sealed before scored execution: yes
Evidence bound to artifact excerpts: yes
Values extracted by a sealed rule: yes
Execution manifest for cited jobs: yes
Semantic re-derivation passed: yes
Observed before the seal: 3 shell; provenance captured
Engine: report built by repro 0.2.1; attempt created by repro 0.2.1

Redaction before release

Before release, the operator's local home-directory path was replaced with ~ in 4 places. Every event hash from event 20 on changed with it. The seal event's bytes changed (its chain hash); its plan hash did not. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now.

Rule: operator home directory prefix → ~. Reason: Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~ Recorded at event 80; the head the operator states for the log before the redaction is 0403d025c8d2151c4165ab7aba0b655d9e303373e7914e23dd1fe5d54908dca7 (not checkable from this log; a copy anchored elsewhere or the operator's kept original can check it).

Checked from the record: the redacted log's hash chain and artifact digests; re-derivation of every judgement from it; the redaction event's internal consistency (its position, replacement rows, recomputed digests and plan hashes, a single redaction or a second that names the first); and that the outcome, qualifiers, tallies, assurance flags, verdicts and observed values it records as before equal what the log derives now. Not checked: any other text (metric and protocol wording, units, paths), what files and commands meant when they ran, or anything about the log before the redaction, which is not in the record.

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-20T02:56:23.198Z → 2026-10-20T03:07:55.781Z
Event log: 81 events, hash-chained; head 6d8fe8bb8cfec5b3ae7018267b9386ac4915947961ec36fda06755c64b5de76b
Verify: python3 openrepro_verify.py arxiv-2205-13504-20261020T025623Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.