Replication record · arXiv:2012.07436

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang · 2021 · doi:10.1609/aaai.v35i12.17325 · arXiv

Partially reproduced

Re-run of attempt arxiv-2012-07436-20261019T132134Z on a dedicated 4-vCPU CPU droplet, with the same pre-registered claims, protocol (authors code at c648cc6) and pinned environment (Python 3.7, torch 1.8.0+cpu, numpy 1.19.4, pandas 0.25.1, scikit-learn 0.21.3), running the authors scripts/ETTh1.sh commands with --itr 1 per iteration. ETTh1 multivariate horizon 24 (Table 2), mean of 4 unseeded runs: MSE 0.5046 and MAE 0.5151 vs paper 0.577 and 0.549 (earlier attempt, 1 run: 0.4941 / 0.5111), consistent and better than reported. ETTh1 multivariate horizon 48 (Table 2), untested in the earlier attempt, single run: MSE 0.6021 and MAE 0.5700 vs paper 0.685 and 0.625, consistent and better than reported. The univariate horizon-24 claims (Table 1, seq_len 720) remain untested: at 27.6 s per training step on this box one iteration needs at least ~7.6 h, beyond the budget. No code was modified.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Every scored check is a one-sided bound, not an equivalence to the published number. At least one scored claim was not tested for compute or memory reasons. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 2 of 3 consistent
Other checks: 2 of 2 consistent, 1 not tested
Deviations: 4 (1 material)
Integrity flags: 1
Logged compute: 9.54 h over 8 commands and 6 jobs
Primary code (declared before the first recorded result): https://github.com/zhouhaoyi/Informer2020 at commit c648cc6 (2021-03-25, "Fix typos"): the last commit on or before 2021-03-28, the date of arXiv v3 (the version tested, the AAAI 2021 camera-ready). Run with the authors own commands in scripts/ETTh1.sh at that commit.
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020-run3
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020-m48
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020-run1
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020-run2

Protocol

Declared before the first recorded result: primary code is https://github.com/zhouhaoyi/Informer2020 at commit c648cc6 (2021-03-25, "Fix typos"): the last commit on or before 2021-03-28, the date of arXiv v3 (the version tested, the AAAI 2021 camera-ready). Run with the authors own commands in scripts/ETTh1.sh at that commit.; earlier versions of the authors' code are not used. The arXiv v3 PDF is the paper under test and c648cc6 is the code the authors had published by that date. The repository history before v3 is short (first public commits were in Dec 2020) and the scripts/ETTh1.sh commands at c648cc6 already match the README reproduction commands, so there is no reason to fall back to earlier code; results come from one declared version only.

Claims

Informer reaches test MSE 0.098 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).headline

Not tested

paper 0.098

Paper 0.098 MSEObserved —Not tested: compute/memory limit (deviation, event 36)Accepted if at most 0.118

ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389

Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · quote verified on p. 5 · bound check · registered at event 7, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed (the authors script runs 5 iterations)

Why this tolerance. The paper reports no variance and the code fixes no random seed; run-to-run spread for Informer on ETTh1 is known to be sizeable. 0.02 is about 20% of the paper value; an MSE above 0.118 would also lose to several baselines in the same table (LogTrans 0.103, DeepAR 0.107, ARIMA 0.108, LSTMa 0.114), so it is the point where the paper claim (Informer wins this cell) stops holding.

Informer reaches test MAE 0.247 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).

Not tested

paper 0.247

Paper 0.247 MAEObserved —Not tested: compute/memory limit (deviation, event 36)Accepted if at most 0.297

ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389

Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · quote verified on p. 5 · bound check · registered at event 8, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed

Why this tolerance. About 20% of the paper value, matching the MSE tolerance rationale (no reported variance, no fixed seed). MAE above 0.297 would also lose to LogTrans (0.259), LSTMa (0.272), Prophet (0.275), DeepAR (0.280), ARIMA (0.284) in the same table.

Informer reaches test MSE 0.577 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).headline

Consistent

paper 0.577observed 0.5046

Paper 0.577 MSEObserved 0.5046 MSEAccepted if at most 0.692

ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754

Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · quote verified on p. 6 · bound check · observed/published 0.8745 · registered at event 9, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 multivariate (features M, all 7 variables), pred_len 24, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value (no reported variance, no fixed seed). An MSE above 0.692 would also lose to Informer-dagger (0.620), LSTMa (0.650) and LogTrans (0.686) in the same table, i.e. the paper claim for this cell stops holding.

Event 61. Observed 0.5046, delta -0.0724, consistent. Evidence events [60, 31, 46, 53, 57] (value matched by the legacy rule; not bound to an excerpt). Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 48, label_len 48, pred_len 24, e_layers 2, d_layers 1, attn prob, factor 3) on CPU as 4 independent --itr 1 iterations (jobs etth1-multi-24-run0..run3, events 31, 46, 53, 57); each early-stopped after epoch 4-6. Observed value is the arithmetic mean of the four test MSEs printed by exp.test() (0.5298, 0.4705, 0.4906, 0.5273), computed by the logged command at event 60 and rounded to 4 decimals. n=4 (the authors script runs 5), no seed fixed; per-run range 0.4705-0.5298. Observed mean MSE is 0.072 below the paper value (better than reported). The earlier attempt observed 0.4941 from a single run, which lies inside this run-to-run range.

Informer reaches test MAE 0.549 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).

Consistent

paper 0.549observed 0.5151

Paper 0.549 MAEObserved 0.5151 MAEAccepted if at most 0.659

ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754

Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · quote verified on p. 6 · bound check · observed/published 0.9383 · registered at event 10, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 multivariate (features M), pred_len 24, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value, same rationale as the MSE claims. MAE above 0.659 would lose to Informer-dagger (0.577), LogTrans (0.604) and LSTMa (0.624) in the same table.

Event 62. Observed 0.5151, delta -0.0339, consistent. Evidence events [60, 31, 46, 53, 57] (value matched by the legacy rule; not bound to an excerpt). Same 4 iterations as etth1-multi-24-mse: arithmetic mean of the four test MAEs printed by exp.test() (0.5318, 0.5066, 0.4925, 0.5295), computed by the logged command at event 60 and rounded to 4 decimals. n=4, no seed fixed; per-run range 0.4925-0.5318. Observed mean MAE is 0.034 below the paper value (better than reported). The earlier attempt observed 0.5111 from a single run.

Informer reaches test MSE 0.685 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).headline

Consistent

paper 0.685observed 0.6021

Paper 0.685 MSEObserved 0.6021 MSEAccepted if at most 0.822

48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906

Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · quote verified on p. 6 · bound check · observed/published 0.879 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value (no reported variance, no fixed seed). MSE above 0.822 would also lose to Informer-dagger (0.692), LSTMa (0.702) and LogTrans (0.766) in the same table.

Event 44. Observed 0.6021, delta -0.0829, consistent. Evidence events [40, 42] (value matched by the legacy rule; not bound to an excerpt). Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 96, label_len 48, pred_len 48, e_layers 2, d_layers 1, attn prob, default factor 5) on CPU with --itr 1; early stopping triggered after epoch 5 (best validation loss at epoch 2); exp.test() printed mse:0.6021124720573425 on the test split (normalized data), rounded here to 4 decimals (event 42 prints the saved metrics.npy as mse=0.6021). Single run (n=1), no seed fixed. Observed MSE is 0.083 below the paper value (better than reported). This claim was untested in the earlier attempt.

Informer reaches test MAE 0.625 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).

Consistent

paper 0.625observed 0.57

Paper 0.625 MAEObserved 0.57 MAEAccepted if at most 0.75

48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906

Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · quote verified on p. 6 · bound check · observed/published 0.912 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value, same rationale as the MSE claims. MAE above 0.750 would lose to Informer-dagger (0.671) and LSTMa (0.675) in the same table.

Event 45. Observed 0.57, delta -0.055, consistent. Evidence events [40, 42] (value matched by the legacy rule; not bound to an excerpt). Same run as etth1-multi-48-mse: exp.test() printed mae:0.5700394511222839 on the test split, rounded to 4 decimals (event 42 prints the saved metrics.npy as mae=0.5700). Single run (n=1), no seed fixed. Observed MAE is 0.055 below the paper value (better than reported). This claim was untested in the earlier attempt.

Deviations from the paper's protocol

minor
All training and evaluation on CPU (dedicated 4 vCPU DigitalOcean droplet s-4vcpu-8gb-intel, 3.5-CPU container quota, torch 1.8.0+cpu) instead of the single Nvidia V100 32GB GPU the paper states. The authors code falls back to CPU automatically when CUDA is unavailable (main_informer.py: args.use_gpu = torch.cuda.is_available() and args.use_gpu); no code was changed.
The sandbox is CPU-only (same as the earlier attempt). event 23
none
OMP_NUM_THREADS=1 for each training run; the horizon-24 and horizon-48 multivariate runs execute concurrently in two separate clean copies of the code at c648cc6.
Micro-benchmark (event 19) on this box: 4 threads were slower than 2 for every config (m24 4.14 vs 2.48 s/step, m48 6.59 vs 4.22), and 1 thread gave 3.51 / 5.99 s/step, so two concurrent 1-thread runs give the most throughput. Thread count affects floating-point reduction order only. event 24
material
Claims etth1-uni-24-mse/mae (univariate, features S, seq_len 720, label_len 168, pred_len 24) were registered but not run on this dedicated box either: they remain untested.
Micro-benchmark on this box (event 19): one training step of the univariate seq_len 720 configuration takes 27.6 s at the best thread count (2 threads; 40.1 s at 1, 34.7 s at 4). With 247 steps per epoch that is about 1.9 h per epoch before validation, and the authors early stopping (patience 3) needs at least 4 epochs, i.e. at least ~7.6 h for one iteration, which exceeds the 7 h wall-time and $0.80 cloud budget of this re-run. Cutting --train_epochs or --seq_len was rejected because it would change the result rather than test the claim. event 36
minor
Fewer training iterations than the --itr 5 in the authors script: ETTh1 multivariate horizon 24 was run as 4 independent --itr 1 iterations (observed value = their mean), and multivariate horizon 48 as a single --itr 1 iteration (n=1). --itr is the scripts own option; no other hyperparameter was changed.
Compute budget (7 h wall time, $0.80): on this CPU box one horizon-24 iteration took 77-113 min and the horizon-48 iteration took 190 min (job events 31, 46, 53, 57, 40), so a second horizon-48 iteration or a fifth horizon-24 iteration could not finish in time. event 63

Integrity flags

flag
At least one deviation from the paper's protocol is marked material.

Notes from the attempt

event 5
Re-run of attempt arxiv-2012-07436-20261019T132134Z on a dedicated box, to test the claims that went untested there for compute reasons.
event 64
Configuration source, as in the earlier attempt: the authors commands from scripts/ETTh1.sh at c648cc6 are run verbatim apart from --itr. The paper text (p.5) describes a 3-layer plus 1-layer encoder, a 2-layer decoder and 8 epochs, whereas the released scripts use --e_layers 2 --d_layers 1 and the code defaults (6 epochs, early-stopping patience 3, batch size 32); the released commands are treated as the authors reproduction recipe. Comparison with the earlier attempt (arxiv-2012-07436-20261019T132134Z): multivariate horizon 24 there gave MSE 0.4941 / MAE 0.5111 from one run; here 4 runs gave 0.4705-0.5298 MSE (mean 0.5046) and 0.4925-0.5318 MAE (mean 0.5151), consistent with it. Multivariate horizon 48, untested there, is tested here. Univariate horizon 24 remains untested here too: this dedicated s-4vcpu-8gb-intel box was not faster per step than the earlier shared one (m24 2.5 s/step at best vs 1.9 s there).

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-19T20:20:50.671Z → 2026-10-20T01:47:06.748Z
Event log: 69 events, hash-chained; head cf28ac49bd513cc6b2d4438a5e99cbbfd7f7210680dec7afd136f609a40513f6
Verify: python3 openrepro_verify.py arxiv-2012-07436-20261019T202050Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1).
Right of reply: Authors may submit a response, which is published alongside this report unedited.