Replication record · arXiv:2012.07436

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang · 2021 · doi:10.1609/aaai.v35i12.17325 · arXiv

Partially reproduced

Informer (arXiv 2012.07436 v3) was run from the authors repository at commit c648cc6 (the last commit before the v3 date) with the exact pinned dependencies (Python 3.7, torch 1.8.0 CPU build, numpy 1.19.4, pandas 0.25.1, scikit-learn 0.21.3) on the public ETTh1 dataset, on a 4-vCPU CPU-only sandbox. Of six registered claims, the two for ETTh1 multivariate horizon 24 (Table 2) were tested with the authors own command (--itr 1 instead of 5, early stopping after epoch 5): observed MSE 0.4941 and MAE 0.5111 against the paper 0.577 and 0.549, both within the pre-registered at-most tolerance and better than reported, from a single unseeded run. The four other claims (univariate horizon 24, multivariate horizon 48) were not run: at the measured ~2-4 s per training step the seq_len 96 and seq_len 720 configurations did not fit the compute budget, so they remain untested. No code was modified.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Every scored check is a one-sided bound, not an equivalence to the published number. At least one scored claim was not tested for compute or memory reasons. Some claims were registered after command output had already been observed. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 1 of 3 consistent
Other checks: 1 of 1 consistent, 2 not tested
Deviations: 5 (1 material)
Integrity flags: 1
Logged compute: 3.11 h over 17 commands and 4 jobs
Primary code (declared before the first recorded result): https://github.com/zhouhaoyi/Informer2020 at commit c648cc6 (2021-03-25, "Fix typos"): the last commit on or before 2021-03-28, the date of arXiv v3 (the version tested, the AAAI 2021 camera-ready). Run with the authors own commands in scripts/ETTh1.sh at that commit.
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020-run1
Work tree when the attempt ended: zhouhaoyi/Informer2020@c648cc6c9fba in /work/Informer2020

Protocol

Declared before the first recorded result: primary code is https://github.com/zhouhaoyi/Informer2020 at commit c648cc6 (2021-03-25, "Fix typos"): the last commit on or before 2021-03-28, the date of arXiv v3 (the version tested, the AAAI 2021 camera-ready). Run with the authors own commands in scripts/ETTh1.sh at that commit.; earlier versions of the authors' code are not used. The arXiv v3 PDF is the paper under test and c648cc6 is the code the authors had published by that date. The repository history before v3 is short (first public commits were in Dec 2020) and the scripts/ETTh1.sh commands at c648cc6 already match the README reproduction commands, so there is no reason to fall back to earlier code; results come from one declared version only.

Claims

Informer reaches test MSE 0.098 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).headline

Not tested

paper 0.098

Paper 0.098 MSEObserved —Not tested: compute/memory limit (deviation, event 42)Accepted if at most 0.118

ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389

Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · quote verified on p. 5 · bound check · registered at event 14, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed (the authors script runs 5 iterations)

Why this tolerance. The paper reports no variance and the code fixes no random seed; run-to-run spread for Informer on ETTh1 is known to be sizeable. 0.02 is about 20% of the paper value; an MSE above 0.118 would also lose to several baselines in the same table (LogTrans 0.103, DeepAR 0.107, ARIMA 0.108, LSTMa 0.114), so it is the point where the paper claim (Informer wins this cell) stops holding.

Informer reaches test MAE 0.247 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).

Not tested

paper 0.247

Paper 0.247 MAEObserved —Not tested: compute/memory limit (deviation, event 42)Accepted if at most 0.297

ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389

Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column) · quote verified on p. 5 · bound check · registered at event 15, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed

Why this tolerance. About 20% of the paper value, matching the MSE tolerance rationale (no reported variance, no fixed seed). MAE above 0.297 would also lose to LogTrans (0.259), LSTMa (0.272), Prophet (0.275), DeepAR (0.280), ARIMA (0.284) in the same table.

Informer reaches test MSE 0.577 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).headline

Consistent

paper 0.577observed 0.4941

Paper 0.577 MSEObserved 0.4941 MSEAccepted if at most 0.692

ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754

Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · quote verified on p. 6 · bound check · observed/published 0.8563 · registered at event 16, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 multivariate (features M, all 7 variables), pred_len 24, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value (no reported variance, no fixed seed). An MSE above 0.692 would also lose to Informer-dagger (0.620), LSTMa (0.650) and LogTrans (0.686) in the same table, i.e. the paper claim for this cell stops holding.

Event 70. Observed 0.4941, delta -0.0829, consistent. Evidence events [65, 67] (value matched by the legacy rule; not bound to an excerpt). Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 48, label_len 48, pred_len 24, e_layers 2, d_layers 1, attn prob, factor 3) on CPU with --itr 1; early stopping triggered after epoch 5 (best validation loss at epoch 2); exp.test() printed mse:0.49413058161735535 on the 2848-window test split (normalized data), rounded here to 4 decimals (event 67 prints the saved metrics.npy as mse=0.4941). Single run (n=1), no seed fixed. Observed MSE is 0.083 below the paper value (better than reported).

Informer reaches test MAE 0.549 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).

Consistent

paper 0.549observed 0.5111

Paper 0.549 MAEObserved 0.5111 MAEAccepted if at most 0.659

ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754

Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column) · quote verified on p. 6 · bound check · observed/published 0.931 · registered at event 17, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 multivariate (features M), pred_len 24, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value, same rationale as the MSE claims. MAE above 0.659 would lose to Informer-dagger (0.577), LogTrans (0.604) and LSTMa (0.624) in the same table.

Event 71. Observed 0.5111, delta -0.0379, consistent. Evidence events [65, 67] (value matched by the legacy rule; not bound to an excerpt). Same run as etth1-multi-24-mse: exp.test() printed mae:0.51107257604599 on the test split, rounded to 4 decimals (event 67 prints the saved metrics.npy as mae=0.5111). Single run (n=1), no seed fixed. Observed MAE is 0.038 below the paper value (better than reported).

Informer reaches test MSE 0.685 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).headline

Not tested

paper 0.685

Paper 0.685 MSEObserved —Not tested: compute/memory limit (deviation, event 42)Accepted if at most 0.822

48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906

Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · quote verified on p. 6 · bound check · registered at event 18, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MSE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value (no reported variance, no fixed seed). MSE above 0.822 would also lose to Informer-dagger (0.692), LSTMa (0.702) and LogTrans (0.766) in the same table.

Informer reaches test MAE 0.625 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).

Not tested

paper 0.625

Paper 0.625 MAEObserved —Not tested: compute/memory limit (deviation, event 42)Accepted if at most 0.75

48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906

Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column) · quote verified on p. 6 · bound check · registered at event 19, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test MAE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs

Why this tolerance. About 20% of the paper value, same rationale as the MSE claims. MAE above 0.750 would lose to Informer-dagger (0.671) and LSTMa (0.675) in the same table.

Deviations from the paper's protocol

minor
All training and evaluation on CPU (4 vCPU DigitalOcean droplet, torch 1.8.0+cpu) instead of the single Nvidia V100 32GB GPU the paper states. The authors code falls back to CPU automatically when CUDA is unavailable (main_informer.py: args.use_gpu = torch.cuda.is_available() and args.use_gpu); no code was changed.
The sandbox is CPU-only. event 13
minor
ETTh1 multivariate horizon-24: 2 training iterations instead of the 5 in the authors script (--itr 5 -> --itr 1, launched twice as independent runs in two copies of the code). The observed value is the mean of these 2 runs.
On this CPU sandbox one training step takes ~1.9 s (job etth1-multi-24, event 31: 1.9755 s/iter), i.e. ~55 min per iteration, so 5 iterations (~4.5 h) exceed the compute budget. --itr is the scripts own option; no other hyperparameter was changed. event 40
none
OMP_NUM_THREADS=2 set for each of the two concurrent training runs (the sandbox has a 3.5-CPU quota).
A micro-benchmark (event 36) showed 2 threads at 2.1 s/iter vs 4 threads at 1.86 s/iter, so two concurrent 2-thread runs nearly double throughput within the budget. Thread count affects floating-point reduction order only. event 41
material
Claims etth1-uni-24-mse/mae (univariate, seq_len 720) and etth1-multi-48-mse/mae (seq_len 96) were registered but not run: they remain untested.
At the measured ~1.9 s per step for seq_len 48, the seq_len 96 run would take roughly 2 h per iteration and the seq_len 720 univariate run several hours; neither fits the budget after the horizon-24 multivariate run. Cutting --train_epochs was rejected because it would change the result rather than test the claim. event 42
minor
Observed values for etth1-multi-24-mse/mae come from a single training run (n=1), not the mean of the 5 iterations in the authors script; the second concurrent iteration was stopped after epoch 3 (see note).
Compute budget: per-step time on the shared-CPU droplet degraded to ~7.5 s with two concurrent runs. This supersedes the earlier 2-run deviation. event 60

Integrity flags

flag
At least one deviation from the paper's protocol is marked material.

Notes from the attempt

event 26
Configuration source: the attempt runs the authors commands from scripts/ETTh1.sh at c648cc6 verbatim. Note that the paper text (p.5, Experimental Details) describes Informer as a 3-layer stack plus a 1-layer stack encoder, a 2-layer decoder and 8 training epochs, whereas the released scripts use --e_layers 2 --d_layers 1 and the code default of 6 epochs with early-stopping patience 3 and batch size 32. The released commands are treated as the authors intended reproduction recipe; no hyperparameter was changed by this attempt. The code fixes no random seed and the scripts run --itr 5; the recorded observed value is the arithmetic mean over the iterations actually run, computed by a logged command from the per-run mse/mae printed by exp.test().
event 59
Iteration B (job etth1-multi-24-run1) was stopped after its 3rd epoch because the two concurrent runs had slowed to ~7.5 s/step (epoch 1: 758 s, epoch 2: 1125 s, epoch 3: 1943 s for 267 steps); finishing both would have taken roughly 2.5 more hours of droplet time. Its partial per-epoch Vali/Test losses are in its harvested output but are not used for any result. Iteration A (job etth1-multi-24-run0) continues alone, so the recorded observed values will come from a single training run (n=1 instead of the scripts 5).

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-19T13:21:34.534Z → 2026-10-19T15:19:39.005Z
Event log: 76 events, hash-chained; head 2dae23f4f4cbeb84c499e8c8f597c5eee0b2032119327d6dcc6d0fbef070080e
Verify: python3 openrepro_verify.py arxiv-2012-07436-20261019T132134Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.