{ "schema": "repro.report/v3", "engine_version": "0.2.1", "attempt_id": "arxiv-2012-07436-20261019T202050Z", "legacy": true, "paper": { "arxiv_id": "2012.07436", "title": "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting", "authors": [ "Haoyi Zhou", "Shanghang Zhang", "Jieqi Peng", "Shuai Zhang", "Jianxin Li", "Hui Xiong", "Wancai Zhang" ], "year": 2021, "doi": "10.1609/aaai.v35i12.17325", "venue": "Proceedings of the AAAI Conference on Artificial Intelligence", "openalex_id": "https://openalex.org/W3177318507" }, "code": [ { "commit": "c648cc6c9fba4b42b35b891e1837c98a5c0f17f2", "full_name": "zhouhaoyi/Informer2020", "modified_files": 0, "path": "/work/Informer2020-run3", "url": "https://github.com/zhouhaoyi/Informer2020", "matches_primary": true }, { "commit": "c648cc6c9fba4b42b35b891e1837c98a5c0f17f2", "full_name": "zhouhaoyi/Informer2020", "modified_files": 0, "path": "/work/Informer2020-m48", "url": "https://github.com/zhouhaoyi/Informer2020", "matches_primary": true }, { "commit": "c648cc6c9fba4b42b35b891e1837c98a5c0f17f2", "full_name": "zhouhaoyi/Informer2020", "modified_files": 0, "path": "/work/Informer2020-run1", "url": "https://github.com/zhouhaoyi/Informer2020", "matches_primary": true }, { "commit": "c648cc6c9fba4b42b35b891e1837c98a5c0f17f2", "full_name": "zhouhaoyi/Informer2020", "modified_files": 0, "path": "/work/Informer2020", "url": "https://github.com/zhouhaoyi/Informer2020", "matches_primary": true }, { "commit": "c648cc6c9fba4b42b35b891e1837c98a5c0f17f2", "full_name": "zhouhaoyi/Informer2020", "modified_files": 0, "path": "/work/Informer2020-run2", "url": "https://github.com/zhouhaoyi/Informer2020", "matches_primary": true } ], "code_used": [], "primary_code_source": "protocol", "outcome": "partially_reproduced", "outcome_label": "Partially reproduced", "qualifiers": [ "bound_checks_only", "untested_for_budget", "plan_not_sealed", "evidence_not_bound", "agent_selected_extraction" ], "qualifier_details": [], "evidence_gate_failed": [], "assurance": { "plan_sealed": false, "sealed_before_scored_execution": false, "evidence_bound": false, "extraction_sealed": false, "execution_manifest": false, "semantic_rederivation_passed": true, "predates": [ "plan_seal", "evidence_binding", "execution_manifest", "paper_text_hash", "scored_job_flag", "retry_reasons", "extraction_rule" ], "engine_version": "0.2.1", "attempt_engine_version": "0.1.0", "setup_outputs_before_seal": null, "observations_before_seal": null, "observation_provenance": null, "plan_hash": null, "paper_text_sha256_ok": null }, "plan": null, "not_attemptable_reason": null, "summary": "Re-run of attempt arxiv-2012-07436-20261019T132134Z on a dedicated 4-vCPU CPU droplet, with the same pre-registered claims, protocol (authors code at c648cc6) and pinned environment (Python 3.7, torch 1.8.0+cpu, numpy 1.19.4, pandas 0.25.1, scikit-learn 0.21.3), running the authors scripts/ETTh1.sh commands with --itr 1 per iteration. ETTh1 multivariate horizon 24 (Table 2), mean of 4 unseeded runs: MSE 0.5046 and MAE 0.5151 vs paper 0.577 and 0.549 (earlier attempt, 1 run: 0.4941 / 0.5111), consistent and better than reported. ETTh1 multivariate horizon 48 (Table 2), untested in the earlier attempt, single run: MSE 0.6021 and MAE 0.5700 vs paper 0.685 and 0.625, consistent and better than reported. The univariate horizon-24 claims (Table 1, seq_len 720) remain untested: at 27.6 s per training step on this box one iteration needs at least ~7.6 h, beyond the budget. No code was modified.", "claims": [ { "claim_id": "etth1-uni-24-mse", "comparison": "at_most", "headline": true, "location": "Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column)", "metric": "test MSE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed (the authors script runs 5 iterations)", "paper_value": 0.098, "quote": "ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389", "statement": "Informer reaches test MSE 0.098 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).", "tolerance": 0.02, "tolerance_rationale": "The paper reports no variance and the code fixes no random seed; run-to-run spread for Informer on ETTh1 is known to be sizeable. 0.02 is about 20% of the paper value; an MSE above 0.118 would also lose to several baselines in the same table (LogTrans 0.103, DeepAR 0.107, ARIMA 0.108, LSTMa 0.114), so it is the point where the paper claim (Informer wins this cell) stops holding.", "unit": "MSE", "registered_event": 7, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.20408163265306123, "extraction_registered": false, "results": [], "withdrawn_after_seal": false, "final": null, "earlier_final": null, "verdict": "untested", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true, "untested_reason": { "label": "not tested: compute/memory limit", "records": [ { "kind": "deviation", "event": 36 } ] } }, { "claim_id": "etth1-uni-24-mae", "comparison": "at_most", "headline": false, "location": "Table 1, p. 5 (Univariate, ETTh1, horizon 24, Informer column)", "metric": "test MAE on ETTh1 univariate (features S, target OT), pred_len 24, normalized data, mean over the runs executed", "paper_value": 0.247, "quote": "ETTh1 24 0.098 0.247 0.092 0.246 0.103 0.259 0.222 0.389", "statement": "Informer reaches test MAE 0.247 on ETTh1 univariate forecasting with prediction horizon 24 (Table 1).", "tolerance": 0.05, "tolerance_rationale": "About 20% of the paper value, matching the MSE tolerance rationale (no reported variance, no fixed seed). MAE above 0.297 would also lose to LogTrans (0.259), LSTMa (0.272), Prophet (0.275), DeepAR (0.280), ARIMA (0.284) in the same table.", "unit": "MAE", "registered_event": 8, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.20242914979757087, "extraction_registered": false, "results": [], "withdrawn_after_seal": false, "final": null, "earlier_final": null, "verdict": "untested", "result_revisions": 0, "ratio_to_published": null, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true, "untested_reason": { "label": "not tested: compute/memory limit", "records": [ { "kind": "deviation", "event": 36 } ] } }, { "claim_id": "etth1-multi-24-mse", "comparison": "at_most", "headline": true, "location": "Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column)", "metric": "test MSE on ETTh1 multivariate (features M, all 7 variables), pred_len 24, normalized data, mean over the 5 iterations the authors script runs", "paper_value": 0.577, "quote": "ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754", "statement": "Informer reaches test MSE 0.577 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).", "tolerance": 0.115, "tolerance_rationale": "About 20% of the paper value (no reported variance, no fixed seed). An MSE above 0.692 would also lose to Informer-dagger (0.620), LSTMa (0.650) and LogTrans (0.686) in the same table, i.e. the paper claim for this cell stops holding.", "unit": "MSE", "registered_event": 9, "quote_verified": true, "quote_page": 6, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.19930675909878684, "extraction_registered": false, "results": [ { "event": 61, "observed_value": 0.5046, "delta": -0.07239999999999991, "consistent": true, "evidence_events": [ 60, 31, 46, 53, 57 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 60", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 48, label_len 48, pred_len 24, e_layers 2, d_layers 1, attn prob, factor 3) on CPU as 4 independent --itr 1 iterations (jobs etth1-multi-24-run0..run3, events 31, 46, 53, 57); each early-stopped after epoch 4-6. Observed value is the arithmetic mean of the four test MSEs printed by exp.test() (0.5298, 0.4705, 0.4906, 0.5273), computed by the logged command at event 60 and rounded to 4 decimals.", "notes": "n=4 (the authors script runs 5), no seed fixed; per-run range 0.4705-0.5298. Observed mean MSE is 0.072 below the paper value (better than reported). The earlier attempt observed 0.4941 from a single run, which lies inside this run-to-run range.", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 61, "observed_value": 0.5046, "delta": -0.07239999999999991, "consistent": true, "evidence_events": [ 60, 31, 46, 53, 57 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 60", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 48, label_len 48, pred_len 24, e_layers 2, d_layers 1, attn prob, factor 3) on CPU as 4 independent --itr 1 iterations (jobs etth1-multi-24-run0..run3, events 31, 46, 53, 57); each early-stopped after epoch 4-6. Observed value is the arithmetic mean of the four test MSEs printed by exp.test() (0.5298, 0.4705, 0.4906, 0.5273), computed by the logged command at event 60 and rounded to 4 decimals.", "notes": "n=4 (the authors script runs 5), no seed fixed; per-run range 0.4705-0.5298. Observed mean MSE is 0.072 below the paper value (better than reported). The earlier attempt observed 0.4941 from a single run, which lies inside this run-to-run range.", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.8745233968804161, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "etth1-multi-24-mae", "comparison": "at_most", "headline": false, "location": "Table 2, p. 6 (Multivariate, ETTh1, horizon 24, Informer column)", "metric": "test MAE on ETTh1 multivariate (features M), pred_len 24, normalized data, mean over the 5 iterations the authors script runs", "paper_value": 0.549, "quote": "ETTh1 24 0.577 0.549 0.620 0.577 0.686 0.604 0.991 0.754", "statement": "Informer reaches test MAE 0.549 on ETTh1 multivariate forecasting with prediction horizon 24 (Table 2).", "tolerance": 0.11, "tolerance_rationale": "About 20% of the paper value, same rationale as the MSE claims. MAE above 0.659 would lose to Informer-dagger (0.577), LogTrans (0.604) and LSTMa (0.624) in the same table.", "unit": "MAE", "registered_event": 10, "quote_verified": true, "quote_page": 6, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.20036429872495445, "extraction_registered": false, "results": [ { "event": 62, "observed_value": 0.5151, "delta": -0.03390000000000004, "consistent": true, "evidence_events": [ 60, 31, 46, 53, 57 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 60", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Same 4 iterations as etth1-multi-24-mse: arithmetic mean of the four test MAEs printed by exp.test() (0.5318, 0.5066, 0.4925, 0.5295), computed by the logged command at event 60 and rounded to 4 decimals.", "notes": "n=4, no seed fixed; per-run range 0.4925-0.5318. Observed mean MAE is 0.034 below the paper value (better than reported). The earlier attempt observed 0.5111 from a single run.", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 62, "observed_value": 0.5151, "delta": -0.03390000000000004, "consistent": true, "evidence_events": [ 60, 31, 46, 53, 57 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 60", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Same 4 iterations as etth1-multi-24-mse: arithmetic mean of the four test MAEs printed by exp.test() (0.5318, 0.5066, 0.4925, 0.5295), computed by the logged command at event 60 and rounded to 4 decimals.", "notes": "n=4, no seed fixed; per-run range 0.4925-0.5318. Observed mean MAE is 0.034 below the paper value (better than reported). The earlier attempt observed 0.5111 from a single run.", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9382513661202185, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "etth1-multi-48-mse", "comparison": "at_most", "headline": true, "location": "Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column)", "metric": "test MSE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs", "paper_value": 0.685, "quote": "48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906", "statement": "Informer reaches test MSE 0.685 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).", "tolerance": 0.137, "tolerance_rationale": "About 20% of the paper value (no reported variance, no fixed seed). MSE above 0.822 would also lose to Informer-dagger (0.692), LSTMa (0.702) and LogTrans (0.766) in the same table.", "unit": "MSE", "registered_event": 11, "quote_verified": true, "quote_page": 6, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.2, "extraction_registered": false, "results": [ { "event": 44, "observed_value": 0.6021, "delta": -0.08290000000000008, "consistent": true, "evidence_events": [ 40, 42 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 40, 42", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 96, label_len 48, pred_len 48, e_layers 2, d_layers 1, attn prob, default factor 5) on CPU with --itr 1; early stopping triggered after epoch 5 (best validation loss at epoch 2); exp.test() printed mse:0.6021124720573425 on the test split (normalized data), rounded here to 4 decimals (event 42 prints the saved metrics.npy as mse=0.6021).", "notes": "Single run (n=1), no seed fixed. Observed MSE is 0.083 below the paper value (better than reported). This claim was untested in the earlier attempt.", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 44, "observed_value": 0.6021, "delta": -0.08290000000000008, "consistent": true, "evidence_events": [ 40, 42 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 40, 42", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Ran the authors command from scripts/ETTh1.sh at c648cc6 (features M, seq_len 96, label_len 48, pred_len 48, e_layers 2, d_layers 1, attn prob, default factor 5) on CPU with --itr 1; early stopping triggered after epoch 5 (best validation loss at epoch 2); exp.test() printed mse:0.6021124720573425 on the test split (normalized data), rounded here to 4 decimals (event 42 prints the saved metrics.npy as mse=0.6021).", "notes": "Single run (n=1), no seed fixed. Observed MSE is 0.083 below the paper value (better than reported). This claim was untested in the earlier attempt.", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.8789781021897809, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "etth1-multi-48-mae", "comparison": "at_most", "headline": false, "location": "Table 2, p. 6 (Multivariate, ETTh1, horizon 48, Informer column)", "metric": "test MAE on ETTh1 multivariate (features M), pred_len 48, normalized data, mean over the 5 iterations the authors script runs", "paper_value": 0.625, "quote": "48 0.685 0.625 0.692 0.671 0.766 0.757 1.313 0.906", "statement": "Informer reaches test MAE 0.625 on ETTh1 multivariate forecasting with prediction horizon 48 (Table 2).", "tolerance": 0.125, "tolerance_rationale": "About 20% of the paper value, same rationale as the MSE claims. MAE above 0.750 would lose to Informer-dagger (0.671) and LSTMa (0.675) in the same table.", "unit": "MAE", "registered_event": 12, "quote_verified": true, "quote_page": 6, "preregistered": true, "registered_after_output": false, "sealed": false, "check_kind": "bound", "broad_bound": false, "tolerance_ratio": 0.2, "extraction_registered": false, "results": [ { "event": 45, "observed_value": 0.57, "delta": -0.05500000000000005, "consistent": true, "evidence_events": [ 40, 42 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 40, 42", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Same run as etth1-multi-48-mse: exp.test() printed mae:0.5700394511222839 on the test split, rounded to 4 decimals (event 42 prints the saved metrics.npy as mae=0.5700).", "notes": "Single run (n=1), no seed fixed. Observed MAE is 0.055 below the paper value (better than reported). This claim was untested in the earlier attempt.", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 45, "observed_value": 0.57, "delta": -0.05500000000000005, "consistent": true, "evidence_events": [ 40, 42 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 40, 42", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": null, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Same run as etth1-multi-48-mse: exp.test() printed mae:0.5700394511222839 on the test split, rounded to 4 decimals (event 42 prints the saved metrics.npy as mae=0.5700).", "notes": "Single run (n=1), no seed fixed. Observed MAE is 0.055 below the paper value (better than reported). This claim was untested in the earlier attempt.", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9119999999999999, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true } ], "counts": { "claims_scored": 3, "consistent": 2, "consistent_earlier_code": 0, "inconsistent": 0, "untested": 1, "withdrawn_after_seal": 0, "other": { "claims": 3, "consistent": 2, "inconsistent": 0, "untested": 1 } }, "withdrawn": [], "protocol": { "earlier_code": "not_allowed", "primary_code": "https://github.com/zhouhaoyi/Informer2020 at commit c648cc6 (2021-03-25, \"Fix typos\"): the last commit on or before 2021-03-28, the date of arXiv v3 (the version tested, the AAAI 2021 camera-ready). Run with the authors own commands in scripts/ETTh1.sh at that commit.", "rationale": "The arXiv v3 PDF is the paper under test and c648cc6 is the code the authors had published by that date. The repository history before v3 is short (first public commits were in Dec 2020) and the scripts/ETTh1.sh commands at c648cc6 already match the README reproduction commands, so there is no reason to fall back to earlier code; results come from one declared version only." }, "deviations": [ { "event": 23, "description": "All training and evaluation on CPU (dedicated 4 vCPU DigitalOcean droplet s-4vcpu-8gb-intel, 3.5-CPU container quota, torch 1.8.0+cpu) instead of the single Nvidia V100 32GB GPU the paper states. The authors code falls back to CPU automatically when CUDA is unavailable (main_informer.py: args.use_gpu = torch.cuda.is_available() and args.use_gpu); no code was changed.", "impact": "minor", "reason": "The sandbox is CPU-only (same as the earlier attempt)." }, { "event": 24, "description": "OMP_NUM_THREADS=1 for each training run; the horizon-24 and horizon-48 multivariate runs execute concurrently in two separate clean copies of the code at c648cc6.", "impact": "none", "reason": "Micro-benchmark (event 19) on this box: 4 threads were slower than 2 for every config (m24 4.14 vs 2.48 s/step, m48 6.59 vs 4.22), and 1 thread gave 3.51 / 5.99 s/step, so two concurrent 1-thread runs give the most throughput. Thread count affects floating-point reduction order only." }, { "event": 36, "description": "Claims etth1-uni-24-mse/mae (univariate, features S, seq_len 720, label_len 168, pred_len 24) were registered but not run on this dedicated box either: they remain untested.", "impact": "material", "reason": "Micro-benchmark on this box (event 19): one training step of the univariate seq_len 720 configuration takes 27.6 s at the best thread count (2 threads; 40.1 s at 1, 34.7 s at 4). With 247 steps per epoch that is about 1.9 h per epoch before validation, and the authors early stopping (patience 3) needs at least 4 epochs, i.e. at least ~7.6 h for one iteration, which exceeds the 7 h wall-time and $0.80 cloud budget of this re-run. Cutting --train_epochs or --seq_len was rejected because it would change the result rather than test the claim." }, { "event": 63, "description": "Fewer training iterations than the --itr 5 in the authors script: ETTh1 multivariate horizon 24 was run as 4 independent --itr 1 iterations (observed value = their mean), and multivariate horizon 48 as a single --itr 1 iteration (n=1). --itr is the scripts own option; no other hyperparameter was changed.", "impact": "minor", "reason": "Compute budget (7 h wall time, $0.80): on this CPU box one horizon-24 iteration took 77-113 min and the horizon-48 iteration took 190 min (job events 31, 46, 53, 57, 40), so a second horizon-48 iteration or a fifth horizon-24 iteration could not finish in time." } ], "notes": [ { "event": 5, "text": "Re-run of attempt arxiv-2012-07436-20261019T132134Z on a dedicated box, to test the claims that went untested there for compute reasons." }, { "event": 64, "text": "Configuration source, as in the earlier attempt: the authors commands from scripts/ETTh1.sh at c648cc6 are run verbatim apart from --itr. The paper text (p.5) describes a 3-layer plus 1-layer encoder, a 2-layer decoder and 8 epochs, whereas the released scripts use --e_layers 2 --d_layers 1 and the code defaults (6 epochs, early-stopping patience 3, batch size 32); the released commands are treated as the authors reproduction recipe. Comparison with the earlier attempt (arxiv-2012-07436-20261019T132134Z): multivariate horizon 24 there gave MSE 0.4941 / MAE 0.5111 from one run; here 4 runs gave 0.4705-0.5298 MSE (mean 0.5046) and 0.4925-0.5318 MAE (mean 0.5151), consistent with it. Multivariate horizon 48, untested there, is tested here. Univariate horizon 24 remains untested here too: this dedicated s-4vcpu-8gb-intel box was not faster per step than the earlier shared one (m24 2.5 s/step at best vs 1.9 s there)." } ], "flags": [ "At least one deviation from the paper's protocol is marked material." ], "compute": { "shell_commands": 8, "sandbox_seconds": 819.3, "jobs": 6, "job_seconds": 33539.1, "sandbox": { "container": "repro-arxiv-2012-07436-20261019t202050z", "docker_daemon": { "mem_bytes": 8326942720, "ncpu": 4 }, "effective": { "cpus": 3.5, "memory_bytes": 6442450944 }, "image": "repro-sandbox:0.2", "image_id": "sha256:0f62397ca0ed1a51f35ab9f6dc755d3f012e5a0d2aff0721a359c88aed8febb2", "limits": { "cpus": 3.5, "gpu": null, "memory": "6g", "network": true, "shm": "2g" }, "target": "cloud", "target_detail": { "box": "arxiv-2012-07436-20261019t202050z", "droplet": "repro-arxiv-2012-07436-20261019-2021", "image": "ubuntu-24-04-x64", "price_hourly_usd": 0.08333, "provider": "digitalocean", "region": "nyc3", "size": "s-4vcpu-8gb-intel" }, "volume": "repro-arxiv-2012-07436-20261019t202050z-work", "warnings": [] }, "environment_event": 66 }, "drivers": [ "operator:claude-opus-5-5 (Claude Code subagent, AI-driven)" ], "model_usage": {}, "model_cost_usd": null, "started_at": "2026-10-19T20:20:50.671Z", "ended_at": "2026-10-20T01:47:06.748Z", "log": { "events": 69, "head_hash": "cf28ac49bd513cc6b2d4438a5e99cbbfd7f7210680dec7afd136f609a40513f6" }, "right_of_reply": "Authors may submit a response, which is published alongside this report unedited." }