{ "schema": "repro.report/v3", "engine_version": "0.2.1", "attempt_id": "arxiv-2205-13504-20261020T025623Z", "legacy": false, "paper": { "arxiv_id": "2205.13504", "title": "Are Transformers Effective for Time Series Forecasting?", "authors": [ "Ailing Zeng", "Muxi Chen", "Lei Zhang", "Qiang Xu" ], "year": 2023, "doi": "10.1609/aaai.v37i9.26317", "venue": "Proceedings of the AAAI Conference on Artificial Intelligence", "openalex_id": "https://openalex.org/W4382203079" }, "code": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear", "matches_primary": false }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear", "matches_primary": true } ], "code_used": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "repo": "cure-lab/LTSF-Linear", "code_version": "primary", "source": "harness", "results": [ 52, 53, 54, 55, 56, 57, 58, 59 ] }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "repo": "cure-lab/LTSF-Linear", "code_version": "primary", "source": "harness", "results": [ 52, 53, 54, 55, 56, 57, 58, 59 ] }, { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "repo": "cure-lab/LTSF-Linear", "code_version": "earlier", "source": "harness", "results": [ 60, 61, 62, 63, 64, 65, 66, 67 ] }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "repo": "cure-lab/LTSF-Linear", "code_version": "earlier", "source": "harness", "results": [ 60, 61, 62, 63, 64, 65, 66, 67 ] } ], "primary_code_source": "protocol", "outcome": "reproduced_with_earlier_code", "outcome_label": "Reproduced only with earlier code", "qualifiers": [], "qualifier_details": [], "evidence_gate_failed": [], "assurance": { "plan_sealed": true, "sealed_before_scored_execution": true, "evidence_bound": true, "extraction_sealed": true, "execution_manifest": true, "semantic_rederivation_passed": true, "predates": [], "engine_version": "0.2.1", "attempt_engine_version": "0.2.1", "setup_outputs_before_seal": 3, "observations_before_seal": { "shell": 3, "job_finished": 0, "job_status": 0, "read_file": 0 }, "observation_provenance": "captured", "plan_hash": "59e7a2da58ab424b26f27eaccc311fbc35d71d1e778efa4df0b7911d6958de23", "paper_text_sha256_ok": true }, "plan": { "event": 22, "setup_outputs_observed": 3, "observations_before_seal": { "shell": 3, "job_finished": 0, "job_status": 0, "read_file": 0 }, "observation_provenance": "captured", "reruns": "allowed", "rerun_policy": "At most one retry per fit, only after a documented external interruption (host loss or transient I/O), with identical inputs, seed and command under a new job name; both invocations are retained and the first complete invocation is scored. Low performance, early stopping, OOM, NaNs and optimizer failure are not grounds for a rerun.", "rerun_rule": "prior_job_failed", "run_policy": "Exactly eight fits: commit {721ab5879ba72fe67fab0846ba2933c3074a0c53 (earlier code), 6f6cce8bc8fa52704059a663181f100d94fc75bf (primary)} x horizon {96,192,336,720}; seed 2021 set by the authors' runner (Python random, NumPy, torch), itr=1, one repetition per fit, one scored job per fit, each a fresh Python process, one at a time, in this order: fit-721ab58-h96, -h192, -h336, -h720, then fit-6f6cce8-h96, -h192, -h336, -h720; all eight run regardless of scores. Job command: bash /work/settle/run_fit.sh /work/settle/fit.py . Arguments = the commit's own ETTh1 script (721ab58: scripts/EXP-LongForecasting/DLinear/etth1.sh, batch 8, runner-default lr 0.0001; 6f6cce8: scripts/EXP-LongForecasting/Linear/etth1.sh with model DLinear, batch 32, lr 0.005), features M, seq_len 336, label_len 48, enc_in 7, individual False, kernel 25, 10 max epochs, patience 3, Adam, lradj type1, authors' early stopping/best checkpoint, test loader drop_last True; plus the single declared loader override --num_workers 0 for both (torch 1.9 draws the iterator base seed and the RandomSampler seed from the global generator identically for 0 and >0 workers; inspected, event 16). Cap: 45 wall-minutes from the start of the first fit, enforced by run_fit.sh with timeout; no pilots, extra epochs, learning-rate sweeps or other seeds. Bound inputs (SHA-256): plan 1ca775a18cee300e242d5a35f97e7857048c4f04eae61a87ed5c02917d709ec8; fit.py 0c9db45ec1dac400afd290f4dbdbd33436ec81968325cb34caae8edea3132cf1; run_fit.sh 455f5970b0141d1b4626569177dff040a58e9b6dc1ca3e4629ffacff98eb211a; manifest_data.py 52119b7395bbb4e3219d7c5c59681be793b7def4dd51176f135fe1968336c5a3; data-manifest.json 3b8d3bf8a099360c6c8df623b38b6e6a7c3730b86cf146eb68d1ee71e30ca58b; ETTh1.csv f18de3ad269cef59bb07b5438d79bb3042d3be49bdeecf01c1cd6d29695ee066; requirements.lock.txt 4839181cb5b75de9cdf676eba44fee2ee7164f730d20872e2171d173f12f9705; env-freeze.txt 1ff64db86b473d6769c9772f50598809f3641ba22f64d4daf3fc5a9003b46f94; env-records.sha256 c9e89a411df7f93cb95873d58b427194c75857895ecc1206901cd21c9a983f57; git trees 721ab58 27aebe2f91c036087007a296c5173bac6e7cff58, 6f6cce8 2583889e5111f2a1a8c59495084eb7e1b8050477 (clean); sandbox image sha256:f5d3ab8203a84c807ae92399a5a0d7fbf2f4c83f32838bb0e52739f80bbc9a05. Environment: Python 3.9.25, torch 1.9.0+cpu, NumPy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2 (full freeze bound), CPU only, OMP/MKL 4 threads, 4-vCPU droplet.", "aggregation": "No aggregation across runs. Each claim's observed value is the single fit's unrounded test MSE or MAE returned by the authors' utils/metrics.py over all retained test windows x horizon x 7 standardized variables (no inverse transform, no percent conversion, no per-batch averaging), re-scored by fit.py from the saved pred/true arrays and required to equal the runtime value exactly (otherwise the job prints no key lines and exits 3). Primary results come from the 6f6cce8 jobs; earlier-code results (code_ref 721ab5879ba72fe67fab0846ba2933c3074a0c53) from the 721ab58 jobs; both are recorded for all eight claims. Judgement per cell: |observed - paper| <= 0.0005 absolute, unrounded (half the last printed decimal). An exact |gap| = 0.0005 is reported as rounding-boundary/inconclusive in a note, whatever the harness's <= rule says. Per version, 'all selected cells match printed precision' is the conjunction of its eight comparisons; every mismatch is listed with its signed gap. No multiplicity adjustment and no standard error: this is a fixed-seed numerical reconstruction, not a sampling study. The plan's section-4 branch is selected mechanically from the two conjunctions (earlier all-eight / primary all-eight).", "stopping_rule": "One pass over the eight-fit matrix in the fixed order; stop at completion or at the 45-minute cap, whichever comes first; never stop early because a result passes; no further runs after seeing results. A required fit that does not complete leaves its two cells evidence-incomplete; no averaging over survivors.", "plan_hash": "59e7a2da58ab424b26f27eaccc311fbc35d71d1e778efa4df0b7911d6958de23", "claims": [ "dlinear-etth1-h96-mse", "dlinear-etth1-h96-mae", "dlinear-etth1-h192-mse", "dlinear-etth1-h192-mae", "dlinear-etth1-h336-mse", "dlinear-etth1-h336-mae", "dlinear-etth1-h720-mse", "dlinear-etth1-h720-mae" ], "claims_registered_after_output": [] }, "not_attemptable_reason": null, "summary": "Prospective settling test of eight ETTh1 DLinear cells of Table 2 (MSE and MAE at horizons 96/192/336/720), sealed before any training: two author code versions x four horizons, seed 2021, one fit each, judged unrounded within 0.0005. The paper-date code 6f6cce8 (primary) misses all eight cells by 0.005 to 0.038; the earlier author code 721ab58 (2022-08-06, fixed in advance as the only allowed earlier version) matches all eight. Both sets of values are bit-identical to the earlier attempt arxiv-2205-13504-20261018T015559Z. This confirms reproduction with pre-update code and version-dependent numerical drift; it does not identify which code change caused the drift.", "claims": [ { "claim_id": "dlinear-etth1-h96-mse", "comparison": "within", "extraction": { "key": "dlinear_etth1_h96_mse", "occurrence": "only", "transform": "none", "units": "MSE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=96, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 96, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 96 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.375, "quote": "96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 96 is 0.375 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MSE (standardized)", "registered_event": 5, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.0013333333333333333, "extraction_registered": true, "results": [ { "event": 52, "observed_value": 0.3840945363044739, "delta": 0.009094536304473877, "consistent": false, "evidence_events": [ 41 ], "evidence": { "artifact": "00041-job-fit-6f6cce8-h96-stdout.txt", "event": 41, "excerpt": "dlinear_etth1_h96_mse 0.3840945363044739", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 40, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 60, "observed_value": 0.3748247027397156, "delta": -0.00017529726028442383, "consistent": true, "evidence_events": [ 24 ], "evidence": { "artifact": "00024-job-fit-721ab58-h96-stdout.txt", "event": 24, "excerpt": "dlinear_etth1_h96_mse 0.3748247027397156", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 23, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 52, "observed_value": 0.3840945363044739, "delta": 0.009094536304473877, "consistent": false, "evidence_events": [ 41 ], "evidence": { "artifact": "00041-job-fit-6f6cce8-h96-stdout.txt", "event": 41, "excerpt": "dlinear_etth1_h96_mse 0.3840945363044739", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 40, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 60, "observed_value": 0.3748247027397156, "delta": -0.00017529726028442383, "consistent": true, "evidence_events": [ 24 ], "evidence": { "artifact": "00024-job-fit-721ab58-h96-stdout.txt", "event": 24, "excerpt": "dlinear_etth1_h96_mse 0.3748247027397156", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 23, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0242520968119304, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h96-mae", "comparison": "within", "extraction": { "key": "dlinear_etth1_h96_mae", "occurrence": "only", "transform": "none", "units": "MAE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=96, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 96, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 96 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.399, "quote": "96 0.80% 0.375 0.397 0.374 0.394 0.375 0.399", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 96 is 0.399 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MAE (standardized)", "registered_event": 6, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.0012531328320802004, "extraction_registered": true, "results": [ { "event": 53, "observed_value": 0.4046996533870697, "delta": 0.005699653387069681, "consistent": false, "evidence_events": [ 41 ], "evidence": { "artifact": "00041-job-fit-6f6cce8-h96-stdout.txt", "event": 41, "excerpt": "dlinear_etth1_h96_mae 0.4046996533870697", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 40, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 61, "observed_value": 0.39947351813316345, "delta": 0.00047351813316343083, "consistent": true, "evidence_events": [ 24 ], "evidence": { "artifact": "00024-job-fit-721ab58-h96-stdout.txt", "event": 24, "excerpt": "dlinear_etth1_h96_mae 0.39947351813316345", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 23, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 53, "observed_value": 0.4046996533870697, "delta": 0.005699653387069681, "consistent": false, "evidence_events": [ 41 ], "evidence": { "artifact": "00041-job-fit-6f6cce8-h96-stdout.txt", "event": 41, "excerpt": "dlinear_etth1_h96_mae 0.4046996533870697", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 40, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h96: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 61, "observed_value": 0.39947351813316345, "delta": 0.00047351813316343083, "consistent": true, "evidence_events": [ 24 ], "evidence": { "artifact": "00024-job-fit-721ab58-h96-stdout.txt", "event": 24, "excerpt": "dlinear_etth1_h96_mae 0.39947351813316345", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 23, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h96: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0142848455816282, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h192-mse", "comparison": "within", "extraction": { "key": "dlinear_etth1_h192_mse", "occurrence": "only", "transform": "none", "units": "MSE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=192, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 192, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 192 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.405, "quote": "192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 192 is 0.405 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MSE (standardized)", "registered_event": 7, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.0012345679012345679, "extraction_registered": true, "results": [ { "event": 54, "observed_value": 0.44340065121650696, "delta": 0.03840065121650693, "consistent": false, "evidence_events": [ 44 ], "evidence": { "artifact": "00044-job-fit-6f6cce8-h192-stdout.txt", "event": 44, "excerpt": "dlinear_etth1_h192_mse 0.44340065121650696", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 43, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 62, "observed_value": 0.40548211336135864, "delta": 0.00048211336135861593, "consistent": true, "evidence_events": [ 27 ], "evidence": { "artifact": "00027-job-fit-721ab58-h192-stdout.txt", "event": 27, "excerpt": "dlinear_etth1_h192_mse 0.40548211336135864", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 26, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 54, "observed_value": 0.44340065121650696, "delta": 0.03840065121650693, "consistent": false, "evidence_events": [ 44 ], "evidence": { "artifact": "00044-job-fit-6f6cce8-h192-stdout.txt", "event": 44, "excerpt": "dlinear_etth1_h192_mse 0.44340065121650696", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 43, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 62, "observed_value": 0.40548211336135864, "delta": 0.00048211336135861593, "consistent": true, "evidence_events": [ 27 ], "evidence": { "artifact": "00027-job-fit-721ab58-h192-stdout.txt", "event": 27, "excerpt": "dlinear_etth1_h192_mse 0.40548211336135864", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 26, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0948164227568071, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h192-mae", "comparison": "within", "extraction": { "key": "dlinear_etth1_h192_mae", "occurrence": "only", "transform": "none", "units": "MAE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=192, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 192, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 192 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.416, "quote": "192 3.57% 0.418 0.429 0.408 0.415 0.405 0.416", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 192 is 0.416 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MAE (standardized)", "registered_event": 8, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.001201923076923077, "extraction_registered": true, "results": [ { "event": 55, "observed_value": 0.44990822672843933, "delta": 0.03390822672843935, "consistent": false, "evidence_events": [ 44 ], "evidence": { "artifact": "00044-job-fit-6f6cce8-h192-stdout.txt", "event": 44, "excerpt": "dlinear_etth1_h192_mae 0.44990822672843933", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 43, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 63, "observed_value": 0.4158572554588318, "delta": -0.0001427445411681938, "consistent": true, "evidence_events": [ 27 ], "evidence": { "artifact": "00027-job-fit-721ab58-h192-stdout.txt", "event": 27, "excerpt": "dlinear_etth1_h192_mae 0.4158572554588318", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 26, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 55, "observed_value": 0.44990822672843933, "delta": 0.03390822672843935, "consistent": false, "evidence_events": [ 44 ], "evidence": { "artifact": "00044-job-fit-6f6cce8-h192-stdout.txt", "event": 44, "excerpt": "dlinear_etth1_h192_mae 0.44990822672843933", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 43, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h192: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 63, "observed_value": 0.4158572554588318, "delta": -0.0001427445411681938, "consistent": true, "evidence_events": [ 27 ], "evidence": { "artifact": "00027-job-fit-721ab58-h192-stdout.txt", "event": 27, "excerpt": "dlinear_etth1_h192_mae 0.4158572554588318", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 26, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h192: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0815101604049022, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h336-mse", "comparison": "within", "extraction": { "key": "dlinear_etth1_h336_mse", "occurrence": "only", "transform": "none", "units": "MSE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=336, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 336, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 336 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.439, "quote": "336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 336 is 0.439 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MSE (standardized)", "registered_event": 9, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.0011389521640091116, "extraction_registered": true, "results": [ { "event": 56, "observed_value": 0.44690412282943726, "delta": 0.007904122829437255, "consistent": false, "evidence_events": [ 47 ], "evidence": { "artifact": "00047-job-fit-6f6cce8-h336-stdout.txt", "event": 47, "excerpt": "dlinear_etth1_h336_mse 0.44690412282943726", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 46, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 64, "observed_value": 0.43851277232170105, "delta": -0.00048722767829895153, "consistent": true, "evidence_events": [ 30 ], "evidence": { "artifact": "00030-job-fit-721ab58-h336-stdout.txt", "event": 30, "excerpt": "dlinear_etth1_h336_mse 0.43851277232170105", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 29, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 56, "observed_value": 0.44690412282943726, "delta": 0.007904122829437255, "consistent": false, "evidence_events": [ 47 ], "evidence": { "artifact": "00047-job-fit-6f6cce8-h336-stdout.txt", "event": 47, "excerpt": "dlinear_etth1_h336_mse 0.44690412282943726", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 46, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 64, "observed_value": 0.43851277232170105, "delta": -0.00048722767829895153, "consistent": true, "evidence_events": [ 30 ], "evidence": { "artifact": "00030-job-fit-721ab58-h336-stdout.txt", "event": 30, "excerpt": "dlinear_etth1_h336_mse 0.43851277232170105", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 29, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0180048356023628, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h336-mae", "comparison": "within", "extraction": { "key": "dlinear_etth1_h336_mae", "occurrence": "only", "transform": "none", "units": "MAE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=336, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 336, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 336 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.443, "quote": "336 6.54% 0.479 0.476 0.429 0.427 0.439 0.443", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 336 is 0.443 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MAE (standardized)", "registered_event": 10, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.001128668171557562, "extraction_registered": true, "results": [ { "event": 57, "observed_value": 0.4481101632118225, "delta": 0.005110163211822505, "consistent": false, "evidence_events": [ 47 ], "evidence": { "artifact": "00047-job-fit-6f6cce8-h336-stdout.txt", "event": 47, "excerpt": "dlinear_etth1_h336_mae 0.4481101632118225", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 46, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 65, "observed_value": 0.4427293539047241, "delta": -0.0002706460952758838, "consistent": true, "evidence_events": [ 30 ], "evidence": { "artifact": "00030-job-fit-721ab58-h336-stdout.txt", "event": 30, "excerpt": "dlinear_etth1_h336_mae 0.4427293539047241", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 29, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 57, "observed_value": 0.4481101632118225, "delta": 0.005110163211822505, "consistent": false, "evidence_events": [ 47 ], "evidence": { "artifact": "00047-job-fit-6f6cce8-h336-stdout.txt", "event": 47, "excerpt": "dlinear_etth1_h336_mae 0.4481101632118225", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 46, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h336: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 65, "observed_value": 0.4427293539047241, "delta": -0.0002706460952758838, "consistent": true, "evidence_events": [ 30 ], "evidence": { "artifact": "00030-job-fit-721ab58-h336-stdout.txt", "event": 30, "excerpt": "dlinear_etth1_h336_mae 0.4427293539047241", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 29, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h336: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.011535357137297, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h720-mse", "comparison": "within", "extraction": { "key": "dlinear_etth1_h720_mse", "occurrence": "only", "transform": "none", "units": "MSE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=720, DLinear MSE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MSE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 720, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MSE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 720 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.472, "quote": "720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MSE at forecast horizon 720 is 0.472 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MSE (standardized)", "registered_event": 11, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.001059322033898305, "extraction_registered": true, "results": [ { "event": 58, "observed_value": 0.5031593441963196, "delta": 0.031159344196319605, "consistent": false, "evidence_events": [ 50 ], "evidence": { "artifact": "00050-job-fit-6f6cce8-h720-stdout.txt", "event": 50, "excerpt": "dlinear_etth1_h720_mse 0.5031593441963196", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 49, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 66, "observed_value": 0.4719056785106659, "delta": -9.432148933408158e-05, "consistent": true, "evidence_events": [ 38 ], "evidence": { "artifact": "00038-job-fit-721ab58-h720-stdout.txt", "event": 38, "excerpt": "dlinear_etth1_h720_mse 0.4719056785106659", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 37, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 58, "observed_value": 0.5031593441963196, "delta": 0.031159344196319605, "consistent": false, "evidence_events": [ 50 ], "evidence": { "artifact": "00050-job-fit-6f6cce8-h720-stdout.txt", "event": 50, "excerpt": "dlinear_etth1_h720_mse 0.5031593441963196", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 49, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 66, "observed_value": 0.4719056785106659, "delta": -9.432148933408158e-05, "consistent": true, "evidence_events": [ 38 ], "evidence": { "artifact": "00038-job-fit-721ab58-h720-stdout.txt", "event": 38, "excerpt": "dlinear_etth1_h720_mse 0.4719056785106659", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 37, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MSE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.0660155597379652, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "dlinear-etth1-h720-mae", "comparison": "within", "extraction": { "key": "dlinear_etth1_h720_mae", "occurrence": "only", "transform": "none", "units": "MAE, standardized" }, "headline": true, "location": "Table 2, p. 5, ETTh1 row T=720, DLinear MAE column (columns: horizon, improvement, Linear MSE/MAE, NLinear MSE/MAE, DLinear MSE/MAE); look-back L=336 per Appendix p. 9", "metric": "Test MAE of DLinear on ETTh1, features M, seq_len 336, label_len 48, pred_len 720, shared channels (individual=False), seed 2021, one fit: the value returned by the authors' utils/metrics.py MAE over all retained test windows (test loader shuffle False, drop_last True, batch = the version's batch size) x 720 steps x 7 variables, on the standardized scale (scaler fit on train rows only), no inverse transform, no percent conversion, no averaging of per-batch errors; best-validation checkpoint via the authors' early stopping (patience 3, max 10 epochs). Judged unrounded.", "paper_value": 0.49, "quote": "720 13.04% 0.624 0.592 0.440 0.453 0.472 0.490", "statement": "On ETTh1 (multivariate, look-back L=336), DLinear's test MAE at forecast horizon 720 is 0.490 (Table 2).", "tolerance": 0.0005, "tolerance_rationale": "Half of the paper's last displayed decimal unit (values printed to 3 decimals), per the sealed settling plan; fixed-seed numerical reconstruction, no seed/split SE estimable from one run. An exact |gap| = 0.0005 tie is reported as rounding-boundary/inconclusive.", "unit": "MAE (standardized)", "registered_event": 12, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": false, "sealed": true, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.0010204081632653062, "extraction_registered": true, "results": [ { "event": 59, "observed_value": 0.5137903690338135, "delta": 0.023790369033813485, "consistent": false, "evidence_events": [ 50 ], "evidence": { "artifact": "00050-job-fit-6f6cce8-h720-stdout.txt", "event": 50, "excerpt": "dlinear_etth1_h720_mae 0.5137903690338135", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 49, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, { "event": 67, "observed_value": 0.49024033546447754, "delta": 0.00024033546447754794, "consistent": true, "evidence_events": [ 38 ], "evidence": { "artifact": "00038-job-fit-721ab58-h720-stdout.txt", "event": 38, "excerpt": "dlinear_etth1_h720_mae 0.49024033546447754", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 37, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] } ], "withdrawn_after_seal": false, "final": { "event": 59, "observed_value": 0.5137903690338135, "delta": 0.023790369033813485, "consistent": false, "evidence_events": [ 50 ], "evidence": { "artifact": "00050-job-fit-6f6cce8-h720-stdout.txt", "event": 50, "excerpt": "dlinear_etth1_h720_mae 0.5137903690338135", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 49, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-6f6cce8-h720: authors' run_longExp.py at 6f6cce8 (unmodified) with the Linear/etth1.sh arguments (model DLinear, batch 32, lr 0.005) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule.", "notes": "", "code_version": "primary", "code_ref": "", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "earlier_final": { "event": 67, "observed_value": 0.49024033546447754, "delta": 0.00024033546447754794, "consistent": true, "evidence_events": [ 38 ], "evidence": { "artifact": "00038-job-fit-721ab58-h720-stdout.txt", "event": 38, "excerpt": "dlinear_etth1_h720_mae 0.49024033546447754", "transform": "none" }, "evidence_binding": "bound", "evidence_verified": true, "evidence_reason": "ok", "evidence_after_seal": true, "evidence_from_unscored_job": false, "evidence_scored_execution": true, "extraction": "registered", "job_start_event": 37, "tree_modified_during_job": false, "manifest_complete": true, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Scored job fit-721ab58-h720: authors' run_longExp.py at 721ab58 (unmodified) with the DLinear/etth1.sh arguments (batch 8, runner-default lr 0.0001, 1/seq_len weight init) plus --num_workers 0, seed 2021, CPU torch 1.9.0; test MAE from the authors' utils/metrics.py, re-scored from the saved arrays (equal to the runtime value); value read by the registered key rule. Run before the primary jobs per the sealed order; recorded after them because the engine records earlier code after the primary result.", "notes": "", "code_version": "earlier", "code_ref": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "code_heads": [ { "commit": "721ab5879ba72fe67fab0846ba2933c3074a0c53", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-721ab58", "url": "https://github.com/cure-lab/LTSF-Linear" }, { "commit": "6f6cce8bc8fa52704059a663181f100d94fc75bf", "full_name": "cure-lab/LTSF-Linear", "modified_files": 0, "path": "/work/src-6f6cce8", "url": "https://github.com/cure-lab/LTSF-Linear" } ] }, "verdict": "consistent_earlier_code", "result_revisions": 0, "ratio_to_published": 1.048551773538395, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true } ], "counts": { "claims_scored": 8, "consistent": 0, "consistent_earlier_code": 8, "inconsistent": 0, "untested": 0, "withdrawn_after_seal": 0, "other": { "claims": 0, "consistent": 0, "inconsistent": 0, "untested": 0 } }, "withdrawn": [], "protocol": { "earlier_code": "allowed", "primary_code": "github.com/cure-lab/LTSF-Linear at commit 6f6cce8bc8fa52704059a663181f100d94fc75bf (2022-08-15, the paper-date snapshot: latest code commit before arXiv v3 of 2022-08-17), DLinear with the settings of scripts/EXP-LongForecasting/Linear/etth1.sh (batch 32, learning rate 0.005, released random initialization).", "rationale": "Settling test per office/plans/2205.13504-settling-plan.md. The earlier code is fixed in advance to exactly one author-produced version, 721ab5879ba72fe67fab0846ba2933c3074a0c53 (2022-08-06, before the 2022-08-15 'Update scripts and models' commit), with the settings of scripts/EXP-LongForecasting/DLinear/etth1.sh (batch 8, runner-default learning rate 0.0001, 1/seq_len weight initialization). It was selected on existing evidence: a post-hoc diagnostic (event 44 of attempt arxiv-2205-13504-20261018T015559Z) matched all eight cells with it, while the paper-date code missed DLinear (that attempt's verdict stands as recorded). This attempt is a prospective confirmation; the plan calls 721ab58 the historical-paper hypothesis and 6f6cce8 the version-drift control. Both versions are run for all eight cells regardless of the primary results, and both are reported equally; no other commit may be used. The comparison is between version bundles and does not isolate learning rate from initialization." }, "deviations": [ { "event": 70, "description": "Full-precision pred.npy/true.npy arrays and selected checkpoints were written on the box but are not retained in the attempt after the box is destroyed; the log keeps their SHA-256 digests, the metrics.csv row, resolved args, train/validation history and the test-index manifest (all printed in each scored job's stdout).", "impact": "minor", "reason": "Arrays are up to ~43 MB per fit and the engine stores only command output as artifacts; the plan asks for these artifacts per fit." }, { "event": 71, "description": "The plan asks for one externally anchored plan before execution. The sealed plan (protocol, claims, run/aggregation/stopping/rerun policy, with the SHA-256 of the plan document, drivers, input manifest, environment freeze, data file, source trees and image) is hash-chained in this log at the seal event, but was not anchored outside OpenRepro before the scored jobs ran.", "impact": "minor", "reason": "No external anchoring step is available before execution in engine 0.2.1; anchors are written only for released attempts." }, { "event": 72, "description": "CPU execution (4-vCPU droplet, 4 OMP/MKL threads) with Python 3.9.25, torch 1.9.0+cpu, NumPy 1.21.6, pandas 1.3.5, scikit-learn 1.0.2; the authors' README sets up Python 3.6.9 and the paper ran on GPU.", "impact": "minor", "reason": "Environment pinned by the settling plan; CPU-only sandbox. torch 1.9.0 is the version in the repository's requirements.txt." }, { "event": 73, "description": "Loader override --num_workers 0 for both versions (the runner default is 10); ./dataset/ is a symlink to /work/data in each fit's fresh output directory.", "impact": "none", "reason": "Declared resource override in the plan; torch 1.9 draws the iterator base seed and sampler seed from the global generator identically for 0 and >0 workers (inspected before the seal), and the values reproduce the earlier attempt's 10-worker values bit for bit." }, { "event": 74, "description": "Engine roles: the plan calls 721ab58 the primary historical-paper hypothesis and 6f6cce8 the version-drift control; this attempt records 6f6cce8 as code_version primary and 721ab58 as earlier (code_ref 721ab58...), as the operator was instructed. The jobs ran in the sealed order (721ab58 first); results were recorded primary first because the engine records an earlier-code result only after the claim's primary result.", "impact": "none", "reason": "Engine semantics of primary/earlier code." }, { "event": 75, "description": "The plan's tie rule (an exact |gap| = 0.0005 is inconclusive) cannot be expressed in the harness, whose 'within' rule counts a tie as consistent; it was checked by hand and no tie occurred.", "impact": "none", "reason": "Harness comparison vocabulary." } ], "notes": [ { "event": 13, "text": "Disclosure: a setup-only attempt, arxiv-2205-13504-20261020T024800Z, was abandoned just before this one (no claims, no seal, no training, no inference, no results). It cloned the two commits, built the pinned environment, fetched ETTh1.csv and drafted the collector; it was abandoned because its setup ran before the claims were registered. The same setup is repeated here after registration. The driver files written below were drafted there; every one is logged here before the seal." }, { "event": 68, "text": "Sealed judgement, applied after all eight fits: 721ab58 (earlier code) matches all eight cells within 0.0005 (signed gaps: h96 MSE -0.000175, MAE +0.000474; h192 MSE +0.000482, MAE -0.000143; h336 MSE -0.000487, MAE -0.000271; h720 MSE -0.000094, MAE +0.000240). 6f6cce8 (paper-date, primary) misses all eight (h96 MSE +0.009095, MAE +0.005700; h192 MSE +0.038401, MAE +0.033908; h336 MSE +0.007904, MAE +0.005110; h720 MSE +0.031159, MAE +0.023790). No |gap| equals 0.0005 exactly, so no cell is a rounding-boundary tie. Plan section 4 branch, selected mechanically: 'Earlier all-eight match; newer has misses.'" }, { "event": 69, "text": "All sixteen values are bit-identical to the earlier attempt arxiv-2205-13504-20261018T015559Z: the 721ab58 values equal its post-hoc diagnostic (event 44) and the 6f6cce8 values equal its paper-era run (event 38), on a different droplet, with --num_workers 0 instead of the default 10. Each fit's re-score from the saved arrays equalled the runtime metric exactly, and every retained test window matched the dataset's own windows (fit.py checks). All eight jobs exited 0; total fit wall time about 6 minutes, well inside the 45-minute cap; no retry was needed or used." } ], "flags": [], "compute": { "shell_commands": 3, "sandbox_seconds": 18.2, "jobs": 8, "job_seconds": 200.0, "sandbox": { "container": "repro-arxiv-2205-13504-20261020t025623z", "docker_daemon": { "mem_bytes": 8326955008, "ncpu": 4 }, "effective": { "cpus": 3.5, "memory_bytes": 6442450944 }, "image": "repro-sandbox:0.2", "image_id": "sha256:f5d3ab8203a84c807ae92399a5a0d7fbf2f4c83f32838bb0e52739f80bbc9a05", "limits": { "cpus": 3.5, "gpu": null, "memory": "6g", "network": true, "shm": "2g" }, "target": "cloud", "target_detail": { "box": "arxiv-2205-13504-20261020t025623z", "droplet": "repro-arxiv-2205-13504-20261020-0257", "image": "ubuntu-24-04-x64", "price_hourly_usd": 0.08333, "provider": "digitalocean", "region": "tor1", "size": "s-4vcpu-8gb-intel" }, "volume": "repro-arxiv-2205-13504-20261020t025623z-work", "warnings": [] }, "environment_event": 77 }, "drivers": [ "operator:claude-opus-5-5 (Claude Code subagent, AI-driven)" ], "model_usage": {}, "model_cost_usd": null, "started_at": "2026-10-20T02:56:23.198Z", "ended_at": "2026-10-20T03:07:55.781Z", "log": { "events": 81, "head_hash": "6d8fe8bb8cfec5b3ae7018267b9386ac4915947961ec36fda06755c64b5de76b" }, "right_of_reply": "Authors may submit a response, which is published alongside this report unedited.", "redactions": [ { "event": 80, "rule": "operator home directory prefix → ~", "reason": "Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~", "replacements": 4, "pre_redaction_head": "0403d025c8d2151c4165ab7aba0b655d9e303373e7914e23dd1fe5d54908dca7", "pre_redaction_events": 80, "chain_rehashed_from_event": 20, "plan_hash_events": [], "seal_bytes_changed": true, "results_before_match": true, "results_before_status": "operator assertion, checked for internal consistency against the current log", "disclosure": "Before release, the operator's local home-directory path was replaced with ~ in 4 places. Every event hash from event 20 on changed with it. The seal event's bytes changed (its chain hash); its plan hash did not. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now." } ] }