{ "schema": "repro.report/v3", "engine_version": "0.2.1", "attempt_id": "arxiv-1606-00931-20261018T225505Z", "legacy": true, "paper": { "arxiv_id": "1606.00931", "title": "DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network", "authors": [ "Jared Katzman", "Uri Shaham", "Alexander Cloninger", "Jonathan Bates", "Tingting Jiang", "Yuval Kluger" ], "year": 2018, "doi": "10.1186/s12874-018-0482-1", "venue": "BMC Medical Research Methodology", "openalex_id": "https://openalex.org/W2753919178" }, "code": [ { "commit": "079dee71e3b20084047b9b022f64ed6b430d3df8", "full_name": "jaredleekatzman/DeepSurv", "modified_files": 0, "path": "/work/ds", "url": "https://github.com/jaredleekatzman/DeepSurv", "matches_primary": true } ], "code_used": [ { "commit": "079dee7", "repo": null, "code_version": "primary", "source": "method_text", "results": [ 51, 52, 53, 54, 55, 56 ] } ], "primary_code_source": "protocol", "outcome": "reproduced", "outcome_label": "Reproduced", "qualifiers": [ "claims_registered_after_output", "plan_not_sealed", "evidence_not_bound", "agent_selected_extraction" ], "qualifier_details": [], "evidence_gate_failed": [], "assurance": { "plan_sealed": false, "sealed_before_scored_execution": false, "evidence_bound": false, "extraction_sealed": false, "execution_manifest": false, "semantic_rederivation_passed": true, "predates": [ "plan_seal", "evidence_binding", "execution_manifest", "paper_text_hash", "scored_job_flag", "retry_reasons", "extraction_rule" ], "engine_version": "0.2.1", "attempt_engine_version": "0.1.0", "setup_outputs_before_seal": null, "observations_before_seal": null, "observation_provenance": null, "plan_hash": null, "paper_text_sha256_ok": null }, "plan": null, "not_attemptable_reason": null, "summary": "Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored.", "claims": [ { "claim_id": "support-cindex", "comparison": "within", "headline": true, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on SUPPORT", "paper_value": 0.618308, "quote": "SUPPORT 0.582870 (0.581,0.585) 0.618308 (0.616,0.620)", "statement": "DeepSurv reaches test C-index 0.618308 on SUPPORT.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 23, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.016173169358960262, "extraction_registered": false, "results": [ { "event": 51, "observed_value": 0.618582, "delta": 0.00027399999999999647, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 51, "observed_value": 0.618582, "delta": 0.00027399999999999647, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0004431448404356, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "metabric-cindex", "comparison": "within", "headline": true, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on METABRIC", "paper_value": 0.643374, "quote": "METABRIC 0.630618 (0.627,0.635) 0.643374 (0.639,0.647)", "statement": "DeepSurv reaches test C-index 0.643374 on METABRIC.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 24, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.015543058936170874, "extraction_registered": false, "results": [ { "event": 52, "observed_value": 0.643492, "delta": 0.00011799999999995148, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 52, "observed_value": 0.643492, "delta": 0.00011799999999995148, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0001834080954468, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "gbsg-cindex", "comparison": "within", "headline": true, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on Rotterdam (train) and GBSG (test)", "paper_value": 0.668402, "quote": "0.657750 (0.654, 0.661) 0.668402 (0.665,0.671)", "statement": "DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 25, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.014961056370268191, "extraction_registered": false, "results": [ { "event": 53, "observed_value": 0.672011, "delta": 0.0036089999999999733, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 53, "observed_value": 0.672011, "delta": 0.0036089999999999733, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 1.0053994452440298, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "whas-cindex", "comparison": "within", "headline": true, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on WHAS", "paper_value": 0.86262, "quote": "WHAS 0.817620 (0.814, 0.821) 0.862620 (0.859,0.866)", "statement": "DeepSurv reaches test C-index 0.862620 on WHAS.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 26, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.011592590016461478, "extraction_registered": false, "results": [ { "event": 54, "observed_value": 0.858003, "delta": -0.004617000000000093, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 54, "observed_value": 0.858003, "delta": -0.004617000000000093, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9946477011893996, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "sim-linear-cindex", "comparison": "within", "headline": false, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on the simulated linear survival data", "paper_value": 0.774019, "quote": "Simulated Linear 0.773677 (0.772,0.775) 0.774019 (0.772,0.776)", "statement": "DeepSurv reaches test C-index 0.774019 on simulated linear data.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 27, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.012919579493526645, "extraction_registered": false, "results": [ { "event": 55, "observed_value": 0.773885, "delta": -0.00013399999999996748, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 55, "observed_value": 0.773885, "delta": -0.00013399999999996748, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9998268776347868, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true }, { "claim_id": "sim-nonlinear-cindex", "comparison": "within", "headline": false, "location": "Table 1, p. 5 (DeepSurv column)", "metric": "test-set C-index of DeepSurv on the simulated nonlinear (Gaussian) survival data", "paper_value": 0.648902, "quote": "0.506951 (0.505,0.509) 0.648902 (0.647, 0.651)", "statement": "DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.", "tolerance": 0.01, "tolerance_rationale": "The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.", "unit": "C-index", "registered_event": 28, "quote_verified": true, "quote_page": 5, "preregistered": true, "registered_after_output": true, "sealed": false, "check_kind": "equivalence", "broad_bound": false, "tolerance_ratio": 0.015410647524587689, "extraction_registered": false, "results": [ { "event": 56, "observed_value": 0.645797, "delta": -0.0031050000000000244, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" } ], "withdrawn_after_seal": false, "final": { "event": 56, "observed_value": 0.645797, "delta": -0.0031050000000000244, "consistent": true, "evidence_events": [ 49 ], "evidence": null, "evidence_binding": "legacy", "evidence_verified": true, "evidence_reason": "value matched by the legacy rule in events 49", "evidence_after_seal": null, "evidence_from_unscored_job": false, "evidence_scored_execution": null, "extraction": "legacy", "job_start_event": null, "tree_modified_during_job": null, "manifest_complete": false, "retry_reason": null, "rerun_rule_checked": false, "rederivation_problems": [], "method": "Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).", "notes": "", "code_version": "primary", "code_ref": "" }, "earlier_final": null, "verdict": "consistent", "result_revisions": 0, "ratio_to_published": 0.9952149939436155, "optional_stopping": false, "evidence_problems": [], "evidence_ok": true } ], "counts": { "claims_scored": 4, "consistent": 4, "consistent_earlier_code": 0, "inconsistent": 0, "untested": 0, "withdrawn_after_seal": 0, "other": { "claims": 2, "consistent": 2, "inconsistent": 0, "untested": 0 } }, "withdrawn": [], "protocol": { "earlier_code": "not_allowed", "primary_code": "github.com/jaredleekatzman/DeepSurv (first author; the paper links it) at commit 079dee7, the last commit on or before 2017-08-09, the date of arXiv v3 (the version tested), run with the documented command in experiments/deepsurv/Dockerfile. for each experiment", "rationale": "arXiv v3 is the PDF fetched and its Table 3 hyper-parameters match the model JSON files at 079dee7 for SUPPORT, METABRIC, GBSG and the two simulated sets. Later commits (2018-01 Merge revisions) belong to the journal revision and use different configs and regenerated simulated data, so they are not the arXiv v3 code. No commit before 079dee7 contains the experiment configs, so there is no earlier-code fallback." }, "deviations": [ { "event": 57, "description": "Python 2.7.15 from conda-forge with numpy 1.13.3, scipy 0.19.1, pandas 0.20.3, h5py 2.7.1 and matplotlib 2.2.3 (only for plots), plus Theano 0.9.0, Lasagne master (0.2.dev1) and lifelines 0.9.4, logger 1.4, tensorboard-logger 0.0.3 from PyPI, on CPU with float32, instead of the floydhub/dl-docker:cpu image.", "impact": "minor", "reason": "The floydhub image is not reproducible today and its library versions are not recorded. The Dockerfile pins lifelines 0.9.4, h5py 2.7.0 and matplotlib 2.0.0; h5py 2.7.1 and matplotlib 2.2.3 were the nearest available conda-forge builds for Python 2.7, and matplotlib is used only for figures." }, { "event": 58, "description": "The documented commands were run with explicit paths to the model, data and results directories instead of the Docker mounts (/models, /shared/data, /shared/results), with /DeepSurv/deepsurv symlinked to the repository, and the WHAS run used the repository model file at 079dee7 even though it differs from the WHAS hyper-parameters in Table 3.", "impact": "none", "reason": "Path mapping only; no code or hyper-parameter was changed. For GBSG the Dockerfile at 079dee7 misspells the data file (gsbg), and the bundled file gbsg_cancer_train_test.h5 was passed." } ], "notes": [ { "event": 29, "text": "Scope and observations, declared before any run: the six DeepSurv C-index values of Table 1 (four real datasets, two simulated survival sets) are tested; CPH and RSF columns, the simulated-treatment row and the treatment-recommendation results (Table 2) are not scored. Each claim is scored on the mean test C-index over 10 independent runs of the documented command, because the code fixes no seed. At 079dee7 the WHAS model file (whas_model_selu_0.json: 3 SELU layers of 40, LR 0.0238) does not match the WHAS column of the paper Table 3 (2 ReLU layers of 48, LR 0.067); the documented command uses the repository file and is run as is. The GBSG Dockerfile at 079dee7 names the data file gsbg_cancer_train_test.h5 while the file in the repository is gbsg_cancer_train_test.h5 (fixed upstream in 6a085af); the correct path is passed. Paths: the Dockerfiles mount the code at /DeepSurv/deepsurv and data at /shared/data; the same layout is recreated with symlinks and the model, data and results paths are passed as arguments." }, { "event": 44, "text": "Every run exits non-zero at its very last step (event 42): deepsurv_run.py at 079dee7 rebinds the name model to the JSON file name (line 162) and then calls model.save_weights, which raises AttributeError. The test C-index has already been computed and printed by then, so the scored values are unaffected; only the saving of trained weights fails. Upstream fixed this in a011cd7 (2017-12-07, [Bug] Fix saving model error). The code is run unmodified, so the FAILED lines in the job output are this error." } ], "flags": [], "compute": { "shell_commands": 14, "sandbox_seconds": 89.8, "jobs": 3, "job_seconds": 6537.5, "sandbox": { "container": "repro-arxiv-1606-00931-20261018t225505z", "docker_daemon": { "mem_bytes": 8326946816, "ncpu": 4 }, "effective": { "cpus": 3.5, "memory_bytes": 6442450944 }, "image": "repro-sandbox:0.2", "image_id": "sha256:f4b241036e33f995f431b595fba27a8b03cf1f26e83ee872595f9957605f656b", "limits": { "cpus": 3.5, "gpu": null, "memory": "6g", "network": true, "shm": "2g" }, "target": "cloud", "target_detail": { "droplet": "repro-sandbox-20261018-2235", "image": "ubuntu-24-04-x64", "price_hourly_usd": 0.08333, "provider": "digitalocean", "region": "nyc3", "size": "s-4vcpu-8gb-intel" }, "volume": "repro-arxiv-1606-00931-20261018t225505z-work", "warnings": [] }, "environment_event": 60 }, "drivers": [ "operator:claude-opus-5-5 (Claude Code session, no human in the loop)" ], "model_usage": {}, "model_cost_usd": null, "started_at": "2026-10-18T22:55:05.668Z", "ended_at": "2026-10-19T00:51:39.239Z", "log": { "events": 63, "head_hash": "b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f" }, "right_of_reply": "Authors may submit a response, which is published alongside this report unedited." }