Replication record · arXiv:1606.00931

DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network

Jared Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, Yuval Kluger · 2018 · doi:10.1186/s12874-018-0482-1 · arXiv

Reproduced

Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Some claims were registered after command output had already been observed. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 4 of 4 consistent
Other checks: 2 of 2 consistent
Deviations: 2 (0 material)
Integrity flags: 0
Logged compute: 1.84 h over 14 commands and 3 jobs
Primary code (declared before the first recorded result): github.com/jaredleekatzman/DeepSurv (first author; the paper links it) at commit 079dee7, the last commit on or before 2017-08-09, the date of arXiv v3 (the version tested), run with the documented command in experiments/deepsurv/Dockerfile.<experiment> for each experiment
Commits the recorded results name: 079dee7: 6 results, named in the result's method text
Work tree when the attempt ended: jaredleekatzman/DeepSurv@079dee71e3b2 in /work/ds

Protocol

Declared before the first recorded result: primary code is github.com/jaredleekatzman/DeepSurv (first author; the paper links it) at commit 079dee7, the last commit on or before 2017-08-09, the date of arXiv v3 (the version tested), run with the documented command in experiments/deepsurv/Dockerfile.<experiment> for each experiment; earlier versions of the authors' code are not used. arXiv v3 is the PDF fetched and its Table 3 hyper-parameters match the model JSON files at 079dee7 for SUPPORT, METABRIC, GBSG and the two simulated sets. Later commits (2018-01 Merge revisions) belong to the journal revision and use different configs and regenerated simulated data, so they are not the arXiv v3 code. No commit before 079dee7 contains the experiment configs, so there is no earlier-code fallback.

Claims

DeepSurv reaches test C-index 0.618308 on SUPPORT.headline

Consistent

paper 0.618308observed 0.618582

Paper 0.618308 C-indexObserved 0.618582 C-indexAccepted if within ±0.01

SUPPORT 0.582870 (0.581,0.585) 0.618308 (0.616,0.620)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 1 · registered at event 23, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on SUPPORT

Why this tolerance. The paper reports one training run per dataset with a test-set bootstrap CI (half-width about 0.002 to 0.004), which covers test sampling but not training randomness: the code sets no seed, so weight initialisation and dropout masks vary run to run. The observed value is the mean test C-index over 10 independent runs of the documented command; a band of 0.01 C-index allows for training-seed variance plus a reconstructed 2017 environment, and is still narrow enough to separate DeepSurv from the CPH baseline on SUPPORT, WHAS and the simulated nonlinear data.

Event 51. Observed 0.618582, delta +0.000274, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

DeepSurv reaches test C-index 0.643374 on METABRIC.headline

Consistent

paper 0.643374observed 0.643492

Paper 0.643374 C-indexObserved 0.643492 C-indexAccepted if within ±0.01

METABRIC 0.630618 (0.627,0.635) 0.643374 (0.639,0.647)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 1 · registered at event 24, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on METABRIC

Event 52. Observed 0.643492, delta +0.000118, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG.headline

Consistent

paper 0.668402observed 0.672011

Paper 0.668402 C-indexObserved 0.672011 C-indexAccepted if within ±0.01

0.657750 (0.654, 0.661) 0.668402 (0.665,0.671)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 1.005 · registered at event 25, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on Rotterdam (train) and GBSG (test)

Event 53. Observed 0.672011, delta +0.003609, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

DeepSurv reaches test C-index 0.862620 on WHAS.headline

Consistent

paper 0.86262observed 0.858003

Paper 0.86262 C-indexObserved 0.858003 C-indexAccepted if within ±0.01

WHAS 0.817620 (0.814, 0.821) 0.862620 (0.859,0.866)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 0.9946 · registered at event 26, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on WHAS

Event 54. Observed 0.858003, delta -0.004617, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

DeepSurv reaches test C-index 0.774019 on simulated linear data.

Consistent

paper 0.774019observed 0.773885

Paper 0.774019 C-indexObserved 0.773885 C-indexAccepted if within ±0.01

Simulated Linear 0.773677 (0.772,0.775) 0.774019 (0.772,0.776)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 0.9998 · registered at event 27, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on the simulated linear survival data

Event 55. Observed 0.773885, delta -0.000134, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.

Consistent

paper 0.648902observed 0.645797

Paper 0.648902 C-indexObserved 0.645797 C-indexAccepted if within ±0.01

0.506951 (0.505,0.509) 0.648902 (0.647, 0.651)

Table 1, p. 5 (DeepSurv column) · quote verified on p. 5 · equivalence check · observed/published 0.9952 · registered at event 28, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. test-set C-index of DeepSurv on the simulated nonlinear (Gaussian) survival data

Event 56. Observed 0.645797, delta -0.003105, consistent. Evidence events [49] (value matched by the legacy rule; not bound to an excerpt). Authors deepsurv_run.py at commit 079dee7 with the documented Dockerfile command (model JSON, epochs, optimizer) on the bundled data, Python 2.7 / Theano 0.9.0 / Lasagne master / lifelines 0.9.4 on CPU; mean of the test C-index printed by 10 independent unseeded runs (job summary line).

Deviations from the paper's protocol

minor
Python 2.7.15 from conda-forge with numpy 1.13.3, scipy 0.19.1, pandas 0.20.3, h5py 2.7.1 and matplotlib 2.2.3 (only for plots), plus Theano 0.9.0, Lasagne master (0.2.dev1) and lifelines 0.9.4, logger 1.4, tensorboard-logger 0.0.3 from PyPI, on CPU with float32, instead of the floydhub/dl-docker:cpu image.
The floydhub image is not reproducible today and its library versions are not recorded. The Dockerfile pins lifelines 0.9.4, h5py 2.7.0 and matplotlib 2.0.0; h5py 2.7.1 and matplotlib 2.2.3 were the nearest available conda-forge builds for Python 2.7, and matplotlib is used only for figures. event 57
none
The documented commands were run with explicit paths to the model, data and results directories instead of the Docker mounts (/models, /shared/data, /shared/results), with /DeepSurv/deepsurv symlinked to the repository, and the WHAS run used the repository model file at 079dee7 even though it differs from the WHAS hyper-parameters in Table 3.
Path mapping only; no code or hyper-parameter was changed. For GBSG the Dockerfile at 079dee7 misspells the data file (gsbg), and the bundled file gbsg_cancer_train_test.h5 was passed. event 58

Notes from the attempt

event 29
Scope and observations, declared before any run: the six DeepSurv C-index values of Table 1 (four real datasets, two simulated survival sets) are tested; CPH and RSF columns, the simulated-treatment row and the treatment-recommendation results (Table 2) are not scored. Each claim is scored on the mean test C-index over 10 independent runs of the documented command, because the code fixes no seed. At 079dee7 the WHAS model file (whas_model_selu_0.json: 3 SELU layers of 40, LR 0.0238) does not match the WHAS column of the paper Table 3 (2 ReLU layers of 48, LR 0.067); the documented command uses the repository file and is run as is. The GBSG Dockerfile at 079dee7 names the data file gsbg_cancer_train_test.h5 while the file in the repository is gbsg_cancer_train_test.h5 (fixed upstream in 6a085af); the correct path is passed. Paths: the Dockerfiles mount the code at /DeepSurv/deepsurv and data at /shared/data; the same layout is recreated with symlinks and the model, data and results paths are passed as arguments.
event 44
Every run exits non-zero at its very last step (event 42): deepsurv_run.py at 079dee7 rebinds the name model to the JSON file name (line 162) and then calls model.save_weights, which raises AttributeError. The test C-index has already been computed and printed by then, so the scored values are unaffected; only the saving of trained weights fails. Upstream fixed this in a011cd7 (2017-12-07, [Bug] Fix saving model error). The code is run unmodified, so the FAILED lines in the job output are this error.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T22:55:05.668Z → 2026-10-19T00:51:39.239Z
Event log: 63 events, hash-chained; head b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f
Verify: python3 openrepro_verify.py arxiv-1606-00931-20261018T225505Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1).
Right of reply: Authors may submit a response, which is published alongside this report unedited.