Reproducibility report: DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network

OpenRepro

Medical statistics · 2018

DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network

Does DeepSurv (Katzman et al., 2018) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).

Jared Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, Yuval Kluger. BMC Medical Research Methodology, 2018. arXiv 1606.00931 · doi:10.1186/s12874-018-0482-1

Does “DeepSurv” reproduce?

Reproduced

Yes, on its headline claims. 4 of 4 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

How far the re-run landed from the published number, in tolerances6 claims re-run, 4 headline

Held (large: headline claim; small: other claim)

The data behind this chart

Claim	Published	Observed	Accepted if	Deviation ÷ tolerance	Result
DeepSurv reaches test C-index 0.618308 on SUPPORT. (headline)	0.618308	0.618582	within ±0.01	+0.03	Held
DeepSurv reaches test C-index 0.643374 on METABRIC. (headline)	0.643374	0.643492	within ±0.01	+0.01	Held
DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG. (headline)	0.668402	0.672011	within ±0.01	+0.36	Held
DeepSurv reaches test C-index 0.862620 on WHAS. (headline)	0.86262	0.858003	within ±0.01	-0.46	Held
DeepSurv reaches test C-index 0.774019 on simulated linear data.	0.774019	0.773885	within ±0.01	-0.01	Held
DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.	0.648902	0.645797	within ±0.01	-0.31	Held

Headline claims are the results the paper is chiefly known for; they decide the outcome. Other claims are reported alongside.

Registration. Claims and tolerances registered at events 23–28; first result recorded at event 51. Registration came before results were recorded, which does not show that no output had been seen (a legacy attempt: no sealed plan). Event numbers refer to the attempt's log.

Checked 19 October 2026.

How this report was made. The re-run was carried out by an AI agent (Claude, made by Anthropic) in a sandbox. Each run is driven by the AI with no person in the loop, and the log's actor field records this. A person oversees the project. No domain expert has reviewed this report. The authors have a right of reply, and their notes can be published alongside it.

What the attempt found

Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored.

Assurance

Level: Legacy (what the levels mean)
Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites
Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
Report re-derived from the log: Passed: the engine re-derived the report from the log
Engine version: 0.2.1
Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Qualifiers: some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it

Headline claims

These decide the outcome. Values are as published and as observed by re-running the code; the accepted-if rule was registered before results were recorded. Each row has a ¶ link of its own.

Claim	Published	Observed	Accepted if	Result
DeepSurv reaches test C-index 0.618308 on SUPPORT.¶Table 1, p. 5 (DeepSurv column) · registered at event 23 · quote verified on p. 5 · equivalence check	0.618308C-index	0.618582	within ±0.01	Held
DeepSurv reaches test C-index 0.643374 on METABRIC.¶Table 1, p. 5 (DeepSurv column) · registered at event 24 · quote verified on p. 5 · equivalence check	0.643374C-index	0.643492	within ±0.01	Held
DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG.¶Table 1, p. 5 (DeepSurv column) · registered at event 25 · quote verified on p. 5 · equivalence check	0.668402C-index	0.672011	within ±0.01	Held
DeepSurv reaches test C-index 0.862620 on WHAS.¶Table 1, p. 5 (DeepSurv column) · registered at event 26 · quote verified on p. 5 · equivalence check	0.86262C-index	0.858003	within ±0.01	Held

Other claims checked

Reported alongside the headline claims; they do not change the outcome.

Claim	Published	Observed	Accepted if	Result
DeepSurv reaches test C-index 0.774019 on simulated linear data.¶Table 1, p. 5 (DeepSurv column) · registered at event 27 · quote verified on p. 5 · equivalence check	0.774019C-index	0.773885	within ±0.01	Held
DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.¶Table 1, p. 5 (DeepSurv column) · registered at event 28 · quote verified on p. 5 · equivalence check	0.648902C-index	0.645797	within ±0.01	Held

Deviations from the paper's setup

minorPython 2.7.15 from conda-forge with numpy 1.13.3, scipy 0.19.1, pandas 0.20.3, h5py 2.7.1 and matplotlib 2.2.3 (only for plots), plus Theano 0.9.0, Lasagne master (0.2.dev1) and lifelines 0.9.4, logger 1.4, tensorboard-logger 0.0.3 from PyPI, on CPU with float32, instead of the floydhub/dl-docker:cpu image.
noneThe documented commands were run with explicit paths to the model, data and results directories instead of the Docker mounts (/models, /shared/data, /shared/results), with /DeepSurv/deepsurv symlinked to the repository, and the WHAS run used the repository model file at 079dee7 even though it differs from the WHAS hyper-parameters in Table 3.

The run

Wall time: 1 h 56 min, 18 October 2026 to 19 October 2026
Job compute: 1 h 48 min across 3 jobs
Code the results name: jaredleekatzman/DeepSurv at 079dee7
Work tree at the end: jaredleekatzman/DeepSurv at 079dee7
Container image: repro-sandbox:0.2 sha256:f4b241036e33…
Sandbox: 3.5 CPUs, 6 GB, no GPU
Run by: Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt: arxiv-1606-00931-20261018T225505Z
Log: 63 events, head b6c148f9

Evidence

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 63 hash-chained events

b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f

The head hash of the attempt's append-only log. The engine generated this report from that log; the verifier checks the log and its artifacts against this hash and that the report names it, but not the report's contents. Download the evidence bundle (the log, every artifact and the report; SHA-256 a3ec5893321c321b…) and check it with the verifier, which needs only Python.

Released 19 October 2026. No author notice was sent. Reply status: No courtesy note: no author address is printed in the arXiv version and the journal page needs a login. No reply on record.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. Reproduced; 4 of 4 headline claims held (some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1606-00931-20261018T225505Z, log head b6c148f9a6936f84, checked 19 Oct 2026. https://openrepro.org/papers/1606.00931/

BibTeX

@techreport{openrepro-1606.00931,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/1606.00931/},
  note        = {Attempt arxiv-1606-00931-20261018T225505Z; log head b6c148f9a6936f84; checked 19 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

Badge

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Markdown

[![OpenRepro: Reproduced. 4 of 4 headline claims held; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.](https://openrepro.org/badge/1606.00931.svg)](https://openrepro.org/papers/1606.00931/)

HTML

<a href="https://openrepro.org/papers/1606.00931/"><img src="https://openrepro.org/badge/1606.00931.svg" alt="OpenRepro: Reproduced. 4 of 4 headline claims held; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="517" height="20"></a>

Right of reply, and reporting an error

Authors may submit a response, which is published alongside this report unedited. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.