Replication record · arXiv:2004.01105

Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions

Jonas Dehning, Johannes Zierenberg, F. Paul Spitzner, Michael Wibral, João Pinheiro Neto, Michael Wilczek, Viola Priesemann · 2020 · doi:10.1126/science.abb9789 · arXiv

Reproduced

Fitting the authors three-change-point SIR model with weekly reporting modulation (repository state on the date of arXiv v3) to German case data until 21 April 2020, using the JHU snapshot bundled with the repository, reproduces all registered values: spreading rates 0.43, 0.25, 0.15 and 0.09 (posterior medians 0.428, 0.251, 0.154, 0.094) and change points around March 6, 15 and 23 (posterior medians March 5.8, 15.2 and 22.6). The credible intervals also match Fig. 3 closely. Only the main three-change-point model was refitted; the 0-2 change-point comparisons and forecasts were not.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 4 of 4 consistent
Other checks: 3 of 3 consistent
Deviations: 2 (0 material)
Integrity flags: 0
Logged compute: 0.84 h over 4 commands and 2 jobs
Primary code (declared before the first recorded result): github.com/Priesemann-Group/covid19_inference_forecast (linked in the paper) at the last commit on or before 2020-05-04 (arXiv v3), using its Germany three-change-point model with the data end date pinned to 2020-04-21 and the earliest JHU snapshot available in the repository at that commit, rather than the current, back-revised JHU series
Commits the recorded results name: 2960f286: 7 results, named in the result's method text
Work tree when the attempt ended: Priesemann-Group/covid19_inference_forecast@2960f286f4ed in /work/cif

Protocol

Declared before the first recorded result: primary code is github.com/Priesemann-Group/covid19_inference_forecast (linked in the paper) at the last commit on or before 2020-05-04 (arXiv v3), using its Germany three-change-point model with the data end date pinned to 2020-04-21 and the earliest JHU snapshot available in the repository at that commit, rather than the current, back-revised JHU series; earlier versions of the authors' code may be tried as a fallback. The paper results state they use data until April 21. If a claim misses with the paper-date code, earlier commits may be tried as a declared fallback and are reported separately.

Claims

Before the first change point the spreading rate in Germany was 0.43 per day.headline

Consistent

paper 0.43observed 0.4275

Paper 0.43 per dayObserved 0.4275 per dayAccepted if within ±0.02

(95 % credible interval (CI: [0.35, 0.51]))

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 0.9942 · registered at event 7, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median of lambda_0, three-change-point model, data until 2020-04-21

Why this tolerance. Posterior medians from MCMC vary between runs, and the case series may differ slightly from the one the authors downloaded; 0.02 is about a third of the reported credible-interval half-widths.

Event 31. Observed 0.4275, delta -0.0025, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

After the first change point the spreading rate was 0.25 per day.headline

Consistent

paper 0.25observed 0.2509

Paper 0.25 per dayObserved 0.2509 per dayAccepted if within ±0.02

to λ1 = 0.25 (CI: [0.20, 0.30]), and occurred around March 6

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 1.004 · registered at event 8, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median of lambda_1

Event 32. Observed 0.2509, delta +0.0009, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

After the second change point the spreading rate was 0.15 per day.headline

Consistent

paper 0.15observed 0.1544

Paper 0.15 per dayObserved 0.1544 per dayAccepted if within ±0.02

the second change point resulted in λ2 = 0.15 (CI: [0.12, 0.20]), and occurred around March 15

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 1.029 · registered at event 9, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median of lambda_2

Event 33. Observed 0.1544, delta +0.0044, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

After the third change point the spreading rate was 0.09 per day.headline

Consistent

paper 0.09observed 0.0937

Paper 0.09 per dayObserved 0.0937 per dayAccepted if within ±0.02

the third change point that resulted in λ3 = 0.09 (CI: [0.06, 0.12]) around March 23

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 1.041 · registered at event 10, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median of lambda_3

Event 34. Observed 0.0937, delta +0.0037, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

The first change point occurred around March 6, 2020.

Consistent

paper 6observed 5.79

Paper 6 day of MarchObserved 5.79 day of MarchAccepted if within ±2

to λ1 = 0.25 (CI: [0.20, 0.30]), and occurred around March 6

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 0.965 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median date of change point 1, as day of March 2020

Why this tolerance. Change-point dates are posterior medians in days; 2 days is within the reported credible intervals (about plus or minus 2-4 days).

Event 35. Observed 5.79, delta -0.21, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

The second change point occurred around March 15, 2020.

Consistent

paper 15observed 15.22

Paper 15 day of MarchObserved 15.22 day of MarchAccepted if within ±2

the second change point resulted in λ2 = 0.15 (CI: [0.12, 0.20]), and occurred around March 15

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 1.015 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median date of change point 2, as day of March 2020

Why this tolerance. Change-point dates are posterior medians in days; 2 days is within the reported credible intervals (about plus or minus 2-4 days).

Event 36. Observed 15.22, delta +0.22, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

The third change point occurred around March 23, 2020.

Consistent

paper 23observed 22.63

Paper 23 day of MarchObserved 22.63 day of MarchAccepted if within ±2

the third change point that resulted in λ3 = 0.09 (CI: [0.06, 0.12]) around March 23

Abstract, p. 1 · quote verified on p. 1 · equivalence check · observed/published 0.9839 · registered at event 13, before the first recorded result

Method, tolerance rationale and evidence

Metric. posterior median date of change point 3, as day of March 2020

Why this tolerance. Change-point dates are posterior medians in days; 2 days is within the reported credible intervals (about plus or minus 2-4 days).

Event 37. Observed 22.63, delta -0.37, consistent. Evidence events [29] (value matched by the legacy rule; not bound to an excerpt). Authors model code at commit 2960f286 (2020-05-04), the paper notebook model cell (three change points, abs_sine weekly modulation) with data until 2020-04-21 from the bundled 2020-04-28 JHU snapshot; PyMC3 3.8, init=advi, 2000 draws, 2 chains; posterior median printed by run_dehning.py.

Deviations from the paper's protocol

minor
Data end pinned to 2020-04-21 and data taken from the repository bundled JHU snapshot (confirmed_global_fallback_2020-04-28.csv) instead of the live JHU download; only the three-change-point model is fitted; pm.sample uses a fixed random_seed.
The notebook uses the last date of the live JHU series, which is now March 2023 and back-revised; the paper states data until April 21. The 0-2 change-point models in the notebook loop do not enter the reported lambda values. The seed only makes this run repeatable. event 24
minor
Python 3.7 with PyMC3 3.8, Theano 1.0.4, numpy 1.18.5, pandas 1.0.5, scipy 1.4.1 and arviz 0.7.0 from conda-forge, CPU only.
The repository pins no versions; these are the releases current in May 2020. event 39

Notes from the attempt

event 38
The paper itself reports the lambda_3 interval two ways: [0.06, 0.12] in the abstract and [0.06, 0.13] in Fig. 3 and the Results text (p. 7-8). This run gives [0.063, 0.139]. The reproduced intervals and change-point times are also close to Fig. 3 (e.g. t1 5.8 [2.5, 8.9] vs 5.7 [2.5, 8.8]; t3 22.6 [20.9, 24.4] vs 22.6 [20.8, 24.5]); these were not registered claims.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T03:30:25.492Z → 2026-10-18T04:24:05.955Z
Event log: 44 events, hash-chained; head 90e4d597839451c42440893ea4dea7451ef11eaedc3887da05aa3c80dcc11e78
Verify: python3 openrepro_verify.py arxiv-2004-01105-20261018T033025Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1).
Right of reply: Authors may submit a response, which is published alongside this report unedited.