Replication record · arXiv:1803.09015

Difference-in-Differences with multiple time periods

Brantly Callaway, Pedro H. C. Sant'Anna · 2020 · doi:10.1016/j.jeconom.2020.12.001 · arXiv

Reproduced

The minimum-wage application (Section 5, Tables 2-3) was rerun with the did package at commit 5e32efd (the last commit before arXiv v4), R 4.0.3 and 2020-12-01 CRAN dependencies, on the 2284-county 2001-2007 panel that co-author Sant'Anna publishes in the CS_RR repository (the bundled mpdta is only a 500-county subset). All 7 pre-registered claims landed within tolerance: the unconditional simple average (-0.051581 vs -0.052), event-study average (-0.089810 vs -0.090), ATT(2004,2007) (-0.135614 vs -0.136), the conditional doubly robust simple average (-0.033117 vs -0.033), both bootstrap SEs, and the Table 2 treated population mean. A post-hoc, unscored comparison found that Table 3's calendar-time rows mix two weightings: Panel (a) matches the paper's eq. (3.8) group weights, while Panel (b) matches the unweighted mean the 2020 package computed.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Some claims were registered after command output had already been observed. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 4 of 4 consistent
Other checks: 3 of 3 consistent
Deviations: 5 (0 material)
Integrity flags: 0
Logged compute: 0.32 h over 15 commands and 2 jobs
Primary code (declared before the first recorded result): github.com/bcallaway11/did at commit 5e32efd9675578f4662aacfb90695dfd0906c4f0 (2020-11-30T22:37:02Z, DESCRIPTION Version 2.0.0), the last commit on or before the tested arXiv version v4 (updated 2020-12-01T16:15:59Z per the arXiv API). Installed from source with R 4.0.3 (Posit build) and CRAN dependencies from the P3M snapshot of 2020-12-01. Data: data/min_wage_CS.rds from github.com/pedrohcgs/CS_RR (commit a9a25d4f), published by co-author Pedro Sant'Anna and described there as the data used in the Callaway and Sant'Anna (2021) application.
Work tree when the attempt ended: pedrohcgs/CS_RR@a9a25d4f7672 in /work/CS_RR
Work tree when the attempt ended: bcallaway11/did@5e32efd96755 in /work/did

Protocol

Declared before the first recorded result: primary code is github.com/bcallaway11/did at commit 5e32efd9675578f4662aacfb90695dfd0906c4f0 (2020-11-30T22:37:02Z, DESCRIPTION Version 2.0.0), the last commit on or before the tested arXiv version v4 (updated 2020-12-01T16:15:59Z per the arXiv API). Installed from source with R 4.0.3 (Posit build) and CRAN dependencies from the P3M snapshot of 2020-12-01. Data: data/min_wage_CS.rds from github.com/pedrohcgs/CS_RR (commit a9a25d4f), published by co-author Pedro Sant'Anna and described there as the data used in the Callaway and Sant'Anna (2021) application.; earlier versions of the authors' code may be tried as a fallback. The paper's Table 3 was produced by the did package; the rule 'closest release or commit not after the arXiv version' selects 5e32efd, a pre-release commit of did 2.0.0 (CRAN release 2020-12-11, 10 days after v4). The last CRAN release before v4 is did 1.2.3 (2020-02-18, tag v1.2.3, commit 0cc3a89); it may be tried as a declared fallback only for claims whose primary result is inconsistent, and would never replace the primary result.

Claims

Under conditional parallel trends with the doubly robust estimator, the simple weighted average of post-treatment ATT(g,t) is -0.033.headline

Consistent

paper -0.033observed -0.033117

Paper -0.033 log pointsObserved -0.033117 log pointsAccepted if within ±0.0005

Simple Weighted Average -0.033

Table 3 Panel (b), p. 30 · quote verified on p. 30 · equivalence check · observed/published 1.004 · registered at event 23, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. aggte(type='simple') overall ATT, DR estimator, covariates region, pop, medinc, pop^2, medinc^2, white, hs, pov

Why this tolerance. Point estimates are deterministic given the data and code (the bootstrap does not affect them); the only allowed gap is the paper's rounding to 3 decimals, so tolerance = 0.0005. The DR first steps (logit propensity score, linear outcome regression) are deterministic.

Event 44. Observed -0.033117, delta -0.000117, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, xformla ~ region + pop + medinc + I(pop^2) + I(medinc^2) + white + hs + pov; aggte(type='simple') overall.att.

The clustered multiplier-bootstrap standard error of the conditional DR simple weighted average is 0.007.

Consistent

paper 0.007observed 0.006506

Paper 0.007 log pointsObserved 0.006506 log pointsAccepted if within ±0.00155

Simple Weighted Average -0.033 (0.007)

Table 3 Panel (b), p. 30 · quote verified on p. 30 · equivalence check · observed/published 0.9294 · registered at event 24, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. bootstrap standard error of aggte(type='simple'), DR conditional PT

Why this tolerance. As for uncond-simple-se: 15% of the paper value (0.00105) for bootstrap randomness plus 0.0005 for 3-decimal rounding.

Event 46. Observed 0.006506, delta -0.000494, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, DR with covariates; aggte(type='simple') overall.se (bootstrap).

Data identity check: treated counties have mean 2000 population of 94.32 thousand in the main dataset.

Consistent

paper 94.32observed 94.3213

Paper 94.32 thousandsObserved 94.3213 thousandsAccepted if within ±0.005

Population (1000s) 94.32 53.43 40.89 0.00

Table 2, p. 28 · quote verified on p. 28 · equivalence check · observed/published 1 · registered at event 25, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. mean of pop across counties with first.treat > 0 (one row per county)

Why this tolerance. A descriptive mean of fixed data; only the paper's 2-decimal rounding (0.005) is allowed. This checks the released file is the paper's sample.

Event 47. Observed 94.3213, delta +0.0013, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). Mean of pop over the 907 counties with first.treat > 0, one row per county, in /work/run_table3.R.

Under unconditional parallel trends, the group-time ATT for the 2004 group (Illinois) in 2007 is -0.136.headline

Consistent

paper -0.136observed -0.135614

Paper -0.136 log pointsObserved -0.135614 log pointsAccepted if within ±0.0005

In 2005, teen employment is estimated to be 7.1% lower; in 2006, 12.5% lower; and in 2007, 13.6% lower.

Section 5.1, p. 28 (also Table 3 Panel (a) Event Study e=3, p. 30) · quote verified on p. 28 · equivalence check · observed/published 0.9972 · registered at event 26, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. att_gt ATT(g=2004, t=2007), unconditional PT

Event 43. Observed -0.135614, delta +0.000386, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, xformla ~1; att_gt estimate for g=2004, t=2007.

Under unconditional parallel trends, the event-study single parameter (average of e=0..3 dynamic effects) is -0.090.headline

Consistent

paper -0.09observed -0.08981

Paper -0.09 log pointsObserved -0.08981 log pointsAccepted if within ±0.0005

-0.027 -0.071 -0.125 -0.136 -0.090

Table 3 Panel (a), row 'Event Study', Single Parameters column, p. 30 · quote verified on p. 30 · equivalence check · observed/published 0.9979 · registered at event 27, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. aggte(type='dynamic') overall ATT, unconditional PT

Event 42. Observed -0.08981, delta +0.00019, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, xformla ~1; aggte(type='dynamic', min_e=0) overall.att.

Under unconditional parallel trends with never-treated controls, the simple (group-size weighted) average of post-treatment ATT(g,t) is -0.052 log points of teen employment.headline

Consistent

paper -0.052observed -0.051581

Paper -0.052 log pointsObserved -0.051581 log pointsAccepted if within ±0.0005

The simple average (weighted only by group size) is 5.2% lower teen employment

Section 5.1 p. 28; Table 3 Panel (a) 'Simple Weighted Average', p. 30 · quote verified on p. 28 · equivalence check · observed/published 0.9919 · registered at event 28, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. aggte(type='simple') overall ATT, unconditional PT, never-treated comparison

Event 41. Observed -0.051581, delta +0.000419, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, xformla ~1; aggte(type='simple') overall.att.

The clustered multiplier-bootstrap standard error of the unconditional simple weighted average is 0.006.

Consistent

paper 0.006observed 0.006595

Paper 0.006 log pointsObserved 0.006595 log pointsAccepted if within ±0.0014

Simple Weighted Average -0.052 (0.006)

Table 3 Panel (a), p. 30 · quote verified on p. 30 · equivalence check · observed/published 1.099 · registered at event 29, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. bootstrap standard error of aggte(type='simple'), unconditional PT

Why this tolerance. Bootstrap SEs are random (paper seed unknown); with 1000 multiplier draws the SE estimate itself has Monte Carlo noise of a few percent. Allow 15% of the paper value (0.0009) plus the paper's 3-decimal rounding (0.0005).

Event 45. Observed 0.006595, delta +0.000595, consistent. Evidence events [35] (value matched by the legacy rule; not bound to an excerpt). did @ 5e32efd att_gt (never-treated controls, est_method dr, 1000-draw multiplier bootstrap clustered by county, seed 20201201) on min_wage_CS.rds, run by /work/run_table3.R, xformla ~1; aggte(type='simple') overall.se (bootstrap).

Deviations from the paper's protocol

minor
Application data taken from data/min_wage_CS.rds in github.com/pedrohcgs/CS_RR (co-author repository, file added 2022-01-02), not from a replication package dated with the paper.
The paper's 2284-county 2001-2007 panel is not shipped with did (mpdta is a 500-county 2003-2007 subset). The CS_RR file is published by co-author Sant'Anna as the data used in the application and matches the paper's sample size, years, groups and the Table 2 population mean (event 47), but it postdates arXiv v4, so exact identity with the file behind Table 3 cannot be proven. event 48
none
Environment: R 4.0.3 (Posit build) on x86_64 Debian 13 with GCC 14 and R reference BLAS, on a 4-vCPU DigitalOcean droplet; packages compiled from source with build flags only (/env/Makevars.gcc14: -include cstddef -include cstdint for C++, -Wno-error for three C diagnostics). No source file of did or any dependency was changed.
The authors' platform (a 2.8 GHz i5 laptop, per Section 5) and R version are not stated; R 4.0.3 was the current release on 2020-12-01. The build flags are needed because GCC 14 is stricter than 2020 compilers. event 49
none
testthat 3.0.4 and isoband 0.2.5 came from the P3M snapshot of 2021-08-01; every other dependency came from the 2020-12-01 snapshot.
testthat 3.0.0 in the 2020-12-01 snapshot vendors a catch.h that uses SIGSTKSZ as a compile-time constant, which glibc >= 2.34 no longer provides (event 18); isoband links to it. Neither package enters the estimation path (isoband is a ggplot2 contour dependency). event 50
minor
Bootstrap standard errors use our own seed (20201201) with the package default 1000 multiplier draws; the paper's seed is not reported.
Bootstrap SEs are random draws; they are scored with a loose pre-registered tolerance while point estimates are scored at rounding precision. event 51
none
The TWFE row of Table 3 and the Supplementary Appendix results (not-yet-treated controls, anticipation) were not attempted.
TWFE is not produced by the did package, and the commit 5e32efd predates did's anticipation argument; the main-text never-treated, no-anticipation specification is the one tested. event 52

Notes from the attempt

event 22
Data decision. The paper application (Section 5, Table 2 note) uses 2284 counties observed 2001-2007. The did package bundled mpdta is a 500-county subset for 2003-2007 and cannot reproduce Table 3. The co-author Pedro Sant Anna publishes data/min_wage_CS.rds in github.com/pedrohcgs/CS_RR (added 2022-01-02 in commit 9d66424, file sha256 ba4497c4...f61a7), introduced there as the data used in Callaway and Sant Anna (2021) application. Event 21 shows it has 2284 counties x 7 years (2001-2007), first.treat groups 0/2004/2006/2007, and the covariates named in the paper (region, pop, medinc, white, hs, pov). It is therefore author-distributed data that can test Table 3; claims are registered against it. Caveat recorded as a deviation: it was released 13 months after arXiv v4 in a separate repository, so it is not provably the exact file behind Table 3; a Table 2 summary statistic is registered as a data-identity check. The simulation tables were not used because no simulation code is distributed by the authors in did or the other author repositories listed in events 9-10.
event 53
Unregistered context (event 35): beyond the 7 registered claims, every other Table 3 point estimate produced by the did package was compared to 3 decimals. Panel (a) group effects (-0.091, -0.047, -0.028, overall -0.039), event study e=0..3, and balanced event study (-0.027, -0.071, -0.049) all match; all of Panel (b) matches, including group, event-study, calendar and balanced rows, and the group-time range in the text (0.9% to 7.1% lower). The bootstrap SE of the unconditional simple average, 0.006595, rounds to 0.007 against the paper's 0.006, within the pre-registered SE tolerance. The did package at 5e32efd has no anticipation argument; the main-text specification uses none.
event 54
Post-hoc finding (unregistered, events 35-40): Table 3's calendar-time rows use two different weightings. Panel (a) calendar effects (-0.055 in 2006, -0.050 in 2007, overall -0.052) match eq. (3.8)'s P(G=g|G<=t) group weights (our -0.054873, -0.050267, overall -0.052396). Panel (b) calendar effects (-0.030, -0.049, overall -0.033) match an unweighted mean across groups (our -0.029554, -0.049269, overall -0.033440), which is what did at 5e32efd and the CRAN 2.0.0 release compute (R/compute.aggte.R: mean(attt)). Under eq. (3.8) weights Panel (b) would read -0.022 (2006), -0.038 (2007), overall -0.029. The package switched to pg-weighted calendar effects in v2.1 (2021-12). Running did 5e32efd on Panel (a) gives -0.074, -0.078, overall -0.064 for the calendar row. Not scored; reported for the authors' right of reply.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-19T16:14:04.610Z → 2026-10-19T16:43:04.824Z
Event log: 59 events, hash-chained; head 48f311848a704e6fdadc0137322dca91d96cfc105c4e13706dc6e39dbaa4c0d7
Verify: python3 openrepro_verify.py arxiv-1803-09015-20261019T161404Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.