Economics · 2020
Reproducibility report: Difference-in-Differences with multiple time periods
4 of 4 headline claims held
late registrationlegacyunregistered extractionThe full answer and its qualifiers
Does Callaway and Sant'Anna's DiD (Callaway and Sant'Anna, 2020) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 19 Oct 2026).
- Some claims were registered after command output had been observed.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
Does Callaway and Sant'Anna's DiD (Callaway and Sant'Anna, 2020) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded (checked 19 Oct 2026).
- Checked
- Released
- Assurance
- Legacy
- Evidence
- Full report · report.json
- Data
- JSON record · How to cite
- Assistants
- Ask via MCP
Does Callaway and Sant'Anna's DiD reproduce?
Yes, on its headline claims. 4 of 4 headline claims held within tolerances registered before results were recorded.
- Some claims were registered after command output had been observed.
- Legacy attempt: the plan was never sealed; claims were frozen only by their first result.
- At least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
- Held (large: headline claim; small: other claim)
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| Under conditional parallel trends with the doubly robust estimator, the simple weighted average of post-treatment ATT(g,t) is -0.033. (headline) | -0.033 | -0.033117 | within ±0.0005 | -0.23 | Held |
| The clustered multiplier-bootstrap standard error of the conditional DR simple weighted average is 0.007. | 0.007 | 0.006506 | within ±0.00155 | -0.32 | Held |
| Data identity check: treated counties have mean 2000 population of 94.32 thousand in the main dataset. | 94.32 | 94.3213 | within ±0.005 | +0.26 | Held |
| Under unconditional parallel trends, the group-time ATT for the 2004 group (Illinois) in 2007 is -0.136. (headline) | -0.136 | -0.135614 | within ±0.0005 | +0.77 | Held |
| Under unconditional parallel trends, the event-study single parameter (average of e=0..3 dynamic effects) is -0.090. (headline) | -0.09 | -0.08981 | within ±0.0005 | +0.38 | Held |
| Under unconditional parallel trends with never-treated controls, the simple (group-size weighted) average of post-treatment ATT(g,t) is -0.052 log points of teen employment. (headline) | -0.052 | -0.051581 | within ±0.0005 | +0.84 | Held |
| The clustered multiplier-bootstrap standard error of the unconditional simple weighted average is 0.006. | 0.006 | 0.006595 | within ±0.0014 | +0.43 | Held |
What the attempt found
The minimum-wage application (Section 5, Tables 2-3) was rerun with the did package at commit 5e32efd (the last commit before arXiv v4), R 4.0.3 and 2020-12-01 CRAN dependencies, on the 2284-county 2001-2007 panel that co-author Sant'Anna publishes in the CS_RR repository (the bundled mpdta is only a 500-county subset). All 7 pre-registered claims landed within tolerance: the unconditional simple average (-0.051581 vs -0.052), event-study average (-0.089810 vs -0.090), ATT(2004,2007) (-0.135614 vs -0.136), the conditional doubly robust simple average (-0.033117 vs -0.033), both bootstrap SEs, and the Table 2 treated population mean. A post-hoc, unscored comparison found that Table 3's calendar-time rows mix two weightings: Panel (a) matches the paper's eq. (3.8) group weights, while Panel (b) matches the unweighted mean the 2020 package computed.
Assurance
- Level
- Legacy (what the levels mean)
- Plan
- Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
- Evidence
- Legacy matching: each observed value was matched against the text of the logged output it cites
- Execution manifest
- No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
- Predates
- plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| Under conditional parallel trends with the doubly robust estimator, the simple weighted average of post-treatment ATT(g,t) is -0.033.¶Table 3 Panel (b), p. 30 · registered at event 23 · quote verified on p. 30 · equivalence check | -0.033log points | -0.033117 | within ±0.0005 | Held |
| Under unconditional parallel trends, the group-time ATT for the 2004 group (Illinois) in 2007 is -0.136.¶Section 5.1, p. 28 (also Table 3 Panel (a) Event Study e=3, p. 30) · registered at event 26 · quote verified on p. 28 · equivalence check | -0.136log points | -0.135614 | within ±0.0005 | Held |
| Under unconditional parallel trends, the event-study single parameter (average of e=0..3 dynamic effects) is -0.090.¶Table 3 Panel (a), row 'Event Study', Single Parameters column, p. 30 · registered at event 27 · quote verified on p. 30 · equivalence check | -0.09log points | -0.08981 | within ±0.0005 | Held |
| Under unconditional parallel trends with never-treated controls, the simple (group-size weighted) average of post-treatment ATT(g,t) is -0.052 log points of teen employment.¶Section 5.1 p. 28; Table 3 Panel (a) 'Simple Weighted Average', p. 30 · registered at event 28 · quote verified on p. 28 · equivalence check | -0.052log points | -0.051581 | within ±0.0005 | Held |
Other claims checked
3 claims: 3 held
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| The clustered multiplier-bootstrap standard error of the conditional DR simple weighted average is 0.007.¶Table 3 Panel (b), p. 30 · registered at event 24 · quote verified on p. 30 · equivalence check | 0.007log points | 0.006506 | within ±0.00155 | Held |
| Data identity check: treated counties have mean 2000 population of 94.32 thousand in the main dataset.¶Table 2, p. 28 · registered at event 25 · quote verified on p. 28 · equivalence check | 94.32thousands | 94.3213 | within ±0.005 | Held |
| The clustered multiplier-bootstrap standard error of the unconditional simple weighted average is 0.006.¶Table 3 Panel (a), p. 30 · registered at event 29 · quote verified on p. 30 · equivalence check | 0.006log points | 0.006595 | within ±0.0014 | Held |
Deviations from the paper's setup
5 recorded: 2 minor, 3 none
- minorApplication data taken from data/min_wage_CS.rds in github.com/pedrohcgs/CS_RR (co-author repository, file added 2022-01-02), not from a replication package dated with the paper.
- noneEnvironment: R 4.0.3 (Posit build) on x86_64 Debian 13 with GCC 14 and R reference BLAS, on a 4-vCPU DigitalOcean droplet; packages compiled from source with build flags only (/env/Makevars.gcc14: -include cstddef -include cstdint for C++, -Wno-error for three C diagnostics). No source file of did or any dependency was changed.
- nonetestthat 3.0.4 and isoband 0.2.5 came from the P3M snapshot of 2021-08-01; every other dependency came from the 2020-12-01 snapshot.
- minorBootstrap standard errors use our own seed (20201201) with the package default 1000 multiplier draws; the paper's seed is not reported.
- noneThe TWFE row of Table 3 and the Supplementary Appendix results (not-yet-treated controls, anticipation) were not attempted.
The run
- Wall time
- 29 min, 19 October 2026 to 19 October 2026
- Job compute
- 17 min 51 s across 2 jobs
- Work tree at the end
- pedrohcgs/CS_RR at a9a25d4
bcallaway11/did at 5e32efd - Container image
- repro-sandbox:0.2 sha256:23ad4ea1cf8d…
- Sandbox
- 3.5 CPUs, 6 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-1803-09015-20261019T161404Z
- Log
- 59 events, head 48f31184
Evidence
log seal, 59 hash-chained events
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 59 hash-chained events
48f311848a704e6fdadc0137322dca91d96cfc105c4e13706dc6e39dbaa4c0d7
How to cite this report
OpenRepro (2026). Reproducibility report: Difference-in-Differences with multiple time periods. Reproduced; 4 of 4 headline claims held (some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1803-09015-20261019T161404Z, log head 48f311848a704e6f, checked 19 Oct 2026. https://openrepro.org/papers/1803.09015/
APA style
OpenRepro. (2026). Reproducibility report: Difference-in-Differences with multiple time periods (Attempt arxiv-1803-09015-20261019T161404Z; log head 48f311848a704e6f). https://openrepro.org/papers/1803.09015/
BibTeX
@techreport{openrepro-1803.09015,
author = {{OpenRepro}},
title = {Reproducibility report: Difference-in-Differences with multiple time periods},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/1803.09015/},
note = {Attempt arxiv-1803-09015-20261019T161404Z; log head 48f311848a704e6f; checked 19 Oct 2026}
}
References
1 paper
- Brantly Callaway, Pedro H. C. Sant'Anna (2020). Difference-in-Differences with multiple time periods. Journal of Econometrics. arXiv 1803.09015 · doi:10.1016/j.jeconom.2020.12.001
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
Copy the badge as Markdown or HTML
[](https://openrepro.org/papers/1803.09015/)<a href="https://openrepro.org/papers/1803.09015/"><img src="https://openrepro.org/badge/1803.09015.svg" alt="OpenRepro: Reproduced. 4 of 4 headline claims held; some claims were registered after command output had been observed; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="517" height="20"></a>Right of reply, and reporting an error
Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.