Reproducibility report: Synthetic Difference-in-Differences

OpenRepro

Economics · 2021

Synthetic Difference-in-Differences

Does Synthetic Difference-in-Differences (Arkhangelsky et al., 2021) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

Dmitry Arkhangelsky, Susan Carleton Athey, David A. Hirshberg, Guido W. Imbens, Stefan Wager. American Economic Review, 2021. arXiv 1812.09970 · doi:10.1257/aer.20190159

Does “Synthetic Difference-in-Differences” reproduce?

Reproduced

Yes, on its headline claims. 4 of 4 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

How far the re-run landed from the published number, in tolerances10 claims re-run, 4 headline

Held (large: headline claim; small: other claim)

The data behind this chart

Claim	Published	Observed	Accepted if	Deviation ÷ tolerance	Result
The SDID estimate of the average effect of Proposition 99 on California per-capita cigarette sales over the 12 post-treatment years is -15.6 packs. (headline)	-15.6	-15.6038	within ±0.1	-0.04	Held
The placebo-method standard error of the SDID estimate is 8.4. (headline)	8.4	8.36799	within ±0.84	-0.04	Held
The synthetic control estimate is -19.6. (headline)	-19.6	-19.6197	within ±0.1	-0.20	Held
The difference-in-differences estimate is -27.3. (headline)	-27.3	-27.3491	within ±0.1	-0.49	Held
The matrix completion estimate is -20.2.	-20.2	-20.2181	within ±0.5	-0.04	Held
The synthetic control with intercept (DIFP) estimate is -11.1.	-11.1	-11.1046	within ±0.1	-0.05	Held
The placebo-method standard error of the SC estimate is 9.9.	9.9	9.91743	within ±0.99	+0.02	Held
The placebo-method standard error of the DID estimate is 17.7.	17.7	17.7403	within ±1.77	+0.02	Held
The placebo-method standard error of the MC estimate is 11.5.	11.5	11.5099	within ±1.15	+0.01	Held
The placebo-method standard error of the DIFP estimate is 9.5.	9.5	9.45566	within ±0.95	-0.05	Held

Headline claims are the results the paper is chiefly known for; they decide the outcome. Other claims are reported alongside.

Registration. Claims and tolerances registered at events 9–18; first result recorded at event 45. Registration came before results were recorded, which does not show that no output had been seen (a legacy attempt: no sealed plan). Event numbers refer to the attempt's log.

Checked 18 October 2026.

How this report was made. The re-run was carried out by an AI agent (Claude, made by Anthropic) in a sandbox. Each run is driven by the AI with no person in the loop, and the log's actor field records this. A person oversees the project. No domain expert has reviewed this report. The authors have a right of reply, and their notes can be published alongside it.

What the attempt found

Running the authors own Table 1 code (vignettes/paper-results.Rmd) at the synthdid commit named in their session-info, with R 4.0.5 and the pinned MCPanel and glmnet commits, reproduced every entry of Table 1 at the reported precision: SDID -15.6 (8.4), SC -19.6 (9.9), DID -27.3 (17.7), MC -20.2 (11.5), DIFP -11.1 (9.5). Point estimates differ from the paper by at most 0.05, and placebo standard errors by at most 0.05. Deviations were limited to platform (x86_64 Linux instead of an M1 Mac) and one non-estimating build dependency. The paper-era synthdid commit is no longer on any branch of the repository but is still retrievable from GitHub by its hash.

Assurance

Level: Legacy (what the levels mean)
Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites
Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
Report re-derived from the log: Passed: the engine re-derived the report from the log
Engine version: 0.2.1
Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Qualifiers: legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it

Headline claims

These decide the outcome. Values are as published and as observed by re-running the code; the accepted-if rule was registered before results were recorded. Each row has a ¶ link of its own.

Claim	Published	Observed	Accepted if	Result
The SDID estimate of the average effect of Proposition 99 on California per-capita cigarette sales over the 12 post-treatment years is -15.6 packs.¶Table 1, p. 8 · registered at event 9 · quote verified on p. 8 · equivalence check	-15.6packs per capita	-15.6038	within ±0.1	Held
The placebo-method standard error of the SDID estimate is 8.4.¶Table 1, p. 8 (placebo method, Section 5, Algorithm 4) · registered at event 10 · quote verified on p. 8 · equivalence check	8.4packs per capita	8.36799	within ±0.84	Held
The synthetic control estimate is -19.6.¶Table 1, p. 8 · registered at event 11 · quote verified on p. 8 · equivalence check	-19.6packs per capita	-19.6197	within ±0.1	Held
The difference-in-differences estimate is -27.3.¶Table 1, p. 8 · registered at event 12 · quote verified on p. 8 · equivalence check	-27.3packs per capita	-27.3491	within ±0.1	Held

Other claims checked

Reported alongside the headline claims; they do not change the outcome.

Claim	Published	Observed	Accepted if	Result
The matrix completion estimate is -20.2.¶Table 1, p. 8 · registered at event 13 · quote verified on p. 8 · equivalence check	-20.2packs per capita	-20.2181	within ±0.5	Held
The synthetic control with intercept (DIFP) estimate is -11.1.¶Table 1, p. 8 · registered at event 14 · quote verified on p. 8 · equivalence check	-11.1packs per capita	-11.1046	within ±0.1	Held
The placebo-method standard error of the SC estimate is 9.9.¶Table 1, p. 8 · registered at event 15 · quote verified on p. 8 · equivalence check	9.9packs per capita	9.91743	within ±0.99	Held
The placebo-method standard error of the DID estimate is 17.7.¶Table 1, p. 8 · registered at event 16 · quote verified on p. 8 · equivalence check	17.7packs per capita	17.7403	within ±1.77	Held
The placebo-method standard error of the MC estimate is 11.5.¶Table 1, p. 8 · registered at event 17 · quote verified on p. 8 · equivalence check	11.5packs per capita	11.5099	within ±1.15	Held
The placebo-method standard error of the DIFP estimate is 9.5.¶Table 1, p. 8 · registered at event 18 · quote verified on p. 8 · equivalence check	9.5packs per capita	9.45566	within ±0.95	Held

Deviations from the paper's setup

noneisoband 0.2.5 (from the 2021-08-01 CRAN snapshot) instead of the 0.2.4 in the 2021-05-01 snapshot used for all other CRAN packages.
minorPlatform differs from the authors Table 1 run: x86_64 Debian 13 with R reference BLAS/LAPACK on a shared-CPU DigitalOcean droplet, versus an M1 MacBook Air with OpenBLAS 0.3.13. R 4.0.5 and the pinned synthdid, MCPanel and glmnet commits match their session-info.
nonefuture.batchtools is not installed or loaded; the vignette imports it alongside rngtools, future and doFuture.

The run

Wall time: 32 min 53 s, 18 October 2026 to 18 October 2026
Job compute: 25 min 13 s across 4 jobs
Code the results name: 2408a92
Work tree at the end: synth-inference/synthdid at 70c1ce3
susanathey/MCPanel at 6b2706f
cran/glmnet at f4fc95a
synth-inference/synthdid at 2408a92
Container image: repro-sandbox:0.2 sha256:d65217b9b8e6…
Sandbox: 3.5 CPUs, 6 GB, no GPU
Run by: Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt: arxiv-1812-09970-20261018T010328Z
Log: 59 events, head 8b90233f

Evidence

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 59 hash-chained events

8b90233f1aba9384bcb8d580e05cbaaf9a5947ebf64d2d62c4f0f4b157f1f7cb

The head hash of the attempt's append-only log. The engine generated this report from that log; the verifier checks the log and its artifacts against this hash and that the report names it, but not the report's contents. Download the evidence bundle (the log, every artifact and the report; SHA-256 b5ba1fe35f4c450b…) and check it with the verifier, which needs only Python.

Released 19 October 2026. Authors notified: courtesy note 19 October 2026. Reply status: Courtesy note sent at publication; one of five addresses bounced. No reply on record.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: Synthetic Difference-in-Differences. Reproduced; 4 of 4 headline claims held (legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1812-09970-20261018T010328Z, log head 8b90233f1aba9384, checked 18 Oct 2026. https://openrepro.org/papers/1812.09970/

BibTeX

@techreport{openrepro-1812.09970,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: Synthetic Difference-in-Differences},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/1812.09970/},
  note        = {Attempt arxiv-1812-09970-20261018T010328Z; log head 8b90233f1aba9384; checked 18 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

Badge

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Markdown

[![OpenRepro: Reproduced. 4 of 4 headline claims held; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.](https://openrepro.org/badge/1812.09970.svg)](https://openrepro.org/papers/1812.09970/)

HTML

<a href="https://openrepro.org/papers/1812.09970/"><img src="https://openrepro.org/badge/1812.09970.svg" alt="OpenRepro: Reproduced. 4 of 4 headline claims held; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="416" height="20"></a>

Right of reply, and reporting an error

Authors may submit a response, which is published alongside this report unedited. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.