Reproducibility report: From Louvain to Leiden: guaranteeing well-connected communities

OpenRepro

Network science · 2019

From Louvain to Leiden: guaranteeing well-connected communities

Does From Louvain to Leiden (Traag et al., 2019) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 4 of 4 headline claims held within tolerances registered before results were recorded; DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).

V. A. Traag, L. Waltman, Nees Jan van Eck. Scientific Reports, 2019. arXiv 1810.08473 · doi:10.1038/s41598-019-41695-z

Does “From Louvain to Leiden” reproduce?

Reproduced

Yes, on its headline claims. 4 of 4 headline claims held within tolerances registered before results were recorded; DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

How far the re-run landed from the published number, in tolerances6 claims re-run, 4 headline

Held (large: headline claim; small: other claim)

Reading the marks

Tail: a one-sided claim (at most, at least); the tail points to the open side, which is shaded faintly

The data behind this chart

Claim	Published	Observed	Accepted if	Deviation ÷ tolerance	Result
On DBLP, the maximal modularity found by the Leiden algorithm over 10 replications of 10 iterations is 0.8387. (headline)	0.8387	0.838377	within ±0.002	-0.16	Held
On Amazon, the maximal modularity found by the Leiden algorithm is 0.9341. (headline)	0.9341	0.934022	within ±0.002	-0.04	Held
On DBLP, the maximal modularity found by the Louvain algorithm is 0.8262.	0.8262	0.826653	within ±0.002	+0.23	Held
On Amazon, the maximal modularity found by the Louvain algorithm is 0.9301.	0.9301	0.929846	within ±0.002	-0.13	Held
On DBLP, Leiden finds higher maximal modularity than Louvain (paper: +0.0125). (headline)	0.0125	0.011724	at least 0	-0.06	Held
On Amazon, Leiden finds higher maximal modularity than Louvain (paper: +0.0040). (headline)	0.004	0.004176	at least 0	+0.04	Held

Headline claims are the results the paper is chiefly known for; they decide the outcome. Other claims are reported alongside.

Registration. Claims and tolerances registered at events 10–15; first result recorded at event 29. Registration came before results were recorded, which does not show that no output had been seen (a legacy attempt: no sealed plan). Event numbers refer to the attempt's log.

Checked 18 October 2026.

How this report was made. The re-run was carried out by an AI agent (Claude, made by Anthropic) in a sandbox. Each run is driven by the AI with no person in the loop, and the log's actor field records this. A person oversees the project. No domain expert has reviewed this report. The authors have a right of reply, and their notes can be published alongside it.

What the attempt found

Using the authors Java implementation at the release tag matching the paper (networkanalysis 1.0.0) on the SNAP DBLP and Amazon networks, with the paper protocol of 10 random starts of 10 iterations each, all tested Table II values reproduced within 0.0005: DBLP Louvain 0.8267 vs 0.8262 and Leiden 0.8384 vs 0.8387; Amazon Louvain 0.9298 vs 0.9301 and Leiden 0.9340 vs 0.9341. Leiden found higher modularity than Louvain on both networks, as claimed. The other four networks in Table II were out of scope (download blocked, proprietary, or too large for the sandbox), and the badly connected community percentages in Fig. 4 were not tested.

Assurance

Level: Legacy (what the levels mean)
Plan: Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
Evidence: Legacy matching: each observed value was matched against the text of the logged output it cites
Execution manifest: No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
Report re-derived from the log: Passed: the engine re-derived the report from the log
Engine version: 0.2.1
Predates: plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
Qualifiers: DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it

Headline claims

These decide the outcome. Values are as published and as observed by re-running the code; the accepted-if rule was registered before results were recorded. Each row has a ¶ link of its own.

Claim	Published	Observed	Accepted if	Result
On DBLP, the maximal modularity found by the Leiden algorithm over 10 replications of 10 iterations is 0.8387.¶Table II, p. 6 (text p. 9) · registered at event 10 · quote verified on p. 9 · equivalence check	0.8387modularity	0.838377	within ±0.002	Held
On Amazon, the maximal modularity found by the Leiden algorithm is 0.9341.¶Table II, p. 6 (text p. 9) · registered at event 11 · quote verified on p. 9 · equivalence check	0.9341modularity	0.934022	within ±0.002	Held
On DBLP, Leiden finds higher maximal modularity than Louvain (paper: +0.0125).¶Table II, p. 6 (text p. 9) · registered at event 14 · quote verified on p. 9 · broad bound, observed 0.938× the published value	0.0125modularity	0.011724	at least 0	Held
On Amazon, Leiden finds higher maximal modularity than Louvain (paper: +0.0040).¶Table II, p. 6 (text p. 9) · registered at event 15 · quote verified on p. 9 · broad bound, observed 1.04× the published value	0.004modularity	0.004176	at least 0	Held

Other claims checked

Reported alongside the headline claims; they do not change the outcome.

Claim	Published	Observed	Accepted if	Result
On DBLP, the maximal modularity found by the Louvain algorithm is 0.8262.¶Table II, p. 6 (text p. 9) · registered at event 12 · quote verified on p. 9 · equivalence check	0.8262modularity	0.826653	within ±0.002	Held
On Amazon, the maximal modularity found by the Louvain algorithm is 0.9301.¶Table II, p. 6 (text p. 9) · registered at event 13 · quote verified on p. 9 · equivalence check	0.9301modularity	0.929846	within ±0.002	Held

Deviations from the paper's setup

noneBuilt and run with OpenJDK 21 instead of a 2018-era JDK.
minorSNAP node ids were renumbered to 0..n-1 in sorted order, and runs used a fixed seed (20261018) rather than an unspecified random one.

The run

Wall time: 9 min, 18 October 2026 to 18 October 2026
Job compute: 2 min 51 s across 1 job
Code the results name: CWTSLeiden/networkanalysis at 2d38563
Work tree at the end: CWTSLeiden/networkanalysis at 2d38563
Container image: repro-sandbox:0.2 sha256:134b91122c63…
Sandbox: 3.5 CPUs, 6 GB, no GPU
Run by: Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt: arxiv-1810-08473-20261018T013946Z
Log: 42 events, head 578432e9

Evidence

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 42 hash-chained events

578432e9062b72e8e2576db5eeed0bf5ac96e6f5b649204a2e77964f446edc6c

The head hash of the attempt's append-only log. The engine generated this report from that log; the verifier checks the log and its artifacts against this hash and that the report names it, but not the report's contents. Download the evidence bundle (the log, every artifact and the report; SHA-256 44a9835769081ead…) and check it with the verifier, which needs only Python.

Released 19 October 2026. Authors notified: courtesy note 19 October 2026. Reply status: Courtesy note sent at publication. No reply on record.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: From Louvain to Leiden: guaranteeing well-connected communities. Reproduced; 4 of 4 headline claims held (DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1810-08473-20261018T013946Z, log head 578432e9062b72e8, checked 18 Oct 2026. https://openrepro.org/papers/1810.08473/

BibTeX

@techreport{openrepro-1810.08473,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: From Louvain to Leiden: guaranteeing well-connected communities},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/1810.08473/},
  note        = {Attempt arxiv-1810-08473-20261018T013946Z; log head 578432e9062b72e8; checked 18 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

Badge

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Markdown

[![OpenRepro: Reproduced. 4 of 4 headline claims held; DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.](https://openrepro.org/badge/1810.08473.svg)](https://openrepro.org/papers/1810.08473/)

HTML

<a href="https://openrepro.org/papers/1810.08473/"><img src="https://openrepro.org/badge/1810.08473.svg" alt="OpenRepro: Reproduced. 4 of 4 headline claims held; DBLP Leiden gain over Louvain 0.938× the published value; passes a registered floor of 0, a sign check; Amazon Leiden gain over Louvain 1.04× the published value; passes a registered floor of 0, a sign check; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="513" height="20"></a>

Right of reply, and reporting an error

Authors may submit a response, which is published alongside this report unedited. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.