Machine learning · 2017

Reproducibility report: Semi-Supervised Classification with Graph Convolutional Networks

Report by OpenRepro, released

Paper by Thomas N. Kipf, Max Welling. International Conference on Learning Representations (ICLR), 2017. arXiv 1609.02907

InconclusiveA registered validity/precision condition was not satisfied
The full answer

Does GCN (Kipf and Welling, 2017) reproduce? Inconclusive: a registered validity/precision condition was not satisfied. A condition on the test itself, sealed before the scored runs, was not satisfied or not tested, so no reproduction outcome is given. This is not a finding about the paper (0 agent-designated controls excluded; 1 gate) (checked 20 Oct 2026).

Does GCN (Kipf and Welling, 2017) reproduce? Inconclusive: a registered validity/precision condition was not satisfied, so no outcome is given (0 agent-designated controls excluded; 1 gate). This is not a finding about the paper (checked 20 Oct 2026).

Checked
Released
Assistants
Ask via MCP

Does GCN reproduce?

Inconclusive. A registered validity/precision condition was not satisfied: a condition on the test itself, sealed before the scored runs, was not satisfied or not tested (see Gates below), so no reproduction outcome is given and nothing is counted as held or missed. This is not a finding about the paper; the claims below are shown as recorded. Roles: 0 agent-designated controls excluded; 1 gate (see Gates below).

Looking for a replication of GCN? This report is a computational reproduction: the authors' own published code was re-run and its results compared with the numbers in the paper. It does not set out to test the findings on new data, and it is not peer reviewed. What an attempt checks.

How far the re-run landed from the published number, in tolerances1 claim re-run, 1 headline
  1. GCN: every claim checked1 claim: not counted (inconclusive).
  • Held (headline claims)
The data behind this chart
ClaimPublishedObservedAccepted ifDeviation ÷ toleranceResult
GCN (fixed Planetoid split, mean of 100 random weight initializations) reaches 70.3% test accuracy on Citeseer; judged by whether the final uncertainty interval of a fresh 100-initialization mean overlaps the paper's rounding interval [70.25, 70.35]. (headline)70.370.3within ±0.05+0.00Held

What the attempt found

Prospective re-run of the GCN Citeseer settling test (Table 2, 70.3%) with the plan's precision gate registered as a gate: the authors' train.py at 193c099 (TF 1.0.1 CPU, released early stopping) on a fresh cohort of 100 initializations, seeds 201-300 (101-200 already have results), judged by the sealed interval rule. The fresh mean is 70.372% (SD 4.545, SE 0.454); the enclosing t/bootstrap interval [69.38, 71.27] overlaps the paper's rounding interval, but it extends 0.993 points from the mean, above the 0.731 precision gate, so the result is inconclusive: the cohort does not estimate the mean precisely enough to settle the disputed gap. One of the 100 runs (seed 241, 26.0%, stopped after 15 epochs) is low and stays included.

Assurance

Level
Sealed, bound and re-derived (what the levels mean)
Plan
Sealed before scored execution; 2 outputs observed before the seal (all of it in the log)
Evidence
Bound: each value was extracted from the cited run output by a rule fixed before the seal
Execution manifest
Yes: for each scored run the command, the Git checkout state (HEAD, status, diff hash) and the hashes of the driver files the command names were recorded when it started and when it ended. Data files and the environment are not bound by this
Report re-derived from the log
Passed: the engine re-derived the report from the log
Engine version
0.2.1
Qualifiers
None

Headline claims

These decide the outcome. Each row gives the published value, the value from the re-run and the accepted-if rule registered before results were recorded. Each row has its own ¶ link.

ClaimPublishedObservedAccepted ifResult
GCN (fixed Planetoid split, mean of 100 random weight initializations) reaches 70.3% test accuracy on Citeseer; judged by whether the final uncertainty interval of a fresh 100-initialization mean overlaps the paper's rounding interval [70.25, 70.35].Table 2, p. 7, row 'GCN (this paper)', Citeseer column; protocol on p. 7 ('report mean accuracy of 100 runs with random weight initializations') · registered at event 7 · quote verified on p. 7 · equivalence check70.3percent (test accuracy, percentage points)70.3within ±0.05Held

Gates

A gate is a validity/precision condition on the test itself. The AI agent that ran the attempt designated it and sealed it with the plan. The rationale shown is the agent's, and no reviewer has checked that the condition is justified. A gate is not a claim about the paper: it is satisfied or not satisfied, and it is not counted among the paper's claims. If a gate is not satisfied or not tested, the outcome is inconclusive: a registered validity/precision condition was not satisfied ("the test was not precise enough" only when every unmet gate is a precision gate). Its value column is the criterion, not a published value.

ClaimCriterionObservedAccepted ifResult
Precision gate of the settling plan for the Citeseer 70.3% cell: the final uncertainty interval of the fresh 100-initialization mean extends no more than 0.731 percentage points from that mean (the size of the historical 70.3 - 69.569 gap the test is meant to resolve); otherwise the comparison is inconclusive.Table 2, p. 7, row 'GCN (this paper)', Citeseer column; gate = 70.3 minus our prior mean 69.569 (attempt arxiv-1609-02907-20261018T022718Z, event 29); settling plan section 4 · registered at event 8 · quote verified on p. 7 · bound check, observed 1.36× the published valueAgent-designated gate. Rationale: not recorded in this report.0.731percentage points (precision gate, settling plan section 4)0.993at most 0.731Not satisfied

Deviations from the paper's setup

7 recorded: 5 minor, 2 none
  • minorSeed list deviates from the settling plan: the fresh headline cohort is seeds 201..300 inclusive (100 fits, released default stopping), not the plan's 101..200, and the plan's diagnostic replay of seeds 21-23 (six fits) is not run, so this attempt has 100 fits rather than 106.
  • minorCPU execution on a dedicated 4-vCPU droplet (container capped at 3.5 CPUs) with Python 3.6.15, TensorFlow 1.0.1 CPU, NumPy 1.12.1, SciPy 0.19.0, NetworkX 1.11; TF intra-op 4 / inter-op 1 and OMP_NUM_THREADS=4. The paper ran on GPU; the original kernel, RNG and thread configuration are unknown.
  • minorThe pip freeze (SHA-256 60c1655e...) and the conda-meta package list (SHA-256 523f2973...) equal those of arxiv-1609-02907-20261020T031257Z, but the conda-forge base libraries are the builds served on 2026-10-20 and the first attempt (arxiv-1609-02907-20261018T022718Z) never recorded its conda-level list, so the full original environment cannot be proven identical. The sandbox image was rebuilt for this box (sha256:5b400120..., previous attempt sha256:14183b46...).
  • minorFull per-node logits (test_predictions.csv) and the per-fit epochs.csv/final.csv files were written on the box and are not retained after it is destroyed. The job output retains every fit's unrounded per-epoch train/validation losses and accuracies, per-node predicted and true class strings, the full final row, stderr, exit code, initialization digest, logits digest and SHA-256 of each file; the analysis output retains the per-seed table and summary.
  • minorThe plan asks for one externally anchored plan before execution. The sealed plan (protocol, claim and gate registrations, run/aggregation/stopping policy with the SHA-256 of the plan document, drivers, input manifest, environment freezes and image) is hash-chained in this log at the seal event, but was not anchored outside OpenRepro before the scored jobs ran.
  • noneThe verifier rederives the claim and gate values from the printed analysis output, not the bootstrap from the 100 rows, so per the plan section 'Seal and execution contract' this is a qualified statistical analysis.
  • noneInstrumentation beyond the plan's list: the collector also records the number of hidden units that are zero on every node at the final weights and the graph op count before and after instrumentation.

The run
Wall time
9 h 5 min, 20 October 2026 to 20 October 2026
Job compute
18 min 28 s across 2 jobs
Code the results name
tkipf/gcn at 193c099
Work tree at the end
tkipf/gcn at 193c099
Container image
repro-sandbox:0.2 sha256:5b40012093ba…
Sandbox
3.5 CPUs, 6 GB, no GPU
Run by
Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
Attempt
arxiv-1609-02907-20261020T035706Z
Log
44 events, head 1ef063a3

Evidence

log seal, 44 hash-chained events, 2 earlier versions

The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.

Log seal · 44 hash-chained events

1ef063a357fbc913ab4a383e0b30ce717a5ce0a6051a021d0fcc90d517955917

The head hash of the attempt's append-only log. The engine generated this report from that log. The verifier checks the log and its artifacts against this hash, and that the report names it. It does not check the report's contents. The evidence bundle for this attempt is not published yet.

Released 4 November 2026 · Settling update sent 20 October 2026 to one author (the other address bounced), with a request to forward it. The report was regenerated after the update, adding qualifiers; the outcome did not change. No reply received.

Earlier versions of this report

The report above is the current version. Each earlier attempt stays public, unedited, with the reason it was replaced.

  • arxiv-1609-02907-20261020T031257Z, 20 October 2026: Partially reproduced. Superseded by arxiv-1609-02907-20261020T035706Z: It registered the settling plan's precision gate as a headline claim, so the engine labelled an inconclusive result “partially reproduced”; the re-run registered the gate as a gate, on a fresh cohort of 100 seeds.
  • arxiv-1609-02907-20261018T022718Z, 18 October 2026: Partially reproduced. Superseded by arxiv-1609-02907-20261020T035706Z: Its fixed 0.5-point band on the Citeseer mean took no account of run-to-run spread; the sealed settling re-run judged a fresh 100-seed cohort by an interval rule with a precision gate.

Share image (1200×630), drawn from this report.

How to cite this report

OpenRepro (2026). Reproducibility report: Semi-Supervised Classification with Graph Convolutional Networks. Inconclusive: a registered validity/precision condition was not satisfied. Attempt arxiv-1609-02907-20261020T035706Z, log head 1ef063a357fbc913, checked 20 Oct 2026. https://openrepro.org/papers/1609.02907/

APA style

OpenRepro. (2026). Reproducibility report: Semi-Supervised Classification with Graph Convolutional Networks (Attempt arxiv-1609-02907-20261020T035706Z; log head 1ef063a357fbc913). https://openrepro.org/papers/1609.02907/

BibTeX
@techreport{openrepro-1609.02907,
  author      = {{OpenRepro}},
  title       = {Reproducibility report: Semi-Supervised Classification with Graph Convolutional Networks},
  institution = {OpenRepro},
  year        = {2026},
  url         = {https://openrepro.org/papers/1609.02907/},
  note        = {Attempt arxiv-1609-02907-20261020T035706Z; log head 1ef063a357fbc913; checked 20 Oct 2026}
}

To cite the whole ledger, use CITATION.cff on the data page.

References

1 paper
  1. Thomas N. Kipf, Max Welling (2017). Semi-Supervised Classification with Graph Convolutional Networks. International Conference on Learning Representations (ICLR). arXiv 1609.02907

Badge

OpenRepro: Inconclusive: a registered validity/precision condition was not satisfied. No outcome is given; this is not a finding about the paper.

Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.

Copy the badge as Markdown or HTML

Markdown

[![OpenRepro: Inconclusive: a registered validity/precision condition was not satisfied. No outcome is given; this is not a finding about the paper.](https://openrepro.org/badge/1609.02907.svg)](https://openrepro.org/papers/1609.02907/)

HTML

<a href="https://openrepro.org/papers/1609.02907/"><img src="https://openrepro.org/badge/1609.02907.svg" alt="OpenRepro: Inconclusive: a registered validity/precision condition was not satisfied. No outcome is given; this is not a finding about the paper." width="391" height="20"></a>

Right of reply, and reporting an error

Authors of any paper on this site may submit a response for publication. Replies are published with the author's permission, in full or in the parts they choose, and any omission is marked. The published text preserves their wording apart from disclosed edits: whitespace is normalized, omitted passages are marked, and any removal of signature email addresses and adjoining separators is disclosed. Write to hello@openrepro.org.

Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.