Replication record · arXiv:1609.02907

Semi-Supervised Classification with Graph Convolutional Networks

Thomas Kipf, Max Welling · 2016 · arXiv

Partially reproduced

Running the authors train.py at the last commit before arXiv v4 (TensorFlow 1.0.1, CPU) over many random seeds reproduces GCN accuracy on Cora (81.53 vs 81.5, 100 runs) and Pubmed (79.16 vs 79.0, 20 runs). On Citeseer the 100-run mean is 69.57 against 70.3 reported, outside the pre-registered 0.5-point tolerance, because 2 of 100 runs collapsed to near or below chance (15.9% and 9.3%); the other 98 runs average 70.73. NELL, the random-split row and the timings were not tested.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 2 of 3 consistent
Deviations: 3 (0 material)
Integrity flags: 0
Logged compute: 1.17 h over 4 commands and 2 jobs
Primary code (declared before the first recorded result): github.com/tkipf/gcn (linked in the paper) at the last commit on or before 2017-02-22, the date of arXiv v4, using its train.py with the hyperparameters stated in Section 6.1
Commits the recorded results name: 193c099: 3 results, named in the result's method text
Work tree when the attempt ended: tkipf/gcn@193c09920076 in /work/gcn

Protocol

Declared before the first recorded result: primary code is github.com/tkipf/gcn (linked in the paper) at the last commit on or before 2017-02-22, the date of arXiv v4, using its train.py with the hyperparameters stated in Section 6.1; earlier versions of the authors' code may be tried as a fallback. The paper version under test is arXiv v4. If a claim misses with that code, earlier commits of the same repository may be tried as a fallback and are reported separately; they never replace the primary result.

Claims

GCN reaches 70.3% mean test accuracy on Citeseer (Planetoid split, mean of 100 random weight initializations).headline

Inconsistent

paper 70.3observed 69.569

Paper 70.3 percentObserved 69.569 percentAccepted if within ±0.5

GCN (this paper) 70.3 (7s) 81.5 (4s) 79.0 (38s)

Table 2, p. 7 · quote verified on p. 7 · equivalence check · observed/published 0.9896 · registered at event 10, before the first recorded result

Method, tolerance rationale and evidence

Metric. mean test accuracy over 100 runs with different random seeds, train.py defaults (dropout 0.5, L2 5e-4, 16 hidden units)

Why this tolerance. Mean of 100 runs with random weight initializations on fixed splits; per-run standard deviation is about 0.5-0.7 points, so the mean itself is stable to about 0.1, and 0.5 points allows for framework version and CPU-versus-GPU numerical differences.

Event 29. Observed 69.569, delta -0.731, inconsistent. Evidence events [24] (value matched by the legacy rule; not bound to an excerpt). Authors train.py at commit 193c099 (2017-02-20) with Python 3.6.15 and TensorFlow 1.0.1 on CPU; only the hard-coded seed line changed so seeds 1-N could be run; mean of the printed test accuracies. 100 runs. Two of the 100 runs collapsed (seed 21: 15.9%, seed 23: 9.3%, event 26). Excluding them, the mean is 70.73 over 98 runs; that figure is post hoc and is not the pre-registered metric.

GCN reaches 81.5% mean test accuracy on Cora (Planetoid split, mean of 100 random weight initializations).headline

Consistent

paper 81.5observed 81.532

Paper 81.5 percentObserved 81.532 percentAccepted if within ±0.5

GCN (this paper) 70.3 (7s) 81.5 (4s) 79.0 (38s)

Table 2, p. 7 · quote verified on p. 7 · equivalence check · observed/published 1 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. mean test accuracy over 100 runs with different random seeds, train.py defaults (dropout 0.5, L2 5e-4, 16 hidden units)

Event 27. Observed 81.532, delta +0.032, consistent. Evidence events [24] (value matched by the legacy rule; not bound to an excerpt). Authors train.py at commit 193c099 (2017-02-20) with Python 3.6.15 and TensorFlow 1.0.1 on CPU; only the hard-coded seed line changed so seeds 1-N could be run; mean of the printed test accuracies. 100 runs.

GCN reaches 79.0% mean test accuracy on Pubmed (Planetoid split, mean of 100 random weight initializations).headline

Consistent

paper 79observed 79.155

Paper 79 percentObserved 79.155 percentAccepted if within ±0.5

GCN (this paper) 70.3 (7s) 81.5 (4s) 79.0 (38s)

Table 2, p. 7 · quote verified on p. 7 · equivalence check · observed/published 1.002 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. mean test accuracy over 100 runs with different random seeds, train.py defaults (dropout 0.5, L2 5e-4, 16 hidden units)

Event 28. Observed 79.155, delta +0.155, consistent. Evidence events [24, 26] (value matched by the legacy rule; not bound to an excerpt). Authors train.py at commit 193c099 (2017-02-20) with Python 3.6.15 and TensorFlow 1.0.1 on CPU; only the hard-coded seed line changed so seeds 1-N could be run; mean of the printed test accuracies. 20 runs (seeds 1-20; declared deviation at event 20).

Deviations from the paper's protocol

minor
Pubmed uses 20 runs (seeds 1-20) instead of the paper 100; Citeseer and Cora use the full 100.
With TensorFlow 1.0.1 on this CPU-only sandbox a Pubmed run takes about 6 minutes, so 100 runs would take about 10 hours. Decided after observing only run counts and timings, before any accuracy value was seen. With a per-run SD of about 0.7 points the 20-run mean has a standard error of about 0.15, well inside the 0.5 tolerance. event 20
minor
CPU instead of GPU, with Python 3.6.15 (conda-forge) and TensorFlow 1.0.1, numpy 1.12.1, scipy 0.19.0, networkx 1.11.
CPU sandbox; versions chosen to match the paper-era commit, whose last change was a fix for TensorFlow 1.0. event 31
none
train.py line 11 (seed = 123) was replaced to read the seed from an environment variable, to run seeds 1 to N.
The paper reports the mean of 100 random weight initializations, but the released script fixes one seed. No other line was changed (the diff is in the job output). event 32

Notes from the attempt

event 13
Scope, declared before any run: the Citeseer, Cora and Pubmed accuracies of the GCN row in Table 2. Not tested: NELL (data not bundled with the code), the GCN (rand. splits) row (no split-generation script in the repository), and the wall-clock times in brackets (GPU timings). Planned deviation: train.py hard-codes seed = 123, so the 100 runs set that one line from an environment variable (seeds 1 to 100); nothing else in the code changes.
event 23
Event 21 exited 143: its pgrep pattern matched the command line of the shell running it, so it terminated itself and did not stop the Pubmed loop. The loop was stopped at the next event. The Pubmed result uses seeds 1-20 only, as declared at event 20, regardless of how many rows were written before the loop stopped.
event 30
Citeseer: 98 of 100 runs lie between about 68% and 72% (run SD without the two collapsed runs is under 1 point), but seeds 21 and 23 collapsed to 15.9% and 9.3%, below chance for six classes. These are genuine outcomes of the authors code at the paper-version commit in this environment (TensorFlow 1.0.1, CPU). The paper does not say whether any runs were excluded. The earlier-code fallback was not used: the miss comes from rare collapsed runs, not from a code-version difference, and rerunning older code would only draw new seeds.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T02:27:18.810Z → 2026-10-18T03:10:34.733Z
Event log: 36 events, hash-chained; head 0bb78a3ac2ad93cb611296a83ff9121aed1634053b3597da503fa7afddeaf1ff
Verify: python3 openrepro_verify.py arxiv-1609-02907-20261018T022718Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.