Replication record · arXiv:1609.02907

Semi-Supervised Classification with Graph Convolutional Networks

Thomas Kipf, Max Welling · 2016 · arXiv

Partially reproduced

Prospective settling test of the GCN Citeseer cell of Table 2 (70.3%), sealed before any training: the authors train.py at 193c099 (TF 1.0.1 CPU, released early stopping), a diagnostic replay of seeds 21-23 with default vs full-length training, and a fresh cohort of 100 initializations (seeds 101-200), judged by a sealed interval rule. The fresh mean is 70.256% (SE 0.581); its enclosing t/bootstrap interval [69.02, 71.41] overlaps the paper rounding interval but extends 1.24 points from the mean, above the 0.731 precision gate, so under the plan the result is inconclusive: the cohort cannot settle the disputed gap. One of the 100 fresh runs (13.1%) is low and stays included. The two prior low seeds replay exactly (15.9%, 9.3%, both stopped after 12 epochs) and reach 71.6% and 70.2% at full length: strong stopping sensitivity in these selected seeds, not an explanation of the population mean.

Plan. Sealed before scored execution; 6 outputs observed before sealing (all of it in the log).

Qualifiers. A scored run is not the direct source of any recorded value: it may be an upstream stage whose output a later scored job read, or a run that was not used (or more scored runs started than the sealed plan declares); the details name each one. 3 scored jobs started after the seal; not the direct source of any recorded value: replay (event 24), fresh (event 27).

Scored (headline): 1 of 2 consistent
Deviations: 6 (0 material)
Integrity flags: 0
Logged compute: 0.30 h over 6 commands and 3 jobs
Primary code (declared before the first recorded result): github.com/tkipf/gcn at commit 193c0992007610b8ff65f3d3f5d6c418b13a343b (2017-02-20, 'Fix for tf 1.0.0'; last commit before arXiv v4 of 2017-02-22), gcn/train.py with its default flags (model gcn, 16 hidden units, dropout 0.5, weight decay 5e-4 on the first layer, Adam learning rate 0.01, at most 200 epochs, --early_stopping 10), dataset citeseer, bundled gcn/data/ind.citeseer.* files and the authors' utils.load_data split
Commits the recorded results name: tkipf/gcn@193c09920076: 2 results, work-tree HEAD logged when the result was recorded
Work tree when the attempt ended: tkipf/gcn@193c09920076 in /work/src-gcn

Protocol

Sealed with the plan: primary code is github.com/tkipf/gcn at commit 193c0992007610b8ff65f3d3f5d6c418b13a343b (2017-02-20, 'Fix for tf 1.0.0'; last commit before arXiv v4 of 2017-02-22), gcn/train.py with its default flags (model gcn, 16 hidden units, dropout 0.5, weight decay 5e-4 on the first layer, Adam learning rate 0.01, at most 200 epochs, --early_stopping 10), dataset citeseer, bundled gcn/data/ind.citeseer.* files and the authors' utils.load_data split; earlier versions of the authors' code are not used. Settling test per office/plans/1609.02907-settling-plan.md section 2: the pinned commit is the only code version; the plan finds no evidence-based earlier-code rescue arm and forbids a calendar/version search (an earlier revision would need a separately registered hypothesis).

Run policy (sealed at event 23, plan hash dad80ab045161ddcdf0ff51d09f699a0494cdff3fea61cc3abcf46ad11104c04): Settling plan office/plans/1609.02907-settling-plan.md (SHA-256 ce660079b1301ff588f1d6260f1a2e4e2764d0b82f0c70c6a93150143da40d67, bound at event 17). Exactly 106 fits of the authors' train.py at 193c0992007610b8ff65f3d3f5d6c418b13a343b on citeseer with default flags, through the collector /work/settle/train_settle.py (SHA-256 09042e262a5cd9e29ee7186c1228a12e3b13ba001821c203ef6c884d9cf76d91; only SETTLE-marked lines differ, diff at event 22), one fresh Python process and TF graph/session per fit, NumPy and TF seed = the seed, PYTHONHASHSEED=0, OMP_NUM_THREADS=4, TF intra-op 4 / inter-op 1, one fit at a time. Scored job 'replay' (bash /work/settle/run_phase.sh replay, run_fit.sh SHA-256 26066328368fb729e2227e21506d7a4796b44b162d1dd6d5395ff717a95d68d0, run_phase.sh SHA-256 ca73865525c7ad30cb8a125c7f56da26ec0eb5fcdeefd4ca01ee1a297ce8356b): seeds 21, 22, 23 in that order, each with --early_stopping 10 (arm replay_default) then --early_stopping 200 (arm replay_full, never stops before the 200-epoch maximum); then scored job 'fresh': seeds 101..200 inclusive with --early_stopping 10 (arm fresh_default); all fits run regardless of earlier results; then scored job 'analysis' (/work/settle/analyze.py SHA-256 736426a91f0e8a5972831104a8da67c194926d3f9af184bf42d4ca8a0b8362c7, Python 3.11.17, NumPy 1.26.4, SciPy 1.13.1). Inputs: the eight gcn/data/ind.citeseer.* files, manifest digest f6d99979e783fbab2987faf2d2d9a03e719561d6e8a17e11f91b9e8bc3450f91 (equal to the plan's, event 12); training env Python 3.6.15 (conda-forge, micromamba 1.5.10), tensorflow 1.0.1 CPU, numpy 1.12.1, scipy 0.19.0, networkx 1.11, protobuf 3.19.6, six 1.17.0, decorator 5.1.1 (pip freeze SHA-256 60c1655ef28d2c0f5ba3aac236efecb9a817dea91816cdef5440380b3d96674a, conda-meta list SHA-256 523f2973c9359c20a1f195b73261344b773b3dc09de199cf71e82fca7f5e2c3c); sandbox image sha256:14183b46fae66b1cfcd5bdca3017b44b7af804553d78822a0cf209825d37e8d8 on a dedicated 4-vCPU droplet (container capped at 3.5 CPUs). Each fit writes epochs.csv, test_predictions.csv and final.csv to a fresh per-job directory /work/out/<job>/gcn/<arm>/seedNNN and echoes its full-precision records, stderr, exit code and file digests into the job output. No pilots, no unscored training, no per-epoch test evaluation, no checkpoint selection. Resource cap: 120 wall-minutes from the first fit's start (environment setup excluded); no fit starts after the cap and a fit is killed at it. Job retry: one retry of a training job is allowed only for a documented external interruption (host loss or transient I/O), with identical inputs, seeds and command under a new job name and fresh output directory; both are retained and the first complete invocation is scored. Low accuracy, early stopping, OOM, NaNs or optimizer failure never justify a rerun or another seed. Aggregation: Canonical per-fit accuracy = 100*correct/test_n percentage points, correct = agreement of argmax final logits and argmax original y_test over the original 1,000-node test mask, test_n must be 1000; it must agree with the authors' TF masked accuracy within 1e-6 (extraction consistency check only; both values kept; a failure makes that row invalid). A required row is invalid if its final.csv is missing, duplicated, has the wrong key or row count, a non-zero fit exit, test_n != 1000, a non-finite value, or fails the consistency check; any invalid fresh row makes the fresh comparison evidence-incomplete (no claim value is printed, no survivor mean). Fresh cohort (seeds 101-200 only; replay seeds never pooled): mu = unweighted mean of all 100 accuracies, s = SD (ddof=1), SE = s/10; t interval mu +/- t_99(0.975)*SE; bootstrap: RandomState(160902), idx = randint(0, 100, size=(50000, 100)), resample means, 2.5/97.5 percentiles (linear interpolation); final interval [L, H] = [min of lower ends, max of upper ends]. Claim gcn-citeseer-compat observed value = min(max(70.3, L), H) (within 0.05 of 70.3 iff [L, H] overlaps [70.25, 70.35]); claim gcn-citeseer-precision observed value = max(mu - L, H - mu) (gate: <= 0.731). Both are printed by the analysis job on their own lines as 'gcn_citeseer_compat_point <repr>' and 'gcn_citeseer_precision_halfwidth <repr>' and read by the registered key rules. Branch (plan section 5): gate fails -> 'Precision gate fails' (inconclusive); gate met and overlap -> 'Compatible, precision gate met'; gate met, no overlap -> 'Precise but disjoint'; invalid evidence -> 'Missing/invalid evidence'. Replay diagnostic (descriptive only): the six canonical accuracies and per-seed full-minus-default differences; low-accuracy replay = default < 20%; strong stopping sensitivity = a low-accuracy replay reaching >= 60% at full length; a prior low seed (21, 23) no longer below 20% is reported as not repeating. No exclusion, clipping or survivor-only mean anywhere. Stopping rule: One pass over the sealed 106-fit matrix and one analysis job, in the order replay, fresh, analysis; stop at the 120-minute cap, not when a result passes; no seeds are added if the precision gate fails; any changed scientific setting requires a new plan citing this attempt. Reruns: not allowed.

Claims

GCN (fixed Planetoid split, mean of 100 random weight initializations) reaches 70.3% test accuracy on Citeseer; judged by whether the final uncertainty interval of a fresh 100-initialization mean overlaps the paper's rounding interval [70.25, 70.35].headline

Consistent

paper 70.3observed 70.3

Paper 70.3 percent (test accuracy, percentage points)Observed 70.3 percent (test accuracy, percentage points)Accepted if within ±0.05

GCN (this paper) 70.3 (7s) 81.5 (4s) 79.0 (38s) 66.0 (48s)

Table 2, p. 7, row 'GCN (this paper)', Citeseer column; protocol on p. 7 ('report mean accuracy of 100 runs with random weight initializations') · quote verified on p. 7 · equivalence check · observed/published 1 · registered at event 8, sealed before scored execution

Method, tolerance rationale and evidence

Metric. The point of the final interval [L, H] nearest to 70.3, i.e. min(max(70.3, L), H), in percentage points. It lies within 0.05 of 70.3 exactly when [L, H] overlaps [70.25, 70.35] (the settling plan's compatibility rule, section 4). Fresh cohort: seeds 101..200 inclusive (100 fits, one fresh process and TF graph/session per seed, NumPy and TF seed = the seed, released default stopping --early_stopping 10, final model, no checkpoint restoration). Per-seed canonical accuracy a_i = 100*correct/test_n in percentage points, where correct = number of the 1,000 original test-mask nodes whose argmax logit equals argmax of the original y_test row, test_n must be 1000. mu = unweighted mean of the 100 a_i; s = sample SD (ddof=1); SE = s/10; t interval mu +/- t_99(0.975)*SE (scipy.stats.t.ppf(0.975, 99)); percentile bootstrap: numpy 1.26.4 RandomState(160902), idx = rs.randint(0, 100, size=(50000, 100)), resample means, np.percentile(means, [2.5, 97.5]) with linear interpolation; final interval [L, H] = [min(t_low, boot_low), max(t_high, boot_high)] (the enclosing interval, never the narrower one). All 100 runs are included; no survivor-only mean, no clipping, no replacement of low runs.

Why this tolerance. Fixed in advance by the settling plan (section 4): 70.3 is printed to one decimal, so its rounding interval is [70.25, 70.35]; observed seed uncertainty enters only through the sealed interval formula, never through a scalar tolerance chosen after seeing the SE. Overlap is compatibility, not proof of equivalence.

Event 33. Observed 70.3, delta +0, consistent. Evidence: 00031-job-analysis-stdout.txt (event 31), excerpt 'gcn_citeseer_compat_point 70.3'; verified. Job start event 30. Sealed analysis job over the 100 fresh fits (seeds 101-200, scored job event 28) of the authors train.py at 193c099 (TF 1.0.1 CPU, default stopping): mean 70.256, SE 0.5812; t interval [69.1028, 71.4092], bootstrap [69.019, 70.932]; enclosing interval [69.019, 71.4092] contains 70.3, so its nearest point to 70.3 is 70.3; value read by the registered key rule.

Precision gate of the settling plan for the Citeseer 70.3% cell: the final uncertainty interval of the fresh 100-initialization mean extends no more than 0.731 percentage points from that mean (the size of the historical 70.3 - 69.569 gap the test is meant to resolve); otherwise the comparison is inconclusive.headline

Inconsistent

paper 0.731observed 1.237

Paper 0.731 percentage points (precision gate, settling plan section 4)Observed 1.237 percentage points (precision gate, settling plan section 4)Accepted if at most 0.731

GCN (this paper) 70.3 (7s) 81.5 (4s) 79.0 (38s) 66.0 (48s)

Table 2, p. 7, row 'GCN (this paper)', Citeseer column; gate = 70.3 minus our prior mean 69.569 (attempt arxiv-1609-02907-20261018T022718Z, event 29) · quote verified on p. 7 · bound check · observed/published 1.692 · registered at event 9, sealed before scored execution

Method, tolerance rationale and evidence

Metric. max(mu - L, H - mu), in percentage points, for the fresh cohort mean mu and final interval [L, H]. 'paper_value' here is the plan's gate 0.731 (not a number printed in the paper). Fresh cohort: seeds 101..200 inclusive (100 fits, one fresh process and TF graph/session per seed, NumPy and TF seed = the seed, released default stopping --early_stopping 10, final model, no checkpoint restoration). Per-seed canonical accuracy a_i = 100*correct/test_n in percentage points, where correct = number of the 1,000 original test-mask nodes whose argmax logit equals argmax of the original y_test row, test_n must be 1000. mu = unweighted mean of the 100 a_i; s = sample SD (ddof=1); SE = s/10; t interval mu +/- t_99(0.975)*SE (scipy.stats.t.ppf(0.975, 99)); percentile bootstrap: numpy 1.26.4 RandomState(160902), idx = rs.randint(0, 100, size=(50000, 100)), resample means, np.percentile(means, [2.5, 97.5]) with linear interpolation; final interval [L, H] = [min(t_low, boot_low), max(t_high, boot_high)] (the enclosing interval, never the narrower one). All 100 runs are included; no survivor-only mean, no clipping, no replacement of low runs.

Why this tolerance. Fixed in advance by the settling plan (section 4): the gate is met when the maximum distance from mu to an interval endpoint is <= 0.731 points; no seeds are added if it fails.

Event 34. Observed 1.237, delta +0.506, inconsistent. Evidence: 00031-job-analysis-stdout.txt (event 31), excerpt 'gcn_citeseer_precision_halfwidth 1.2369999999999806'; verified. Job start event 30. Sealed analysis job: max distance from the fresh mean 70.256 to the ends of the enclosing interval [69.019, 71.4092] is 1.237 (lower end, bootstrap), above the 0.731 gate; value read by the registered key rule.

Deviations from the paper's protocol

minor
CPU execution on a 4-vCPU droplet (container capped at 3.5 CPUs) with Python 3.6.15, TensorFlow 1.0.1 CPU, NumPy 1.12.1, SciPy 0.19.0, NetworkX 1.11; TF intra-op 4 / inter-op 1 and OMP_NUM_THREADS=4. The paper ran on GPU; the original kernel, RNG and thread configuration are unknown.
Environment pinned by the settling plan; the thread configuration is a declared new reproducibility setting, not asserted to be the historical one. event 39
minor
The pip-level environment equals the previous attempt exactly (same ten packages and versions), but the conda-forge base libraries (libgcc, openssl, sqlite and others under Python 3.6.15) are the builds conda-forge served on 2026-10-20 via micromamba 1.5.10; the previous attempt did not record its conda-level list, so those cannot be proven identical.
The plan asks for the full previous environment pinned; only its recorded part could be pinned. event 40
minor
Full per-node logits (test_predictions.csv) and the per-fit epochs.csv/final.csv files were written on the box and are not retained after it is destroyed. The job outputs retain every fit's unrounded per-epoch train/validation losses and accuracies, per-node predicted and true class strings, the full final row, stderr, exit code, initialization digest, logits digest and SHA-256 of each file; the analysis output retains the per-seed table and summary.
The engine stores command output as artifacts, not /work files; printing all logits for 106 fits would add several MB. event 41
minor
The plan asks for one externally anchored plan before execution. The sealed plan (protocol, claims, run/aggregation/stopping policy with the SHA-256 of the plan document, drivers, input manifest, environment freezes and image) is hash-chained in this log at the seal event, but was not anchored outside OpenRepro before the scored jobs ran.
No external anchoring step is available before execution in engine 0.2.1. event 42
none
The plan's three-way verdict (compatible / precise but disjoint / inconclusive) is encoded as two headline claims, compatibility and the precision gate, mapped before any output (event 10). With the gate failing, the engine outcome word is not the plan's word: the plan's verdict for this attempt is inconclusive. The verifier rederives the claim values from the printed analysis output, not the bootstrap from the 100 rows, so per the plan this is a qualified statistical analysis.
Harness comparison vocabulary has no inconclusive verdict. event 43
none
Instrumentation beyond the plan's list: the collector also records the number of hidden units that are zero on every node at the final weights and the graph op count before and after instrumentation (equal, 353, in every fit).
Descriptive diagnostics for the collapse question; they fetch existing tensors only and never enter any criterion. event 44

Notes from the attempt

event 10
Scope and verdict mapping, declared before any command output. Settling test per office/plans/1609.02907-settling-plan.md (bound into this log before the seal). Only the Citeseer cell of Table 2 (70.3) is scored; Cora, Pubmed, NELL and timings are not. The plan has three branches; the engine has only consistent/inconsistent per claim, so both plan conditions are headline claims: compatibility (gcn-citeseer-compat: final interval overlaps [70.25,70.35]) and precision (gcn-citeseer-precision: max distance from the mean to an interval end <= 0.731). Mapping: both consistent = plan branch "Compatible, precision gate met"; compat inconsistent and precision consistent = "Precise but disjoint" (a remaining pinned-pipeline difference, not a refutation); precision inconsistent (whatever compat says) = "Precision gate fails" (inconclusive); any missing/invalid required row = "Missing/invalid evidence" (no fresh verdict; claim values are then not printed). Because the precision branch is inconclusive in the plan, an engine word such as Partially reproduced or Could not reproduce caused by a failed precision gate means inconclusive, not a negative finding. The diagnostic replay of seeds 21, 22, 23 (default stopping vs --early_stopping 200) is descriptive only, never a scored claim, exclusion or pass criterion; a low-accuracy replay is default accuracy < 20%, strong stopping sensitivity is such a seed reaching >= 60% at full length.
event 35
Sealed judgement, applied mechanically after all 106 fits (analysis job event 31): all 106 required rows valid (every fit exit 0, test_n 1000, canonical vs TF accuracy within 1e-6 for every row; largest difference about 3e-7). Fresh cohort seeds 101-200: mean 70.256, SD 5.8120, SE 0.5812 percentage points; t_99 interval [69.1028, 71.4092]; bootstrap interval [69.0190, 70.9320]; enclosing final interval [69.0190, 71.4092]. It overlaps [70.25, 70.35] (compat claim consistent), but its maximum distance from the mean is 1.2370 > 0.731 (precision claim inconsistent). Plan section 4/5 branch, selected mechanically: 'Precision gate fails' - inconclusive: 'The fresh cohort does not estimate the mean precisely enough to settle the disputed gap. All low-accuracy runs remain included.' Per the mapping declared at event 10, the engine outcome word produced by this failed gate means inconclusive, not a negative finding. Seeds were not added.
event 36
Fresh cohort detail (descriptive, not a criterion): 1 of 100 runs is low (seed 163: 13.1%, stopped by the released rule after 15 epochs, predicting class 0 for 933 of 1,000 test nodes); the other 99 lie between 69.3% and 72.1% (mean of those 99: 70.83, reported only as description, never as an estimate). 93 runs reached the 200-epoch maximum and 7 stopped early. No run had non-finite logits or a dead hidden unit at the final weights.
event 37
Diagnostic replay (scored job event 25; selected seeds, not a population estimate): seed 21 default 15.9% (stopped after 12 epochs) vs full length 71.6% (difference +55.7); seed 22 default 70.0% vs full 70.0% (difference 0.0; the default run met the stop condition only at epoch 200, identical final state); seed 23 default 9.3% (stopped after 12 epochs) vs full 70.2% (difference +60.9). The prior attempt's values for these seeds (15.9, 70.0, 9.3) repeated exactly. Under the predeclared diagnostic rule: 'Seeds 21, 23 showed strong stopping sensitivity under the predeclared diagnostic rule.' This is strong stopping sensitivity in these selected seeds, not a statement that early stopping explains the population mean. Both low default runs stopped at the first epoch the released rule can fire (epoch > 10) and predicted one class for most nodes (910 and 982 of 1,000 as class 0); no hidden unit was dead at the final weights, so 'dead activations' is not supported by this evidence.
event 38
Correction to the earlier attempt arxiv-1609-02907-20261018T022718Z, per plan section 5 (to be appended to its interpretation, not replacing it): 'Our original Citeseer mean was 69.569%, with sample SD 8.2239 and SE 0.8224 percentage points across 100 initializations. The 0.731-point difference from 70.3 is smaller than one observed SE. Our fixed-band miss did not establish a statistically persuasive discrepancy. Two low runs depressed the mean, but their cause was not recorded and remains unproved.' This attempt adds: the two low runs replay exactly and both recover to 70-72% when the released stopping rule is disabled.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: yes
Sealed before scored execution: yes
Evidence bound to artifact excerpts: yes
Values extracted by a sealed rule: yes
Execution manifest for cited jobs: yes
Semantic re-derivation passed: yes
Observed before the seal: 6 shell; provenance captured
Engine: report built by repro 0.2.1; attempt created by repro 0.2.1

Redaction before release

Before release, the operator's local home-directory path was replaced with ~ in 4 places. Every event hash from event 17 on changed with it. The seal event's bytes changed (its chain hash); its plan hash did not. The redaction records the outcome, verdicts and values from before it. That record is an operator assertion, checked for internal consistency against the current log, not proof that the original log was unchanged: the original is not part of the record. It matches what the log derives now.

Rule: operator home directory prefix → ~. Reason: Before release: the operator's local home-directory path, copied from a settling-plan file, was replaced with ~ Recorded at event 49; the head the operator states for the log before the redaction is 45a46fa3ff077bcf25816b6bf89a926c61507472d97ef4163c829084386c5f1f (not checkable from this log; a copy anchored elsewhere or the operator's kept original can check it).

Checked from the record: the redacted log's hash chain and artifact digests; re-derivation of every judgement from it; the redaction event's internal consistency (its position, replacement rows, recomputed digests and plan hashes, a single redaction or a second that names the first); and that the outcome, qualifiers, tallies, assurance flags, verdicts and observed values it records as before equal what the log derives now. Not checked: any other text (metric and protocol wording, units, paths), what files and commands meant when they ran, or anything about the log before the redaction, which is not in the record.

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-20T03:12:57.609Z → 2026-10-20T03:42:20.151Z
Event log: 50 events, hash-chained; head 0aa037fda235eef42779ef345008a4a32558c63eb2cbef9cc63bd6794a7698c1
Verify: python3 openrepro_verify.py arxiv-1609-02907-20261020T031257Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1) (the engine that does this is not yet public; the standalone verifier at /verify/ runs the chain and artifact checks).
Right of reply: Authors may submit a response, which is published alongside this report unedited.