Replication record · arXiv:1708.05031

Neural Collaborative Filtering

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, Tat-Seng Chua · 2017 · arXiv

Reproduced

The authors Keras code (commit 312aff0, identical in code to the current default branch) was run on MovieLens-1M with the exact README environment (Python 2.7.15, Keras 1.0.7, Theano 0.8.0) on CPU, using the README commands with 8 predictive factors and 20 epochs. All six pre-registered Table 2/3/4 values at 8 factors landed within 0.005 of the paper: NeuMF without pre-training HR@10 0.6848 vs 0.688 and NDCG@10 0.4088 vs 0.410; NeuMF with pre-training (initialised from the authors bundled GMF/MLP weights rather than retrained ones) 0.6798 vs 0.684 and 0.4054 vs 0.403; MLP-3 0.6707 vs 0.671 and 0.3979 vs 0.399. Each value is one run reported at the best-HR epoch, which is the scripts own convention. Mid-attempt the shared cloud droplet was destroyed by another process; the environment and code were rebuilt with the same recipe and only the in-flight MLP run was repeated.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. Some claims were registered after command output had already been observed. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 4 of 4 consistent
Other checks: 2 of 2 consistent
Deviations: 4 (0 material)
Integrity flags: 0
Logged compute: 2.25 h over 5 commands and 5 jobs
Primary code (declared before the first recorded result): github.com/hexiangnan/neural_collaborative_filtering at commit 312aff0 (2017-08-21, 'Change the name of Data'), the last commit on or before the arXiv v2 date 2017-08-26 (v1 was 2017-08-16; the PDF tested is v2). Event 9 shows HEAD 4aab159 differs from 312aff0 only in README.md, LICENSE and Dockerfile, so the code is identical to the current default branch.
Commits the recorded results name: 312aff0: 3 results, named in the result's method text
Work tree when the attempt ended: hexiangnan/neural_collaborative_filtering@312aff0ee6f4 in /work/ncf

Protocol

Declared before the first recorded result: primary code is github.com/hexiangnan/neural_collaborative_filtering at commit 312aff0 (2017-08-21, 'Change the name of Data'), the last commit on or before the arXiv v2 date 2017-08-26 (v1 was 2017-08-16; the PDF tested is v2). Event 9 shows HEAD 4aab159 differs from 312aff0 only in README.md, LICENSE and Dockerfile, so the code is identical to the current default branch.; earlier versions of the authors' code are not used. The only earlier code is the 2017-03-04 'Raw codes (before clearning)' snapshot and the 2017-08-01/02 cleanup commits; the 2017-08-01 commit 'Fix bug in evaluation' means pre-fix code evaluates differently from what the paper's camera-ready/arXiv numbers describe, so falling back to it would not be a faithful test. Primary code only.

Claims

NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches HR@10 = 0.688 on MovieLens-1M.headline

Consistent

paper 0.688observed 0.6848

Paper 0.688 ratioObserved 0.6848 ratioAccepted if within ±0.02

8 0.684 0.403 0.688 0.410

Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' HR@10 column), p. 7 · quote verified on p. 7 · equivalence check · observed/published 0.9953 · registered at event 12, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. HR@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M, leave-one-out with 99 sampled negatives

Why this tolerance. HR@10 is the mean of 6,040 per-user binary hits, so its binomial standard error is about sqrt(0.69*0.31/6040) = 0.006; run-to-run variation from random initialisation and negative sampling (the paper reports a single run) is of similar size. 0.02 absolute is roughly three combined standard errors and is also the smallest step that separates adjacent factor sizes in the paper's own tables.

Event 38. Observed 0.6848, delta -0.0032, consistent. Evidence events [34] (value matched by the legacy rule; not bound to an excerpt). README NeuMF command without pre-training (8 factors, layers [64,32,16,8], 20 epochs, batch 256, 4 negatives, Adam 0.001) on the primary commit 312aff0 with Keras 1.0.7/Theano 0.8.0 on CPU; value is the scripts End-of-run best-epoch HR@10 (epoch 18 of 20). Per-epoch HR@10 over the last 6 epochs ranged 0.6738-0.6848; the final epoch (19) gave 0.6844. Single run, single seed.

NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches NDCG@10 = 0.410 on MovieLens-1M.headline

Consistent

paper 0.41observed 0.4088

Paper 0.41 ratioObserved 0.4088 ratioAccepted if within ±0.02

8 0.684 0.403 0.688 0.410

Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' NDCG@10 column), p. 7 · quote verified on p. 7 · equivalence check · observed/published 0.9971 · registered at event 13, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. NDCG@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M

Why this tolerance. NDCG@10 averages per-user values in [0,1] over 6,040 users with standard error about 0.006, plus seed/negative-sampling variation of similar size for a single reported run. 0.02 absolute is roughly three combined standard errors and matches the HR tolerance so both metrics are judged on the same footing.

Event 39. Observed 0.4088, delta -0.0012, consistent. Evidence events [34] (value matched by the legacy rule; not bound to an excerpt). Same run as neumf-nopre-8-hr10 (job neumf-nopre-8, event 34): NDCG@10 at the best-HR epoch (18) as reported by the scripts End line. The final epoch (19) reached NDCG@10 = 0.4109, slightly higher, but the authors code reports NDCG at the best-HR epoch and that convention is used here. Single run.

NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches HR@10 = 0.684 on MovieLens-1M.headline

Consistent

paper 0.684observed 0.6798

Paper 0.684 ratioObserved 0.6798 ratioAccepted if within ±0.02

8 0.684 0.403 0.688 0.410

Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' HR@10 column), p. 7 · quote verified on p. 7 · equivalence check · observed/published 0.9939 · registered at event 14, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. HR@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M

Event 42. Observed 0.6798, delta -0.0042, consistent. Evidence events [40] (value matched by the legacy rule; not bound to an excerpt). README NeuMF command with pre-training (8 factors, layers [64,32,16,8], 20 epochs, batch 256, 4 negatives, Adam 0.001, alpha 0.5 via the scripts load_pretrain_model) initialised from the repositorys bundled Pretrain/ GMF and MLP weights, primary commit 312aff0, Keras 1.0.7/Theano 0.8.0, CPU; value is the scripts End-of-run best-epoch HR@10 (epoch 9 of 20). Before any fine-tuning (Init line) the pre-trained initialisation already scored HR@10 = 0.6710, NDCG@10 = 0.3969; the final epoch (19) gave 0.6720 / 0.3990. As the paper notes for MovieLens at 8 factors, the pre-trained NeuMF (0.6798) did not beat NeuMF from scratch (0.6848, event 34). Pre-trained weights are the authors files, not retrained here (see deviation at event 26).

NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches NDCG@10 = 0.403 on MovieLens-1M.headline

Consistent

paper 0.403observed 0.4054

Paper 0.403 ratioObserved 0.4054 ratioAccepted if within ±0.02

8 0.684 0.403 0.688 0.410

Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' NDCG@10 column), p. 7 · quote verified on p. 7 · equivalence check · observed/published 1.006 · registered at event 15, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. NDCG@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M

Event 43. Observed 0.4054, delta +0.0024, consistent. Evidence events [40] (value matched by the legacy rule; not bound to an excerpt). Same run as neumf-pre-8-hr10 (job neumf-pre-8, event 40): NDCG@10 at the best-HR epoch (9) as reported by the scripts End line. Single run; authors bundled pre-trained GMF/MLP weights used as the initialisation.

MLP with three hidden layers (32->16->8, embedding 16; the README's --layers [64,32,16,8]) reaches HR@10 = 0.671 on MovieLens-1M at 8 predictive factors.

Consistent

paper 0.671observed 0.6707

Paper 0.671 ratioObserved 0.6707 ratioAccepted if within ±0.02

8 0.452 0.628 0.655 0.671 0.678

Table 3 (MovieLens rows, Factors = 8, MLP-3 column), p. 8 · quote verified on p. 8 · equivalence check · observed/published 0.9996 · registered at event 16, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. HR@10 of MLP-3, 8 predictive factors, MovieLens-1M

Event 61. Observed 0.6707, delta -0.0003, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). README MLP command (layers [64,32,16,8], 20 epochs, batch 256, 4 negatives, Adam 0.001) on the primary commit 312aff0 with Keras 1.0.7/Theano 0.8.0 on CPU, re-run on a fresh droplet after the first run was lost (events 44-48); value is the scripts End-of-run best-epoch HR@10 (epoch 18 of 20). Per-epoch HR@10 over the last 6 epochs ranged 0.6573-0.6707; final epoch (19) gave 0.6674. Single run, single seed.

MLP with three hidden layers reaches NDCG@10 = 0.399 on MovieLens-1M at 8 predictive factors.

Consistent

paper 0.399observed 0.3979

Paper 0.399 ratioObserved 0.3979 ratioAccepted if within ±0.02

8 0.253 0.359 0.383 0.399 0.406

Table 4 (MovieLens rows, Factors = 8, MLP-3 column), p. 9 · quote verified on p. 9 · equivalence check · observed/published 0.9972 · registered at event 17, before the first recorded result (after command output had been observed)

Method, tolerance rationale and evidence

Metric. NDCG@10 of MLP-3, 8 predictive factors, MovieLens-1M

Event 62. Observed 0.3979, delta -0.0011, consistent. Evidence events [59] (value matched by the legacy rule; not bound to an excerpt). Same run as mlp3-8-hr10 (job mlp3-8-rerun, event 59): NDCG@10 at the best-HR epoch (18) as reported by the scripts End line. Final epoch (19) reached NDCG@10 = 0.3985. Single run.

Deviations from the paper's protocol

none
All training and evaluation run on CPU (Theano device=cpu, 3.5-CPU/6 GB container on a shared 4-vCPU DigitalOcean droplet, OMP_NUM_THREADS=1), not on whatever hardware the authors used.
The engine is CPU-only. The paper does not state its hardware; the models are small (8 predictive factors) and the authors code contains no GPU-specific logic, so the device changes speed, not the computation. event 24
none
Python 2.7.15 with Keras 1.0.7 and Theano 0.8.0 exactly as the README specifies, but with numpy 1.16.5, scipy 1.2.1 and h5py 2.10.0 (the last Python 2 releases from conda-forge/PyPI) rather than the unspecified 2017-era versions; Theano compiles with the sandbox system g++ 14.2.
The README pins only Keras and Theano; those pins installed exactly. The scientific-stack versions are not stated anywhere in the paper or repository. event 25
minor
For the NeuMF with pre-training claims, the pre-trained GMF (8 factors) and MLP ([64,32,16,8]) weights are the authors files bundled in the repository (Pretrain/ml-1m_GMF_8_1501651698.h5 and Pretrain/ml-1m_MLP_[64,32,16,8]_1501652038.h5, as named in the README command) rather than GMF and MLP models retrained in this attempt.
Time budget: retraining GMF and MLP first (about 45-70 min each on CPU) and then NeuMF would exceed the 2-3 hour window with at most two concurrent training processes on the shared box. The README with-pre-training command itself uses these files. The with-pre-training result therefore tests the NeuMF fine-tuning stage on the authors initialisation, not the full GMF->MLP->NeuMF pipeline. event 26
minor
The cloud sandbox was destroyed mid-attempt by another process (shared droplet torn down at ~15:17Z) and rebuilt on a fresh droplet: the Python 2.7 / Keras 1.0.7 / Theano 0.8.0 environment and the authors repository were set up a second time with the same recipe and commit, and the MLP-3 run (job mlp3-8) was restarted from scratch. The NeuMF results were complete before the loss and were not re-run.
Operator error on a shared box, not anything in the paper or code. The environment recipe is deterministic (pinned versions), so the second setup is equivalent; the restarted MLP run is a fresh single run like the original would have been. event 48

Notes from the attempt

event 27
Plan: GMF timing (event 21) gives ~128 s/epoch train + ~10 s eval at 8 factors, so 20 epochs of GMF is ~46 min and NeuMF/MLP are slower. With at most two concurrent training processes on the shared droplet, the order is: NeuMF without pre-training and NeuMF with pre-training now (20 epochs each, README commands), then MLP-3 (README MLP command) when a slot frees. GMF at 8 factors has no table value in the paper (Figure 4 only) and is not a registered claim. Observed values will be taken from the scripts own End-of-run line (best epoch by HR@10 over the 20 evaluated epochs), which is how the authors code reports a run; the paper does not say whether Table 2 reports the last or the best iteration.
event 47
Infrastructure incident: the shared DigitalOcean droplet (repro-sandbox-20261019-1321, hosting this attempts container and its /work volume alongside another attempts container) was destroyed by another process (the other attempts operator ran repro cloud down) at about 15:17Z, while job mlp3-8 was at iteration ~12 of 20. doctl now lists no droplets and SSH to the box times out (event 44 error, this note). The sandbox, the /env/py27 environment, the cloned code and the in-flight MLP job were lost. Results recorded before this (events 34 and 40, claims at events 38, 39, 42, 43) were produced and logged before the loss and remain valid evidence. Plan: bring up a fresh box, rebuild the environment with the same recipe, re-clone the primary commit and re-run only the lost MLP-3 job within the remaining budget.

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude (Claude Code subagent, AI-driven)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-19T13:52:18.131Z → 2026-10-19T16:00:36.780Z
Event log: 67 events, hash-chained; head 04ac3824830007550ce23545b33c04ec2f8c290dd7107cca03355bd8ce4ddfc6
Verify: python3 openrepro_verify.py arxiv-1708-05031-20261019T135218Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1).
Right of reply: Authors may submit a response, which is published alongside this report unedited.