Replication record · arXiv:1607.01759

Bag of Tricks for Efficient Text Classification

Armand Joulin, Édouard Grave, Piotr Bojanowski, Tomáš Mikolov · 2017 · doi:10.18653/v1/e17-2068 · arXiv

Reproduced

With fastText at the commit of arXiv v3 and its classification-results.sh, test accuracy with 10 hidden units and bigrams reproduces Table 1 on all eight datasets: 92.2, 96.7, 98.6, 95.7, 63.9, 72.4, 60.2 and 94.6 against 92.5, 96.8, 98.6, 95.7, 63.9, 72.3, 60.2 and 94.6. The unigram row also reproduces, within 0.2 points everywhere. The script own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs, preprocessed with the script own function. Training speed (Table 2) was not tested.

Plan. Registered before results were recorded; not sealed (legacy).

Qualifiers. The plan was never sealed (legacy attempt): claims were frozen only by their first result. At least one result was matched by the legacy rule rather than bound to an excerpt. At least one scored value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.

Scored (headline): 8 of 8 consistent
Other checks: 8 of 8 consistent
Deviations: 3 (0 material)
Integrity flags: 0
Logged compute: 1.06 h over 2 commands and 1 jobs
Primary code (declared before the first recorded result): github.com/facebookresearch/fastText (linked in the paper) at the last commit on or before 2016-08-09 (arXiv v3), building with make and running classification-results.sh, whose header says it produces Table 1
Commits the recorded results name: 206179d: 16 results, named in the result's method text
Work tree when the attempt ended: facebookresearch/fastText@206179d64c17 in /work/fastText

Protocol

Declared before the first recorded result: primary code is github.com/facebookresearch/fastText (linked in the paper) at the last commit on or before 2016-08-09 (arXiv v3), building with make and running classification-results.sh, whose header says it produces Table 1; earlier versions of the authors' code may be tried as a fallback. arXiv v3 is the version tested. Data: the script own Google Drive download is used if it still works; otherwise the public s3.amazonaws.com/fast-ai-nlp mirror of the same dataset tarballs (declared here, before any download). Earlier commits may be tried as a declared fallback for misses.

Claims

fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News.headline

Consistent

paper 92.5observed 92.2

Paper 92.5 percentObserved 92.2 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9968 · registered at event 10, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], AG News, fastText h=10 with bigrams

Why this tolerance. Hogwild multithreaded SGD makes accuracy vary by about 0.1 point between runs, and the code version and data copy may differ slightly; values are reported to 0.1, so 0.3 points is used.

Event 33. Observed 92.2, delta -0.3, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News.

Consistent

paper 91.5observed 91.4

Paper 91.5 percentObserved 91.4 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9989 · registered at event 11, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], AG News, fastText h=10 without bigrams

Event 41. Observed 91.4, delta -0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News.headline

Consistent

paper 96.8observed 96.7

Paper 96.8 percentObserved 96.7 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.999 · registered at event 12, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Sogou News, fastText h=10 with bigrams

Event 34. Observed 96.7, delta -0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News.

Consistent

paper 93.9observed 93.7

Paper 93.9 percentObserved 93.7 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9979 · registered at event 13, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Sogou News, fastText h=10 without bigrams

Event 42. Observed 93.7, delta -0.2, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia.headline

Consistent

paper 98.6observed 98.6

Paper 98.6 percentObserved 98.6 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1 · registered at event 14, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], DBpedia, fastText h=10 with bigrams

Event 35. Observed 98.6, delta +0, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia.

Consistent

paper 98.1observed 98.3

Paper 98.1 percentObserved 98.3 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1.002 · registered at event 15, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], DBpedia, fastText h=10 without bigrams

Event 43. Observed 98.3, delta +0.2, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity.headline

Consistent

paper 95.7observed 95.7

Paper 95.7 percentObserved 95.7 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1 · registered at event 16, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yelp Polarity, fastText h=10 with bigrams

Event 36. Observed 95.7, delta +0, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity.

Consistent

paper 93.8observed 93.9

Paper 93.8 percentObserved 93.9 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1.001 · registered at event 17, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yelp Polarity, fastText h=10 without bigrams

Event 44. Observed 93.9, delta +0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full.headline

Consistent

paper 63.9observed 63.9

Paper 63.9 percentObserved 63.9 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1 · registered at event 18, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yelp Full, fastText h=10 with bigrams

Event 37. Observed 63.9, delta +0, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full.

Consistent

paper 60.4observed 60.3

Paper 60.4 percentObserved 60.3 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9983 · registered at event 19, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yelp Full, fastText h=10 without bigrams

Event 45. Observed 60.3, delta -0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers.headline

Consistent

paper 72.3observed 72.4

Paper 72.3 percentObserved 72.4 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1.001 · registered at event 20, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yahoo! Answers, fastText h=10 with bigrams

Event 38. Observed 72.4, delta +0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers.

Consistent

paper 72observed 71.8

Paper 72 percentObserved 71.8 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9972 · registered at event 21, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Yahoo! Answers, fastText h=10 without bigrams

Event 46. Observed 71.8, delta -0.2, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full.headline

Consistent

paper 60.2observed 60.2

Paper 60.2 percentObserved 60.2 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1 · registered at event 22, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Amazon Full, fastText h=10 with bigrams

Event 39. Observed 60.2, delta +0, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full.

Consistent

paper 55.8observed 55.6

Paper 55.8 percentObserved 55.6 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9964 · registered at event 23, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Amazon Full, fastText h=10 without bigrams

Event 47. Observed 55.6, delta -0.2, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity.headline

Consistent

paper 94.6observed 94.6

Paper 94.6 percentObserved 94.6 percentAccepted if within ±0.3

fastText, h = 10, bigram 92.5 96.8 98.6 95.7 63.9 72.3 60.2 94.6

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 1 · registered at event 24, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Amazon Polarity, fastText h=10 with bigrams

Event 40. Observed 94.6, delta +0, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity.

Consistent

paper 91.2observed 91.1

Paper 91.2 percentObserved 91.1 percentAccepted if within ±0.3

fastText, h = 10 91.5 93.9 98.1 93.8 60.4 72.0 55.8 91.2

Table 1, p. 3 · quote verified on p. 3 · equivalence check · observed/published 0.9989 · registered at event 25, before the first recorded result

Method, tolerance rationale and evidence

Metric. test accuracy [%], Amazon Polarity, fastText h=10 without bigrams

Event 48. Observed 91.1, delta -0.1, consistent. Evidence events [31] (value matched by the legacy rule; not bound to an excerpt). fastText at commit 206179d (2016-08-09) built with make; classification-results.sh unchanged for bigrams (-dim 10 -wordNgrams 2 -epoch 5 -thread 4, per-dataset learning rates from the script), the same command with -wordNgrams 1 for the unigram row; test P@1 printed by fasttext test, as a percentage.

Deviations from the paper's protocol

minor
Datasets came from the public s3.amazonaws.com/fast-ai-nlp mirror (tarball hashes in the job output) instead of the script own googledrive.com/host download, then preprocessed with the script own normalize_text function.
googledrive.com/host returned HTTP 404 (event 31); this fallback was declared in the protocol before any download. Train/test sizes match the standard splits (e.g. AG 120,000/7,600). event 49
none
The unigram row was produced with the same command as the script but -wordNgrams 1; the script itself only runs the bigram configuration.
The paper reports fastText with and without bigrams under the same parameters. event 50
minor
Built with g++ 14 on Debian 13 and trained with 4 threads on a 4 vCPU shared droplet.
Current toolchain; Hogwild training is nondeterministic across thread schedules, which the tolerance allows for. event 51

Assurance

What the harness enforced for this attempt, and what it predates. A legacy attempt is labelled, never rewritten.

Plan sealed: no
Sealed before scored execution: no
Evidence bound to artifact excerpts: no
Values extracted by a sealed rule: no
Execution manifest for cited jobs: no
Semantic re-derivation passed: yes
Predates: a sealed plan before scored execution, evidence bound to an exact artifact excerpt, an execution manifest per job (HEAD, status, diff hash, driver file hashes), the paper text hashed at create, jobs declared scored or unscored, a declared reason for every repeated result, an extraction rule per claim, fixed by the seal
Engine: report built by repro 0.2.1; attempt created by repro 0.1.0

Provenance

Driven by: operator:claude-opus-5-5 (Claude Code session, no human in the loop)
Sandbox: 3.5 CPUs, 6.00 GiB memory, no GPU · image repro-sandbox:0.2
Period: 2026-10-18T22:37:13.283Z → 2026-10-18T23:44:49.124Z
Event log: 56 events, hash-chained; head 87b5ac6197ec5f210cb74ba6aaa5412fe8e2237bdf845f1afee95be9043de61d
Verify: python3 openrepro_verify.py arxiv-1607-01759-20261018T223713Z checks the chain and every artifact with the Python standard library alone; repro verify also recomputes every quote match, evidence check and verdict and re-derives this report (engine 0.2.1).
Right of reply: Authors may submit a response, which is published alongside this report unedited.