Natural language processing · 2017
Bag of Tricks for Efficient Text Classification
Does fastText (Joulin et al., 2017) reproduce? Yes, on its headline claims: when OpenRepro re-ran the authors' code, 8 of 8 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it (checked 18 Oct 2026).
Does “fastText” reproduce?
Yes, on its headline claims. 8 of 8 headline claims held within tolerances registered before results were recorded; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it.
- Held (large: headline claim; small: other claim)
The data behind this chart
| Claim | Published | Observed | Accepted if | Deviation ÷ tolerance | Result |
|---|---|---|---|---|---|
| fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News. (headline) | 92.5 | 92.2 | within ±0.3 | -1.00 | Held |
| fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News. | 91.5 | 91.4 | within ±0.3 | -0.33 | Held |
| fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News. (headline) | 96.8 | 96.7 | within ±0.3 | -0.33 | Held |
| fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News. | 93.9 | 93.7 | within ±0.3 | -0.67 | Held |
| fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia. (headline) | 98.6 | 98.6 | within ±0.3 | +0.00 | Held |
| fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia. | 98.1 | 98.3 | within ±0.3 | +0.67 | Held |
| fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity. (headline) | 95.7 | 95.7 | within ±0.3 | +0.00 | Held |
| fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity. | 93.8 | 93.9 | within ±0.3 | +0.33 | Held |
| fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full. (headline) | 63.9 | 63.9 | within ±0.3 | +0.00 | Held |
| fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full. | 60.4 | 60.3 | within ±0.3 | -0.33 | Held |
| fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers. (headline) | 72.3 | 72.4 | within ±0.3 | +0.33 | Held |
| fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers. | 72 | 71.8 | within ±0.3 | -0.67 | Held |
| fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full. (headline) | 60.2 | 60.2 | within ±0.3 | +0.00 | Held |
| fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full. | 55.8 | 55.6 | within ±0.3 | -0.67 | Held |
| fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity. (headline) | 94.6 | 94.6 | within ±0.3 | +0.00 | Held |
| fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity. | 91.2 | 91.1 | within ±0.3 | -0.33 | Held |
What the attempt found
With fastText at the commit of arXiv v3 and its classification-results.sh, test accuracy with 10 hidden units and bigrams reproduces Table 1 on all eight datasets: 92.2, 96.7, 98.6, 95.7, 63.9, 72.4, 60.2 and 94.6 against 92.5, 96.8, 98.6, 95.7, 63.9, 72.3, 60.2 and 94.6. The unigram row also reproduces, within 0.2 points everywhere. The script own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs, preprocessed with the script own function. Training speed (Table 2) was not tested.
Assurance
- Level
- Legacy (what the levels mean)
- Plan
- Not sealed (legacy): claims were registered before results were recorded, which does not show that no output had been seen
- Evidence
- Legacy matching: each observed value was matched against the text of the logged output it cites
- Execution manifest
- No: not recorded for every cited run, or incomplete (no Git checkout, a driver file that could not be hashed)
- Report re-derived from the log
- Passed: the engine re-derived the report from the log
- Engine version
- 0.2.1
- Predates
- plan seal, evidence binding, execution manifest, paper text hash, scored job flag, retry reasons, extraction rule
- Qualifiers
- legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it
Headline claims
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News.¶Table 1, p. 3 · registered at event 10 · quote verified on p. 3 · equivalence check | 92.5percent | 92.2 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News.¶Table 1, p. 3 · registered at event 12 · quote verified on p. 3 · equivalence check | 96.8percent | 96.7 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia.¶Table 1, p. 3 · registered at event 14 · quote verified on p. 3 · equivalence check | 98.6percent | 98.6 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity.¶Table 1, p. 3 · registered at event 16 · quote verified on p. 3 · equivalence check | 95.7percent | 95.7 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full.¶Table 1, p. 3 · registered at event 18 · quote verified on p. 3 · equivalence check | 63.9percent | 63.9 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers.¶Table 1, p. 3 · registered at event 20 · quote verified on p. 3 · equivalence check | 72.3percent | 72.4 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full.¶Table 1, p. 3 · registered at event 22 · quote verified on p. 3 · equivalence check | 60.2percent | 60.2 | within ±0.3 | Held |
| fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity.¶Table 1, p. 3 · registered at event 24 · quote verified on p. 3 · equivalence check | 94.6percent | 94.6 | within ±0.3 | Held |
Other claims checked
| Claim | Published | Observed | Accepted if | Result |
|---|---|---|---|---|
| fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News.¶Table 1, p. 3 · registered at event 11 · quote verified on p. 3 · equivalence check | 91.5percent | 91.4 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News.¶Table 1, p. 3 · registered at event 13 · quote verified on p. 3 · equivalence check | 93.9percent | 93.7 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia.¶Table 1, p. 3 · registered at event 15 · quote verified on p. 3 · equivalence check | 98.1percent | 98.3 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity.¶Table 1, p. 3 · registered at event 17 · quote verified on p. 3 · equivalence check | 93.8percent | 93.9 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full.¶Table 1, p. 3 · registered at event 19 · quote verified on p. 3 · equivalence check | 60.4percent | 60.3 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers.¶Table 1, p. 3 · registered at event 21 · quote verified on p. 3 · equivalence check | 72percent | 71.8 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full.¶Table 1, p. 3 · registered at event 23 · quote verified on p. 3 · equivalence check | 55.8percent | 55.6 | within ±0.3 | Held |
| fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity.¶Table 1, p. 3 · registered at event 25 · quote verified on p. 3 · equivalence check | 91.2percent | 91.1 | within ±0.3 | Held |
Deviations from the paper's setup
- minorDatasets came from the public s3.amazonaws.com/fast-ai-nlp mirror (tarball hashes in the job output) instead of the script own googledrive.com/host download, then preprocessed with the script own normalize_text function.
- noneThe unigram row was produced with the same command as the script but -wordNgrams 1; the script itself only runs the bigram configuration.
- minorBuilt with g++ 14 on Debian 13 and trained with 4 threads on a 4 vCPU shared droplet.
The run
- Wall time
- 1 h 7 min, 18 October 2026 to 18 October 2026
- Job compute
- 1 h 3 min across 1 job
- Code the results name
- facebookresearch/fastText at 206179d
- Work tree at the end
- facebookresearch/fastText at 206179d
- Container image
- repro-sandbox:0.2 sha256:f4b241036e33…
- Sandbox
- 3.5 CPUs, 6 GB, no GPU
- Run by
- Claude (claude-opus-5-5), an AI agent made by Anthropic, in a Claude Code session
- Attempt
- arxiv-1607-01759-20261018T223713Z
- Log
- 56 events, head 87b5ac61
Evidence
The full evidence report has the exact paper quotes, the method behind every number, the reason for each tolerance, notes made during the attempt and the environment it ran in. The raw report.json is the unmodified data this page is built from.
Log seal · 56 hash-chained events
87b5ac6197ec5f210cb74ba6aaa5412fe8e2237bdf845f1afee95be9043de61d
How to cite this report
OpenRepro (2026). Reproducibility report: Bag of Tricks for Efficient Text Classification. Reproduced; 8 of 8 headline claims held (legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it). Attempt arxiv-1607-01759-20261018T223713Z, log head 87b5ac6197ec5f21, checked 18 Oct 2026. https://openrepro.org/papers/1607.01759/
BibTeX
@techreport{openrepro-1607.01759,
author = {{OpenRepro}},
title = {Reproducibility report: Bag of Tricks for Efficient Text Classification},
institution = {OpenRepro},
year = {2026},
url = {https://openrepro.org/papers/1607.01759/},
note = {Attempt arxiv-1607-01759-20261018T223713Z; log head 87b5ac6197ec5f21; checked 18 Oct 2026}
}
Badge
Authors and maintainers may embed this badge; it always reflects the current report, and a correction updates it.
[](https://openrepro.org/papers/1607.01759/)<a href="https://openrepro.org/papers/1607.01759/"><img src="https://openrepro.org/badge/1607.01759.svg" alt="OpenRepro: Reproduced. 8 of 8 headline claims held; legacy attempt: the plan was never sealed; claims were frozen only by their first result; at least one value was read from an excerpt the agent chose after seeing the output, not by a rule fixed before it." width="416" height="20"></a>Right of reply, and reporting an error
Authors may submit a response, which is published alongside this report unedited. Write to hello@openrepro.org.
Anyone, not only an author, can report an error in an attempt to the same address. The same address takes suggestions of papers to check and questions about the method.