I test published claims about small language models on hardware most people have: a 6 GB GTX 1660 Ti and free Kaggle T4 GPUs. So far the questions are how much a model forgets when trained on a new language, what 2-bit compression costs rare words, and whether targeted repairs can recover it.
Matthew MacKinnon·Now running: checking whether interpretability findings transfer to a larger model · Updated Nov 1, 2026
Show notebook sections
Learning Korean with less forgetting
Training Pythia-160M on 5M tokens of Korean raised its loss on English and code. Using the paper's larger Adam ε on the output layer and replaying 1% old text with a KL loss that keeps predictions near the original model cut that rise by 81.4% (95% CI 80.5–82.3%) across five paired seeds.
This measures loss retention in one setup; residual forgetting remains.
After 2-bit compression, both Granite-1B and Qwen3-1.7B scored lower on rare-word definitions than on common ones at the same uncompressed score: gaps of −1.47 [−2.23, −0.75] and −1.11 [−2.10, −0.16] nats.
These teacher-forced scores show an association, not generated-definition quality or a causal effect of word frequency.
Biology-inspired rules failed against tuned baselines; a memory proposal stopped at prior art. The work then moved to mapping the field, reproducing a forgetting benchmark and testing specific fixes.
Finding. At each model’s pooled mean BF16 score, rare-minus-common gaps were −1.47 [−2.23, −0.75] for Granite-1B and −1.11 [−2.10, −0.16] for Qwen3-1.7B; this association does not show that frequency caused the difference.
Criterion not met. Neither targeted group beat size-matched random restoration on rare words. With clean inputs, 57.4% [49.4, 64.9] of the damage was recovered; this does not show that any one component caused that share.
Inconclusive. At two removal strengths that gave usable comparisons, the extra rare-versus-common score loss was −0.045 [−0.25, +0.18] and +0.14 [−0.18, +0.48]; both intervals include zero.
Show setup, results and limitsSeeds, predictions and caveats for the paper comparison
Method
Seed
Training
Loss rise (nats/token)
No fix
51
0M
0.0000
No fix
51
5.00531M
1.2099
No fix
51
10.0024M
1.2107
No fix
51
20.0049M
1.0391
No fix
51
30.0073M
0.8589
No fix
51
40M
0.8258
No fix
52
0M
0.0000
No fix
52
5.00531M
1.3114
No fix
52
10.0024M
1.2112
No fix
52
20.0049M
1.0334
No fix
52
30.0073M
0.8963
No fix
52
40M
0.8779
Output-layer ε
51
0M
0.0000
Output-layer ε
51
5.00531M
0.4114
Output-layer ε
51
10.0024M
0.4472
Output-layer ε
51
20.0049M
0.4644
Output-layer ε
51
30.0073M
0.4605
Output-layer ε
51
40M
0.4485
Output-layer ε
52
0M
0.0000
Output-layer ε
52
5.00531M
0.4753
Output-layer ε
52
10.0024M
0.5049
Output-layer ε
52
20.0049M
0.4610
Output-layer ε
52
30.0073M
0.4677
Output-layer ε
52
40M
0.4330
ε + 1% KL replay
51
0M
0.0000
ε + 1% KL replay
51
5.00531M
0.2534
ε + 1% KL replay
51
10.0024M
0.2819
ε + 1% KL replay
51
20.0049M
0.2843
ε + 1% KL replay
51
30.0073M
0.2527
ε + 1% KL replay
51
40M
0.2407
ε + 1% KL replay
52
0M
0.0000
ε + 1% KL replay
52
5.00531M
0.2761
ε + 1% KL replay
52
10.0024M
0.3154
ε + 1% KL replay
52
20.0049M
0.2746
ε + 1% KL replay
52
30.0073M
0.2736
ε + 1% KL replay
52
40M
0.2521
No fix
Output-layer ε
ε + 1% KL replay
Seed 51 Seed 52
Figure 4. Pythia-160M, Korean continued pretraining, lr 3e-5, Kaggle T4; two paired seeds. Solid circles mark seed 51; dashed squares mark seed 52. Forgetting is the mean held-out English and code loss rise, in nats per token. Data (CSV)
Earlier benchmark check: SmolLM2 on SST-2 and AG News
Across three seeds and two answer-token conditions, the reproduced benchmark matched the paper’s qualitative pattern: old answers lost probability mass, old-task accuracy fell, and resetting optimizer momentum—the running average of recent gradients—reduced forgetting. With separate answer tokens for each task, forgetting was about three times larger than reported, so this is a qualitative rather than magnitude match. Each run took about six minutes on the GTX 1660 Ti, and the benchmark now serves as a fixed scorer with held-out test texts.
What I tested
I took Pythia-160M, a small open model trained on English and code (the Pile), and kept training it on Korean Wikipedia. I measured forgetting as the rise in English and code validation loss, in nats per token. A rise of 0.1 nats means about 10% higher perplexity, a measure of how poorly the model predicts text.
The first round ran for 5M tokens with five paired seeds per method, using the same random starting conditions across methods. I compared no fix, the paper’s larger ε in the Adam optimizer (a numerical stabilizer) on the output layer, resetting its stored momentum, replaying 1% old text with a Kullback–Leibler (KL) loss that keeps predictions near the original model, and ε plus replay.
Output-layer ε cut forgetting by 70.7% (95% CI 69.0–72.3%), momentum reset by 3.7% (2.0–5.4%), replay by 75.1% (73.1–77.0%), and ε plus replay by 81.4% (80.5–82.3%). Korean validation loss fell in all 25 runs, and the combined fix improved Korean learning by 1.38% over no fix (95% CI 0.19–2.56%).
These five seeds measure run-to-run variation for this one model, corpus, learning rate and token budget, not across setups. Three of the fifteen momentum resets missed my planned 0.1% norm-matching tolerance (the worst was off by 1.03%), so that arm is a less clean test of the reset than I intended.
Method
Seed 51
Seed 52
Seed 53
Seed 54
Seed 55
Mean
No fix
0.9674
0.9604
0.9613
0.9543
0.9777
0.9642
Output-layer ε
0.2887
0.2889
0.2947
0.2601
0.2824
0.2829
Momentum reset
0.9267
0.9238
0.9484
0.9154
0.9284
0.9285
1% replay + KL
0.2338
0.2486
0.2573
0.2413
0.2213
0.2404
Output-layer ε + replay
0.1830
0.1794
0.1856
0.1781
0.1704
0.1793
Table 4. Loss rise by seed; each cell is the mean over the two old domain evaluations.
What I predicted
I wrote my predictions down before the runs. Momentum reset did little, as I expected, and ε landed inside my predicted 50–85% reduction. Replay beat my predicted 15–50%. I expected ε to leave Korean learning within the baseline spread, but it improved by 1.28% on average, so I count that prediction as missed. The combined fix still left 0.1793 nats of forgetting, about 20% higher perplexity on English and code than the original model.
After the pilot, I clarified that “learning change inside the baseline spread” means a fix should not learn Korean worse than baseline by more than the seed spread; a learning gain does not fail the rule. I recorded this after seeing pilot results. The original frozen prediction remains unchanged and counts as missed.
At 40M tokens
At 40M tokens, Korean held-out loss was 1.2905 → 1.2769 (1.05% lower) in seed 51 and 1.2991 → 1.2840 (1.16% lower) in seed 52.
Arm
Seed 51 (nats)
Seed 52 (nats)
Reduction vs baseline (51 / 52)
No fix
0.826
0.878
0% / 0%
Output-layer ε
0.449
0.433
45.7% / 50.7%
ε + 1% KL replay
0.241
0.252
70.9% / 71.3%
Table 5. Forgetting is mean held-out English and code loss rise, in nats per token. Each reduction is computed against its baseline within the same seed using unrounded values.
For output-layer ε alone, the paper cut Pythia-160M forgetting by 73.2% (1.448 → 0.388 nats across three seeds); my two seeds gave 45.7% and 50.7%. This is context, not a matched replication: my Korean data and evaluation differ, and my baseline rise was about 0.85 nats. Two seeds do not characterize run-to-run variation reliably.
Rare-word definition scores under 2-bit compression
Quantization stores model weights in lower-bit formats to shrink files, sometimes changing scores. Q2_K is the 2-bit format; Q3_K_M is the 3-bit comparison. For each fresh word chosen before seeing model scores, I ask the model to select a definition from fixed alternatives. The score is the correct synonym’s log-probability advantage over alternatives, in nats, compared at each model’s pooled mean BF16 baseline score. Teacher-forced scoring supplies the answer options and measures their likelihood instead of asking the model to generate definitions.
Show exact score differences
Model
Rare minus common score (nats [95% CI])
Granite-1B
−1.47 [−2.23, −0.75]
Qwen3-1.7B
−1.11 [−2.10, −0.16]
The test was preregistered. This score does not measure generated definitions or language knowledge in general.
Show earlier compression resultsJudge, matcher, paper and three-bit comparisons
The earlier judged test covered 529 words (429 rare, 100 common); at 2 bits, correctly judged rare definitions fell from 423 to 275. The companion paper’s Granite-1B judge reported a 43.6% rare-word and 12.0% common-word drop; my blind judge found 35.0% and 4.0%, and a fixed matcher found 33.1% and 11.5%. These use a different scoring task and are not comparable to the new teacher-forced score.
Format
Labels
Rare net change
Common net change
Common examples scored
Baseline-correct common words
Q2_K
Matcher
−33.1%
−11.5%
100
87
Q2_K
Judge
−35.0%
−4.0%
100
99
Q3_K_M
Matcher
−4.2%
−12.6%
100
87
Q3_K_M
Judge
−2.8%
+1.0%
100
99
Table 6. Both labelers scored 100 common examples; 87 matcher-scored and 99 judged words were baseline-correct. Percentages are net changes from each label set’s baseline-correct words. The judge agreed with a separate pass on 101 of 104 texts; I had seen three texts for one word beforehand.
At 3 bits, the fresh rare-minus-common gap was −0.04 [−0.47, +0.38] for Granite and −1.16 [−1.73, −0.62] for Qwen. These formats use unmatched bit budgets, so the results are context, not a head-to-head comparison. In the earlier judged test, rare definitions fell 2.8% at 3 bits; the paper reported 3.6% under a different judge and evaluation. Score drop ranked which definitions broke, but did not explain why.
Ranking measure
AUC
95% CI
Score drop
0.68
[0.62, 0.73]
Before compression
0.55
Unknown
Remaining headroom
0.73
Unknown
Figure 6. Ranking scores for 528 words, including 154 lost definitions. Area under the receiver operating characteristic curve (AUC) was 0.68 [0.62, 0.73] for score drop, 0.55 for the pre-compression score, and 0.73 for remaining headroom; 0.5 is chance. The 0.05 score-drop versus headroom difference had an interval of −0.02 to +0.11, so the rankings were not separated.
Show weak-activity and other checksMatched-cost removal, shared fragility and calibration
In Qwen3-1.7B, I removed weak activations (internal signals) in the multilayer perceptron (MLP) layers for 192 rare and 64 common words; q controls how much activity is removed.
Two interpretable matched-cost comparisons; both intervals include zero.
Planted score penalty
1.0-nat rare-only penalty estimated at +0.985
The scoring estimator recovered a planted score change; neuron-level sensitivity remains untested.
Shared fragility
Adjusted cross-method r=0.11–0.17 vs 0.30 preregistered bar; Granite–Qwen r=0.17 on 347 shared words
Weak shared fragility, not no relationship.
Calibration text
−0.28 [−0.66, +0.12] vs +1.0 target
The interval includes zero and does not meet the preset target.
Positive weak-activity values mean extra rare-word damage relative to strong-activity removal.
Only q=0.25 and 0.50 gave usable comparisons with strong-activation removal matched within 10% on plain-text loss, not definition-scoring cost. At q=0.1, the random control cost about 120 times more; at q=0.75, strong-activation removal hurt rare target words far more. The table reports extra rare-versus-common score loss under weak removal relative to the control. The planted score penalty checks the estimator, not sensitivity to damage inside neurons. Fragility matching used generic-text perplexity, not definition-scoring cost; the calibration interval still allows a smaller benefit.
Testing selective precision on fresh words
Selected Mamba projection matrices (weights that transform layer inputs) were stored at higher precision to keep file size near ordinary 3-bit compression. Mamba is a recurrent sequence-model design; negative score differences favor this allocation.
Cohort
Selective-precision model minus calibrated Q3_K_S (3-bit), mean [95% confidence interval]
Rare (n=236)
−0.154 [−0.558, +0.267]
Common (n=36)
−1.302 [−2.152, −0.430]
Table 7. Words withheld from model selection. Negative differences favor selective precision; the rule set before this evaluation required the full 95% confidence interval to fall below zero in both cohorts.
Show methods and interpretationHow the model was tested and what the result supports
The initial 128-word evaluation reused items during development; its 52.2% [41.6%, 62.4%] rare-word recovery was exploratory. On 272 held-out words, the common interval met the preset rule but the rare interval crossed zero, so a rare-word benefit was not established.
The hybrid file was 712 MB, versus 695 MB for calibrated Q3_K_S and about 732 MB for Q3_K_M. Related projection protection appears in Quamba and Quamba2; no published paper reports a directly comparable contrast-damage score.
Where does compression damage enter the Mamba layers?
Neither timing-related B+C+dt nor content-gate z+x beat a size-matched random restore on rare words. With clean inputs, 57.44% [49.38%, 64.91%] of rare-word damage recovered; this reference shows a possible recovery when every input is clean, not what any one component caused.
Target and paired comparisons are percentage points of 2-bit score damage recovered per 100 MB restored, rounded to two significant figures. The clean-input row is the share recovered, rounded to three significant figures.
Comparison
Rare-word result
Common-word result
B+C+dt vs size-matched random
+0.23 [−7.2, +8.0]
+11 [−7.2, +30]
z+x vs size-matched random
−0.30 [−0.77, +0.16]
−1.3 [−2.4, −0.44]
Output projection vs size-matched random
+3.0 [+0.33, +5.7]
−11 [−21, −4.0]
B+C+dt minus all of z+x
−2.0 [−9.6, +5.9]
+4.9 [−14, +22]
B+C+dt minus same-size slice of z+x
+1.4 [−5.9, +9.0]
+15 [−3.9, +34]
All Mamba inputs restored to clean BF16 (weights stay 2-bit)
57.4% [49.4%, 64.9%]
23.1%
Show exact restoration results
Target and paired comparisons are percentage points of 2-bit score damage recovered per 100 MB restored, rounded to two significant figures. The clean-input row is the share recovered, rounded to three significant figures.
Comparison
Rare-word result
Common-word result
B+C+dt vs size-matched random
+0.23 [−7.2, +8.0]
+11 [−7.2, +30]
z+x vs size-matched random
−0.30 [−0.77, +0.16]
−1.3 [−2.4, −0.44]
Output projection vs size-matched random
+3.0 [+0.33, +5.7]
−11 [−21, −4.0]
B+C+dt minus all of z+x
−2.0 [−9.6, +5.9]
+4.9 [−14, +22]
B+C+dt minus same-size slice of z+x
+1.4 [−5.9, +9.0]
+15 [−3.9, +34]
All Mamba inputs restored to clean BF16 (weights stay 2-bit)
57.4% [49.4%, 64.9%]
23.1%
Recovery differences are percentage points of 2-bit score damage recovered per 100 MB restored. Bracketed intervals are 95% bootstraps over words. The preregistered localization criterion was not met: neither target’s rare-word interval was wholly above zero versus random. Output projection did beat random, but it was not a timing-versus-content target.
Show further diagnosticsPer-megabyte comparisons and the audit
The timing-over-content estimate is mostly a size artifact: 90.6% comes from dividing the shared clean-input rescue by the 28.1 MB timing group versus the 568 MB content group. On the per-100 MB scale, that estimate is 214 [188, 239] percentage points. Subtracting the clean-input rescue from plain Q2 weights first leaves an exploratory 20.2 [16.6, 24.4] points per 100 MB; absolute incremental recovery favors content, 11.53 versus 6.23 percentage points. In the middle 12 layers, z+x restoration lowered local mixer-output root-mean-square error from 0.347 to 0.241 while total error barely changed. The local and accumulated errors interact rather than add cleanly, so this suggests sensitivity to incoming-signal damage but does not identify one cause.
Independent CPU audits reproduced the stored scores and primary intervals. Fresh GPU swaps and uncached component diagnostics remain unverified. No directly comparable published timing-versus-content localization number was found.
Research map and question selection
The map
Select a paper to open it; zoom between broad regions and topics.
Region names and counts from the research map.
Region
Papers
In scope
Momentum
Architecture & training efficiency
1,258
975
1.04×
Data, distillation & knowledge
901
716
0.95×
Multilingual & translation
267
78
0.72×
Interpretability & cognition
676
456
0.91×
Safety, privacy & unlearning
696
385
1.06×
Code & cybersecurity
563
84
1.17×
Reasoning, RL & agents
1,581
1,060
1.22×
RAG, retrieval & applied NLP
748
371
0.91×
Education, law & social text
492
106
0.85×
Medicine & biology
741
3
0.93×
Quantization & inference hardware
487
185
0.95×
Edge, on-device & federated
861
202
0.86×
Speech & audio
279
169
1.02×
Time series, finance & industry
367
21
1.22×
Vision-language models
514
359
0.73×
Robotics, video & embodied
628
147
1.04×
Figure 1. Each point is a paper, placed near papers with similar titles and abstracts. Color shows its region or publication month; brighter marks are newer. Hover for details and select a paper to open it. The map covers 11,059 papers published January 2025 to Oct 2026.
Show Table 1:16 regions with counts, momentum and hardware mix
Region names and counts from the research map.
Region
Papers
In scope
Momentum
Architecture & training efficiency
1,258
975
1.04×
Data, distillation & knowledge
901
716
0.95×
Multilingual & translation
267
78
0.72×
Interpretability & cognition
676
456
0.91×
Safety, privacy & unlearning
696
385
1.06×
Code & cybersecurity
563
84
1.17×
Reasoning, RL & agents
1,581
1,060
1.22×
RAG, retrieval & applied NLP
748
371
0.91×
Education, law & social text
492
106
0.85×
Medicine & biology
741
3
0.93×
Quantization & inference hardware
487
185
0.95×
Edge, on-device & federated
861
202
0.86×
Speech & audio
279
169
1.02×
Time series, finance & industry
367
21
1.22×
Vision-language models
514
359
0.73×
Robotics, video & embodied
628
147
1.04×
Table 1. Regions sorted by momentum; expand a region for its topics. Momentum compares recent and full-corpus shares. In-scope papers concern general methods, analyses, benchmarks or model releases, not applications, systems or surveys. Hardware fit is the least powerful available GPU that can run the smallest reported experiment.
How I chose the first questions
Paper counts and recent growth show where attention is moving; open problems and hardware limits help decide what I can test. The map is a sample of papers from January 2025 onward, not a complete census, and its categories and hardware estimates depend on paper abstracts and reported configurations.
From topics with at least 20 papers about general methods, analyses, benchmarks, or model releases, I compared seven candidate areas on six criteria: small models, an existing benchmark, evidence at small scale, compute fit, room beyond prior work, and relevant tools or measurements already available. Their scores were within one point, so they organize a judgment rather than choose a winner by themselves. I chose forgetting because three October papers proposed different continued-training fixes without comparing them, making the gap feasible to test without claiming it is the field’s most important problem.
The fastest-growing topics mostly need compute I don’t have
Topics with at least 20 in-scope papers.
Topic
In-scope papers
Momentum
Fits available GPUs
Agentic RL
80
2.01×
17%
Policy optimization methods
72
1.84×
19%
Small-model agents
41
1.72×
75%
Agent memory
126
1.57×
51%
Mixture-of-experts
95
1.40×
39%
On-policy & preference distillation
82
1.38×
20%
RLVR for general & multimodal reasoning
127
1.33×
17%
Attention & architecture analysis
134
1.28×
69%
Memorization & self-consistency
61
1.25×
79%
Distillation & activation steering
86
1.25×
58%
Sparse & long-context attention
146
1.24×
41%
Edge-cloud routing & on-device RAG
34
1.24×
92%
Robot planning & manipulation
32
1.23×
38%
Diffusion language models
69
1.23×
27%
Scaling laws & optimizers
112
1.21×
50%
Privacy & membership inference
119
1.21×
69%
Learning-rate schedules
64
1.19×
41%
Low-rank & parameter-efficient training
104
1.17×
58%
Tokenizers & cross-tokenizer distillation
81
1.15×
44%
Scientific text mining
27
1.15×
54%
Sparse autoencoders & circuits
140
1.13×
87%
Generation diversity & decoding
50
1.13×
43%
Post-training & low-bit quantization
159
1.10×
66%
Hallucination & calibration
113
1.07×
83%
Brain & cognitive alignment
29
1.07×
91%
Text-to-SQL
24
1.06×
67%
Representation geometry
112
1.06×
79%
Federated fine-tuning
42
1.01×
78%
Speech recognition & audio LMs
176
0.98×
39%
Fast inference & distributed training
26
0.98×
57%
RAG & dense retrieval
108
0.97×
61%
Value alignment principles
29
0.97×
64%
Collaborative & distributed inference
35
0.97×
50%
Efficient long-context inference
165
0.96×
53%
Data mixtures & continual pretraining
83
0.95×
38%
Machine unlearning
114
0.94×
51%
Emotion & affect
21
0.94×
50%
Fingerprinting & provenance
67
0.93×
64%
On-device inference & benchmarks
82
0.93×
53%
Reasoning evaluation
108
0.91×
52%
KV-cache compression & pruning
132
0.90×
82%
Serving & energy
22
0.90×
56%
Probing representations
41
0.89×
77%
Spatial reasoning & visual assistance
35
0.89×
25%
Jailbreaks & adversarial robustness
110
0.87×
71%
Deep-research & search agents
49
0.86×
14%
Math RLVR & theorem proving
119
0.85×
42%
Training-data attribution
57
0.84×
75%
Test-time reasoning
135
0.84×
37%
Knowledge graphs
45
0.82×
31%
Knowledge editing & injection
63
0.81×
43%
Code generation & software engineering
103
0.79×
40%
Personalization & customer support
50
0.79×
65%
Information extraction & summarization
40
0.78×
62%
Video understanding
90
0.78×
29%
Image generation & unified multimodal
83
0.78×
28%
Mobile GUI agents
36
0.78×
14%
Efficient on-device VLMs
26
0.76×
47%
Pretraining data selection & synthetic data
128
0.71×
30%
Cross-lingual transfer
32
0.71×
58%
Document OCR & parsing
25
0.68×
53%
Reasoning distillation & test-time scaling
113
0.67×
46%
Language acquisition (BabyLM)
108
0.65×
84%
Data curation & domain corpora
40
0.63×
33%
VLM architecture & visual tokens
140
0.56×
36%
Bias & fairness
36
0.55×
80%
Preference optimization
80
0.53×
37%
Multilingual & low-resource languages
63
0.45×
39%
Multimodal LLM efficiency & adaptation
92
0.43×
34%
Sentiment & misinformation
29
0.29×
54%
Small-LM surveys
21
0.27×
77%
Figure 2. Topics with at least 20 in-scope papers. Momentum compares a topic’s share of papers from the last six months with its share across the full map. Testable is the share whose smallest reported experiment fits a GTX 1660 Ti or Kaggle T4. Area shows in-scope paper count; rings mark candidate topics.
Show Table 2:11 recurring problem groups
Problem group
Papers
Momentum
Testable
Open problem share
Forgetting in continued training and unlearning
59
0.85×
64%
32%
Low-bit quantization breaks reasoning
80
1.19×
63%
27%
Long chain of thought hurts small models
49
0.98×
72%
72%
Confidently wrong instead of abstaining
66
0.76×
56%
10%
Optimizer and initialization instabilities
55
1.51×
48%
34%
Router quality is rarely measured
64
1.45×
65%
30%
Long-context degradation
76
1.19×
66%
48%
Sparse reward signal in RL post-training
77
1.18×
15%
19%
Training-data leakage in fine-tuning
69
1.13×
87%
22%
Coupling to a fixed tokenizer
67
0.90×
52%
17%
Results inconclusive against strong baselines
160
1.71×
67%
67%
Table 2. Papers are those with a statement in the cluster; testable is the share whose smallest experiment fits available GPUs. The final row gathers evidence caveats.
Seven candidates score within a point; forgetting goes first
The field still has more open questions than this project can test, so this first choice reflects the available evidence and compute, not a complete ranking of research value.
Candidate area
Small models
Benchmark
Small-scale
Compute
Novelty
Leverage
Total
Forgetting when a small model is fine-tuned
2
1
2
2
1
1
9
What low-bit quantization takes from small models
2
1
2
2
1
0
8
Rare-task acquisition under interference
2
1
2
1
1
1
8
Unlearning that survives fine-tuning
1
2
2
2
1
0
8
Knowing when a small model should abstain or hand off
2
1
2
2
1
0
8
Data-efficient pretraining (BabyLM)
1
2
2
1
1
1
8
Small models as tool callers
2
2
2
2
0
0
8
Figure 3. Each candidate scored 0–2 on six criteria. Candidate sources and supporting evidence are below.
Show candidate evidence:Table 3, sources and quoted open problems
Small models
the problem matters for models I can run locally.
Benchmark
a standard benchmark and baseline code exist and run on my GPUs.
Small-scale signal
effects show up at 1.5B parameters or fewer.
Compute
active, but not dominated by compute I don't have.
Novelty room
named problems remain open, and few papers address them directly.
Leverage
assets I already hold: tested tools, measurements, or an aimed idea from another field.
Candidate area
Small models
Benchmark
Small-scale
Compute
Novelty
Leverage
Total
Forgetting when a small model is fine-tuned
2
1
2
2
1
1
9
What low-bit quantization takes from small models
2
1
2
2
1
0
8
Rare-task acquisition under interference
2
1
2
1
1
1
8
Unlearning that survives fine-tuning
1
2
2
2
1
0
8
Knowing when a small model should abstain or hand off
2
1
2
2
1
0
8
Data-efficient pretraining (BabyLM)
1
2
2
1
1
1
8
Small models as tool callers
2
2
2
2
0
0
8
RL post-training at small scale
0
0
0
0
0
0
0
Distillation into small models
0
0
0
0
0
0
0
Mixture-of-experts at small scale
0
0
0
0
0
0
0
Memory for LM agents
0
0
0
0
0
0
0
Reasoning length and overthinking
0
0
0
0
0
0
0
Data selection
0
0
0
0
0
0
0
Knowledge editing
0
0
0
0
0
0
0
Architectures and optimizers as areas
0
0
0
0
0
0
0
Interpretability as an area
0
0
0
0
0
0
0
Calibration and routing as separate areas
0
0
0
0
0
0
0
Tokenization
0
0
0
0
0
0
0
Table 3. Candidate areas, with the corpus slices each came from and the set-aside slices below. Expand a row for its evidence and quoted open problems.
Other slices
18 areas
How the work was done
I mapped recent small-model papers by title and abstract, then reviewed full texts to select experiments. Where used, predictions and pass rules were recorded before main runs, seeds were paired, and judges were blinded to conditions.
I chose the questions, decided which experiments to run and when to stop a line, and approved each protocol before its main runs. AI coding agents wrote most of the code, processed papers for the map and drafted analyses; a second model checked each result against its stored records before I accepted it. Where a rule was clarified after seeing data, the experiment says so.
Show method detailsCorpus, labeling, quality checks and hardware
Stage
Papers
Description
Harvested
40,592
From 18 seed releases and 9 searches
Screened
11,059
On-device papers with no size dropped
Labeled
9,098
Rubric v3.1, validated batches
In scope
5,317
General-ML work on a model or pipeline
Gap extraction
5,600
In-scope and borderline papers
Figure 7. The pipeline, with paper counts at each stage.
Corpus
I used mar-source, a citation-following paper search tool extended with 9 Semantic Scholar, arXiv and OpenAlex searches. I followed citations from 18 small-model releases and kept papers from January 2025 on. Citation chasing broadens keyword coverage, but Llama 3 reached Semantic Scholar’s 9,999-citation paging limit, so this is a sample, not a census.
Labels and calibration
Labels record each paper’s contribution, target, domain, modality, model size and compute regime. Regimes range from inference through adaptation: low-rank adaptation (LoRA) adds a compact weight update, while quantized low-rank adaptation (QLoRA) uses quantized base weights; the other categories are full fine-tuning and pretraining. Scope covers general methods, analyses, benchmarks and model releases about a model or pipeline. A second labeler reviewed blind samples; the main disagreement concerned math reasoning and tool use as general or specialized domains. GPU fit is estimated from model size and compute regime.
Round
Rubric
n
Kind
Target
Domain
Modality
Size
Fit
In scope
Calibration: 1
v1
60
57/60
Unknown
Unknown
54/60
59/60
Unknown
Unknown
Calibration: 2
v2
40
33/40
38/40
31/40
38/40
40/40
33/40
32/40
Calibration: 3
v3
30
28/30
29/30
29/30
30/30
29/30
25/30
29/30
Full run vs. reference: 1
60
56/60
Unknown
Unknown
53/60
55/60
Unknown
Unknown
Full run vs. reference: 2
40
34/40
39/40
34/40
38/40
39/40
Unknown
33/40
Full run vs. reference: 3
30
29/30
30/30
30/30
28/30
26/30
24/30
30/30
Table 8. Agreement between the two labelers during calibration and between the full run and blind reference labels on the same papers. The first calibration round used an earlier scope field and is not comparable on scope.
Quality controls
Every batch is validated against its input: paper IDs, field names and allowed values. One batch skipped a paper, shifting later labels; the ID check caught it and the batch was relabeled. A second independent pass relabeled scope for every paper. The two passes agree on scope for 92% of 9,090 papers, and the 694 disagreements are flagged as borderline and kept. A size-only pass reviewed papers without a model size and filled some blanks; most remaining blanks are abstracts that state no size.
Landscape
Qwen3-Embedding-0.6B turns the first 256 title and abstract tokens into vectors on the 1660 Ti. Uniform Manifold Approximation and Projection (UMAP) reduces them to 10 dimensions for hierarchical clustering and 2 for the map. Regions and topics were named from their papers. The map has 16 regions and 90 topics. Momentum adjusts for uneven citation time: recent papers are under-indexed, while older papers have had longer to attract citations.
Gap extraction
For 5,600 papers, I recorded their stated gap, open problems, failures, novelty claims, baselines and headline result. Only 92% state a motivating gap, 34% an open problem and 44% a benchmark.
Hardware
I chose configurations I can access locally or through Kaggle; the table reports measured training and fine-tuning throughput and peak memory.
5.0 min per run; subset-eval seed SD 0.018 bpb (n = 3)
≈250 runs/day
1.4 GiB
GRPO with LoRA, Qwen2.5-0.5B
9.7 s per step; 155 steps in 25 min
0.82 completions/s
1.2 GiB
Full fine-tune, SmolLM2-135M, seq 512
1.7 s per step at batch 2
613 tokens/s
3.3 GiB
LoRA, Qwen2.5-0.5B, seq 512
2.4 s per step at batch 1
210 tokens/s
2.6 GiB
Table 9. Measured on a GTX 1660 Ti, 6 GB (Turing; no bf16 or FlashAttention 2). Kaggle T4s (15 GB, 30 GPU-hours a week) extend this to roughly 1.5B with LoRA and 7B for 4-bit inference; those figures are estimates, not measurements.
Project progress
Show project milestones and log
Biology-inspired rules tested
Switched to starting from the field
Hardware measured
Discovery added to my paper tool
Corpus harvested
Labeler calibrated
Full labeling run
Landscape built
Gaps extracted
Candidate areas shortlisted
Full-text review: forgetting
Baseline reproduced
Forgetting fixes tested in paired runs
Two-seed 40M forgetting comparison completed
Compression result published
Neuron-level tests planned
Neuron-level tests started
Project milestones.Show full log17 entries with details
Neuron-level tests started
Started the weak-activity screen on Qwen3-1.7B and the neuron-steering pilot on Kaggle.
Compression result published
On Granite 1B at 2 bits, correct rare-word definitions fell 35% and common ones 4% under blind judging (paper, rare words: 43.6%). The automatic matcher chosen in advance gives 33.1% and 11.5%. The result shows which words break, not why.
Neuron-level tests planned
Planned a test of whether weak neuron activity carries rare-word meaning, measured against control removals of the same cost, and a Granite study of where 2-bit damage enters the model (neuron layers, Mamba layers or earlier).
Two-seed 40M forgetting comparison completed
At 40M tokens, two paired Kaggle T4 seeds showed the same pattern: ε plus replay cut forgetting by 70.9% in seed 51 and 71.3% in seed 52. Baseline forgetting peaked at 5–10M tokens and fell by 40M, while ε alone and ε plus replay stayed roughly flat after 5M. Final Korean held-out loss was 1.05% lower in seed 51 (1.2905 → 1.2769) and 1.16% lower in seed 52 (1.2991 → 1.2840). The paper’s number is context, not a matched comparison.
Forgetting fixes tested in paired runs
Compared no fix with output layer Adam ε, momentum reset, 1% KL replay, and their combination on Pythia-160M. Across five paired seeds, the combination cut forgetting on English and code by 81.4% (95% CI 80.5–82.3%) while Korean learning stayed positive.
Baseline reproduced
Reproduced the SmolLM2-135M forgetting setup from “Step Towards Forgetting” (SST-2, then AG News) on the GTX 1660 Ti, at about six minutes per run. Across three seeds and two answer-token conditions, every qualitative result matches the paper: old answers lose probability mass, old-task accuracy falls, and resetting the optimizer's momentum reduces forgetting. The magnitudes differ; with separate answer tokens per task, forgetting is about three times larger than reported. It now serves as a fixed scorer with held-out test texts.
Full-text review: forgetting
Read 77 papers in full: 45 from the map and 32 found through their references and citations, each tabulated into a fixed schema, with a sample checked against the source texts. The three October mechanisms mostly don't contradict each other; no paper compares their fixes with each other or with replay at matched cost; and the shortest standard setup (SmolLM2-135M, SST-2 then AG News) fits my GPU at about 20 minutes per run.
Candidate areas shortlisted
Cut the corpus into eighteen keyword slices and scored each on six criteria. An independent review rescored five, corrected several claims and added a seventh candidate. Forgetting during fine-tuning went first. Of the other eleven slices, four feed a candidate and seven were set aside, each with a reason.
Gaps extracted
For 5,600 in-scope and borderline papers: the gap each says motivates it, what it leaves open, and the models, benchmarks and baselines it names.
Landscape built
Local embeddings, UMAP and a Ward tree cut into 16 regions and 90 topics, with a recency-normalized momentum score per group.
Full labeling run
9,098 abstracts labeled in 37 validated batches, followed by an independent second pass on the scope fields and a size-only pass.
Labeler calibrated
Three rounds of blind double labeling (60, 40 and 30 papers); the rubric went from v1 to v3.1 and reached 97% agreement on the scope decision.
Corpus harvested
Initial search: 11,818 papers since January 2025, from citations of 18 small-model releases and 9 keyword searches; 9,098 labeled after screening. The Oct 23 refresh added 1,961 screened papers without D15 labels.
Discovery added to my paper tool
Extended mar-source with Semantic Scholar, arXiv and OpenAlex search, citation chasing and resumable harvests.
Hardware measured
Timed what runs on the GTX 1660 Ti: five-minute pretraining runs at 50M parameters, GRPO with LoRA on Qwen2.5-0.5B, full fine-tuning of SmolLM2-135M.
Switched to starting from the field
Changed approach: map recent work on models small enough for my hardware, choose an area from the field’s own open problems, reproduce its baseline, and only then propose ideas.
Biology-inspired rules tested
Turned eleven biology-inspired mechanisms into continual-learning candidates and screened the three strongest against seven tuned baselines on two benchmarks with verified failure modes. None beat the best baseline, and an independent rerun reproduced every number.
Limitations
Abstracts only. The map uses abstracts. I read full texts only for the areas I test.
A sample, not a census. Citation chasing from 18 seed releases under-covers areas far from them.
Momentum is relative. It depends on this corpus and on how quickly recent papers are indexed.
Imperfect labels. Two independent passes disagree on scope for 8% of papers; those are flagged and kept.
Unknown fit. GPU fit is unknown for 49% of papers, mostly because their abstracts give no model size.