Sub-3B local models — all 34 · V1

Local Model Testing · Tier v1.0

The first Sigma5C local-model observation. Thirty-four models with three billion parameters or fewer were run on a 48-thread Xeon CPU (157 GB RAM, CPU-only, Ollama) against fourteen prompts spanning five categories: coding, reasoning, instruction-following, summarization, math.

S-tier (≥9.17): 7
A-tier: 10
B-tier: 3
C-tier: 4
D-tier: 10

What was observed

Each model received the same 14 prompts. Two task types — coding and math — had execution-verified scoring (Codex CLI ran the model's code against a reference test). The remaining three categories were scored by a four-judge panel.

Coding · 30%

Quicksort, recursion, fizzbuzz

Code is extracted from the response and executed. Either runs and produces the expected output, or it does not.

Reasoning · 25%

Train-distance, riddles

Judged on whether the final answer matches and the chain of reasoning is sound.

Instruction · 20%

Format and follow exact rules

Numbered lists, JSON outputs, multi-step transformations.

Summarization · 15%

Long-text condensation

Faithfulness, coverage, brevity.

Math · 10%

Arithmetic with work shown

847 × 23 = 19,481 is the headline test. Only 7 of 34 models computed it correctly.

What the data shows

The 3B quality wall

A reasonably sharp ceiling at 3B

Average score: 2–3B = 8.7, 1–2B = 7.2, 0.5–1B = 5.5, <500M = 2.0. The jump from 2B to 3B is the largest in the dataset.

Math is the equalizer

Only 7 of 34 models solve 847 × 23

Every math-passer except deepseek-r1 and deepcoder is in the top 10. Multiplication is the cleanest filter between capable and not.

0.5B surprise

qwen2.5:0.5b scores 8.19/10

500 million parameters, 29.5 tok/s on CPU. Observed as suitable for edge or IoT deployment where bandwidth is constrained.

Speed king

smollm:360m at 47.3 tok/s

Highest speed observed — but only 3.73 quality. For speed-and-quality use gemma3:1b (26.4 tok/s, 8.52) or qwen2.5-coder:1.5b (29.6 tok/s, 8.52).

The full Sub-3B observed score table

Sorted by composite overall. Category columns show raw 0–10 scores; tok/s is CPU throughput on the 48-thread Xeon CPU (157 GB RAM). Tier classifications S → D follow the standard distribution: S ≥ 9.0, A 8.0–8.99, B 7.0–7.99, C 5.0–6.99, D < 5.0.

Overall

llama3.2:3b: 9.67
phi3:latest: 9.67
qwen2.5-coder:3b: 9.52
cogito:3b: 9.52
qwen2.5:3b: 9.52
phi4-mini:latest: 9.49
phi3.5:latest: 9.17
falcon3:3b: 8.97
smollm2:1.7b: 8.67
hermes3:3b: 8.67
granite3.1-dense:2b: 8.64
gemma3:1b: 8.52
qwen2.5-coder:1.5b: 8.52
granite4:latest: 8.49
dolphin-phi:latest: 8.49
qwen2.5:1.5b: 8.27
qwen2.5:0.5b: 8.19
llama3.2:1b: 7.89
deepcoder:1.5b: 7.46
qwen2:1.5b: 6.46
deepseek-r1:1.5b: 5.5
qwen2.5-coder:0.5b: 5.3
qwen2:0.5b: 5.25
yi-coder:1.5b: 4.31
smollm2:360m: 3.98
orca-mini:latest: 3.87
smollm:360m: 3.73
qwen3:1.7b: 3.71
tinydolphin:latest: 3.38
qwen:0.5b: 1.96
smollm2:135m: 1.88
tinyllama:latest: 1.86
smollm:135m: 0.43
qwen3:0.6b: 0