The capability ladder — execution-verified · V1

Capability Ladder

Eight small models, thirty levels of progressive coding difficulty, eight programming languages. 240 cells per model. Crucially: code is executed, not LLM-judged. A cell passes only if the compiled output matches the expected output. This is the most rigorous coding-ability observation in the entire Observatory.

top score (65%): 155/240
bottom (qwen3.5:0.8b): 0/240
languages: 8
highest level passed: L29

How a cell is scored

PASS

Compiles + correct output

The model's code compiles cleanly in the target language and produces the exact expected output. No partial credit.

SYNTAX

Compiles + wrong output

The code compiles but produces wrong results. The model knows the language; it doesn't understand the task. Sub-1B models have unusually high SYNTAX counts.

FAIL

Doesn't compile

Syntactically broken code that the compiler rejects. Concentrated in the largest models among the failed set (e.g. qwen3.5:0.8b, qwen3:0.6b).

Python is universally easiest · Perl is the fringe indicator

Mean pass rate across the 4 models in the table

python: 63%
js: 62%
ruby: 48%
go: 45%
c: 42%
java: 40%
perl: 36%
bash: 30%

Model	bash	c	go	java	js	perl	python	ruby
qwen2.5-coder:0.5b	37%	57%	67%	67%	77%	47%	83%	83%
gemma3:1b	40%	50%	67%	50%	83%	53%	83%	57%
qwen2.5:0.5b	43%	53%	43%	43%	73%	43%	83%	47%
smollm2:135m	0%	7%	3%	0%	17%	0%	3%	7%

JavaScript is second-easiest after Python. Bash is surprisingly weak — output-formatting failures dominate. C exhibits the highest SYNTAX rates (models produce valid C with wrong answers). Perl is the fringe indicator: only gemma3:1b and qwen2.5-coder handle it well.

The capability ladder observed score table

PASS

qwen2.5-coder:0.5b: 155
gemma3:1b: 145
qwen2.5:0.5b: 129
qwen3:0.6b: 109
smollm2:360m: 108
granite4:350m: 100
smollm2:135m: 11
qwen3.5:0.8b: 0

8 entries · ordered by PASS as observed in this run · not a ranking

Where every model breaks

L08 While Loop

Best: gemma3:1b at 88%

Basic iteration. Where the leader feels confident. Sub-1B models are still patchy here.

L21 Try/Catch

Best: gemma3:1b at 25%

Error-handling is the universal cliff. The leading model passes only one in four attempts. Below L21, error handling does not work at small scale.

L24 Fibonacci

Best: qwen3:0.6b at 38%

Classic recursion test. Memoization and iterative variants are easier than the textbook recursive form.

L27 Read+Process

Best: qwen2.5-coder at 25%

File I/O with transformation. Combines two competencies that small models struggle to chain.