Thinking-token re-observation — 14B-32B · V2
Local Model Testing · Tier v3.0 · follow-up
A March 10 follow-up to the 14B-32B primary run. Four "thinking models" — models that emit
<think>…</think> blocks before their answer — scored under 3 / 10 in the primary run. On closer look the LLM judges were scoring the literal think blocks rather than the final answer, severely penalizing reasoning-heavy models. Re-running with <think> stripping before judging gave an average +45.4% score improvement.- avg lift after the fix
- +45.4%
- top (qwen3:14b)
- +155%
Re-judged with the think tags stripped
- qwen3:14b
- 4.97
- deepcoder:14b
- 4.45
- qwen3:32b
- 2.4
- magistral:24b
- 0
4 entries · ordered by New Score as observed in this run · not a ranking
qwen3:14b went from 1.95 to 4.97 — a +155% lift simply by stripping the think tags before sending the response to the judge. This finding fed directly into the Wrapper Comparison Observatory's bias-mitigation rules, where strip_thinking_tags: true is now a non-negotiable judge-panel setting.
Thinking-token analysis on deepcoder:14b: 67% of tokens spent thinking. A math prompt produced 1,383 think tokens and 218 answer tokens (ratio 0.86). For CPU thinking models, capping num_predict at 2048 is essential to keep generation under ~14 minutes.