Sigma5C
Request Beta Access

Cloud Models — three-judge autoresearch observation set · V1

Cloud Model Testing

Karpathy-style autonomous evaluation of 35 Ollama-cloud-routed models. Each model is prompted across five categories (code, reasoning, math, instruction following, creative) and judged by three independent large models with non-self-judge rules and thinking-tag stripping. Run on a 48-thread Xeon CPU with 157 GB RAM, March 24, 2026. 945 total observations.
top score (kimi-k2 1T)
9.87
lowest score
8.09
S-tier (≥9.0)
30
A-tier (8.0-8.99)
5
providers
11

How the observation worked

Each model received the same five prompts (one per category), with three trial repeats per prompt. Outputs were then sent to three judges with explicit instructions to score only the final answer after stripping <think> blocks. The composite score weights the five categories at: Code 25%, Reason 25%, Math 20%, Instruction 15%, Creative 15%.

Primary judge

kimi-k2:1t-cloud

Moonshot's 1-trillion-parameter model. Scores all categories.

Code specialist

qwen3-coder:480b-cloud

Alibaba's 480B coder. Weighted 60% on the Code category.

Reason/Math cross-validator

cogito-2.1:671b-cloud

671B DeepSeek-family. Weighted 50% on Reasoning and Math.

Models never judge themselves. The qwen3-coder judge does not vote when qwen3-coder-* is the candidate. Outlier votes > 3.0 from the median are dropped before the average.

The full 35-model observed score table

Every model observed in the autoresearch run, ranked by composite overall. n/a* in the tok/s column denotes entries where the speed metric is suspect due to a divide-by-near-zero in the source CSV — refer to the raw flat results for the trustworthy values.
Overall
kimi-k2:1t-cloud
9.87
deepseek-v3.2:cloud
9.85
gemini-3-pro-preview:latest
9.84
glm-5:cloud
9.83
qwen3-coder-next:cloud
9.83
nemotron-3-super:cloud
9.83
gpt-oss:20b-cloud
9.82
minimax-m2:cloud
9.82
qwen3-vl:235b-instruct-cloud
9.82
qwen3-vl:235b-cloud
9.77
kimi-k2-thinking:cloud
9.76
minimax-m2.5:cloud
9.76
minimax-m2.1:cloud
9.73
qwen3.5:397b-cloud
9.72
qwen3-coder:480b-cloud
9.71
cogito-2.1:671b-cloud
9.7
kimi-k2.5:cloud
9.68
nemotron-3-nano:30b-cloud
9.68
gemini-3-flash-preview:cloud
9.59
deepseek-v3.1:671b-cloud
9.57
gemma3:27b-cloud
9.55
qwen3-next:80b-cloud
9.53
glm-4.7:cloud
9.53
minimax-m2.7:cloud
9.51
mistral-large-3:675b-cloud
9.43
ministral-3:3b-cloud
9.39
gpt-oss:120b-cloud
9.37
devstral-2:123b-cloud
9.27
devstral-small-2:24b-cloud
9.25
ministral-3:14b-cloud
9.2
rnj-1:8b-cloud
8.96
ministral-3:8b-cloud
8.82
gemma3:12b-cloud
8.67
glm-4.6:cloud
8.24
gemma3:4b-cloud
8.09

35 entries · ordered by Overall as observed in this run · not a ranking

Observed values by category

Coding · 25% weight

kimi-k2.5:cloud

9.93 / 10 on code-specific prompts in this observation.

Reasoning · 25% weight

kimi-k2:1t-cloud

10.00 / 10 in the reasoning category under the run's judge panel.

Math · 20% weight

kimi-k2:1t-cloud

10.00 / 10 in the math category under the same protocol.

Instruction · 15% weight

kimi-k2:1t-cloud

10.00 / 10. Thirteen other models also recorded 10.00 on instruction prompts.

Creative · 15% weight

gpt-oss:20b-cloud

9.60 / 10 in the creative category in this observation.

Throughput measurement

gpt-oss:20b-cloud

242.9 tok/s with a 9.82 overall score under this protocol.

Provider context

Related delivery-route references are maintained separately from this dated observation. A catalog reference does not transfer these results to another model or route.

Models

Families

Companies

Related observations

AI-generated & AI-scored. Every observation on this site is produced and judged by AI, and AI makes mistakes. Treat each result as a recorded observation under a declared protocol, as of its date. Method and marker legend