Academic observation — 32 cloud models × Cline and Pi · V1

Academic Observation

A 2×32 factorial observation with sequential execution, 30-second cooldown, 600-second timeout, and a single retry per cell. Both wrappers receive an identical 73,376-character prompt — the Testing Philosophy 2026 corpus with its embedded README (2,075 lines) and QA Standards (421 lines). Outputs are dual-judged by Claude Opus 4.6 and Codex 5.3 xhigh independently, then averaged.

Pi success: 87.5%
Cline success: 40.6%
Codex judge tokens: 504K
judge agreement (±1): 95.9%

Why this comparison matters

Cline

Heavy system overhead

cline -m $MODEL -y --json -c <projdir> "$(cat prompt.txt)" — adds ~8,000 tokens of system context (environment, tool descriptions, task framework) before the user prompt. Wrapper-attributed failures dominated on small/mid models.

Pi (oh-my-pi / omp)

Minimal overhead

omp --model $MODEL -p --no-tools --no-lsp --no-session "$(cat prompt.txt)" — strips tools, language-server features, and session state. Wrapper overhead approaches zero.

The "wrapper effect" is mostly a context-budget effect. Cline's 8K-token system prompt crushed small/mid models — all three Gemma3 variants context-overflowed. The same models worked at 87.5% success under Pi's minimal prompt. The wrapper's per-call overhead is the dominant variable, not the model.

The dual-judge consensus observed score table

Composite score = mean of Claude Opus 4.6 + Codex 5.3 xhigh. Each judge scored six dimensions (coverage, depth, actionability, novelty, accuracy, coherence) on a 1–5 Likert scale, then averaged into a composite.

Consensus

kimi-k2.5:cloud: 5
qwen3.5:cloud: 4.92
gemini-3-flash-preview:cloud: 4.83
kimi-k2.5:cloud: 4.83
kimi-k2-thinking:cloud: 4.83
gemini-3-flash-preview:cloud: 4.75
glm-4.7:cloud: 4.75
glm-5:cloud: 4.75
glm-5:cloud: 4.67
mistral-large-3:675b-cloud: 4.67

10 entries · ordered by Consensus as observed in this run · not a ranking

What the dual-judge data revealed

Wrapper effect

Pi > Cline on every metric

Pi mean response time 62.1s vs Cline 85.6s; words 1,415 vs 783; sections 4.8/5 vs 2.8/5; topics 8.1/10 vs 4.2/10. The exception: Cline's coherence (4.69) beats Pi's (4.50) — when Cline does succeed, its structure is tighter.

Size effect = zero

Spearman ρ = -0.004 (p=0.98)

Across 32 cloud models, model size and topic coverage have zero relationship. The 3B Ministral beat the 1T Kimi on Pi. Parameter count is not a useful predictor of QA-document analysis quality.

Inter-rater agreement

95.9% adjacent agreement

Pearson r = 0.729 across 246 dimension scores. Both judges agreed kimi-k2.5 (Cline) was best; both agreed kimi-k2:1t (Cline) and cogito-2.1:671b (Cline) were near-bottom.

Where they disagree

Accuracy: 36.6% exact agreement

Codex was systematically stricter (mean 2.98 vs Claude's 3.73). Codex penalized models for claiming AI/Agent QA was "missing" when those topics had dedicated README sections.

Six-dimension Claude rubric — Cline vs Pi means

Dimension	Cline mean	Pi mean	Notes
Coverage	4.85	4.89	Section completeness
Depth	4.23	4.36	Analysis nuance
Actionability	4.31	4.50	What / why / how / impact per item
Novelty	3.31	3.93	Genuinely new gaps surfaced
Accuracy	3.77	3.71	Factual correctness — most disputed dimension
Coherence	4.69	4.50	Organization
Composite	4.19	4.31