The agentic wrapper observatory · V1
Wrapper Comparison Observatory
- study 1 runs
- 3,731
- study 2 runs
- 156
- study 3 runs
- 839
- total wall clock
- ~40h
What we mean by wrapper
Tier A · Ollama-native
Direct model routing
OpenCode, Kilo Code, Cline, Pi, Aider, Agent Zero, OpenClaw, Roo Code. Configured to route any request to a chosen Ollama model.
Tier B · Orchestrated
Partial / mediated
Amp (partial), Claude Flow (delegates to Ollama through an orchestration layer).
Tier C · Native-only
Vendor API
Claude Code (Anthropic-only), Codex (OpenAI-only), Gemini CLI (Google-only). These cannot route to Ollama at all.
Quality matrix: wrappers × models × tasks
The headline finding of Study 1 is that only two wrappers — OpenCode and Kilo Code — consistently deliver usable agentic output when paired with Ollama-routed cloud models. Cline shows 100% completion but only 0.72 average quality because of a model-override bug that ignores the requested model and routes to its bundled GPT-5.2 (returning 9,434-char meta-commentary). Claude Flow returns acknowledgement-length responses (69 chars avg). Agent Zero and Pi return near-empty outputs. Aider, Amp, and Roo Code never completed the matrix.
- Kilo Code
- 8.3
- OpenCode
- 7.82
- Claude Flow
- 3.22
- Cline
- 0.72
- Agent Zero
- 0.29
- Pi
- 0.08
- OpenClaw
- 0
7 entries · ordered by Avg Quality as observed in this run · not a ranking
Memory observation — does the wrapper remember?
Binary distribution. There is no wrapper in the 0.10 < recall < 0.81 band. Wrappers cluster at ≥ 0.812 (three Tier-A wrappers — OpenCode, Cline, Kilo Code) or at ≤ 0.104 (everyone else). The gap is 0.826 with zero observations inside it. The observation records memory behavior at the wrapper layer for this protocol.
- OpenCode
- 0.917
- Cline
- 0.917
- Kilo Code
- 0.906
- OpenClaw
- 0.08
- Gemini CLI
- 0.049
- Claude Code
- 0.049
- Claude Flow
- 0.049
- Agent Zero
- 0.028
- Aider
- 0.028
- Codex
- 0.021
- Roo Code
- 0
- Pi
- 0
- Amp
- 0
13 entries · ordered by Mean recall as observed in this run · not a ranking
Thinking-token observation
Universal think-tag stripping. Across all 839 records, thinking_tokens = 0, thinking_ratio = 0.0, and thinking_time_seconds = 0 — every CLI wrapper consumes the streaming response and surfaces only the post-</think> portion. The intended measurement (does thinking length correlate with quality?) was not directly possible — the dependent variable was destroyed in transit. This is the most consequential single finding of the Observatory: the entire ecosystem of CLI wrappers homogenizes thinking behavior away.
- Kilo Code
- 6.98
- OpenCode
- 6.92
- Cline
- 0.18
- Pi
- 0.01
- Aider
- 0
- Roo Code
- 0
- Agent Zero
- 0
- OpenClaw
- 0
8 entries · ordered by Avg Quality as observed in this run · not a ranking
Models
Families
Companies
Related observations
- Academic Observation — 32 models × Cline + Pi · V1
- Aider Deep AI Research — 169 runs · V1
- Tool-Calling Compatibility — 31 cloud models · V1
- Cloud Wrapper Comparison — 31 models × 3 wrappers · V1
- 14B–32B Local + Cloud — 47 models · V1
- Thinking-Token Re-Observation — 14B–32B · V2
- Tool Calling — 9 cloud models × 30 prompts · V1