The agentic wrapper observatory · V1

Wrapper Comparison Observatory

Thirteen coding-agent CLI wrappers, observed against thirty-three Ollama-cloud models, across three sequential studies: a quality matrix, a memory recall observation, and a thinking-token observation. The wrappers under observation include OpenCode, Kilo Code, Cline, Aider, Roo Code, Pi, Agent Zero, OpenClaw, Amp, Claude Flow, Claude Code, Codex, and Gemini CLI.
study 1 runs
3,731
study 2 runs
156
study 3 runs
839
total wall clock
~40h

What we mean by wrapper

A wrapper is a coding-agent CLI tool that sits between you and a language model. The wrapper handles prompts, tool calls, file edits, project context, and the judging of its own progress. Wrappers vary enormously in how much overhead they add (system prompts can range from ~0 tokens to ~29,000), how strictly they enforce a model selection, how they handle context across turns, and how they parse the model's streaming output. The Observatory measures every dimension.

Tier A · Ollama-native

Direct model routing

OpenCode, Kilo Code, Cline, Pi, Aider, Agent Zero, OpenClaw, Roo Code. Configured to route any request to a chosen Ollama model.

Tier B · Orchestrated

Partial / mediated

Amp (partial), Claude Flow (delegates to Ollama through an orchestration layer).

Tier C · Native-only

Vendor API

Claude Code (Anthropic-only), Codex (OpenAI-only), Gemini CLI (Google-only). These cannot route to Ollama at all.

Quality matrix: wrappers × models × tasks

Each wrapper was given the same eight standardized tasks (code generation, code review, bug fix, refactor, test writing, architecture design, context memory, multi-step reasoning) against each of the 33 cloud-routed models, with three trial repeats. Outputs were judged by a weighted four-judge panel (Mistral Large 3 primary, Qwen3 Coder for code, Kimi K2 1T for reasoning, Devstral Small 2 as tiebreaker), with thinking-tag stripping, response normalization, and randomized presentation order to mitigate position and verbosity bias.

The headline finding of Study 1 is that only two wrappers — OpenCode and Kilo Code — consistently deliver usable agentic output when paired with Ollama-routed cloud models. Cline shows 100% completion but only 0.72 average quality because of a model-override bug that ignores the requested model and routes to its bundled GPT-5.2 (returning 9,434-char meta-commentary). Claude Flow returns acknowledgement-length responses (69 chars avg). Agent Zero and Pi return near-empty outputs. Aider, Amp, and Roo Code never completed the matrix.

Avg Quality
Kilo Code
8.3
OpenCode
7.82
Claude Flow
3.22
Cline
0.72
Agent Zero
0.29
Pi
0.08
OpenClaw
0

7 entries · ordered by Avg Quality as observed in this run · not a ranking

Memory observation — does the wrapper remember?

Twelve memory probes across three suites: single-turn recall (6 probes), multi-turn retention (4 probes), context-window stress (2 probes). The probe verifies whether the wrapper preserves cross-turn information when the underlying Ollama model has the capacity to recall it. Hallucinations are tracked separately: a confidently-wrong answer counts as a hallucination, an empty answer does not.

Binary distribution. There is no wrapper in the 0.10 < recall < 0.81 band. Wrappers cluster at ≥ 0.812 (three Tier-A wrappers — OpenCode, Cline, Kilo Code) or at ≤ 0.104 (everyone else). The gap is 0.826 with zero observations inside it. The observation records memory behavior at the wrapper layer for this protocol.

Mean recall
OpenCode
0.917
Cline
0.917
Kilo Code
0.906
OpenClaw
0.08
Gemini CLI
0.049
Claude Code
0.049
Claude Flow
0.049
Agent Zero
0.028
Aider
0.028
Codex
0.021
Roo Code
0
Pi
0
Amp
0

13 entries · ordered by Mean recall as observed in this run · not a ranking

Thinking-token observation

Eight Ollama-native wrappers × five thinking models (qwen3-coder-next, kimi-k2.5, cogito-2.1:671b, deepseek-v3.2, glm-5) × nine complexity levels × three trials. The study was designed to measure whether longer "thinking" (the model's internal reasoning chain before its final answer) correlates with higher final quality.

Universal think-tag stripping. Across all 839 records, thinking_tokens = 0, thinking_ratio = 0.0, and thinking_time_seconds = 0 — every CLI wrapper consumes the streaming response and surfaces only the post-</think> portion. The intended measurement (does thinking length correlate with quality?) was not directly possible — the dependent variable was destroyed in transit. This is the most consequential single finding of the Observatory: the entire ecosystem of CLI wrappers homogenizes thinking behavior away.

Avg Quality
Kilo Code
6.98
OpenCode
6.92
Cline
0.18
Pi
0.01
Aider
0
Roo Code
0
Agent Zero
0
OpenClaw
0

8 entries · ordered by Avg Quality as observed in this run · not a ranking

Models

Families

Companies

Related observations

AI-generated & AI-scored. Every observation on this site is produced and judged by AI, and AI makes mistakes. Treat each result as a recorded observation under a declared protocol, as of its date. Method and marker legend