Harness vs. Wrapper — An Observation (V2)

SIGMA5C MODEL OBSERVATORY · Research Note · Revision 2

observation window: OpenRouter public app data, 2026-06-01 · published by the Sigma5C Observatory · V1 2026-06-04 · V2 2026-06-09

Sigma5C practices Continuous AI Observability: we record how AI systems behave under declared conditions over time. This is one observation in a continuing series — the purpose is to learn each pass and improve, again and again. This is the second pass of this note; the data window is unchanged, the reading is sharper.

TL;DR

A wrapper makes an LLM easier to call. A harness makes an LLM reliable enough to ship. Across the 12 OpenRouter application categories in this observation window, high token-volume applications consistently exhibit the harness pattern (Hermes Agent, Kilo Code, OpenClaw, Cline, Claude Code, pi); applications that exhibit the wrapper pattern (extra.email, HyperWrite AI, GPT Workspace) appear in the lower-volume tail of their respective categories. This is an observed correlation, not a claim about superiority.

The cleanest definition, from the Substack piece cited in the sources, is: if an agent = LLM + loop with tool calls, the harness is everything except the LLM itself.

New in V2: the central finding holds under both readings of the word harness — capability depth and architectural ownership — and gets tighter under the ownership reading. The vocabulary section below now carries both axes side by side, with worked examples; and a closing section turns the small/local-model thread into a concrete experiment design.

1. Wrapper (the thinnest layer)

A wrapper is glue around an LLM API. It does four things, and nothing more:

• Authentication — manage API keys
• Request formatting — shape input to what the model expects
• Response parsing — hand back something clean
• Basic error handling — retry a failed call, surface a clean error

That's it. A wrapper is passive. It doesn't shape what the model does, just how you talk to it.

Examples observed in the OpenRouter data:

• extra.email (3.15B tokens across 4 categories) — calls an LLM to classify inbox messages. A thin LLM call inside a category sorter. The product is the inbox, not the agent.
• HyperWrite AI (174M tokens) — text-completion frontend with no tool loop, no memory across sessions, no iteration.
• GPT Workspace (934M tokens) — Chrome extension that routes prompts to OpenRouter. The harness is the browser context, the extension is the wrapper.

These products use LLMs; they are not defined by their agentic capability. The model is a feature, not the core.

A note on the word. We use wrapper here in its narrow, literal sense — the thin glue layer above — and not as a value judgment. The term is contested: many builders use wrapper to mean the product and feature layer built on top of an engine (see On nomenclature, below), which is substantial work, not thin glue. Where this note says "wrapper pattern," it means the thin-glue end of the spectrum, nothing more.

A useful mental model: a wrapper is a handle on a knife. The knife doesn't change. You just made it easier to hold.

2. Scaffolding (the middle layer)

Scaffolding is what most "AI app" products ship, and is sometimes mislabeled as a "wrapper." It provides structure around the call:

• Prompt templates — pre-structured inputs that guide behavior
• Conversation state — track what's been said across turns
• Tool-calling loops — let the model invoke a function and continue
• Retry logic — decide when to try again and how
• Context window management — summarize, truncate, inject

Scaffolding gives the model a shape to work within. It's still relatively passive. It sets up the environment without actively managing outcomes.

In current practice the word is often drawn more narrowly still: the declarative configuration that steers an agent without being code — system prompts, agents.md, skills, and spec files (for example Spec-Kit or BMAD-style specs). That is a sharper and useful line, because it separates the config that steers an agent from the runtime machinery (tool-calling loops, retry, context management) that some taxonomies place in the engine instead. V2 adopts this sharper line as the primary definition — see On nomenclature, where the reasoning is laid out; it came directly from the V1 field feedback.

Examples observed in the OpenRouter data:

• LibreChat (671M tokens) — a ChatGPT-clone UI with multi-model routing and conversation persistence, but no autonomous tool loop or feedback iteration.
• Chatbox AI (607M tokens) — on-device copilot with a chat surface and some context, no agentic loop.
• T3.Chat (773M tokens) — high-performance chat app; the "harness" is just the chat UI plus model selection.

The key tell: scaffolding tools have a chat box as their primary surface. Wrappers often don't even have a UI (they're embedded); harnesses usually have a task surface, not a chat surface.

A useful mental model: scaffolding is the frame of a building under construction. It holds things in place but doesn't check the work.

3. Harness (the full system)

A harness is the infrastructure that compensates for the model's weaknesses. It catches errors the model makes, enforces constraints the model would ignore, and creates feedback loops so the model can iterate.

A well-designed harness includes:

• Tool access — file system, terminal, browser, APIs the model can call
• Memory and context management — keep relevant state across long tasks and across sessions
• Feedback loops — run the code, check if tests pass, feed the result back
• Guardrails and constraints — linters, structural rules, enforced deterministically, not by the model's judgment
• Observability — logs, traces, metrics the agent can see and use to self-correct
• Multi-agent coordination — routing to specialized subagents, handoffs, shared state

The key observation: the harness actively manages outcomes. It's not setting up an environment and stepping back. It's continuously shaping what the model does next.

Examples observed in the OpenRouter data (high token volume, harness pattern):

• Hermes Agent — 574B tokens, observed in 5 categories. Self-improving agent with persistent memory, 40+ tools, scheduled automations, subagents. Consistent with the harness pattern.
• Kilo Code — 139B tokens, observed in 4 categories. Cross-IDE + CLI coding agent with agentic workflows. Tool calls, file edits, terminal, browser automation.
• OpenClaw — 123B tokens, observed in 5 categories. Real actions: runs commands, browses the web, manages files, sends emails. The harness is the product.
• Claude Code — 51.6B tokens, observed in 3 categories. Reads the codebase, plans, executes, runs tests, iterates. Canonical harness.
• pi — 37.4B tokens, observed in 5 categories. CLI-native harness.
• Cline — 19.8B tokens, observed in 4 categories. Lives in the IDE, autonomously explores, edits, runs terminal commands.
• Roo Code — 8.94B tokens, observed in 4 categories. Team of specialized agents with customizable modes — i.e. the harness includes a routing layer.

A useful mental model: the harness is the full training system. Feedback after every session, corrected form, tracked progress, adjusted plan. The model is the brain in the jar; the harness is everything that makes the brain useful in the real world.

The two layers inside any harness

The MindStudio framework splits the harness into inner and outer layers, which is the cleanest way to design one:

Inner harness — the immediate wrapper around a single model call

• System prompt and instruction structure
• Context assembly logic (retrieval, memory injection, history summarization)
• Tool and function definitions
• Output format specifications (JSON, schema, etc.)
• Output validators — parse and check before acting on it
• Retry and fallback logic

The inner harness is what most developers spend their time on first. Small changes here produce large differences in behavior, not because the model changed, but because what you gave it and how you structured the response handling changed.

Outer harness — the orchestration around the inner harness

• Trigger logic — what starts the agent (webhook, schedule, user action, another agent's output)
• Pre-processing pipelines
• Routing logic — which agent (or version) handles a given input
• Persistent memory systems (separate from in-context memory)
• Multi-agent coordination and handoffs
• Post-processing pipelines
• Monitoring and observability

The outer harness is where architectural decisions live: single agent vs. multi-agent, vector store vs. structured DB, how concurrent runs are handled. These questions have nothing to do with which model was picked.

The insight from the data: the harness matters more than the model

This is the load-bearing observation, supported by the OpenRouter data:

Three signals in the OpenRouter data support this:

1. Same model family, different harness, different volume. These three commonly run the same underlying model family, each behind a different harness:

• Kilo Code (139B) — multi-IDE harness
• Cline (19.8B) — VS Code-native harness
• Roo Code (8.94B) — VS Code multi-agent harness

Same model family, different harnesses, an observed ~15x volume spread within this window.

1. Model switching is cheap; harness redesign is expensive. When teams swap models, they usually don't rebuild the harness. They swap the model and adjust the inner harness slightly. The outer harness stays the same. If the model were the primary driver of performance, model swaps would be high-stakes events. They are configuration changes.

1. Most agent failures are harness failures, not model failures:

• Tool call returned an error, no retry logic caught it
• Model produced output in the wrong format, no validator caught it
• Context window got too full, agent lost track of earlier instructions
• Two agents wrote conflicting state because no coordination layer

None of these are fixed by using a better model. They are fixed by building a better harness.

Why the vocabulary matters

When something that is a harness is called a "wrapper," the people building it under-invest in it. When a harness is called "just scaffolding," the active ingredient is missed: the feedback loop, the guardrails, the observability that makes an agent reliable rather than impressive in demos and broken in production.

A wrapper engineer asks: How do I make this API easier to call? A scaffolding engineer asks: How do I give this model structure to work within? A harness engineer asks: How do I build the environment that catches failures, enforces correctness, and lets this model iterate toward the right answer?

Three different questions. Three different systems. Three very different outcomes.

The vocabulary shapes the design decisions.

On nomenclature — two axes, and the field hasn't aligned

(Rewritten in V2 around field feedback from a team building its own engine layer.)

The industry has not standardized these three words, and reasonable practitioners draw the lines differently. V1 ordered the layers by one axis; the V1 feedback supplied the other. Both are coherent, both are useful, and — this is the load-bearing point — on the two axes the words "harness" and "wrapper" can point in opposite directions. Naming the axis is therefore not pedantry; it is the difference between two engineers thinking they disagree when they are measuring different things.

Axis A — capability depth (V1's reading)

How much each layer compensates for the model and actively manages outcomes. This is the right axis when the subject is reliability, which is the subject of this note.

• Harness — the full system that compensates for the model: tools, memory, feedback loops, guardrails, observability, coordination. The most complete layer.
• Scaffolding — the middle layer: prompt templates, conversation state, tool loops, retry, context management.
• Wrapper — the thin glue above the model: auth, request formatting, response parsing, basic error handling. Passive.

Axis B — architectural ownership (the engine-builder's reading)

Where a layer sits in the dependency stack, and whether you build it or buy it. This is the more practical axis for a team deciding whether to write its own engine or sit on a vendor SDK. The definitions that came back from the field:

• Harness = the core engine. The machinery that connects the development experience to the LLM — loops, tool-call handling, context plumbing — whether model-specific or provider-agnostic and harmonizing across many providers. The substrate others build on. Example: Anthropic's Agent SDK.
• Wrapper = the product and feature layer built on top of a harness, exposing features and results to users — substantial work, not thin glue. Example: products like T3 Code or Conductor that build on Anthropic's Agent SDK instead of writing their own engine.
• Scaffolding = the declarative configuration entities inside the wrapper layer: system prompts, agents.md, skills, spec files. Example: Spec-Kit or BMAD.

The two readings side by side

A worked example: Claude Code

The cleanest illustration of the inversion. Under Axis A, Claude Code is the canonical harness — it reads a codebase, plans, executes, runs tests, and iterates; it is the most complete, outcome-managing layer in the comparison. Under Axis B, Claude Code is a wrapper on a bought engine — it sits on the Agent SDK rather than re-implementing the loop and tool-call substrate itself. Same product, opposite word, depending on the axis. The distinction depends on how we name the product and the layer you mean, not just the word.

Why the central finding survives — and sharpens

The thesis of this note holds under both readings, and the ownership reading makes it tighter. "Most agent failures are engine failures, not model failures" is a stronger claim than the capability-depth version: every failure class catalogued above — an uncaught tool error, an unvalidated output format, a context overflow, an absent coordination layer — lives in the engine, not the model.

And the ownership reading extends the thesis one turn further. As engines themselves become buyable (the Agent SDK and its peers), the differentiation that used to live in building the engine climbs up into the wrapper and the scaffolding — the product layer and the declarative config. The model was always a dependency; increasingly the engine is becoming one too. The open question for any team then becomes: which layer do you still have to build yourself? A team that builds its own engine is betting that the engine boundary is exactly where it does not want to cede control; a team that buys the engine is betting its differentiation lives in the product and the config above it. Neither bet is wrong in general — but they are different bets, and the vocabulary above is how you state which one you are making.

How Sigma5C reads this

Sigma5C is an independent AI evaluation, research, and development company. We observe and measure how AI systems behave under controlled conditions over time. The harness/wrapper distinction is useful to us because it names where the behavior we observe in practice comes from — the orchestration around a model, not the model in isolation. A few readings of the public OpenRouter data, in that frame:

• The harness, not the model, explains most of the volume spread. Several high-volume applications run the same underlying model family behind very different harnesses, and the volumes differ by more than an order of magnitude. Under this reading, the differentiator users responded to is the surrounding system, not the model choice.
• Routing is an outer-harness decision. "Small models route, large models investigate" is an architecture choice — it lives in the orchestration layer. Where it shows up, it works because the system was deliberately designed around model specialization, not because a single model was selected.
• The ~51.6B tokens observed for Claude Code is a harness signal: a first-party agent appearing on a third-party router because users chose that route for cost controls, model flexibility, or multi-model routing — a harness-layer decision made by the user.
• The ~5.17B observed for OpenAI Codex reads as a routing-visibility effect — most users of that agent reach it through its native channels rather than this router. Same kind of agent, different routing, an order-of-magnitude volume spread.
• Pieces.app at ~12.1B tokens is evidence for memory as a standalone harness component — a persistence layer other harnesses can build on, rather than a model in its own right.

In this observation window, across the agent categories in this window's public data, the harness-shaped applications occupy the larger token-volume range and the wrapper-shaped applications the smaller. The model is increasingly a dependency; the observable differentiation lives in inner + outer harness design. This is an observed correlation for one window of public data — we claim no permanent superiority for any application named here.

This is the kind of question Sigma5C works on: Continuous AI Observability — the observation layer that records how AI systems — and the harnesses around them — behave under declared conditions, over time, so the evidence accumulates instead of resetting with every model release.

Comparison table: 10 high-volume agents, scored on inner/outer harness criteria

This is a sample, not a ranking. Scoring rubric (0-2 each, 14 max):

Inner harness (8 pts)

• I1 System prompt / instruction structure (decomposed, focused, role-specific)
• I2 Context assembly (retrieval, memory injection, history management)
• I3 Tool / function definitions (rich, well-described, parameterized)
• I4 Output validation (schema checks, format assertions before action)

Outer harness (6 pts)

• O1 Routing / multi-agent coordination (specialized subagents, handoffs)
• O2 Persistent memory across sessions (not just in-context)
• O3 Observability (logs, traces, self-correction feedback)

Observed pattern: the high-volume sample clusters at 9-14 (harness). The single vertical-specialist (Descript) and the chat-frontend (Janitor AI) score 4-6 despite meaningful token volume, because their product surface is narrow and the agentic loop is shallow. The observation: token volume correlates with harness quality in agent categories, and diverges from it in vertical-product categories (Descript, Janitor) where the LLM is embedded inside a non-agent product. This is a within-sample observation for one window of public data; it should not be generalized beyond it.

System prompts as the next lever: small and local models (V2)

This section follows a thread raised in the V1 field feedback and is the most forward-looking part of the note: it is a hypothesis and a proposed method, not a result.

The field feedback made a point we think is under-appreciated: most coding agents have not yet spent serious effort optimizing for small and local models. The harness conventions that ship by default are tuned for large frontier models; pointed at an 8B model running locally, they leave quality on the table. The hypothesis is that there is meaningful performance and reliability to unlock without changing the model at all — purely at the scaffolding layer (Axis B): the system prompt, the agents.md, the output conventions, the validators.

Our own local-model observations are consistent with this. In the LFM2.5-8B observation series, the model's worst failure mode was emitting literal escape sequences instead of real newlines and then looping on the resulting SyntaxError a dozen times. Decomposed against the two axes:

• It was an engine bug in the ownership sense — the loop had no output validator and no convergence check, so a malformed result was executed and re-executed.
• But a real share of it is recoverable at the scaffolding layer: a system prompt that pins output conventions (real newlines, no escaped literals) plus an output validator that parses before executing would have caught it before the loop ever started.

In other words, the same defect can be attacked from two layers, and for small/local models the scaffolding attack is cheap and has not been seriously tried.

A clean way to in practice measure it

The confound to avoid is changing several things at once. The design that isolates the scaffolding contribution:

1. Hold the model fixed — one small local model, one server version, one host.
2. Hold the engine fixed — the same harness/loop and tool-call machinery for every run.
3. Vary only the scaffolding — N system-prompt / agents.md variants (e.g. baseline, output-convention-pinned, role-decomposed, validator-instructed).
4. Measure — tool-call conformance (does it emit well-formed calls?) and task-satisfaction (did the task get done?), not just process-completion.

Because the model and engine are constant, any difference is attributable to the configuration layer alone — a direct read on how much the config layer unlocks on its own. This is on the Sigma5C V2 observation plan regardless; the harness and raw results are reproducible by design, so the experiment can be re-run and poked at rather than taken on trust.

Anthropic's "Effective Harnesses for Long-Running Agents" (Nov 2025)

Anthropic's piece is the canonical reference for the outer harness problem in particular: how to make an agent make consistent progress across many context windows, not just one.

The problem

The Claude Agent SDK has compaction, but compaction alone is not enough. Out of the box, even Opus 4.5 in a loop across multiple context windows fails to build a production-quality app from a high-level prompt. Two specific failure modes were observed:

1. One-shotting — the agent tries to do much at once, runs out of context mid-implementation, leaves a half-built feature undocumented, and the next session has to guess at what happened.
2. Premature completion — a later agent instance sees that progress has been made and declares the job done.

Both are classic outer-harness failures. The inner harness (compaction, context management) is working. The outer harness (session-to-session continuity, progress tracking, completion criteria) is not.

The two-part solution

Anthropic's answer is a deliberate two-agent pattern within the same underlying harness:

1. Initializer agent (first run only) — a specialized prompt that sets up the environment:

• init.sh script (runs the dev server)
• claude-progress.txt (running log of what agents have done)
• Initial git commit (shows what files were added)
• feature_list.json — a comprehensive list of feature requirements (200+ features in the claude.ai clone example), all marked passes: false

1. Coding agent (every subsequent run) — prompted to:

• Read claude-progress.txt and git log to get bearings
• Run pwd to confirm directory
• Read feature_list.json and pick the highest-priority failing feature
• Work on one feature at a time
• Leave a clean state: descriptive git commit + progress file update
• Only mark a feature passes: true after self-verification with browser automation (Puppeteer MCP), not just unit tests or curl

Why this matters for the framework

This is a textbook outer-harness design:

• Trigger logic — first-run vs. subsequent-run prompts
• Pre-processing — initializer sets up the environment
• Routing — first agent does init, later agents do incremental coding
• Persistent memory — claude-progress.txt + git history + feature_list.json together form a multi-layer memory system (text log + structured state + version control)
• Multi-agent coordination — two agents with different prompts, same harness
• Post-processing — git commits with descriptive messages
• Observability — the progress file is the observability surface; the agent reads it on startup and writes to it on shutdown

The piece is also honest about what it does not solve: vision limitations, browser-native modals (the Puppeteer MCP can't see alert modals), and the open question of whether single general-purpose agents beat multi-agent specialization (e.g. dedicated testing agent, QA agent, cleanup agent).

Mapping to OpenRouter observations

• Claude Code (51.6B tokens) — closest direct implementation of this Anthropic pattern; same SDK, same claude-progress.txt convention observed in user logs.
• Cline (19.8B) and Roo Code (8.94B) — implement similar session-continuity patterns in the IDE surface, not the CLI.
• Hermes Agent (574B) — large aggregate volume; almost certainly runs the longest continuous sessions of any harness in the data. Its "persistent memory across sessions" and "self-improving" framing maps onto Anthropic's claude-progress.txt + feature_list.json pattern, generalized to non-coding tasks.
• GitLawb (172B) — a "live decentralized git network" with the most aggressive session-continuity primitive available (git itself as the memory substrate). Worth follow-up.

The "76% fail on first try" stat

The prodfeat.ai piece cited in the search results claims 76% of AI agent tasks fail on first try — not because of the model, but because of the harness. The Anthropic piece is the practical counter-recipe: design the outer harness so the agent cannot one-shot, cannot declare early completion, and cannot lose state between sessions. Under that harness, the same model that fails 76% of the time in a bad harness is observed to succeed more often — we claim no fixed percentage here, because the original 76% number is one external source.

What changed from V1 to V2 (changelog)

• Field validation recorded — an independent engine-building team concurs with V1's findings (external concurrence, not ground truth).
• Nomenclature rewritten around two explicit axes (capability depth vs. architectural ownership), with a side-by-side table and the Claude Code worked example showing the words can invert.
• Sharper scaffolding definition adopted — declarative config (system prompts, agents.md, skills, specs) pulled up as the primary meaning; runtime machinery pushed down into the engine.
• Engine-as-dependency thesis extended — as engines become buyable, differentiation climbs into the wrapper and scaffolding layers.
• New section — System prompts as the next lever, with a concrete hold-model-and-engine-fixed / vary-scaffolding experiment design for small and local models.
• Data window, comparison table, and scores are unchanged from V1.

Sources

• Gaurang Karia, Wrapper vs. Scaffolding vs. Harness: What's in practice the Difference? — (source URL omitted in public pack) (Mar 2026)
• MindStudio Team, What Is Harness Engineering? Why Your Agent Wrapper Drives More Behavior Relative to the Model — (source URL omitted in public pack) (May 2026)
• OpenAI, Harness Engineering — (source URL omitted in public pack)
• Anthropic, Effective Harnesses for Long-Running Agents (Justin Young, Nov 26 2025) — (source URL omitted in public pack)
• OpenRouter publicly published application token-volume data, observation window 2026-06-01.
• V2 nomenclature and the small/local-model thread: field feedback on V1 from an independent engine-building team (attribution pending consent).

Sigma5C Model Observatory · Continuous AI Observability. We observe and measure how AI systems behave under controlled conditions over time. The observations in this note are time-bound and describe public data for one window; they are not permanent conclusions or rankings of the products named.

Companies

• Anthropic
• OpenAI
• Nous Research

Related observations

• Codex vs Claude — Native audit of 425K LOC · V1
• Agental Wrappers Inventory — 13 source trees · V1
• Cloud Wrapper Comparison — 31 models × 3 wrappers · V1
• Codex CLI 8-Model — speed observation · V1
• Large-Paper Embedding — Apr 28, 2026 · V1
• LFM2.5-8B — agentic tool-calling V2
• LFM2.5-8B — agentic tool-calling V1
• 10-Duo Wave — Feb 9, 2026 · V1
• OpenCode v1.2.1 Internals · V1
• Nemotron-3-Ultra — agentic tool-calling V1
• Ollama Deployment Observation — 27 models (Aug 2025) · V1
• OpenRouter Observability — 56-day campaign · V1
• Scaffolded Observations — 1,934 runs · V1
• Stage 1 — Cloud Reachability and Command Generation · V1
• Pre-retirement tool-use observation: o3-mini and o4-mini by OpenAI · V1
• Pre-retirement tool-use observation: three models through Ollama cloud · V1
• Harness vs. Wrapper — An Observation · V1