Harness vs. Wrapper — An Observation · V1

About this observation

TL;DR

A wrapper makes an LLM easier to call. A harness makes an LLM reliable enough to ship. Across the 12 OpenRouter application categories in this observation window, high token-volume applications consistently exhibit the harness pattern (Hermes Agent, Kilo Code, OpenClaw, Cline, Claude Code, pi); applications that exhibit the wrapper pattern (extra.email, HyperWrite AI, GPT Workspace) appear in the lower-volume tail of their respective categories. This is an observed correlation, not a claim about superiority.

The definition, from the Substack piece cited in the sources, is: if an agent = LLM + loop with tool calls, the harness is everything except the LLM itself.

1. Wrapper (the thinnest layer)

A wrapper is glue around an LLM API. It does four things, and nothing more:

• Authentication — manage API keys
• Request formatting — shape input to what the model expects
• Response parsing — hand back something clean
• Basic error handling — retry a failed call, surface a clean error

That's it. A wrapper is passive. It doesn't shape what the model does, just how you talk to it.

Examples observed in the OpenRouter data:

• extra.email (3.15B tokens across 4 categories) — calls an LLM to classify inbox messages. A thin LLM call inside a category sorter. The product is the inbox, not the agent.
• HyperWrite AI (174M tokens) — text-completion frontend with no tool loop, no memory across sessions, no iteration.
• GPT Workspace (934M tokens) — Chrome extension that routes prompts to OpenRouter. The harness is the browser context, the extension is the wrapper.

These products use LLMs; they are not defined by their agentic capability. The model is a feature, not the core.

A note on the word. We use wrapper here in its narrow, literal sense — the thin glue layer above — and not as a value judgment. The term is contested: many builders use wrapper to mean the product and feature layer built on top of an engine (see On nomenclature, below), which is substantial work, not thin glue. Where this note says "wrapper pattern," it means the thin-glue end of the spectrum, nothing more.

A useful mental model: a wrapper is a handle on a knife. The knife doesn't change. You just made it easier to hold.

2. Scaffolding (the middle layer)

Scaffolding is what most "AI app" products ship, and is sometimes mislabeled as a "wrapper." It provides structure around the call:

• Prompt templates — pre-structured inputs that guide behavior
• Conversation state — track what's been said across turns
• Tool-calling loops — let the model invoke a function and continue
• Retry logic — decide when to try again and how
• Context window management — summarize, truncate, inject

Scaffolding gives the model a shape to work within. It's still relatively passive. It sets up the environment without actively managing outcomes.

In current practice the word is often drawn more narrowly still: the declarative configuration that steers an agent without being code — system prompts, agents.md, skills, and spec files (for example Spec-Kit or BMAD-style specs). That is a sharper and useful line, because it separates the config that steers an agent from the runtime machinery (tool-calling loops, retry, context management) that some taxonomies place in the engine instead. We flag the distinction here rather than legislate it, and pick it up again under On nomenclature.

Examples observed in the OpenRouter data:

• LibreChat (671M tokens) — a ChatGPT-clone UI with multi-model routing and conversation persistence, but no autonomous tool loop or feedback iteration.
• Chatbox AI (607M tokens) — on-device copilot with a chat surface and some context, no agentic loop.
• T3.Chat (773M tokens) — high-performance chat app; the "harness" is just the chat UI plus model selection.

The key tell: scaffolding tools have a chat box as their primary surface. Wrappers often don't even have a UI (they're embedded); harnesses usually have a task surface, not a chat surface.

A useful mental model: scaffolding is the frame of a building under construction. It holds things in place but doesn't check the work.

3. Harness (the full system)

A harness is the infrastructure that compensates for the model's weaknesses. It catches errors the model makes, enforces constraints the model would ignore, and creates feedback loops so the model can iterate.

A well-designed harness includes:

• Tool access — file system, terminal, browser, APIs the model can call
• Memory and context management — keep relevant state across long tasks and across sessions
• Feedback loops — run the code, check if tests pass, feed the result back
• Guardrails and constraints — linters, structural rules, enforced deterministically, not by the model's judgment
• Observability — logs, traces, metrics the agent can see and use to self-correct
• Multi-agent coordination — routing to specialized subagents, handoffs, shared state

The key observation: the harness actively manages outcomes. It's not setting up an environment and stepping back. It's continuously shaping what the model does next.

Examples observed in the OpenRouter data (high token volume, harness pattern):

• Hermes Agent — 574B tokens, observed in 5 categories. Self-improving agent with persistent memory, 40+ tools, scheduled automations, subagents. Consistent with the harness pattern.
• Kilo Code — 139B tokens, observed in 4 categories. Cross-IDE + CLI coding agent with agentic workflows. Tool calls, file edits, terminal, browser automation.
• OpenClaw — 123B tokens, observed in 5 categories. Real actions: runs commands, browses the web, manages files, sends emails. The harness is the product.
• Claude Code — 51.6B tokens, observed in 3 categories. Reads the codebase, plans, executes, runs tests, iterates. Canonical harness.
• pi — 37.4B tokens, observed in 5 categories. CLI-native harness.
• Cline — 19.8B tokens, observed in 4 categories. Lives in the IDE, autonomously explores, edits, runs terminal commands.
• Roo Code — 8.94B tokens, observed in 4 categories. Team of specialized agents with customizable modes — i.e. the harness includes a routing layer.

A useful mental model: the harness is the full training system. Feedback after every session, corrected form, tracked progress, adjusted plan. The model is the brain in the jar; the harness is everything that makes the brain useful in the real world.

The two layers inside any harness

The MindStudio framework splits the harness into inner and outer layers, which this note uses to describe the layers:

Inner harness — the immediate wrapper around a single model call

• System prompt and instruction structure
• Context assembly logic (retrieval, memory injection, history summarization)
• Tool and function definitions
• Output format specifications (JSON, schema, etc.)
• Output validators — parse and check before acting on it
• Retry and fallback logic

The inner harness is what most developers spend their time on first. Small changes here produce large differences in behavior, not because the model changed, but because what you gave it and how you structured the response handling changed.

Outer harness — the orchestration around the inner harness

• Trigger logic — what starts the agent (webhook, schedule, user action, another agent's output)
• Pre-processing pipelines
• Routing logic — which agent (or version) handles a given input
• Persistent memory systems (separate from in-context memory)
• Multi-agent coordination and handoffs
• Post-processing pipelines
• Monitoring and observability

The outer harness is where architectural decisions live: single agent vs. multi-agent, vector store vs. structured DB, how concurrent runs are handled. These questions have nothing to do with which model was picked.

The insight from the data: harness design and model behavior

The note offers the following interpretation of the OpenRouter data:

Three signals in the OpenRouter data support this:

1. Same model family, different harness, different volume. These three commonly run the same underlying model family, each behind a different harness:

   • Kilo Code (139B) — multi-IDE harness
   • Cline (19.8B) — VS Code-native harness
   • Roo Code (8.94B) — VS Code multi-agent harness

   Same model family, different harnesses, an observed ~15x volume spread within this window.

2. Model switching is cheap; harness redesign is expensive. When teams swap models, they usually don't rebuild the harness. They swap the model and adjust the inner harness slightly. The outer harness stays the same. If the model were the primary driver of performance, model swaps would be high-stakes events. They are configuration changes.

3. The note identifies these harness failure classes:

   • Tool call returned an error, no retry logic caught it
   • Model produced output in the wrong format, no validator caught it
   • Context window got too full, agent lost track of earlier instructions
   • Two agents wrote conflicting state because no coordination layer

   These examples concern tool errors, validation, context handling, and coordination in the surrounding system.

Why the vocabulary matters

When something that is a harness is called a "wrapper," the people building it under-invest in it. When a harness is called "just scaffolding," the active ingredient is missed: the feedback loop, the guardrails, the observability that makes an agent reliable rather than impressive in demos and broken in production.

A wrapper engineer asks: How do I make this API easier to call? A scaffolding engineer asks: How do I give this model structure to work within? A harness engineer asks: How do I build the environment that catches failures, checks constraints, and lets this model iterate?

Three different questions. Three different systems. Three very different outcomes.

The vocabulary shapes the design decisions.

On nomenclature — the field hasn't aligned

The industry has not standardized these three words, and reasonable practitioners draw the lines differently. The definitions above order the layers by capability depth — how much each layer compensates for the model and actively manages outcomes. That is the axis used here when the subject is reliability, which is the subject of this note.

Builders who work closer to the engine — particularly teams building agentic IDEs and SDK-first tools — tend to order the same three words by architectural ownership instead: where a layer sits in the dependency stack, and whether you build it or buy it. On that axis the definitions come out roughly like this:

Two things are worth saying plainly. First, "harness" and "wrapper" can point in opposite directions between the two readings. A tool that is the harness under the capability-depth reading can be a wrapper on a bought engine under the architectural-ownership reading. When the words conflict, name the product and the layer you mean, not just the word.

Second, the two axes are close to orthogonal and both are useful. The capability-depth axis answers where does reliability come from. The architectural-ownership axis answers what should we build versus buy, which is the more practical axis for a team deciding whether to write their own engine or sit on a vendor SDK. Neither reading is wrong.

Notably, the central finding of this note survives — and sharpens — under the engine reading. The note describes these failure classes as engine failures: every failure class in the section above (an uncaught tool error, an unvalidated output format, a context overflow, an absent coordination layer) lives in the engine, not the model. And it extends the thesis one turn further: as engines themselves become buyable, the differentiation that used to live in building the engine moves up into the product and configuration layers — the wrapper and the scaffolding, in the ownership reading. The model was always a dependency; increasingly the engine is becoming one too, and the question becomes which layer a given team still has to build itself.

How Sigma5C reads this

Sigma5C is an independent AI evaluation, research, and development company. We observe and measure how AI systems behave under controlled conditions over time. The harness/wrapper distinction is useful to us because it names where the behavior we observe actually comes from — the orchestration around a model, not the model in isolation. A few readings of the public OpenRouter data, in that frame:

• The note interprets the volume spread through harness design. Several high-volume applications run the same underlying model family behind very different harnesses, and the volumes differ by more than an order of magnitude. Under this reading, the differentiator users responded to is the surrounding system, not the model choice.
• Routing is an outer-harness decision. "Small models route, large models investigate" is an architecture choice — it lives in the orchestration layer. Where it shows up, it works because the system was deliberately designed around model specialization, not because a single model was selected.
• The ~51.6B tokens observed for Claude Code is a harness signal: a first-party agent appearing on a third-party router because users chose that route for cost controls, model flexibility, or multi-model routing — a harness-layer decision made by the user.
• The ~5.17B observed for OpenAI Codex reads as a routing-visibility effect — most users of that agent reach it through its native channels rather than this router. Same kind of agent, different routing, an order-of-magnitude volume spread.
• Pieces.app at ~12.1B tokens is evidence for memory as a standalone harness component — a persistence layer other harnesses can build on, rather than a model in its own right.

The recorded reading for this observation window: across the agent categories in this window's public data, the harness-shaped applications occupy the larger token-volume range and the wrapper-shaped applications the smaller. The model is increasingly a dependency; the observable differentiation lives in inner + outer harness design. This is an observed correlation for one window of public data — we claim no permanent superiority for any application named here.

This is the kind of question Sigma5C works on: Continuous AI Observability and the observation layer that records how AI systems — and the harnesses around them — behave under declared conditions, over time, so the evidence accumulates instead of resetting with every model release.

Comparison table: 10 high-volume agents, scored on inner/outer harness criteria

This is a sample, not a ranking. Scoring rubric (0-2 each, 14 max):

Inner harness (8 pts)

• I1 System prompt / instruction structure (decomposed, focused, role-specific)
• I2 Context assembly (retrieval, memory injection, history management)
• I3 Tool / function definitions (rich, well-described, parameterized)
• I4 Output validation (schema checks, format assertions before action)

Outer harness (6 pts)

• O1 Routing / multi-agent coordination (specialized subagents, handoffs)
• O2 Persistent memory across sessions (not just in-context)
• O3 Observability (logs, traces, self-correction feedback)

Observed pattern: the high-volume sample clusters at 9-14 (harness). The single vertical-specialist (Descript) and the chat-frontend (Janitor AI) score 4-6 despite meaningful token volume, because their product surface is narrow and the agentic loop is shallow. The observation: token volume is read alongside the recorded harness criteria in agent categories, and diverges from it in vertical-product categories (Descript, Janitor) where the LLM is embedded inside a non-agent product. This is a within-sample observation for one window of public data; it should not be generalized beyond it.

Anthropic's "Effective Harnesses for Long-Running Agents" (Nov 2025)

Anthropic's piece is the canonical reference for the outer harness problem in particular: how to make an agent make consistent progress across many context windows, not just one.

The problem

The Claude Agent SDK has compaction, but compaction alone is not enough. Out of the box, even Opus 4.5 in a loop across multiple context windows fails to build a production-quality app from a high-level prompt. Two specific failure modes were observed:

1. One-shotting — the agent tries to do much at once, runs out of context mid-implementation, leaves a half-built feature undocumented, and the next session has to guess at what happened.
2. Premature completion — a later agent instance sees that progress has been made and declares the job done.

Both are classic outer-harness failures. The inner harness (compaction, context management) is working. The outer harness (session-to-session continuity, progress tracking, completion criteria) is not.

The two-part solution

Anthropic's answer is a deliberate two-agent pattern within the same underlying harness:

1. Initializer agent (first run only) — a specialized prompt that sets up the environment:

   • init.sh script (runs the dev server)
   • claude-progress.txt (running log of what agents have done)
   • Initial git commit (shows what files were added)
   • feature_list.json — a comprehensive list of feature requirements (200+ features in the claude.ai clone example), all marked passes: false

2. Coding agent (every subsequent run) — prompted to:

   • Read claude-progress.txt and git log to get bearings
   • Run pwd to confirm directory
   • Read feature_list.json and pick the highest-priority failing feature
   • Work on one feature at a time
   • Leave a clean state: descriptive git commit + progress file update
   • Only mark a feature passes: true after self-verification with browser automation (Puppeteer MCP), not just unit tests or curl

Why this matters for the framework

This is a textbook outer-harness design:

• Trigger logic — first-run vs. subsequent-run prompts
• Pre-processing — initializer sets up the environment
• Routing — first agent does init, later agents do incremental coding
• Persistent memory — claude-progress.txt + git history + feature_list.json together form a multi-layer memory system (text log + structured state + version control)
• Multi-agent coordination — two agents with different prompts, same harness
• Post-processing — git commits with descriptive messages
• Observability — the progress file is the observability surface; the agent reads it on startup and writes to it on shutdown

The piece is also honest about what it does not solve: vision limitations, browser-native modals (the Puppeteer MCP can't see alert modals), and the open question of whether single general-purpose agents beat multi-agent specialization (e.g. dedicated testing agent, QA agent, cleanup agent).

Mapping to OpenRouter observations

• Claude Code (51.6B tokens) — closest direct implementation of this Anthropic pattern; same SDK, same claude-progress.txt convention observed in user logs.
• Cline (19.8B) and Roo Code (8.94B) — implement similar session-continuity patterns in the IDE surface, not the CLI.
• Hermes Agent (574B) — large aggregate volume; is discussed in the note through its session-continuity framing. Its "persistent memory across sessions" and "self-improving" framing maps onto Anthropic's claude-progress.txt + feature_list.json pattern, generalized to non-coding tasks.
• GitLawb (172B) — a "live decentralized git network" using git itself as the memory substrate for session continuity.

The "76% fail on first try" stat

The prodfeat.ai piece cited in the search results claims 76% of AI agent tasks fail on first try — not because of the model, but because of the harness. The note discusses Anthropic's session-continuity pattern alongside that external claim. This note does not independently measure the 76% figure or a change in task success under that pattern.

Sources

• Gaurang Karia, Wrapper vs. Scaffolding vs. Harness: What's Actually the Difference? —

  (Mar 2026)

• MindStudio Team, What Is Harness Engineering? Why Your Agent Wrapper Drives More Performance Than the Model —

  (May 2026)

• OpenAI, Harness Engineering —

• Anthropic, Effective Harnesses for Long-Running Agents (Justin Young, Nov 26 2025) —

• OpenRouter publicly published application token-volume data, observation window 2026-06-01.

Sigma5C Model Observatory · Continuous AI Observability. We observe and measure how AI systems behave under controlled conditions over time. The observations in this note are time-bound and describe public data for one window; they are not permanent conclusions or rankings of the products named.

Companies

• Anthropic
• OpenAI
• Nous Research

Related observations

• Harness vs. Wrapper — An Observation · V2