What is an agent harness, and why does it move the score?

When a lab says "model X scores 60% on benchmark Y", the sentence names one piece of software. At least four are involved. The model is one of them. The program that runs the model in a loop, hands it tools and decides what it sees on every turn is another, and it gets far less attention than it deserves.

That program is the agent harness. Claude Code, Codex CLI, OpenCode and Pi are all harnesses. If you build agents, evaluate them or just choose which coding agent to use, the harness is half of what you are choosing.

The short definition

A model is a stateless function. You send it a conversation; it returns the next message, which may contain text, a request to call a tool, or both. It keeps nothing between calls and cannot run anything itself. In OpenAI's description of the Codex loop, the model either produces a final answer or "requests a tool call that the agent is expected to perform" (Bolin, 2026).

The harness is everything that turns that function into something that keeps working toward a goal. Anthropic's docs put it plainly: Claude Code "is the layer around the model that provides the tools and manages the context the model sees. This surrounding layer is what the term agentic harness refers to" (Anthropic).

Two neighbouring terms get mixed up with it:

• A framework (LangGraph, the OpenAI Agents SDK) is a library you import to build an agent. A harness is a program you run on a task. The Claude Agent SDK is the interesting edge case: you import it, but what you get is a complete harness.
• A benchmark is a set of tasks plus a grader. The program that runs it, provisioning containers, enforcing limits and calling the tests, is an evaluation harness wrapped around the agent harness.

The ten decisions every harness makes

Look at enough harnesses and the same design questions keep coming up. The course turns them into a 10-point anatomy rubric:

1. Loop and stop conditions. Who decides the agent is done: the model, a step cap, a budget, the user?
2. Tool set. Forty purpose-built tools, or just a shell?
3. Edit format. How a model's intended code change actually reaches the disk.
4. Context assembly and compaction. What the model sees each turn, and what gets summarised away.
5. Project memory. AGENTS.md, CLAUDE.md, GEMINI.md and how far up the tree they are read.
6. Sandbox and permissions. What the agent may do without asking.
7. Sessions and checkpoints. Whether you can resume, fork or rewind.
8. Extensibility. Hooks, skills, MCP, plugins.
9. Subagents. Whether work can be delegated to a fresh context.
10. Provider layer and UI. Which models it speaks to, and where you meet it.

None of these is about the model's weights. All of them change what the model can accomplish.

The evidence that the harness moves the score

This is not a matter of taste. Several studies changed only the harness, kept the model fixed, and watched the number move:

• Tools and feedback format. SWE-agent's ablation on SWE-bench Lite held GPT-4 Turbo fixed. A shell-only agent resolved 11.0% of tasks; the same model with a purpose-built agent-computer interface resolved 18.0% (Yang et al., 2024).
• Edit format alone. On Aider's refactoring benchmark, switching GPT-4 Turbo from a SEARCH/REPLACE format to unified diffs raised the score from 20% to 61% (Aider, 2023). It is one model on a vendor benchmark, but a 41-point gap from an output format is hard to ignore.
• Whole-scaffold choice. The Holistic Agent Leaderboard ran 21,730 rollouts across 9 models and 9 benchmarks and found real model × scaffold interactions: Claude models did better with one web scaffold, OpenAI models with another (Kapoor et al., 2025).
• Even the outer harness. Anthropic varied only container resource enforcement on Terminal-Bench 2.0 and saw a 6-point spread, concluding that leaderboard gaps under 3 points "deserve skepticism until the eval configuration is documented and matched" (Segato, 2026).

A 2026 position paper pulls this together into an argument for mandatory "harness disclosure cards" (Zhang et al., 2026). You don't have to accept its strongest claim, that harness variance usually beats model variance, to take the practical point: every benchmark score belongs to a model + harness pair, never to a model alone.

What this changes in practice

Three habits follow.

Read scores as pairs. When a model card quotes a number, find out which harness produced it and under what limits. A model that looks weaker on one leaderboard may simply be running in a scaffold that suits it worse.

Change one harness variable at a time. Retry on a failed edit, a different edit format, a tighter context budget: each is a lever you can measure, the same way the SWE-agent and Aider results were obtained.

Pick a harness for what it's for. A minimal bash-only loop is the right baseline for comparing models. A permissioned, checkpointed harness is the right default for running against a repo you care about. Those are different jobs.

Learn it by taking harnesses apart, then building one

Our course Agent Harnesses: Anatomy, Build, Frameworks, Benchmarks is built around this idea. Across 32 lessons you:

• score Claude Code, Codex CLI, OpenCode, Pi and seven more harnesses on the 10-point rubric;
• build your own harness, tinyharness, in Python on your own OpenRouter key, adding one capability per lesson from the loop to sandboxing, sessions, hooks and subagents;
• rebuild it on LangGraph, the OpenAI Agents SDK and the Claude Agent SDK to see what a framework actually buys you;
• learn how SWE-bench, Terminal-Bench and the other headline benchmarks work, and where they rot;
• finish by running tinyharness on a Terminal-Bench slice through Harbor and publishing a harness disclosure card for your result.

Every lesson is citation-dense, and in LearnOS you can ask questions against the text and get answers that point back to the exact section.

Start Agent Harnesses on LearnOS