Model, Harness, Framework, Benchmark — and Why the Harness Moves the Score

When someone says "model X scores 60% on benchmark Y", four different pieces of software are involved, and the sentence names only one. The model is a stateless function from a prompt to a completion. The harness is the program that runs that function in a loop: it gives the model tools, assembles its context, runs commands in a sandbox, saves sessions, and exposes extension points. Claude Code, Codex CLI, OpenCode and Pi are harnesses. A framework is a library you use to build agents or harnesses, such as LangGraph, the OpenAI Agents SDK, or the Claude Agent SDK, which ships a whole harness as a library. A benchmark is a set of tasks, each with an environment and a grader. The benchmark is itself a harness that sits around the agent. This lesson pins down where each boundary sits, classifies real systems against them, and then covers the evidence that changing only the harness, with the model fixed, can move scores by several to tens of percentage points. In some measurements that is more than switching models. The conclusion that drives this course is that every score belongs to a model + harness pair. The AI Engineering lesson Harnesses and Scaffolds (04-03) already lists the harness's responsibilities and builds a minimal loop. This lesson assumes that background and goes deeper on the taxonomy and the measurements.

The four layers, defined precisely

Model. A model endpoint takes the whole conversation as input and returns the next message, which may contain text, tool-call requests or both. It keeps nothing between calls, cannot run anything, and cannot decide to stop. All it can do is emit text that asks for an action or looks like a final answer. In the Codex team's description of their agent loop, the model "either (1) produces a final response to the user's original input, or (2) requests a tool call that the agent is expected to perform", and the agent, not the model, then "executes the tool call" (Bolin, Unrolling the Codex agent loop, 2026).

Harness. The harness is everything else needed to turn that function into an agent that keeps working toward a goal. That includes the loop and stopping rules, the tool set and how tool output is presented, context assembly and compaction, the sandbox and permissions, sessions, and extensibility (hooks, skills, MCP, subagents). Both vendors now use this word for their own products. Anthropic's docs say "Claude Code is the layer around the model that provides the tools and manages the context the model sees. This surrounding layer is what the term agentic harness refers to" (Anthropic, How Claude Code works). OpenAI calls "the Codex harness" the part "which provides the core agent loop and execution logic that underlies all Codex experiences" (Bolin, 2026). One test separates a harness from a framework: a harness is a program you run on a task.

Framework. A framework is a library you import to write an agent or a harness. It gives you building blocks such as graphs, state, handoffs and guardrails, but no finished product. LangGraph describes itself as "a low-level orchestration framework and runtime for building, managing, and deploying long-running, stateful agents" (LangGraph docs). The OpenAI Agents SDK offers "agents", "handoffs" and "guardrails" plus "a built-in loop that continues until the task is complete" (OpenAI Agents SDK docs).

Benchmark. A benchmark is a set of tasks plus a way to grade them. Each Terminal-Bench task includes "a unique environment, human-written solution, and comprehensive tests for verification" (Merrill et al., 2026). Harbor, the runner used for Terminal-Bench-style tasks, defines a task as "an instruction, environment, and test script", where the test script must "produce a numerical reward" in a known file (Harbor task overview). The program that runs a benchmark (it provisions the container, starts the agent, enforces limits and calls the grader) is called the evaluation harness. It is a second harness wrapped around the first.

How the layers nest

These are not four peers. They nest. The model sits inside the agent harness, a framework is optionally the material the harness is built from, and the benchmark's evaluation harness wraps the whole agent together with an environment and a grader.

Read the bottom box as the thesis of this lesson. A reported score is a function of at least four inputs, and the model is only one of them. The Anthropic infrastructure-noise study, covered below, shows that even the outer evaluation harness's resource settings move the number. That is why Env sits inside the benchmark box and not in some neutral zone outside it.

The nesting also shows why the boundaries blur. The Claude Agent SDK is a framework in the sense that you import it, but what you import is a complete harness. Pi is a harness you run, and it also exposes an SDK mode. A benchmark runner is a harness whose "user" is a grader instead of a human. The next section classifies real systems so the edge cases are explicit.

Worked example — classifying eight real systems

Use two questions. Do you run it on a task, or import it to build something? (harness vs framework). Does it contain a grader? (benchmark). Here are eight systems this course returns to:

That last row is a good case study. Harbor's list of pre-integrated agents includes claude-code, codex, opencode, pi, mini-swe-agent, openhands and even langgraph, and you choose an agent with -a and a model with -m (Harbor, pre-integrated agents). So the benchmark runner's command line already treats the two as separate, independently chosen inputs. Module 07 uses exactly that interface to run tinyharness.

Evidence 1 — SWE-agent: the agent-computer interface, model fixed

The first sharp result is from SWE-agent. The authors argued that "LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces". They built an agent-computer interface (ACI), a set of file-viewing, search and edit commands designed for the model with concise, validated feedback, and put it in place of the raw Linux shell (Yang et al., 2024).

With GPT-4 Turbo, SWE-agent resolved 12.47% of the full SWE-bench test set. The previous best resolve rate was 3.8%, achieved by a non-interactive retrieval-augmented system (Yang et al., 2024, §1). That headline mixes two changes, though: going interactive, and the ACI design. The cleaner harness-only comparison is the paper's ablation on SWE-bench Lite. With the model again GPT-4 Turbo, a shell-only agent resolved 11.00% and the full SWE-agent interface resolved 18.00% (Yang et al., 2024, Table 3). The model, the tasks and the grader were the same in both runs. Only the tools and the format of their feedback changed, and that was worth 7 points, roughly a 64% relative gain.

Anthropic reported the same pattern when building its own SWE-bench agent: "we actually spent more time optimizing our tools than the overall prompt". One example was changing a tool to require absolute file paths after "the model would make mistakes with tools using relative filepaths" (Schluntz & Zhang, Building Effective Agents, 2024). Lesson 03-03 builds an ACI for tinyharness using these ideas.

Evidence 2 — Aider: an edit format change, model fixed

The second result is smaller and more surgical. It changes one harness decision, the edit format, which is how the model must write a code change for the harness to apply it. Aider built a benchmark of 89 Python refactoring tasks designed to provoke "lazy" coding, where the model leaves placeholder comments like # ... rest of code here instead of writing the code. With its SEARCH/REPLACE edit format, GPT-4 Turbo "only scored 20% as a baseline". Switching to a unified-diff format "raised the score to 61%", and lazy comments dropped from 12 tasks to 4 (Aider, Unified diffs make GPT-4 Turbo 3X less lazy, 2023).

Aider's explanation is an interface argument: "GPT is quantitatively better at code editing when you reduce the burden of formatting edits by using a familiar, simple, high level and flexible editing format" (Aider, 2023). Keep the caveats in view. It is one model, a vendor-built benchmark aimed at one failure mode, and a blog post rather than a peer-reviewed paper. As far as we know, no peer-reviewed comparison of edit formats exists, and lesson 03-04 measures them for that reason. Still, a 41-point gap from an output format alone shows clearly that "the model's coding ability" depends on the harness it is measured through.

Evidence 3 — HAL and "Just a wrapper?": a models × scaffolds grid

Single-paper ablations change one thing at a time. The Holistic Agent Leaderboard (HAL) varies models and scaffolds ("scaffold" here means harness) together at scale: "21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000", analysed across models, scaffolds and benchmarks (Kapoor et al., 2025). Three of its findings matter for this course (Kapoor et al., 2025, HTML):

• Model × scaffold interactions are real. On Online Mind2Web, "Claude models perform better with BrowserUse, while OpenAI models achieve higher accuracy with SeeAct". No single scaffold is best for every model.
• Specialist scaffolds beat generalist ones. "In cases where the same models are used for both task-specific and generalist scaffolds across three benchmarks, task-specific agents consistently outperform". On CORE-Bench Hard the task-specific agent won "9 of 12 runs".
• Cost diverges more than accuracy. SeeAct with GPT-5 Medium cost $171 and BrowserUse with Claude Sonnet 4 cost $1,577 on the same benchmark, "a 9x difference in cost despite just a two-percentage-point difference in accuracy".

MIT FutureTech's Just a wrapper? How much do scaffolds matter? re-analyses HAL's data. It is a blog post (tier-5 source), not peer-reviewed, so read it as informed commentary. Its headline is that "scaffolds explain more of the variance in price-performance in our data than models do", and that "switching from a generalist to a specialist scaffold can buy ~100× cheaper performance at the same accuracy". It also finds that a scaffold helps models unevenly: "some models see large benefits from a given scaffold — while others see little advantage or may even be hindered". Model rankings are correspondingly unstable. On GAIA, a Kendall τ of 0.17 across scaffolds means "only about a 58% chance that any two models keep their relative order" (Gundlach et al., 2026).

Evidence 4 — infrastructure noise: even the outer harness moves the score

The previous results vary the agent harness. Anthropic's infrastructure-noise study varies the evaluation harness, specifically how strictly the container's resource limits are enforced, while holding the model, the agent harness and the task set fixed. Across six resource configurations on Terminal-Bench 2.0, "the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01)". Infrastructure failures fell monotonically as headroom increased, from 5.8% of tasks under strict enforcement to 0.5% when uncapped (Segato, Quantifying infrastructure noise in agentic coding evals, 2026).

This matters because, as the post notes, "top spots on leaderboards often separated by just a few percentage points". Its practical rule is that "leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched". Its recommendations are to specify both a guaranteed resource allocation and a hard kill threshold per task, report how limits are enforced, and run evaluations at multiple times and days (Segato, 2026).

This is the reason Env sits inside the benchmark box in the nesting diagram. The grader may be objective, a test suite that passes or fails. But whether the agent gets far enough to be graded depends on memory limits, CPU, timeouts and whether the network is flaky, and all of those are harness configuration. Module 06-01 returns to infrastructure noise as one of the main threats to a benchmark's validity.

Evidence 5 — the case for mandatory harness disclosure

Stop Comparing LLM Agents Without Disclosing the Harness (May 2026) collects this evidence into an argument. It is explicitly a position paper. It defines the harness as "the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model". Its "Binding Constraint Thesis" is that, for long-horizon tasks across models of comparable frontier capability, "performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements" (Zhang et al., 2026).

Its evidence comes in two kinds, and they should be weighed differently (Zhang et al., 2026, HTML):

• Leaderboard figures it cites from other sources. On a third-party SWE-bench Pro leaderboard, six frontier models span only 4.9 points under one standardized scaffold. Under that scaffold Claude Opus 4.5 scores 45.9%, and under Claude Code "the same model reaches 55.4%", a 9.5-point swing. Citing HAL on SWE-bench Verified Mini, it reports cross-scaffold gaps (SWE-Agent vs HAL Generalist) "of 34 points for Claude Sonnet 4.5 (68% to 34%), 34 points for GPT-5 Medium (46% to 12%), and nearly 48 points for o4-mini".
• Its own controlled experiment. Three models (GPT-5.4, Kimi K2.6, GLM-5.1) were run with three harness configurations, from a minimal harness (no compression, no retry logic, no verification) to a full one (retrieval-compressed context, structured retry, self-checks, loop detection, checkpoint rollback). The task set was a 100-task subset of SWE-bench Verified, with two runs per cell. Variance across harnesses was 18.48 pp² and variance across models was 2.37 pp², "a ratio of 7.80×".

Hold this paper to the same standard it argues for. The factorial experiment changes many harness features at once and uses three models from one capability tier, so it shows that harness variance can dominate. It does not show that it always does. The paper's practical proposal is a structured harness disclosure card covering seven layers: execution, tools, context, scheduling, observability, verification and governance. The capstone asks you to publish one for tinyharness.

Try it — a toy model scored under two harness policies

The simulation below is a toy with invented numbers. It does not measure any real model or harness. It reproduces the mechanism behind the evidence above. A "model" is two fixed probabilities: how often it produces an edit the harness can apply ($p_{\text{edit}}$), and how often an applied edit actually fixes the bug ($p_{\text{fix}}$). The harness cannot change either number. It decides only one thing: when an edit fails to apply, does it drop the edit silently, or report the error and let the model retry up to $k$ times? The expected score is

$$\text{score}(k) \;=\; p_{\text{fix}}\,\bigl(1 - (1-p_{\text{edit}})^{\,k+1}\bigr).$$

Run it and watch the leader change when only the harness changes.

Under the no-retry harness, model_B leads (about 49% vs 42%) because its edits apply more often. Add a single retry-on-edit-failure and model_A takes the lead, because the harness now turns its formatting weakness into a recoverable error. Its stronger fixing ability then dominates. This is a toy version of HAL's finding that a scaffold helps some models much more than others, and of the rank instability measured on GAIA. A leaderboard built on either harness alone would report a "model ranking" that is really a ranking of model + harness pairs.

What this means for the course — every score is a pair

Put the evidence together and one working rule follows: a benchmark number describes a model + agent harness + evaluation harness configuration, never a model alone. The measured effects span every layer. There are tools and feedback format (SWE-agent, +7 points on Lite with the model fixed), edit format (Aider, 20% → 61%), whole-scaffold choice (HAL's tens-of-points gaps and rank instability), and resource enforcement in the evaluation harness (6 points on Terminal-Bench 2.0). Three habits follow, and the course practices all of them.

1. Read scores as pairs. When a model card or leaderboard reports a number, ask which harness produced it and with what limits. Module 06 does this for each major benchmark.
2. Change one harness variable at a time and measure it. This is how the SWE-agent ablation and Aider's edit-format result were obtained, and it is the method of the capstone.
3. Disclose your harness. Following the disclosure-card proposal and Anthropic's guidance on reporting enforcement settings, every result you publish in this course carries a description of its harness.

That is why the course is built around a harness you write yourself. tinyharness starts in lesson 03-01 as a short loop on your own OpenRouter key and gains one capability per lesson: tools and an ACI, edit formats, context management, a sandbox, sessions, hooks and subagents. Because you own every line, you can toggle any one decision, such as retry-on-edit-failure in the toy above, and measure its effect on a real Terminal-Bench slice via Harbor in Module 07. Before building, though, you need a vocabulary for which decisions a harness makes. The next lesson, 01-02, turns the components in the nesting diagram into the 10-point anatomy rubric used for every teardown and every build step that follows.

Sources

• Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan, Press — SWE-agent (2024, arXiv:2405.15793): LM agents as "a new category of end users"; 12.47% on the SWE-bench test set vs a previous best of 3.8%; SWE-bench Lite ablation with GPT-4 Turbo, shell-only 11.00% vs SWE-agent 18.00% (HTML).
• Aider — Unified diffs make GPT-4 Turbo 3X less lazy (2023): 89-task laziness benchmark; SEARCH/REPLACE 20% → unified diffs 61%; lazy comments 12 → 4 tasks.
• Kapoor et al. — Holistic Agent Leaderboard (2025, arXiv:2510.11977): 21,730 rollouts, 9 models, 9 benchmarks, about $40,000; model × scaffold interactions; task-specific vs generalist scaffolds; the 9× cost gap on Online Mind2Web (HTML).
• Gundlach, Brown, Lynch, Thompson (MIT FutureTech) — Just a Wrapper? How Much Do Scaffolds Matter? (2026): tier-5 blog post, a re-analysis of HAL data; scaffolds explain more price-performance variance than models; ~100× cost gaps; Kendall τ = 0.17 on GAIA.
• Segato (Anthropic) — Quantifying infrastructure noise in agentic coding evals (2026): a 6-point Terminal-Bench 2.0 gap from resource configuration alone (p < 0.01); infrastructure failures 5.8% → 0.5%; skepticism toward gaps under 3 points.
• Zhang, Wang, Ge, Xu, Hamm, Reddy — Stop Comparing LLM Agents Without Disclosing the Harness (2026, arXiv:2605.23950): position paper; Binding Constraint Thesis; cited leaderboard gaps; a 3×3 factorial with harness/model variance ratio 7.80×; the seven-layer disclosure card (HTML).
• Schluntz & Zhang (Anthropic) — Building Effective Agents (2024): agents as "LLMs using tools based on environmental feedback in a loop"; more time spent optimizing tools than the prompt for SWE-bench.
• Bolin (OpenAI) — Unrolling the Codex agent loop (2026): the Codex harness as "the core agent loop and execution logic"; final response vs tool call.
• Anthropic — How Claude Code works: Claude Code as "the layer around the model", the definition of "agentic harness".
• Anthropic — Agent SDK overview: "Claude Code as a library", with the same tools, agent loop and context management.
• LangGraph overview and OpenAI Agents SDK: self-descriptions as frameworks.
• openai/codex, OpenCode docs, pi.dev, mini-swe-agent: self-descriptions as harnesses.
• Merrill et al. — Terminal-Bench (2026, arXiv:2601.11868): task = environment + reference solution + tests.
• Harbor docs, task overview, pre-integrated agents: task = instruction + environment + test script → reward; agent (-a) and model (-m) chosen separately.

Practice

1. Classify and justify. Place each of these in the four-layer taxonomy and state the deciding question you used: Anthropic Managed Agents, the openhands-sdk package, SWE-bench Verified, the Codex Responses API endpoint, and a 40-line Python script that calls OpenRouter in a while loop. Which one is hardest to classify, and why?
2. Separate the confounds. SWE-agent reports both "3.8% → 12.47%" and "11.00% → 18.00%". Explain which comparison better isolates the effect of the ACI, and name the other variable that the headline comparison also changes.
3. Audit a claim. A vendor announces "our model gains 4 points on Terminal-Bench 2.0 over the competitor." Using the infrastructure-noise study and the disclosure-card proposal, list the five specific facts you would need before believing the gain is due to the model.
4. Extend the toy. In the py-run block, add a harness policy that truncates the model's view of the failing test output, as a third probability that the model "sees" the real error, which scales $p_{\text{fix}}$. Find a setting where truncation reverses the leader again. What real harness decision does this mimic?
5. Predict a design trade-off. HAL found specialist scaffolds beat a generalist scaffold, and MIT FutureTech reports cost differences of up to ~100×. Argue for or against making tinyharness a specialist coding harness rather than a general one, and say what you would measure to settle it.