Agent Harnesses — Syllabus & How to Use This Course

This course covers the agent harness: the code around a stateless language model that makes it an agent. That means the loop, the tools, the edit format, context management, the sandbox, sessions, hooks and subagents. You take apart the most popular coding-agent harnesses, build your own from scratch in Python, rebuild it on the leading agent frameworks, and measure it on the benchmarks the frontier labs report today.

It is written for practitioners who have already built a basic tool-calling agent and want to go deeper. The reason to care is that the harness is not a thin wrapper. In the SWE-agent paper's ablation, GPT-4 Turbo stayed the same and only the interface changed, from a plain shell to a purpose-built agent-computer interface. That alone raised SWE-bench Lite resolution from 11.0% to 18.0% (Yang et al., 2024). Many choices that decide whether an agent works are harness choices, not model choices.

The course runs on one project. tinyharness is the agent you build. It starts as a 30-line loop and gains one capability per lesson. Later modules rebuild it on frameworks, benchmark it, and ship it. Every lab uses your own OpenRouter API key, so any model from any provider plugs into the same code.

What you'll be able to do

• Take apart any agent harness with a 10-point rubric: loop, tools/agent-computer interface (ACI), edit format, context, project memory, sandbox and permissions, sessions, extensibility, subagents, and the provider/UI layer. Explain why Claude Code, Codex CLI, OpenCode and Pi made different choices.
• Build a working coding-agent harness in Python on OpenRouter. It will stream tool calls, edit files reliably, compact its own context, run in a sandbox behind permission gates, persist and resume sessions, take hooks, skills and MCP servers, and spawn subagents.
• Choose between a raw API, a framework and a harness-as-library for a given job. You will have ported the same task onto LangGraph, the OpenAI Agents SDK, the Claude Agent SDK and others, and compared the results.
• Read agent benchmark results critically. That covers task + environment + grader, pass@k vs pass^k, contamination, saturation, infrastructure noise, and why every leaderboard entry is a model + harness pair.
• Run your harness on Terminal-Bench through Harbor, read its trajectories, and make a measured improvement to its interface.
• Turn an eval harness into an RL environment. You will understand rollouts, verifiers and rewards well enough to start fine-tuning against your own tasks.

Prerequisites

You should already be comfortable with:

• Python. You will write a few hundred lines of it. The course uses Python 3.12 and uv.

• The terminal, git and Docker. Docker runs the sandbox and Harbor. It works on macOS, Linux and Windows (WSL2).

• Tool calling and the basic agent loop. If those are new, do these lessons in the AI Engineering course first:

  • 02-03 Tool Calling
  • 03-01 What Is an Agent?
  • 03-02 ReAct
  • 03-04 Memory and Context Management
  • 04-03 Harnesses and Scaffolds

  AI Engineering 05-01 Agentic Frameworks and 07-02 The Benchmark Landscape are short introductions to Modules 05 and 06 here.

• An OpenRouter account and API key, which you bring yourself. Lesson 03-01 walks through creating the key and putting a spending limit on it. Each lab gives an approximate cost and a cheap default model.

Python is the only course language. Several of the harnesses you study are written in TypeScript (Claude Code, OpenCode, Pi). Where it helps, a short "In TypeScript" sidebar shows the equivalent. You never need to write TypeScript to finish the course.

The roadmap

The course has four arcs: anatomy → build → frameworks → benchmarks. Each arc uses the one before it.

Modules 02 and 03 can be read in either order once Module 01 is done. Doing the teardowns first gives you more ideas to borrow while you build. Module 06 is mostly reading and can be taken any time after Module 01. You need it before Module 07.

Time commitment

About 30–33 hours in total: about 16 hours of reading and about 15 hours of labs, including the capstone. At two sittings a week, one module per week takes eight to nine weeks. The build modules (03 and 04) are the heaviest. Give each of their lessons a full sitting, because every one ends with a working increment of tinyharness that later lessons build on.

Cost. The labs bill model spend to your OpenRouter key. Every lab states a typical cost per run and a cheap default model. The Terminal-Bench runs in Module 07 and the capstone are the most expensive part, so they come with a stated estimate before you start. The course never holds or spends money on your behalf.

How to study in LearnOS

• Keep tinyharness running. Commit after every lesson. If a later lesson breaks something, you want the last working version one git checkout away.
• Score what you read. For each teardown in Module 02, fill in the 10-point rubric yourself before reading the lesson's scoring. Disagreeing with the lesson is fine. Not having an opinion is not.
• Practice active recall instead of re-reading. After each lesson, close it and explain the key design decision out loud. Retrieval practice improves long-term retention more than re-studying does (Roediger & Karpicke, 2006).
• Review flashcards with spaced repetition. Each module has a deck. Spacing reviews out beats cramming the same total time (Cepeda et al., 2006), so let the scheduler set the pace.
• Chat with the lesson when a section doesn't land. Answers cite the specific section they came from.
• Watch the dates. This field moves monthly. Every lesson has a last_reviewed date, and benchmark scores carry an "as of" date. Treat anything older than a few months as a starting point to re-check.

The capstone

You ship tinyharness as a public repo with a measured result:

1. Run a fixed slice of 10–20 tasks from Terminal-Bench 4.0 through Harbor, with at least two models on your OpenRouter key.
2. Make one change to the agent-computer interface (ACI), such as a new tool, a different edit format or a smarter truncation rule. Re-run the same slice and report the before/after difference honestly, including noise.
3. Publish a harness disclosure card: model, harness version, tools, edit format, context policy, sandbox and resource limits, budget, and run count. With it, anyone can see what your number actually measures.

This makes a strong portfolio piece. It shows you can build an agent, evaluate it the way the labs do, and change it based on evidence rather than vibes.

What this course omits

• Training the model. Module 07 ends at the point where an eval harness becomes an RL environment. Fine-tuning a model against it (SFT, GRPO, agentic RL) is the separate Open Fine-Tuning Stack course.
• IDE integration. IDE extensions, editor plugins and building a polished terminal UI are discussed in the teardowns but not built.
• Hosted agent platforms in depth. Hosted platforms and enterprise deployment (managed agent services, SSO, fleet management) get a mention but no lesson.
• Non-coding agents as primary subjects. Voice, browser-only and customer-support agents appear through their benchmarks (τ³-bench, BrowseComp, OSWorld) rather than as builds.
• A ranking of "the best" harness or framework. The course gives you the rubric and the measurements and lets you reach your own conclusions. Any ranking would be out of date within a quarter.

How this course was made — a disclosure card for the course itself

This course asks you to publish a harness disclosure card for every number you report, so here is one for the course itself. It was written by an AI agent harness and reviewed and approved module by module by its human editor. The figures come from the session's own transcripts, summarized by tools/session_stats.py in the course repository.

Read it the way Module 06 taught you to read a model card. It states what was measured and where the numbers come from, and it names what was not verified.

Sources

• Roediger & Karpicke, 2006: the testing effect, i.e. retrieval practice beats re-study.
• Cepeda et al., 2006: meta-analysis of the distributed-practice (spacing) effect.
• Yang et al., 2024 (SWE-agent): in its ablation, changing only the interface the harness presents moved performance with the model held fixed.