What Is an AI Agent Harness? Why a Finished Fix Became a Blank Answer SheetSame AI, same task, and one harness made the difference between failing and passing. A tour of an agent’s body, following the case.

01Case file: an AI that fixed everything and handed in a blank sheet

Let’s start at the scene. The task was one problem from SWE-bench Verified, a software bug-fixing exam: fixing a bug in Sphinx, a Python documentation tool. The case number is sphinx-doc__sphinx-9658.

The AI edited the source code, and the existing tests passed. To finish, it wrote a neat report that starts with “## Summary of Changes” and ends with “the existing tests all pass.” Here is the problem. The harness accepted that report as the submission, just as it was. Not a single line of code change reached the grader, and the task was marked unresolved.^[1]

The researchers then put the same final model into an evolved harness and had it solve the same task again. Both runs reached the same state: the code was edited and the existing tests passed. But this time the harness held back the first submission and said, “the patch you submitted contains no files.” The AI checked its changes again, built a diff, submitted a valid patch and passed.^[1] Let’s put the two scenes side by side.

The brain was exactly the same. The only difference was one extra look at the moment of submission. So who wrote that warning? Not a person. We will come back to that in Chapter 5. First, the suspects: the model that fixed the bug, and the harness that wrapped it and accepted the answer sheet. What exactly is a harness, if it can flip the ending?

02What the model does, and what the harness does

Let’s start with the model. A large language model (LLM) takes in text and predicts what comes next. The model itself does not keep any memory between calls. When an AI service seems to remember your last chat, that is because some mechanism feeds the earlier conversation and work log back in on the next call. When the model writes “I’d like to open this file,” that same mechanism actually opens the file and returns the result. It also decides when to stop. This execution layer is the harness.^[6]

LangChain put the relationship in one formula: “Agent = Model + Harness.” And they added: “If you’re not the model, you’re the harness.” In other words, all the code, configuration and execution logic outside the model is the harness.^[7]

So the same brain can do very different things depending on the body you put it in. In a chat with no tools connected, it can only tell you the fix in words. Inside a coding harness, it edits the files and runs the tests itself. These days chat screens often use tools too, so the difference is not the screen. It is the harness connected behind it. Claude Code’s docs put it this way: Claude Code is the harness, and Claude is the model inside it.^[8]

The numbers show it too. Terminal-Bench 2.0, which tests terminal skills, lists many agents that use the same model. The Meta-Harness paper from Stanford and other researchers collected the official leaderboard records in a table. With the same Claude Opus 4.6, Claude Code scored 58.0% and Meta-Harness scored 76.4%.^[9] We selected six from that table and drew them as bars.

That is why an experiment where one team kept the model fixed and changed only the harness is so interesting. LangChain kept gpt-5.2-codex as it was and tuned only the harness. Its Terminal Bench 2.0 score went from 52.8 to 66.5, jumping from outside the top 30 into the top 5.^[10] Hugging Face’s terminology guide also explains that two products using the same model can feel completely different because their harnesses make different choices.^[6]

03Anatomy of a harness: workspace, tools, memory, checkpoint

So what is inside a harness? A 2026 survey paper that reviews research in this area splits it into seven layers: execution, tooling, context, lifecycle, observability, verification and governance.^[11] The names sound stiff, but the roles are simple. A workspace to work in, tools that act as hands, a device that keeps memory, a loop that repeats the work, a device that keeps records, a checkpoint that checks results, and an approval step that blocks risky actions. Tap each part.

Back to the case. In this diagram, which slot was empty in the blank answer sheet case?

Here is the core of it. The model decides what to do. The harness decides what it can actually do, and when the work counts as finished.

04The more you give, the more confused it gets: context and a workbench for AI

“So why not just show it everything?” That is a common thought when you first build a harness. But often, the more information you give a model, the harder its job gets. A model can only look at so much text at once (its context window), and Anthropic explains that as the number of tokens in the window grows, the model gets worse at accurately recalling information from it. This is often called context rot. So they stress that context must be treated as a finite resource with diminishing returns.^[12]

Tools work the same way. When tools overlap in function or have a vague purpose, an agent gets confused about which one to use.^[13] If a human engineer cannot say for sure which tool fits a situation, we cannot expect an AI to do better.^[12] That is why a good harness picks well instead of giving a lot. Switch the habits below on and off, and watch how much room is left for new information and for the answer.

Every fix on the right is a technique in real use. Anthropic’s write-up describes compaction (summarize an old conversation and continue in a fresh window), loading files only at the moment they are needed, and handing research to sub-agents that return only a short summary of about 1,000 to 2,000 tokens.^[12] Anthropic’s Agent Skills also use “progressive disclosure” as a core design principle: they show the table of contents first and open the full text only when needed.^[14] The goal is not to empty the window. It is to pick what is needed right now and put it in.

A workbench at AI eye level: SWE-agent’s ACI

A screen can be designed on the same principle. In 2024, Princeton researchers described language model agents as “a new category of end user, with their own needs and abilities,” built an interface just for them, and called it the agent-computer interface (ACI).^[15] A person can skim hundreds of lines spilling across a terminal and spot the one that matters. For a language model, every line that spills out takes up context. So when a search returned more than 50 results, SWE-agent showed none of them and asked the agent to write a more specific query. It showed files at most 100 lines at a time and ran a linter to check syntax after every edit. Press the buttons to compare the two screens, and below that, try guessing the answer first.

It paid off. Using GPT-4 Turbo, SWE-agent resolved 12.47% of the 2,294 SWE-bench tasks. That is more than three times the previous best of 3.8%, from a system that retrieves material once and generates an answer. There is also a same-model comparison: on SWE-bench Lite, an agent with only the default Linux shell solved 11.0%, while SWE-agent with its ACI solved 18.0%. Not a single weight of the model changed.^[15]

The same conclusion in the field

Outside the lab, the harness is treated as real skill too. In 2026, “harness engineering” started to sound like a job title. Here are four snapshots.

The record from OpenAI’s Codex team is especially telling. A team that started with three engineers and grew to seven merged about 1,500 pull requests, and no code was written directly by humans. Their principle is short: “Humans steer. Agents execute.”^[16] They also learned early that knowledge an agent cannot see might as well not exist.

Anthropic reached a similar conclusion. Give an agent a task that takes days, and it starts each new session with no memory of what the last session did. So an initializer agent in the first session leaves a feature list, a progress file and a first git commit, and each later session finishes just one feature at a time.^[17] The memory is not left to the model. The harness keeps it in files.

05The AI started building its own checkpoint: Self-HPO

Most of the harnesses we have seen so far were crafted by people. But in 2026, research on AI editing harnesses itself poured out. If you sort it by who edits the harness, and whether the model’s weights change too, you see four branches. Tap the cards one by one.

The results of the third branch are impressive too. Shanghai AI Laboratory’s Self-Harness kept the model as it was and edited only the harness. On Terminal-Bench 2.0, it raised Qwen3.5-35B-A3B’s overall pass rate from 18.0% to 36.7%. On problems that were kept aside and not used while editing the harness, the rate rose from 23.8% to 38.1%.^[19] Sakana AI and colleagues’ RHI also keeps rewriting a harness written as prompts and does not touch the model’s weights.^[20] But a critique also appeared, arguing that such gains from harness evolution do not clearly beat spending the same budget on more attempts.^[21] If the model stays fixed, it tends to circle inside the same blind spots. If you train only the model, it learns only within the range of behavior that a fixed harness brings out. So the fourth branch emerged: research that grows the harness and the model together. HarnessForge, released in June 2026, goes in this direction.^[22]

Then, on October 16, 2026, researchers from Microsoft Research, the University of Virginia and Carnegie Mellon University released Self-HPO. The name stands for self-evolving coding agents via harness-policy optimization. The blank answer sheet case comes from this paper’s appendix.^[1]

The starting point is modest. The model is the open model Qwen3.5-35B-A3B, which has 35B total parameters and uses only about 3B of them at a time.^[23] The harness is mini-SWE-agent, roughly 100 lines of Python.^[18] From there, two steps alternate.

In the H-step, the model’s weights are frozen while it solves problems. The model reads its own failure logs and writes six proposed harness revisions. Each candidate is actually run, and the harness that solves the most wins. In the P-step, the model is trained with reinforcement learning inside that harness. The only reward is whether the tests pass. Then the tasks it never solved during training are collected and used as the next H-step’s exam. This makes sure harness fixes always aim at the model’s latest weak spots.^[1]

Notice there is no smarter “teacher model” in this loop. The same model fixes the harness and learns inside it.

Now let’s see where the blank answer sheet got caught

So what did the AI write into its own harness? If you expected a grand new reasoning algorithm, it may feel a little plain. According to the authors’ blog, almost everything the AI added was a practical check that runs at the moment of submission. The harness does not accept the patch right away. It looks at it first, tells the model what looks wrong, and gives it one more chance. There are three checks: a missing-file check, which the authors call a coverage check (did every file it touched end up in the patch?), a decorator reminder (does the patch use a decorator such as @cache without importing it?) and a PR intent reminder (does the patch deal with the key words of the issue?).^[24] Choose which checks to switch on at the checkpoint.

This is the very checkpoint that stopped the blank answer sheet in Chapter 1.^[1] Still, a checkpoint only finds problems and sends the work back. The fixed answer has to pass the check again.

Interestingly, a human-built harness put its effort in the same place. LangChain added a checklist step that forces a verification pass before the agent finishes.^[10] Human or AI, “one more check before you finish” protects the score. The team released the evolved harnesses as code, so anyone can compare them with their own.^[25]

06How much better, and how far can we trust it?

We cannot draw a conclusion from one case. Let’s look at the full report card. The researchers ran this round of extra training on Python tasks only, then tested on three benchmarks: SWE-Bench Verified (500 Python tasks), SWE-Bench Multilingual (300 tasks in 9 languages other than Python)^[26] and the longer, harder SWE-Bench Pro.^[27] Switch the tabs.

Self-HPO raised the three-benchmark average resolve rate from 57.8% to 64.8%, up 7.0 points. It beats both harness-only evolution (60.1%) and model-only training (61.9%). Even though this round of extra training used only Python tasks, the resolve rate on the exam made of nine other languages rose 5.5 points, while harness-only changes had barely moved it (+0.1).^[1]

What if you change both?

There is an even more interesting number. Adding only the evolved harness to the base model raised the Verified score by 3.5 points. Putting only the trained model into the base harness raised it by 3.4. So what happens when you use both together? Try guessing before you press the button.

The answer is 8.5 points. That is 1.6 points more than the 6.9 you get by adding the two separate gains. One plus one came out bigger than two.^[1] The secret lies in the learning signal of reinforcement learning. This style of training tries each problem several times and learns by comparing attempts that worked with ones that did not. If every attempt fails, there is nothing to compare and the signal goes dead. With model-only training, these dead groups mostly stayed between 11% and 17% late in training. In Self-HPO’s final training phase, they fell to 6.5%.^[1] We can read it this way: when the harness catches silly mistakes like a blank answer sheet, some attempts start succeeding, and there is more to learn from.

Use a strong model as a teacher, or as a designer?

Here is a comparison that is useful for practitioners. Companies that want to grow a small model often collect a strong model’s solutions and train the small model on them (SFT distillation). That uses the strong model as a “teacher.” The researchers compared three approaches with Claude Opus 4.8. The first is SFT, which trains the small model on the solution records. The second is the Meta-Harness approach, where the small model stays as it is and Opus 4.8 only designs the harness. The third is HPO, which adds the small model’s reinforcement learning on top of Opus 4.8’s harness design.^[1] Press the buttons to compare.

The model trained on solution records rose to 65.0% on the familiar Verified exam, but on the unfamiliar Multilingual (49.6%) and Pro (49.1%) it fell below even the base model. The three-benchmark averages are 54.6% for SFT, 63.5% for Meta-Harness and 66.3% for HPO. Harness design alone scored 8.9 points higher than SFT, and adding the small model’s reinforcement learning widened the gap to 11.7 points.^[1] In this experiment, that means using the strong model as a harness designer worked better than using it as a teacher. The first author put it this way on his blog:

“If you have a strong model, put its knowledge into the harness rather than the weights.”Zhepei Wei, Self-HPO blog ^[24]

The designer did not have to be a strong model, though. Adding Opus 4.8 as the designer lifted Verified from 69.5% to 71.6%, another 2.1 points. But the authors concluded that this helps but is not required.^[1] The checkpoint that caught the blank answer sheet was also not written by Opus. The model that was in training wrote it itself.

Even when the average rises, some problems get worse

It is not all good news. Split the tasks by how often the base agent solved them in 8 tries, and the gains are uneven. Medium tasks, which it solved only sometimes, rose 17.2 points. Easy ones rose 12.9. Hard ones, which it never solved in 8 tries, still rose 6.1. The share of tasks it could solve within 8 tries grew from 80.0% to 84.8%. But on tasks it solved all 8 times, the score dropped 2.2 points.^[1]

The paper does not explain this dip. Since the harness and the model changed together, we cannot tell which one caused it. Just remember that a better average does not mean every problem got better.

The benchmarks themselves deserve careful reading too. In February 2026, OpenAI said SWE-bench Verified was increasingly contaminated, stopped evaluating on it and recommended SWE-bench Pro instead.^[28] This paper’s Pro score uses the authors’ public V2 set (642 tasks), so it cannot be compared directly with the official leaderboard. The harness and the model were improved in turns for two rounds, and scores are averages of three evaluation runs.^[1] It is safer to read this as a sign that the direction works.

07Check the harness of the AI you use

Now let’s turn our eyes to our own screen. Whether it is a coding agent or a work automation tool, the AI we use has a harness too. Use the five questions we learned from the blank answer sheet case to check it. Just switch on the items that apply.

Back to the blank answer sheet case. The model was the same. What changed was one small check that made it look again right before submitting. An AI’s skill does not live only inside the model. It also lives in the world around the model: the harness. So next time you hand work to an AI, start with one question: what does it check before it says “done”? The answer tells you what kind of tack it is wearing.

New to agents and automation? Start with our guide to handing busywork to AI agents, and for coding AI rankings, read our coding AI comparison.

Five questions readers ask next

Are a harness and a framework the same thing?

Not quite. A framework is a code toolkit developers use to assemble an agent. The harness is the whole runtime the assembled agent actually runs in. You can build a harness with a framework, and the same framework can produce very different harnesses.

Isn’t a good prompt enough?

A prompt is just one part of the harness. Where commands run, what the agent may touch, what happens on failure and whether the work is really done cannot be settled by a prompt alone. LangChain kept its model fixed and changed only the harness. Its score went from 52.8 to 66.5 on Terminal Bench 2.0.

Am I already using a harness?

If you use a coding agent, yes. Claude Code’s docs describe Claude Code as the harness and Claude as the model inside it. If you add agents to everyday work, start with four things: an isolated workspace, only the permissions you need, checks such as tests, and run logs.

Isn’t it risky for an AI to edit its own harness?

It can be. Research on self-evolving agents warns that instructions hidden in outside content, once stored in memory, can become a persistent compromise that survives across sessions. Self-HPO ran in an environment and test suite that the researchers controlled. In real use, harness changes deserve the same review and testing as code.

Should I take the Self-HPO numbers at face value?

Read them with care. The harness and the model were improved in turns for two rounds, and the scores average three evaluation runs. OpenAI stopped evaluating on SWE-Bench Verified because of contamination, and the Pro score uses the public V2 set of 642 tasks that the authors used. Treat the results as a direction, not a final verdict.