Reading Notes on Harness Engineering: AI Engineering’s Third Paradigm Shift
One-Sentence Summary
Harness Engineering = building an “office” for the Agent, instead of endlessly polishing the “email wording”
Background: The Evolution Across Three Generations of Engineering Paradigms
| Era | Core Question | Focus | Analogy |
|---|---|---|---|
| Prompt Engineering (2022-2024) | How do I talk to the model | Wording, format, few-shot, CoT | Writing a perfect email |
| Context Engineering (2025) | How do I put the key information in front of the model | RAG, conversation history, tool definitions | Attaching every file the email needs |
| Harness Engineering (2026) | How do I make the model reliably get things done in a real environment | Workflows, constraints, feedback loops, toolchains | Architecting the entire office |
These three generations don’t replace one another — each one expands the layer of abstraction outward. The closer a task is to real production, the more the latter two matter.
Core Argument: Agents aren’t hard; the Harness is hard
This is the honest takeaway from OpenAI Codex team member Ryan Lopopolo after building 1 million lines of code and 1,500 PRs with a GPT-5 agent.
Experimental data:
- Same model, same data, same prompt — changing only the runtime environment (the shell) jumped coding benchmark success rate from 42% to 78%
- Anthropic’s experiment: a simple prompt-and-run setup produced a broken product ($9), while a structured, iteratively managed environment produced a playable game ($200)
Cost difference is irrelevant. Capability difference is everything.
Takeaways as an Agent Developer
1. We Used to Optimize the Wrong Layer
We spent enormous time on prompt templates, few-shot examples, and role-play. But what actually decides success or failure is the design of the agent’s runtime environment.
Anyone who has built agents has been through this: the demo dazzles, production falls apart. The reason is that a demo is “a single interaction under perfect conditions,” while production is “continuous operation in a real environment.”
2. Constraints Paradoxically Increase Productivity
While working on “Self-Driving Codebases,” the Cursor team found that when a model is capable enough, it burns a huge number of tokens on dead ends and nonsense approaches. A well-designed Harness forces the agent to converge on the right answer faster by giving it clear boundaries and a limited set of high-quality tools.
You’re not restricting the agent — you’re giving it a narrow path to success.
3. Architectural Constraints Must Be Enforced by Linters, Not Requested via Prompts
One of the hard rules the OpenAI team learned: Architectural constraints are enforced by linters, not prompts.
Don’t tell the agent “please follow the singleton pattern” — build a linter that enforces the singleton pattern. Don’t request. Constrain.
4. When the Agent Fails, the Harness Is the Problem
The OpenAI team has a principle: If a PR requires significant human intervention, the agent is not the problem — the Harness is.
This is the core mindset shift. When the agent makes a mistake, we don’t scold it — we improve its runtime environment so that the mistake can never happen again. This is exactly Mitchell Hashimoto’s definition of Harness Engineering:
“Every time you discover an agent has made a mistake, you take the time to engineer a solution so that it can never make that mistake again.”
5. Anthropic’s Key Finding: Models Can’t Reliably Evaluate Their Own Output
This points directly to a GAN-style architecture: separate Generator and Evaluator agents. It’s also why a reliable Harness needs an independent verification layer.
In Practice: The Five Pillars of a Harness (from OpenAI + Stripe)
- Single source of truth: the repository is the agent’s only source of information; assume no external knowledge
- Agent-readable code: not just human-readable — it also has to be understandable by the agent (clear comments, structure)
- Linters enforce architectural constraints: rules aren’t prompts, they’re code
- Incremental authorization: the Harness must have stages and gates; the agent can’t start out with full permissions
- Two-strike rule (Stripe): if the agent’s first fix attempt fails, escalate to a human immediately — don’t waste cycles
What This Means for My Current Project
Applied to our iUX Diyuan Health project:
- Tool-calling layer: needs stricter tool definitions and an error-handling harness
- Multi-step workflows: need to introduce stage-level gates and feedback loops
- Code generation: the code an agent outputs needs linter-enforced constraints, not prompt reminders
Harness Engineering isn’t a tool — it’s an upgrade in engineering philosophy, from “training the model” to “building the system.”