AI-generated report — July 2026. This page was produced by an AI assistant drawing on published benchmarks, blog posts, and repository documentation. It should be read as a secondary synthesis, not primary research. The landscape moves fast — check specific numbers against the original source before relying on them.

Context

When someone uses a tool like Claude Code, OpenCode, Codex CLI, or Cursor Agent, they are not using a raw AI model on its own. They are using a harness — the surrounding software that wraps the model in a run loop (how it plans and executes steps), a set of tools (file editing, shell commands, search), permission rules (what actions are allowed), a way of managing context (what information fits in the model’s limited window), and an output format (how changes are applied). The same model placed inside a different harness can produce noticeably different results in correctness, cost, and speed. Choosing the right harness for a task can matter more than picking one frontier model over another.

A growing family of benchmarks tries to measure how much the harness matters, separate from the model. This page summarises the most relevant ones in plain language.

Key Takeaways

1 The harness matters as much as the model. The same model under different harnesses can swing in score by 17–27 points — often larger than the gap between competing frontier models. A state-of-the-art model can look mediocre if wrapped in a poor harness, and a mid-tier model can punch above its weight with a well-designed scaffold. Researchers now treat the harness as a first-class variable, not a detail.

2 When everyone solves the task, speed and cost decide. On straightforward tasks where all good harnesses succeed, efficiency becomes the differentiator. Wall-clock time can vary by 4× and token cost by 8× between harnesses running the same model. The fastest harness finishes in minutes while another takes nearly twenty — and API costs scale accordingly. For teams running agents at scale, this efficiency gap translates directly into real budget differences.

3 Reliability is the hidden axis. A single successful run (pass@1) might look impressive, but it hides how often the agent really fails. The stricter pass^k metric — succeeding on every repeat — tells a different story. State-of-the-art agents that score 60%+ on a single attempt can drop below 25% when required to get it right eight times out of eight. In production, where users do not retry until it works, reliability is the metric that matters most.

4 Benchmarks are not neutral. A reported score reflects the model plus the harness plus the scoring pipeline — not the model alone. A benchmark's native scaffold may favour certain models or tool-calling formats. One study showed that switching from a benchmark's own pipeline to a uniform scaffold reversed model rankings entirely. Always ask: "What scaffold produced this number?" before drawing conclusions.

5 Infrastructure changes scores. Resource configuration alone — CPU, memory, network, timeout settings — can swing benchmark results by about 6 points (p < 0.01). Cold-start latency on serverless providers can shift latency-sensitive benchmarks by 10–20%. A benchmark score is never purely about the agent; it is always about the agent plus the infrastructure it runs on.

6 The field moves fast. Scores saturate (everyone hits the ceiling), benchmarks get contaminated (tasks leak into training data, inflating scores), and new variants appear monthly. The SWE-bench Verified leaderboard, once the gold standard, is now considered contaminated. Any snapshot — including this one — is temporary. Always check the publication date. 7 Using AI to track AI. The pace of change makes it economically impractical for humans to track every benchmark release manually. Hence this auto-generated, human-in-the-loop update document: AI compiles the landscape, a human reviews and decides what matters. Expect periodic refreshes as the field evolves.

Plain-Language Glossary

• Harness — the scaffolding around a model: tools, run loop, permissions, context handling, output format.
• Scaffold — often used interchangeably with harness; the supporting code and prompts that turn a model into an agent.
• pass@k — the task counts as solved if at least one of k attempts succeeds. Optimistic; flatters agents.
• pass^k — the task counts as solved only if all k attempts succeed. Strict; measures consistency.
• Pass@1 — single-trial success rate. The most common headline number.
• Benchmaxxing — when a model or agent looks good because it is over-optimised for public benchmarks or has seen the answers in training data.
• Contamination — benchmark tasks leaking into a model's training data, inflating its score.

The Benchmarks

HarnessBench

HarnessBench compares Codex CLI, Claude Code CLI, and Cursor Agent CLI side-by-side on 27 real debugging tasks drawn from 9 open-source repositories. Each task is scored by deterministic hidden tests (core tests for the required fix, regression tests for surrounding behaviour) — never by an AI judge. This avoids the “LLM-as-a-judge” problem where the grader itself can wobble.

The top condition was Codex / GPT-5.5 / xhigh at 22 out of 27 passes. Runtime differences were clearer than accuracy differences: the fastest Cursor configuration finished in a median of 3.6 minutes, while the slowest Claude configuration took 15.1 minutes. The author cautions that 27 tasks are too few for statistically significant ranking claims, but that harness differences are real and show up in exploration style, timeout handling, and caching behaviour — factors that rarely appear in model-only benchmarks.

Note on fairness: Repository-local steering files (such as AGENTS.md, CLAUDE.md, .codex, .claude) are sanitised before each run to prevent the agent from being quietly steered toward a known answer.

Repository · Blog post

Claw-SWE-Bench

Claw-SWE-Bench is a 350-task multilingual benchmark with an adapter protocol that makes different agent harnesses (called “claws”) comparable under fixed settings — same prompt, same runtime budget, same workspace contract, same evaluator. It spans 8 languages and 43 repositories.

The key finding is remarkably clean: harness choice changes the score by 27.4 percentage points under fixed models, while model choice changes it by 29.4 points under fixed harness — harness and model are nearly equal in effect size. A minimal adapter (asking the model to write a unified diff directly) scores only 19.1%, while a full adapter (letting the model edit repository files through tools and exporting the patch from Git state) reaches 73.4% with the same model. The difference is not the model’s coding ability — it is whether the output format matches what the scoring system expects.

Paper · Repository

OpenBench

OpenBench compares coding-agent harnesses — codex, pi, opencode, cursor, devin, and open-model claude — on self-contained coding tasks. Grading uses checker scripts (exit 0 = solved, optionally with SCORE: for partial credit), never the harness’s own claim of success. This prevents the agent from declaring victory when it has not actually solved the task.

Its Track A pins all harnesses to the same canonical model (gpt-5.5-medium) so that any difference comes from the harness alone. Findings so far: correctness saturates for frontier harnesses on easier tasks — they all solve everything — so efficiency becomes the separator. Wall-clock spread reaches about 4× and token cost up to about 8×. The pi harness is repeatedly the fastest and leanest across panels. Open models are surprisingly close to frontier models: a 72-run open-model matrix cost about $1.02 in total API spend, demonstrating that cost-effective evaluation is feasible.

Repository

SWE-bench Family

SWE-bench is the dominant coding-agent benchmark and the one most cited in model announcements. The agent receives a real GitHub issue and a repository checkout, produces a patch, and is scored by the repository’s own test suite. No judge model, no rubric, no partial credit — either the tests pass or they do not. Variants include:

Variant	Tasks	Notes
SWE-bench Full	2,294	12 Python repos, original set
Verified	500	Human-verified solvable (with OpenAI). Now considered contaminated
Lite	300	Filtered subset, faster eval
Multilingual	300	9 languages
Pro	1,865	Enterprise-difficulty (Scale AI). Top ~23% vs 70%+ on Verified
Live	50+ new/month	Continuously growing (Microsoft Research)
Mobile	Proprietary	iOS production codebase, multi-modal (PRD + Figma)

Key insight: Scores are a property of model + scaffold + harness, not the model alone. On SWE-bench Pro, a basic scaffold scores 23% while an optimised 250-turn scaffold scores 45%+ — a 22-point swing from scaffolding alone, with no model change. On SWE-bench Mobile, the same model (Opus 4.5) achieves 12% on Cursor but only 2% on OpenCode — a 6× gap. The best overall configuration across all agents and models scored only 12%, revealing a large gap between current agent capabilities and the demands of industrial software development. Most failures came from incomplete implementations, not from misunderstanding the task.

Leaderboards · SWE-bench Mobile

Terminal-Bench

Terminal-Bench measures end-to-end terminal agent capability: compiling code, training ML models, configuring servers, reverse engineering binaries, and running scientific workflows. These are open-ended, multi-step tasks that require planning, error recovery, and environment awareness — not just editing a single file.

Harness engineering proof point: LangChain improved from 52.8% to 66.5% (+13.7 points) by changing only the harness (system prompt, tool choice, execution flow) while keeping the model fixed. Anthropic also showed that infrastructure configuration alone swings scores by +6 points (p < 0.01).

Top scores (early 2026): GPT-5.3-Codex at 77.3%, Claude Code Opus 4.6 at 65.4%.

TAU-bench / τ-bench

TAU-bench (τ-bench) evaluates tool-agent-user interaction in customer-service domains — retail, airline, and later telecom and banking/knowledge. The agent must follow a written policy document across a multi-turn conversation while using domain API tools (booking flights, processing refunds, checking policies). Grading compares the final database state to an annotated goal — the transcript quality is irrelevant. The agent cannot bluff its way to a pass with confident-sounding text; it must actually make the correct tool calls that change the database.

The most important metric here is pass^k: the probability that all k independent attempts succeed. State-of-the-art agents drop below 25% at pass^8 in retail, even when single-run scores look healthy in the low-to-mid 60s. For production agents that must get it right every time, pass^k is the more honest signal.

τ²-bench (2025) adds a dual-control telecom domain where the user also holds tools. τ³-bench (2026) expands to voice full-duplex and knowledge retrieval.

Repository · τ²-bench · Leaderboard

AgentBench

AgentBench evaluates agents across eight distinct environments: OS shell, database SQL, knowledge graph queries, digital card game, household simulation, web shopping, web browsing, and lateral-thinking puzzles. It is the broadest major benchmark — a model that crushes coding benchmarks might fail at web navigation or creative reasoning. 2026 community scores: Claude Opus 4.7 ~73%, GPT-5.3 Codex ~70%.

Caveat: Per-environment scores diverge by 30+ points — aggregate scores can mask zeros in individual environments. A 2026 study found all eight major agent benchmarks could be reward-hacked by automated scanning agents, so top scores warrant extra scrutiny.

Repository

OpenCode Harness

OpenCode Harness is a clean-room, model-agnostic harness for evaluating coding agents across providers — DeepSeek, Qwen, Claude, OpenAI, and local endpoints (vLLM, SGLang, Ollama). It standardises the agent loop, tool permissions, trace production, and eval reporting. Its first DeepSeek diagnostic benchmark (4 suites, 12 tasks) exposed concrete failure modes: marker-following drift (the agent stops tracking which files it has edited), tool-loop overrun (the agent repeats unsuccessful tool calls), long-context synthesis gaps (the agent loses track of earlier context), and repair finalisation gaps (the agent cannot finish fixing its own mistakes). Results are presented as diagnostic evidence rather than a leaderboard — the goal is to improve harness design, not to rank models.

Repository

Coder Eval

Coder Eval is an open-source framework that runs real agents (Claude Code, Codex, Gemini Antigravity) in a sandbox against declarative YAML tasks, then scores the files and commands they produced. It is designed for CI gates (fail the build on regressions), A/B experiments between agent configs (model vs. model, prompt vs. prompt, tool-on vs. tool-off), and skill trigger verification (did the agent actually use the skill?). Unlike fixed-dataset benchmarks that rank models on a shared leaderboard, Coder Eval scores your own tasks — the tasks, skills, and workflows you ship — with weighted 0.0–1.0 continuous criteria, per-tool cost telemetry, and a JUnit XML report for CI pipelines. It fills the gap between “research benchmark” and “production CI check”.

Repository

Other Notable Benchmarks

• ICAE-Bench — 480 anonymised tasks across 12 programming languages. The agent receives a deliberately fuzzy Product Requirement Document and must clarify missing requirements with a hidden-spec Oracle (a user agent that knows the ground truth) before implementing. Four-part scoring: dynamic tests, structural similarity, critic review, and interaction quality. This tests whether agents can ask clarifying questions — a critical real-world skill.
• EvoCode-Bench — 26 stateful coding tasks with 5–15 rounds per task. Tests whether agents can keep a project working as user requirements change cumulatively. Later rounds inherit earlier implementation decisions, dependencies, file layouts, and API choices. Runs on the Harbor multi-step framework with per-step verifiers.
• LoCoBench-Agent — 8,000 interactive scenarios across 10 languages and 36 domains, with context ranges from 10K to 1M tokens and multi-turn evaluation up to 50 turns. Nine bias-free metrics rigorously validated to eliminate file count bias — including execution success rate, memory retention across turns, and cross-file consistency.
• OpenCode Bench (anomalyco) — Multi-judge evaluation across five dimensions (API signature compliance, logic equivalence, integration correctness, test coverage, project checks). Uses three independent LLM judges per submission with variance penalties for disagreement. Three isolated episodes per evaluation for statistical reliability.
• FeatureBench — Tests feature development rather than bug fixing. A revealing data point: Claude 4.5 Opus drops from 74.4% on SWE-bench to 11.0% on FeatureBench, showing that current scaffolds are far better at patching existing code than at building new functionality from scratch.

Reliability Metrics in Plain Terms

Metric	Definition	Meaning
pass@k	At least 1 of k attempts succeeds	Optimistic; flatters agents
pass^k	All k attempts succeed	Strict; measures consistency
pass@1	Single-trial success rate	Most common headline metric

Why this matters: pass^k is the more production-relevant signal because it measures consistency, not peak capability. An agent that works 8 of 10 times independently has a pass^2 of 64% and a pass^4 of 41% — far below its pass@1 of 80%. In a workflow that needs the agent right every time — processing refunds, deploying code, responding to customers — the worst run governs the outcome, not the best. If you see only pass@1 in a vendor's report, ask for pass^k before trusting the number.

Maintenance

This landscape changes rapidly. New benchmarks and updated scores appear weekly. The content of this report should be viewed as a snapshot from mid-2026, not a permanently current reference. Scores shift as models update, benchmarks evolve, and contamination accumulates. The SWE-bench Verified leaderboard that was the gold standard in 2024 is now considered contaminated — this will happen to today’s benchmarks too. For live leaderboard data, consult the benchmark-specific sites linked above.

Evolution

The pattern across all agentic harness benchmarks points toward a convergence on multi-axis evaluation: raw accuracy alone is insufficient to determine whether an agent is ready for real-world use. Future benchmarks increasingly pair correctness with cost accounting (dollars per task), latency measurement (time to completion), pass^k reliability (consistency across repeated runs), and policy adherence (did the agent follow the rules?). The harness — not the model — is the variable that separates production-ready from demo-ready: a well-designed harness makes a capable model reliable, while a poor one makes any model unpredictable.

---

This report was generated by an AI assistant on 28 July 2026. It synthesises publicly available sources including the HarnessBench blog and repository, Claw-SWE-Bench, OpenBench, SWE-bench documentation, τ-bench documentation, AgentBench, OpenCode Harness, Coder Eval, ICAE-Bench, EvoCode-Bench, LoCoBench-Agent, and survey articles from Rapid Claw, Prefactor, and others cited in the text. Verbatim claims should be checked against the primary sources before relying on them.