What It Takes to Build a Terminal Agent That Doesn’t Break

A field guide to autonomous terminal work — the failure modes, the design choices, and how SimplAI Agent reached 52.13% on Terminal-Bench 2.0 with Claude Sonnet 4.6: the top Sonnet-class result on the public board.

SimplAI Engineering  ·  89 tasks · 445 trials · ~10h run  ·  June 2026

Reproduced under leaderboard rules: 5 trials per task, no timeout extension, no agent / verifier / environment overrides.

Introduction: Why This Report Exists

An agent that can write code is now ordinary. An agent that can sit at a real terminal, in a real environment, and get something done — compile a stubborn C project, recover a corrupted database, boot a virtual machine, crack a hash — is still rare. The gap between the two is not intelligence. It’s operating discipline.

Terminal-Bench 2.0 measures exactly that gap. It drops an agent into a fresh Docker container with a live shell and one instruction, then grades the result with a verifier that runs the actual artifact. There is nowhere to hide: either the program compiles, the test passes, the service answers — or it doesn’t.

We built a small, opinionated harness around Claude Sonnet 4.6 and reached 52.13% across all 89 tasks — the leading Sonnet-class entry on the public leaderboard. This report is less a victory lap than a field guide: what makes terminal agents fail, what you can do about it, and the path we took to get there.

SECTION 01  ·  THE PROBLEM: WHY TERMINAL AGENTS FAIL

A terminal agent runs one loop, over and over: observe the screen, decide on a command, act by sending keystrokes, then observe again. Simple to describe, unforgiving in practice — because a shell has no undo, no schema, and no patience. A single misread of the screen compounds for the rest of the task.

When we read hundreds of failed runs, almost none of the failures were the model ‘not knowing the answer.’ They were operational. Four archetypes accounted for the overwhelming majority:

The lesson that shaped everything after: terminal-task reliability is mostly an operations problem wearing an AI costume. The model supplies the reasoning. The harness has to supply the discipline.

Published harnesses on the same Sonnet-class model cluster between roughly 40% and 46.5%. That six-point spread isn’t the model thinking harder. It’s the scaffold wasting fewer turns.

SECTION 02  ·  THE SPACE OF SOLUTIONS — AND THE BET WE MADE

Once you accept that the bottleneck is operational, the design choices come into focus. There are four broadly sensible ways to attack the problem. We considered all of them:

The Design Principle

Constrain the model, then recover relentlessly.

Every turn the model emits exactly one structured response — reason about the screen, state a plan, send commands, declare done or not. Everything else is the harness watching for trouble and intervening before a slip becomes a lost task.

SECTION 03  ·  THE AGENT, IN FOUR FILES

The implementation is deliberately small — no framework, nothing to reverse-engineer six months from now. Four files carry the whole agent:

The recovery mechanisms map one-to-one onto the four failure archetypes from Section 01 — that correspondence is the whole design:

• Against the doom loop: a rolling window of recent commands. Repeat the same command, or pile up consecutive errors, and the harness forces a change of approach instead of letting the model dig.
• Against the wedged terminal: watch for identical screen snapshots in a row, then auto-inject recovery keystrokes rather than waiting on an external reset.
• Against context amnesia: cap noisy terminal output and summarise the conversation into a compact handoff before the window fills.
• Against premature ‘done’: a completion guard that double-checks the claim and only accepts it when the work genuinely holds up.

SECTION 04  ·  THE RESULT: 52.13%, AND THE ENGINEERING GAP

We ran the full 89-task suite five times over — 445 trials — under the exact rules the leaderboard validator enforces: no timeout extensions, no task-specific tuning, no overrides. The honest, conservative metric counts infrastructure errors as zero rather than discarding them.

On the same model that other harnesses cap in the low-to-mid 40s, ours reaches 52.13%. The table below is the entire argument of this report in one view: identical model class, very different scaffolds.

Leaderboard Score Comparison — Sonnet Class

Same model class, no model change. The full 11.9-point span from floor to our result is pure harness engineering.

What the Score Is Made Of

The 30 partial tasks are the most telling: the agent clearly can solve them — it just doesn’t do so on every attempt. That band of reliability, not raw capability, is the richest remaining vein of score.

A Clear Strength Profile

Where the harness does best maps neatly onto its design. It excels at bounded, verifiable, operational work — recover this, configure that, exploit this — and struggles where success hinges on long, fragile build chains or heavy ML training pipelines that fail on budget and environment rather than reasoning.

A hand-curated representative slice of the 89 tasks, shown to illustrate the strength profile — not an exhaustive partition.

SECTION 05  ·  THE ROAD THERE: SIX ITERATIONS, SIX LESSONS

We did not arrive at 52.13% in one shot. The score is the product of a tight loop of running the benchmark, reading the failures, fixing one thing, and running again. Each iteration taught something we’d carry into the next — and most of the lessons generalise well beyond this benchmark.

ITERATION 0  ·  BASELINE — THE NAIVE LONG-ROLLOUT AGENT

We started with generous turn budgets — give the model hundreds of turns and let it work. On our earlier hardware the run never even finished: only a handful of tasks completed before resources ran out.

Lesson: More turns is not more capability. An agent with no discipline mostly buys itself more rope to wander with.

ITERATION 1  ·  READ THE FAILURES — STOP GUESSING, START READING LOGS

Instead of theorising, we read the actual episode logs of failed runs and built a taxonomy. The four archetypes in Section 01 fell straight out — and they were repeatable, not random.

Lesson: Your eval logs already contain the roadmap. Categorising real failures beats inventing fixes for problems you imagine you have.

ITERATION 2  ·  TIGHTEN THE LOOP — A SMALLER BUDGET, ENFORCED HARDER

We cut the turn budget dramatically and added the anti-repetition and forced-approach-switch mechanisms. Counterintuitively, a tighter leash raised completion: the agent committed instead of dithering.

Lesson: Constraints are a feature. A hard budget forces decisiveness; an open-ended one rewards procrastination.

ITERATION 3  ·  THE ONE-LINE BUG WORTH POINTS — THE COMPLETION-CONFIRMATION TRAP

Our completion guard had a subtle flaw: it reset its own state whenever the model ran verification commands, trapping the agent in an endless verify-then-confirm oscillation. On a single task this burned 29 of 60 turns doing nothing. The fix was tiny; the payoff was not.

Lesson: One scaffold bug can dominate your score. Measure per-task turn spend — a task that eats its whole budget is a bug report, not a hard task.

ITERATION 4  ·  FAST FEEDBACK — RUN THE QUICK TASKS FIRST

We reordered the suite to run the fastest tasks first. A harness regression that used to take 15+ minutes to surface now showed up in nine — turning every run into an early-warning system for our own bugs.

Lesson: Order your evaluation for feedback speed, not just throughput. The sooner a regression screams, the cheaper it is to fix.

ITERATION 5  ·  THE SUBSTRATE LIES — A TERMINAL BUG MASQUERADING AS A MODEL BUG

Nearly half the trials in one run wedged within two minutes. It looked like the model failing — it wasn’t. Our bundled terminal multiplexer required newer system libraries than the older task images provided, so it silently failed to load and every keystroke errored. We rebuilt it against the oldest target and verified it actually runs before trusting it.

Lesson: Infrastructure bugs disguise themselves as model failures. Verify the substrate end-to-end before you blame the agent’s reasoning.

ITERATION 6  ·  THE FULL RUN — 89 TASKS, 5 TRIALS, 52.13%

With the loop tight, the recovery paths working, and the substrate solid, the full benchmark ran clean end to end — and landed at the top of the Sonnet-class board.

Lesson: The score is a lagging indicator of a hundred small reliability fixes. Win the operations, and the number follows.

SECTION 06  ·  WHAT’S STILL HARD, AND WHAT WE SHIP NEXT

27 tasks scored zero on every trial. Naming the cause precisely is what makes the next round tractable — and none of it is a model-reasoning ceiling:

• Long build & compile chains. Success needs a flawless multi-step toolchain dance inside the budget; one wrong flag early cascades to the end.
• Heavy ML pipelines. Real training and evaluation that strain wall-clock and environment setup — closer to an MLOps task than a terminal puzzle.
• Brittle environments. Heavy emulation and media stacks where one missing dependency is fatal and recovery is costly.

The Roadmap

• Close the partial-task variance first. 30 tasks already pass sometimes — tightening determinism there is worth more than any new capability.
• Budget by task class. A git-recovery task and a multi-hour compile shouldn’t share one turn-and-output budget. Adapt the envelope to the work.
• Better build-chain guidance for the weakest domain, to stop early-step cascades before they start.
• The same harness on frontier models. Operational wins compound on stronger reasoning — the obvious next submission.

What we won’t do: No per-task hacks. The leaderboard rules forbid task-specific overrides, and a harness that needs them isn’t a product. Every fix has to generalise, or it doesn’t ship.

SECTION 07  ·  THE SIMPLAI COMMITMENT

Across everything we build — voice agents at enterprise scale, and now autonomous terminal work graded by an unforgiving verifier — the lesson rhymes: the model is necessary but not sufficient.

Reliability lives in the engineering around it: recovery, budgets, context discipline, and honest measurement. That’s the work that turns a capable model into an agent you can depend on.

52.13% is a checkpoint, not a finish line. The methodology is reproducible, the failures are documented in full, and the next iteration is already scoped.

Building Agents That Can’t Afford to Break?

We’d love to compare notes on terminal autonomy, agent-harness design, and benchmarking at scale.

Talk to the team
Book Demo

Author bio

Sandeep Dinodiya Founder & CEO @ SimplAI

author

The Founder and CEO of SimplAI, an enterprise-grade Agentic AI platform built for regulated industries. With over 16 years of experience in enterprise SaaS, cloud platforms, and AI infrastructure, he leads the vision of making multi-agent orchestration accessible and production-ready for large organizations. Sandeep specializes in compliance-aware AI governance, scalable agentic architectures, and deploying LLM solutions across BFSI, healthcare, and Fortune-level enterprises.

See author's posts