Coding Agents Meet 70% of Requirements but Finish 26% of Tasks

One of the tasks in a new coding benchmark from Alibaba looks almost boring: take the amount from a receipt and update every total in the app that depends on it. Reading the number is easy. The hard part is "every". An agent can fix the main total, leave one dependent field untouched, and still hand back something that looks finished.

That pattern is the real finding of Co-Design Bench, a paper posted to arXiv on 26 October 2026 by researchers from Alibaba's Token Hub group, Fudan, Zhejiang, CUHK, Shanghai Jiao Tong and NTU, and picked up widely since The Decoder covered it on 8 November. The best of the ten tested model endpoints, GPT-6 Astra, met on average 69.9% of the acceptance criteria per task. It fully satisfied only 26.4% of the tasks.

Both numbers come from the same runs. Read together, they say that the agent usually does most of the job and leaves at least one thing undone. According to the paper, 73.6% of tasks kept at least one criterion that was not fully met.

What was actually tested

The 273 tasks were curated from 37,463 deduplicated requests across 26,314 real user coding sessions. The researchers deliberately favoured requests with a history of trouble: users who rephrased the same request two or three times, kept following up, or rolled a change back. Experts then refined each request and attached hidden acceptance criteria, 2,331 in total, or roughly eight and a half per task.

There were two tracks. Localized edits, sometimes with an image, audio, video or document as input, and repository edits that need coordinated changes across a codebase. The second was much harder: the top endpoint fully solved 41.0% of the localized tasks and 17.9% of the repository ones. All ten endpoints did worse on repository edits that combined three or four types of change than on those with one or two.

Every model ran inside the same harness, Claude Code, with the same prompts and tools. An AI judge tested each finished app in a browser and checked the source, and a second AI verifier challenged every positive verdict.

The limits matter as much as the scores. Each request was fixed, so the agents could not ask a clarifying question or get a correction halfway through. The judges are models and can be wrong. And the tasks were selected because they had already caused problems. As one careful summary on DEV puts it, 26.4% should not be quoted as "agents fail three out of four everyday tasks".

Where the cost hides

This is our reading, not the paper's: the expensive part of working with coding agents is the last 30%, because it is invisible at a glance. A change that satisfies seven of eight requirements looks done in a demo. The missing eighth shows up later, usually in a total that no longer adds up or a screen nobody re-checked.

The benchmark could see that gap only because someone had written the criteria down in advance. That is the part companies tend to skip. A request like "add a discount field to the invoice form" carries five or six unstated expectations: where it appears, how it rounds, what happens to VAT, whether old invoices change, what the PDF shows. The person asking knows them. The agent does not, and neither does the reviewer unless they are written somewhere.

The tool around the model matters too

The paper also ran two models through three different harnesses. On repository edits, Qwen3.8-27B scored 0.560 in one harness and 0.749 in another, with the same tasks. The gap between two models changed by up to 16.2 percentage points depending on the harness, and on localized edits the harness ranking flipped completely between the two models.

For anyone choosing a tool, that is a practical warning. A public leaderboard compares models in one setup. What your team will use is a model inside a specific product, with its own way of reading files, running code and deciding when it is finished. The only comparison that tells you much is that combination on your own tasks.

What we would change in practice

If a team already uses coding agents, or is about to, the benchmark points to a few habits that cost little:

• Write the acceptance criteria before the request. Five to ten short, checkable lines per task. They help the agent, and they turn review from "looks fine" into a checklist.
• Split requests that touch several kinds of change. The drop on three-to-four-type repository edits is the clearest signal in the data. Two small requests are easier to verify than one large one.
• Let the agent ask. The benchmark forbade clarifying questions. Real work does not have to. Tools that stop and ask about an ambiguous requirement are worth more than their leaderboard score suggests.
• Check the running result, not the summary. The benchmark judge opened the app in a browser. A reviewer who reads only the agent's description of what it changed is checking the agent's confidence.
• Measure your own full-pass rate. Take twenty real requests from the last month, run them, and count how many were completely right without a human touch-up. That number, not the average, tells you how much review time to plan for.

None of this makes the agents less useful. Meeting 70% of the requirements on hard, real requests is a large amount of work done. It just means the job of the person in the loop shifts from writing the code to defining what "done" means and checking it.

If your team is starting with coding agents and wants to build these habits from the first week, that is part of what we cover in our AI training for technical teams.