AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

{ "@context": "https://schema.org", "@type": "NewsArticle", "headline": "AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test", "datePublished": "2026-11-08T10:50:59+01:00", "dateModified": "2026-11-08T10:51:53+01:00", "author": { "@type": "Person", "name": "Jonathan Kemper" }, "publisher": { "@type": "Organization", "name": "The Decoder" }, "description": "Alibaba's new coding-agent benchmark, Co-Design Bench, tests AI agents on real user requests that already tripped them up in practice. Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks. On older benchmarks, the same models score above 99 percent.", "articleBody": "Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even the best model fully completes only about one in four tasks, putting the near-perfect scores of older tests into perspective.\r\n\r\nThe team developed Co-Design Bench, a new test for coding agents. The agents must rework existing web apps based on user requests, sometimes using images, audio recordings, videos, or documents as input.\r\n\r\nAccording to the paper, GPT-6 Astra, the best of the ten models tested, meets about 70 percent of a task's requirements on average. But the result is fully correct in only 26.4 percent of tasks. In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.\r\n\r\n[video width=\"1280\" height=\"720\" mp4=\"https:\/\/the-decoder.com\/wp-content\/uploads\/2026\/11\/agent-edit-highlights.mp4\"][\/video]\r\nCoding benchmarks have grown too easy for agents\r\nThe authors justify the new test by pointing to the saturation of existing benchmarks. When models can run, check, and fix their own code, they hit scores above 99 percent on those tests, and differences between models can barely be measured.\r\n\r\n[caption id=\"attachment_64227\" align=\"aligncenter\" width=\"1800\"] On older benchmarks like WebCompass-edit, agentic coding models reach near-perfect scores, while Co-Design Bench leaves a gap of more than 35 percentage points. | Image: Wei et al.[\/caption]\r\n\r\nCo-Design Bench draws on real requests from actual coding sessions instead. The researchers favored cases where users had to rephrase their request multiple times or roll back changes, so the test contains mainly tasks that already gave AI agents trouble in practice. They range from small tweaks, like pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls.\r\n\r\n[caption id=\"attachment_64226\" align=\"aligncenter\" width=\"1800\"] In the multimodal track, agents need to extract values from receipts, tables, or videos and update all dependent views consistently. | Image: Wei et al.[\/caption]\r\n\r\nGrading is strict to match. One AI judge tests the finished app in a browser, then a second re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.\r\nMultiple change types in one task overwhelm the models\r\nBehind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max take the next spots, while Grok 4.6 lands in last place. On larger overhauls, the much smaller Qwen3.8-27B climbs to second, but the authors do not derive a general advantage of the model from that result.\r\n\r\nPerformance drops across all models as a task combines more types of changes, including layout, design, interaction, and content. But these intertwined requirements are the norm in real projects\r\n\r\n[caption id=\"attachment_64225\" align=\"alignnone\" width=\"1800\"] Every one of the ten models scores worse once a task demands three or four different types of changes. | Image: Wei et al.[\/caption]\r\n\r\nExtra compute does little to fix this weakness. GLM-5.2 burns through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse.\r\n\r\n[caption id=\"attachment_64224\" align=\"aligncenter\" width=\"1800\"] GPT-6 Astra leads both tracks but fully solves only 41 percent of multimodal tasks and just under 18 percent of repository tasks. | Image: Wei et al.[\/caption]\r\nThe harness shapes results as much as the model\r\nIn a separate test, the researchers ran the same models in different harnesses, the programs through which an agent reads, writes, and executes code. Qwen3.8-27B scored 75 or just 56 percent on larger overhauls depending on the environment. Which of two models came out ahead also depended on the environment.\r\n\r\nLeaderboards that list only the model therefore only tell part of the story. The authors recommend always reporting model and working environment together. For companies picking a coding agent, that means testing both components as a pair.\r\n\r\nThe new benchmark does have limits. It evaluates one fixed request per task; follow-up questions or mid-task corrections from users are not part of the test. The AI judges can also make mistakes.\r\n\r\nA study by the research organization METR already showed how far benchmark results and real-world usability can diverge. The actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified.\r\n\r\nFlaws in test environments can also let agents pass tasks without actually solving them. The effect of the working environment is not clear-cut, though. A recent study on coding agents found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.", "isAccessibleForFree": "False", "hasPart": { "@type": "WebPageElement", "isAccessibleForFree": "False", "cssSelector": ".paywall-active" } }