Creating code with AI can result in working code quickly, but as new features are added repeatedly, it can lead to inconsistencies with old processes or make the code more complex. To investigate such issues, a benchmark called "SlopCodeBench" was developed, which allows AI to incrementally add new specifications to its own generated code while measuring the maintenance of functionality and changes in code quality.
SlopCodeBench
https://www.scbench.ai/
SprocketLab/slop-code-bench: SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
https://github.com/SprocketLab/slop-code-bench
advanced-context-engineering-for-coding-agents/benchmarking-new-models-on-slop-code-bench.md at main · humanlayer/advanced-context-engineering-for-coding-agents
https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md
◆What is SlopCodeBench
SlopCodeBench measures the behavior and code quality of AI-generated programs as new specifications are incrementally added and repeatedly modified.
◆Mechanism of SlopCodeBench
・Iterative specification refinement checkpoints
The "Problems" section on the official website allows viewing 36 types of challenges, such as processing configuration files and evaluating digital circuits, and provides multiple checkpoints for each challenge.
For example, the "cfgpipe" challenge has six checkpoints, starting with a specification that asks the AI to create a program that parses a configuration file and outputs the result.
Since AI extends previous code with new requirements, the initial design choices it makes can influence subsequent modifications.
◆Results of Verifying AI Coding Agents
・It is easy to handle new features but difficult to maintain old ones
When evaluating separately, "Core" measures the ability to handle new features at each stage, while "Strict" determines whether all tests, including past features, are passed. The Core pass rate was 65.82 to 73.98 percent, while the Strict pass rate was only 12.24 to 16.33 percent, showing the difficulty of maintaining old functionality while continuing development.
The "GPT-6 Astra XHigh" which was strict and performed best passed 32 out of 196 checkpoints, recording 16.33%, while the next best "GPT-6 SOL Medium" passed 30, achieving 15.31%.
・Increasing the amount of reasoning does not necessarily improve the pass rate
"Opus 5.5" passed 28 tasks when the reasoning amount was Medium, but decreased to 25 when it was XHigh, and the estimated execution cost increased by about 4.22 times, from $193.47 (about 39,000 yen) to $817.04 (about 135,000 yen).
However, in Astra, XHigh performed better, and the number of executions for each condition was only once, so it is not certain that increasing the reasoning amount caused the performance degradation.
・Pass rate of the finally completed program
When focusing on the final stage of 36 tasks, the maximum number of tasks that passed all tests was only three, indicating that even AI that showed results along the way found it difficult to maintain functionality until completion.
◆Quality of generated code
SlopCodeBench examines not only whether the functionality works but also "structural degradation" where complex functions increase and "redundancy" where duplicated processing increases. In the October verification, the proportion of code lines that matched the problem patterns of Opus 5.5 was 77.75% for XHigh and 94.25% for Medium among the final code of the 28 common tasks.
This is not the actual proportion of bugs, but the proportion of code lines that matched the detection rules, showing that a higher pass rate in tests does not necessarily mean the code is more concise.
The results of SlopCodeBench indicate that the ability for AI to create new features and the ability to maintain existing features and code quality despite repeated modifications are separate capabilities.
◆Check the results on the official website
On the official website's "Leaderboard", you can check the pass rates and code quality indicators for each model.
The benchmark software itself can be freely obtained and executed from Github's official repository, but because there are many check items and it consumes a large amount of credits, it is necessary to pay attention to cost management.
◆Evaluation of SlopCodeBench
On the social news site Hacker News, there are discussions about SlopCodeBench, with some opinions stating that "defensive programming by LLM leads to complex code" and others posting evaluations that "it is an excellent benchmark and sometimes a lower reasoning setting is a wise choice."
In response to this, Dex Horthy, the author of the verification article, explained, "The pass rate for Fable was 33% when tested with only six tasks, whereas it was around 14-15% when all tasks were considered," and explained .
Using SlopCodeBench allows you to confirm not only the model's pass rate but also execution cost and code degradation, making it useful as an evaluation method when entrusting long-term development to AI.
