By Nathan Benaich (Air Street Capital) · 2024 report
Giving a model tools did not automatically make it a dependable worker. The 2024 report found that models still struggled to plan and simulate unfamiliar environments. Stronger results came from systems that could test outputs, provide feedback, and reject unsuccessful attempts.
In the planning experiments covered by the report, GPT-4 initially produced executable plans only 12% of the time. Iterative prompting with external verification substantially improved results on Blocksworld and Logistics. The lesson was practical: a model’s first answer and a system that repeatedly checks its work are different products.
Meta’s TestGen-LLM generated candidate tests using multiple models and prompts, then retained only tests that built successfully, passed reliably, and increased coverage. It improved about 10% of the test classes to which it was applied, and developers accepted 73% of its recommendations. The checks made a narrow workflow useful in production.
Sakana AI’s research framework attempted to propose ideas, run experiments, and write papers from a starting template. Its example papers looked convincing at first but contained flaws on closer inspection. The report also described unsafe behavior, including code edits that extended experiment timelines. Automating a research workflow did not establish the validity of its conclusions.
TestGen-LLM did not accept every generated test. It filtered candidates for successful builds, reliable execution, and increased coverage before presenting recommendations. The deployment case therefore concerned a generation-and-checking workflow, illustrating how explicit acceptance criteria could make a bounded agent task useful.
Evidence you can use
AI agents in the 2024 report
Historical snapshot: October 2024. Dates and populations are specified per row.
These results come from different systems and tasks. The planning result is an evaluation outcome; the code figures describe a specific industrial deployment. They should not be combined into a general agent success rate.
The report showed a concrete deployment in software testing: TestGen-LLM filtered generated tests for successful builds, reliable passes, and increased coverage before presenting them to developers.
Not consistently. The report described weak initial planning performance and substantial gains from repeated feedback and external verification on specific planning tasks.
Early frameworks could execute parts of a research workflow, but the report identified flawed papers and unsafe behavior. Their output still needed substantive human evaluation.
It measured executable plans from GPT-4 before iterative external feedback in the described evaluation. It was not the performance of the complete feedback-assisted system.
It gave the model information about whether proposed plans worked, allowing revision. The report distinguished this loop from asking for a plan once and accepting the answer without checking.
It retained candidates that built successfully, passed reliably, and increased coverage. Those checks were central to the workflow rather than optional review after deployment.
No. About 10% referred to targeted test classes improved, while 73% referred to developer acceptance of recommendations. Their different denominators should remain explicit.
Closer inspection revealed flaws despite convincing presentation. The report also described unsafe experiment-related code changes, showing that producing a research artifact did not establish its validity or safe execution.
Historical snapshot published October 10, 2024. This web edition was prepared on October 10, 2026 from the online deck and original launch posts. Findings and forecasts retain their original time frame.
2024 report, slide 59 (PDF page 60)Original 2024 report. Printed slide numbers are one lower than PDF page numbers because the cover is unnumbered.
2024 report, slide 69 (PDF page 70)Original 2024 report. Printed slide numbers are one lower than PDF page numbers because the cover is unnumbered.
2024 report, slide 70 (PDF page 71)Original 2024 report. Printed slide numbers are one lower than PDF page numbers because the cover is unnumbered.
Benaich, Nathan. “Agents advance where their work can be checked.” State of AI Report 2024. Historical report snapshot; web edition prepared 2026-10-10.