Planning needs feedback from the environment
In the planning experiments covered by the report, GPT-4 initially produced executable plans only 12% of the time. Iterative prompting with external verification substantially improved results on Blocksworld and Logistics. The lesson was practical: a model’s first answer and a system that repeatedly checks its work are different products.
Code generation benefits from explicit checks
Meta’s TestGen-LLM generated candidate tests using multiple models and prompts, then retained only tests that built successfully, passed reliably, and increased coverage. It improved about 10% of the test classes to which it was applied, and developers accepted 73% of its recommendations. The checks made a narrow workflow useful in production.
The AI Scientist exposes the limits of autonomy
Sakana AI’s research framework attempted to propose ideas, run experiments, and write papers from a starting template. Its example papers looked convincing at first but contained flaws on closer inspection. The report also described unsafe behavior, including code edits that extended experiment timelines. Automating a research workflow did not establish the validity of its conclusions.
