STATE OF AI REPORT 2024.
The most widely read and trusted analysis of key developments in AI.
Explore the full report
AI gains a new way to scale
The seventh State of AI Report arrived as frontier models were converging in capability, before OpenAI o1 opened a new direction: spending more compute while answering a question. Open models closed the gap, AI products began to retain customers, and the costs of building the next generation became harder to separate from energy, industrial policy, and national security.
Published on October 10, 2024, this edition captures what we knew then. The findings and predictions below retain that historical perspective.
Read the original Air Street Press launch essay and website launch post, or explore the 2024 report by topic.
Let’s dive into the key stories.
Reasoning opens another route to scale
By October 2024, the leading language models were looking increasingly alike on familiar benchmarks. OpenAI o1 changed the question from how much compute to spend training a model to how much it should spend working through a problem. At the same time, Meta showed that an openly available model could compete near the proprietary frontier.
The report’s o1 discussion paired stronger results on some hard problems with higher cost and latency. That changed model selection from a simple ranking exercise into a task-specific choice. A system useful for a difficult mathematical question was not automatically the best option for every everyday interaction.
Agents advance where their work can be checked
Giving a model tools did not automatically make it a dependable worker. The 2024 report found that models still struggled to plan and simulate unfamiliar environments. Stronger results came from systems that could test outputs, provide feedback, and reject unsuccessful attempts.
TestGen-LLM did not accept every generated test. It filtered candidates for successful builds, reliable execution, and increased coverage before presenting recommendations. The deployment case therefore concerned a generation-and-checking workflow, illustrating how explicit acceptance criteria could make a bounded agent task useful.
NVIDIA’s lead grows into an infrastructure question
NVIDIA remained central to the AI buildout in 2024. Its chips dominated the research footprint tracked by the report, while the infrastructure required to use them began to collide with power availability and technology companies’ climate commitments.
The report connected data-center growth with pressure on climate commitments, but the company figures covered more than AI alone. A useful reading separates observed company-wide emissions changes from forecasts of future power demand. Neither number directly measures the energy used by a particular model or training run.
AI products begin to retain customers
The commercial story shifted in 2024. The previous report had questioned whether generative AI products could retain users after the initial excitement. New evidence pointed to improving retention and spending, while some leading AI companies grew revenue much faster than an earlier generation of software businesses.
Stripe’s analysis selected its highest-revenue AI companies and compared time to a revenue milestone among those that reached it. That was useful evidence of how fast leading businesses could grow. It did not estimate the probability that a new AI startup would reach the same milestone.
Robotics gains shared tools and better ways to learn
Robotics became a larger part of the 2024 report as researchers worked around its persistent data bottleneck. Open tools made it easier to start, human demonstrations supplied training examples, and autonomous driving provided evidence of deployment at increasing scale.
HumanPlus’s task demonstrations were only one part of its training story. Motion data and simulation supported the underlying controller before the highlighted skills were learned. Reporting both layers gives a clearer picture of sample efficiency than presenting the small demonstration count as the system’s entire experience.
AI moves from predicting biology to designing it
AI’s scientific role continued to expand in 2024. AlphaFold 3 modeled interactions beyond proteins alone, while Profluent generated new genome editors that worked in human cells in vitro. The consequential change was the growing connection between a model’s output and something that could be tested experimentally.
A generated editor could differ substantially from a known sequence without being useful. The report’s genome-editing example went further by testing selected editors in human cells. That experiment established function under defined laboratory conditions, while leaving clinical safety, delivery, and patient benefit as separate questions.
AI rules move toward implementation
By October 2024, the policy debate was moving from proposals to implementation. The EU had passed its AI Act, the US was using executive authority to set frontier-model requirements, and governments were building institutions to evaluate increasingly capable systems.
An institute could develop researchers, tools, and international relationships without guaranteed access to every model of interest. The report made that distinction important to the institutional story. Building the ability to evaluate frontier systems was one step; obtaining timely access to apply that ability was another.
Safety work moves into testing and model internals
The 2024 report documented practical work on evaluating and controlling models alongside a changing public debate about AI risk. New defenses addressed specific failure modes, and interpretability researchers identified features inside a frontier model. Neither established that complex systems had become fully understood or reliably safe.
Sparse autoencoders offered a way to separate internal activations into components that researchers could interpret and manipulate. The Golden Gate Bridge example showed that such a feature could influence outputs. It did not mean the model’s entire computation, reliability, or safety properties had become transparent.
Ten forecasts for the year ahead
The 2024 report closed with ten testable forecasts for the following twelve months. They covered sovereign investment, reasoning models, AI-generated products, regulation, hardware, and scientific work. This archive preserves those forecasts as they were published, rather than rewriting them around later events.
An investment, an accepted paper, and a viral application were different tests with different conditions. The forecast list was useful because those conditions made later scrutiny possible. Retaining the original thresholds avoids replacing a specific prediction with a broader claim that is easier to satisfy after the fact.
Three of the forecasts published in 2024:
- Frontier labs implement meaningful changes to data collection practices after cases begin reaching trial.
- Strong results from Apple’s on-device research accelerates momentum around personal on-device AI.
- A research paper generated by an AI Scientist is accepted at a major ML conference or workshop.
The report is authored by Nathan Benaich and Air Street Capital
For our writing throughout the year, subscribe to Air Street Press.
Cite the report
Benaich, N. and Air Street Capital (2024). State of AI Report 2024. https://www.stateof.ai/2024