How Accurate Is the State of AI Report?.
Every prediction we have made since 2018, graded in public: 59 graded calls, 31 hits, 8 partials, 20 misses. This page explains the numbers, the grading rules, and what the misses say about our judgment.
In brief
Across 59 graded predictions made between 2018 and 2024, the State of AI Report scored 31 hits, 8 partials, and 20 misses: a 53% strict hit rate, or 59% counting partials as half credit. The report is strongest on industry and research calls and weakest on geopolitics and deal-driven predictions. Ten predictions from the 2025 report remain open and will be graded in October 2026.
Most year-ahead commentary in AI is never checked. Ours is. Each annual State of AI Report closes with roughly ten falsifiable predictions for the next twelve months, and each following report grades them in public.
This page is the audit. The underlying data, every prediction with its verdict and justification, lives on the scorecard.
How do we grade predictions?
Every prediction is written to be checkable within about a year: a named event, a threshold, a deadline. The following year's report grades each one as a hit, a partial, or a miss, with a one-line justification. The original wording is never edited after publication, and verdicts are never quietly revised.
Grading involves judgment, and we do it ourselves, so readers should treat partials with appropriate suspicion. That is why every verdict ships with its justification: if you think we were generous, the evidence to disagree is on the page.
What is the overall hit rate?
As of July 2026, 59 predictions from the 2018 through 2024 reports have been graded: 31 hits, 8 partials, and 20 misses. Strictly, that is a 53% hit rate. Counting partials as half credit, 59%.
The honest benchmark is not a coin flip. Most of these calls are low base-rate events: a specific acquisition collapsing, a specific benchmark falling, a specific dollar threshold being crossed within twelve months. Predicting that nothing unusual happens would score well and say nothing. We prefer calls that are wrong in an informative way.
| Report year | Graded | Hits | Partials | Misses | Accuracy |
|---|---|---|---|---|---|
| 2018 | 8 | 3 | 1 | 4 | 44% |
| 2019 | 6 | 4 | 1 | 1 | 75% |
| 2020 | 8 | 5 | 1 | 2 | 69% |
| 2021 | 8 | 4 | 0 | 4 | 50% |
| 2022 | 9 | 5 | 1 | 3 | 61% |
| 2023 | 10 | 5 | 2 | 3 | 60% |
| 2024 | 10 | 5 | 2 | 3 | 60% |
| Total | 59 | 31 | 8 | 20 | 59% |
Where are we strongest and weakest?
The pattern across topics is consistent: we are better at predicting what the technology and its builders will do than what dealmakers and governments will do on a twelve-month clock.
Industry calls, on adoption, products, and company behavior, run at 75% accuracy. Research calls run at 63%. The weak spots are geopolitics and investment, both at 38%, and hardware at 44%. The failure mode is usually specificity about transactions: we predicted semiconductor consolidation that never closed, an ASML market cap that took years longer to approach, and acquisition waves that stayed rumors.
| Topic | Graded | Hits | Partials | Misses | Accuracy |
|---|---|---|---|---|---|
| Industry | 10 | 7 | 1 | 2 | 75% |
| Policy | 6 | 3 | 2 | 1 | 67% |
| Bio | 3 | 2 | 0 | 1 | 67% |
| Safety | 3 | 2 | 0 | 1 | 67% |
| Media | 3 | 2 | 0 | 1 | 67% |
| Research | 16 | 9 | 2 | 5 | 63% |
| Hardware | 8 | 3 | 1 | 4 | 44% |
| Geopolitics | 4 | 1 | 1 | 2 | 38% |
| Investment | 4 | 1 | 1 | 2 | 38% |
What do the misses have in common?
Timing, more than direction. In 2018 we predicted that access to Taiwanese and South Korean semiconductor firms would become an explicit part of the US-China trade war. Graded a miss in 2019; by 2022, chips were the center of the export-control regime. In 2023 we predicted a group would spend more than $1 billion training a single model. Graded a miss a year later, then overtaken by events. A one-year clock punishes calls that are early, and it should: a forecast that cannot be wrong on time is not a forecast.
The second failure mode is underweighting narrative gravity in capital markets. Our worst recent miss, that humanoid investment would trail off without product-market fit, is dissected in its own post-mortem.
The third is picking the right trend and the wrong protagonist. Our Apple on-device AI call is the cleanest example, and it has its own post-mortem too.
What is open for 2026?
Ten predictions from the 2025 report resolve in the State of AI Report 2026, published in October 2026. They cover agentic commerce, open-sourcing at the frontier, AI scientific discovery, data center politics, generative media, and a Chinese frontier leaderboard moment. Each one is open for a live crowd forecast on the scorecard.
When they are graded, this page's tables will be updated, whichever way the numbers move.
Frequently Asked Questions
How many predictions has the State of AI Report made?
69 public predictions across the 2018 to 2025 reports: 59 graded and 10 still open. Each report typically makes around ten predictions for the year ahead.
Who grades the predictions?
The report's authors, in public. Every verdict is published with a justification in the following year's report and on the scorecard, so anyone can audit or dispute a grade.
What does a partial mean?
The prediction was directionally right but failed a specific condition: the right event at the wrong size, the right trend through the wrong actor, or a threshold that was approached but not crossed. Partials count as half credit in our accuracy figures and are 8 of 59 graded calls.