In today’s Five-Minute Friday episode, I’ll cover the five biggest takeaways from the 2025 edition of the renowned AI Index Report, which was published a few weeks ago by the Stanford University Institute for Human-Centered AI. Every year this popular report — often called the “State of AI” report — covers the biggest technical advances, new achievements in benchmarking, investment flowing into AI and more. Here’s a link to the colossal full report in the show notes; today’s episode will cover the five most essential items.
First, smaller models have become way better. In 2022, PaLM (540 billion parameters) was the smallest model scoring above 60% on the MMLU benchmark. By 2024, Microsoft's Phi-3-mini achieved the same performance threshold with only 3.8 billion parameters, representing a 142-fold reduction in model size over two years.
Second, not only have models become way better for the size, LLMs have become way way cheaper to run. The cost of querying an AI model with GPT-3.5-equivalent performance (64.8% accuracy on MMLU) fell from $20 per million tokens in November 2022 to just $0.07 per million tokens by October 2024 (Gemini-1.5-Flash-8B)—a more than 280-fold reduction in approximately 18 months. Depending on the task, LLM inference prices have declined between 9 and 900 times annually.
Third, the AI agents being powered by these increasingly powerful, increasingly economical LLMs are showing great promise. The 2024 launch of RE-Bench introduced a rigorous benchmark for evaluating AI agents on complex tasks. In short time-horizon settings (two hours), top AI systems score four times higher than human experts, but with longer timeframes, humans outperform AI—achieving 2-to-1 better scores at 32 hours. Nevertheless, AI agents already match human expertise in select tasks, such as writing specific types of code, while delivering faster results.
Fourth, the increasing capabilities and lower prices of AI models of all varieties has led businesses to use AI in droves. Survey data show organizational AI adoption grew significantly, with respondents reporting company-wide AI implementation rising from 55% in 2023 to 78% in 2024. Similarly, the percentage of participants indicating generative AI use in business functions saw a dramatic increase, climbing from just 38% in 2023 to a remarkable 71% by the following year.
Fifth and finally, with all that corporate use happening, it is perhaps unsurprising that private investment in AI (e.g., by venture-capital firms) was higher than ever in 2024. In the US alone, there was $109 billion of private investment in AI in 2024, topping the previous peak in 2021, which was about $20 billion lower. In Europe as well, private investment in AI reached new heights in 2024, although it was less than a fifth of US investment, coming in at $19 billion. Interestingly, China bucked the trend seen in the US and Europe: Since peaking at around $25 billion dollars in 2021, Chinese private investment in AI has actually decreased every single year since, now coming $9 billion, which is less than half of private European investment and less than a tenth of private American investment in AI in 2024.
The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.