Artificial intelligence (AI) reasoning models have been touted as the next big leap towards achieving artificial general intelligence (AGI). However, recent research from Apple suggests that these models may not be as intelligent as previously thought. According to Apple’s findings, reasoning models like Meta’s Claude, OpenAI’s o3, and DeepSeek’s R1 do not actually reason and their accuracy diminishes with increasing task complexity.
The Rise of Reasoning Models
Reasoning models are a subset of large language models (LLMs) that allocate more computational resources to produce accurate responses. These models have led tech giants to claim they are nearing the creation of AGI, systems that can outsmart humans in most tasks. However, Apple’s study, published on June 7 on their Machine Learning Research site, challenges these claims by showing that reasoning models fail to maintain accuracy as task complexity increases.
Apple’s Study Findings
Apple researchers conducted extensive experiments with various puzzles to test reasoning models. They found that the accuracy of these models collapses beyond a certain level of complexity. The researchers noted, “Through extensive experimentation across diverse puzzles, we show that frontier LRMs face a complete accuracy collapse beyond certain complexities.” They also observed a counterintuitive scaling limit where the reasoning effort increases with complexity up to a point, then declines despite having adequate resources.
How Reasoning Models Work
Reasoning models attempt to enhance AI accuracy through a process called “chain-of-thought.” This involves tracing patterns in data using multi-step responses, mimicking human logic. This process allows chatbots to reevaluate their reasoning and tackle complex tasks more accurately. However, this statistical approach often leads to ‘hallucinations’ or erroneous responses.
The Hallucination Problem
OpenAI’s technical report highlights that reasoning models are prone to hallucinations more than their generic counterparts. For instance, OpenAI’s o3 and o4-mini models produced erroneous information 33% and 48% of the time, respectively, compared to the 16% error rate of the o1 model. OpenAI admits they don’t fully understand why this occurs, indicating the need for further research.
Peeking Inside the Black Box
Apple’s study involved setting generic and reasoning bots, including OpenAI’s o1 and o3 models, DeepSeek R1, and others, to solve classic puzzles like river crossing and The Tower of Hanoi. The complexity of these puzzles was adjusted to test the models’ capabilities. Interestingly, generic models outperformed reasoning models in low-complexity tasks without the extra computational costs. As complexity increased, reasoning models initially had an advantage but failed when the puzzles became highly complex.
Limitations and Criticisms
The study authors acknowledge that their research represents only a “narrow slice” of potential reasoning tasks. Apple’s position in the AI race has also come under scrutiny, with some suggesting the company is lagging behind its competitors. However, some AI researchers see the study as a necessary reality check on the exaggerated claims about AI’s potential.
Conclusion
Apple’s research indicates that current reasoning models rely heavily on pattern recognition rather than genuine logic, challenging the notion of imminent machine intelligence. While the study has its limitations, it highlights the need for a more realistic understanding of AI’s capabilities.
Note: This article is inspired by content from https://www.livescience.com/technology/artificial-intelligence/ai-reasoning-models-arent-as-smart-as-they-were-cracked-up-to-be-apple-study-claims . It has been rephrased for originality. Images are credited to the original source.