The Sequence Research #663: The Illusion of Thinking, Inside the Most Controversial AI Paper of Recent Weeks
Can LLMs models really reason?
I had different plans for this week’s research section but that Apple Research paper completely changed the schedule. The Illusion of Thinking is causing quite a bit of controversy in the AI community by challenging some of the core assumptions about LLMs: are they able to reason?
Recent progress in LLMs has introduced a new class of systems known as Large Reasoning Models (LRMs). These models explicitly generate intermediate thinking steps—such as Chain-of-Thought (CoT) reasoning and self-reflection—before providing an answer. While they outperform standard LLMs on some benchmarks, this paper, "The Illusion of Thinking" challenges prevailing assumptions about their reasoning abilities.
Current evaluation frameworks often rely on math and code benchmarks, many of which suffer from data contamination and do not assess the structure or quality of the reasoning process itself. To address these gaps, the authors introduce controllable puzzle environments that allow precise manipulation of problem complexity while maintaining logical consistency. These include Tower of Hanoi, River Crossing, Checker Jumping, and Blocks World.