Large Reasoning Models: Thinking Capabilities and Performance Limitations

Investigating LRM Thinking Capabilities

• 💡 The study explores whether Large Reasoning Models (LRMs) genuinely think or if their performance is due to pre-training knowledge or data contamination.
• 🧠 Unlike standard Large Language Models (LLMs), LRMs generate detailed reasoning traces before providing a final answer.

Puzzle-Based Evaluation Methodology

• 🎯 Researchers used puzzle-based thinking problems like Tower of Hanoi, Checkers Jumping, River Crossing, and Blocks World to ensure no pre-training data contamination.
• 🔬 These puzzles allow for precise manipulation of compositional complexity by varying parameters like the number of disks or tokens.

Three Performance Regimes

• 📊 Low-complexity tasks: Standard LLMs surprisingly outperform LRMs, which use significantly more computational effort (tokens) due to overthinking.
• 📈 Medium-complexity tasks: LRMs demonstrate a clear advantage, with their additional thinking processes leading to better accuracy.
• ⚠️ High-complexity tasks: Both LLMs and LRMs experience a complete accuracy collapse, failing to solve problems effectively.

Reasoning Effort and Overthinking

• 📉 For low-complexity problems, LRMs exhibit overthinking, using excessive tokens and often exploring incorrect alternatives even after finding a correct path.
• 🧠 Counter-intuitively, as problem complexity increases beyond a certain point, LRMs refuse to use more tokens, leading to a decline in reasoning effort despite having an adequate budget.
• 🚫 In failed high-complexity cases, LRMs tend to fixate on early wrong answers, wasting their token budget without reaching correct solutions.

Algorithmic Understanding Limitations

• ❌ A critical finding is that providing the explicit algorithmic solution as input does not improve LRM performance for high-complexity problems.
• 🧩 This suggests LRMs struggle with generalizable algorithmic thinking and cannot consistently execute algorithms for specific problem instances.