Whatever happened to BABA is AI from 2024? [D]

A 2024 paper presented at the ICML conference by MIT and Virginia Tech researchers found that state-of-the-art multi-modal large language models like GPT-4o, Gemini-1.5-Pro, and Gemini-1.5-Flash "fail dramatically" when generalization requires manipulating and combining game rules. This research, potentially important for benchmarks like ARC-AGI-4, suggests current LLMs struggle with complex, rule-based puzzles, though some believe agentic swarms could solve them.

This report highlights a specific failure mode for state-of-the-art LLMs like GPT-4o and Gemini 1.5, unlike other benchmarks that focus on general capabilities.