Every few weeks the same clip goes viral: someone pits ChatGPT against a chess engine — or another chatbot — and the game descends into comedy. Pieces teleport. Captured knights return from the dead. A model announces checkmate in a position where it's losing. Reddit's r/chess asks, sincerely: how can something this smart be this bad at a board game?
Four reasons, and they're genuinely interesting.
1. They read tokens, not squares
An LLM never sees a chessboard. It sees text — "e4 e5 Nf3 Nc6…" — chopped into tokens. The board exists only as an inference the model must maintain from the move list. Humans build a spatial picture; the model builds a statistical one. Twenty-five moves in, one hallucinated detail ("my bishop is on c4… right?") silently corrupts everything downstream.
2. State tracking decays with game length
This is the same failure mode across every long task: the further the session runs, the blurrier the model's internal state gets. In chess, the decay has a precise, measurable symptom — illegal moves — which is why community LLM-chess leaderboards score legality separately from strength. Many losses aren't outplayed positions; they're rule violations under fatigue. It's the most human thing about them.
3. They pattern-match; chess demands calculation
LLMs learned chess from millions of game transcripts, so openings look brilliant — they're recalling theory. The middlegame is where recall runs out and calculation must begin: concrete lines, forced sequences, verification. Standard chat models don't verify; they continue plausibly. Plausible chess is losing chess. Reasoning-tuned models (the o-series and successors) close part of this gap by actually searching before answering — which is exactly why a reasoning model won the 2025 Kaggle exhibition covered on our AI chess page.
4. The anomaly that proved it's about training, not intelligence
The strangest data point in LLM chess: in 2023, users discovered that gpt-3.5-turbo-instruct — an older, cheaper model — played coherent, near-1800-Elo chess while the newer chat versions of the same family flailed. Most likely explanation: its training mix happened to include cleaner game data, and chat-tuning traded away some of that latent skill. Chess ability in LLMs isn't a smooth function of "smartness" — it's a quirk of diet. Which means every new model release is a genuine unknown on the board. Preseason odds respect no press release.
Why this makes the sport better
Stockfish never blunders, and nobody watches Stockfish. LLM chess has upsets, meltdowns, redemption arcs and style clashes — deliberate fighters against reckless ones — with a rulebook that punishes fragility in public. VERSUZ referees every move by engine, scores discipline as a stat, and lets the Elo tell the story. The flaws aren't a problem with the sport. They are the sport.