AI Comes Full Circle on the Chessboard

Many YouTube videos involve humans demonstrating the limitations of AI, usually by asking it silly questions. These videos are beloved, especially by teenagers, but my own teenage son recently showed me a fascinating example of the genre involving chess. It caught my attention because the history of modern AI is closely associated with the ancient board game.

Computer scientists have been building chess-playing robots—now called chess engines—since the late 1950s. Limitations in computing power needed to deal with the astronomical number of possible moves meant that these early systems performed poorly. But by the 1970s, they had improved to the point where they could beat skilled amateurs. As a kid, I recall my older brother, an avid player, struggling against a computer chess system built right into a plastic chess board with a simple LED display.

In 1997, the Deep Blue system, developed by computer scientists at Carnegie Mellon University and IBM, beat chess world champion Garry Kasparov. Today, a sense of inferiority to computers in certain realms is commonplace, but Kasparov’s defeat was a watershed moment. People began to realize the potentially boundless power of AI systems.

Deep Blue is considered to be a "supercomputer expert system" rather than artificial intelligence, since it could only play chess. There is only one set of rules in chess, and they never change. The rules were hard-coded into Deep Blue, and it was explicitly trained on 700,000 historical games played by grandmasters.

Modern chess engines work in essentially the same way. Stockfish, a free, open-source system that can beat any human player with ease, also has the rules of the game built in, and was trained on past matches. At any given point in the game, it "simply" evaluates a huge range of possible moves and counter-moves several steps into the future, then chooses the move most likely to bring victory. This is hardly a simple process: the number of possible chess games that could be played is a one followed by 120 zeros—a number much more than a google (one followed by one hundred zeros). But modern chess engines have solved this challenge.

Today's AI systems are different. They can find the best driving route to New York or the visual attributes that define the class of objects called "sofa" even though the best solution to these tasks is constantly changing—as are the range of tasks the systems are asked to perform. In contrast, chess engines have no capacity for recognizing objects or providing travel routes—and they never will. Whereas Deep Blue and other first-generation AIs were built as "symbolic" systems, where rules and logic are built in, ChatGPT and other modern AIs are "connectionist" in design. They learn from the world and are constantly adapting to it, much as the brain does.

Given its impressive and ever-growing capabilities, I would have thought that chess would be something ChatGPT is good at. For one thing, it has presumably ingested many, if not most, of the hundreds of thousands of books and websites devoted to chess as part of its training.

Moreover, ChatGPT has vastly more computing resources available to it. The system uses tens of thousands of networked computer chips, which each perform more than 10 trillion calculations per second. In contrast, the entirety of the Stockfish program contains less than 100 megabytes of data, and it can be run efficiently on a single desktop machine.

Yet ChatGPT-3 doesn't even comprehend the rules of chess. Demonstration matches by International Chess Master Levy Rozman (aka Gotham Chess) make this plain. In one of a series of his amusing videos from 2023, ChatGPT-3 confidently makes pawns and rooks jump over other pieces. It reminds me of playing chess against my son when he was a preschooler (but he has beaten me in every chess game we have played since he was eight).

In the videos, Rozman patiently tries to correct the AI and explains via chat why the moves were illegal, but to no avail. At the end of one game, the chatbot had magically accumulated five rooks—and it still managed to lose. Worse, the chatbot tried to tell Rozman that he was making illegal moves when this was not the case. Rozman got similarly hilarious results playing against Google's Bard AI—and when ChatGPT-3 played against itself. When Rozman pitted ChatGPT-3 against a different species of robot—Stockfish—the game descended into nonsense as the chatbot made illegal moves and still managed to lose in spectacular fashion.

Artificial Intelligence Essential Reads

Do You Have Bixonimania?

Can We Trust the Recommendations of AI?

On one hand, this is not surprising. ChatGPT-3 is bad at many things, especially those that require strict adherence to rules and logic. As sci-fi author Ted Chiang famously showed in a 2023 New Yorker piece, ChatGPT-3 fails even at simple arithmetic. It is designed to be adaptable, and to learn from examples on the fly, rather than to rigidly apply pre-programmed rules. If it has not encountered a given example during training, it fails.

Yet this may all soon change. In a recent video, Rozman played against ChatGPT-4, a newer version of the chatbot. It played well for more than 30 moves, using sophisticated attacks and defenses, before it started to go haywire. The chatbot made a few poor endgame moves and then resigned. But Rozman was clearly troubled by its performance. “Not only can ChatGPT play chess now,” Rozman said, “it nearly defeated me.”

Ironically, it may very well be Rozman—and his many followers in the chess world—who provided some of the examples and training that allowed ChatGPT-4 to improve on its predecessor. This process may well accelerate.

For now, humanity, as Rozman says, "still answers to only one robotic chess entity." Indeed, I suspect that, if granted the same computing resources as ChatGPT, Stockfish and other symbolic AI will always be superior at chess compared to connectionist AI systems.

But it is fascinating that chess—the archetypal AI use case that paved the way for the ChatGPT era—has so far been a stumbling block for the most advanced of models of today.

Special thanks to Wesley for inspiring this post!

Copyright © 2024 Daniel Graham. All rights are reserved, including those for text and data mining, AI training, and similar technologies. For reprint requests, email reprints@internetinyourhead.com.