In a series of three televised events starting tonight, an IBM supercomputer named Watson is set to compete against two human champions on the game show “Jeopardy!” If the computer wins—and in test matches that IBM has held, Watson has been dominating the humans—it will inspire comparisons to Deep Blue, the chess computer that shocked the world and prompted existential hand-wringing about the nature of human consciousness by beating Garry Kasparov in 1997.
But here’s the rub: “Jeopardy!”is actually a terrible way of proving that Watson is more intelligent than its opponents. A successful contest would certainly show that Watson can interpret complex sentences and recall esoteric information, and it would certainly show that IBM has designed a pretty smart computer. However, if the supercomputer triumphs, it will probably be for another reason entirely: because it can activate the buzzer most quickly.
This is how the “Jeopardy!” rules work: Whoever buzzes in first—using a clicking device usually compared to a large pen—gets the first chance at answering the question. The wrinkle, however, is that the contestants have to wait until Alex Trebek is completely finished reading the question before they are allowed to buzz in. Buzz too soon, and your buzzer is “locked out” for a quarter of a second, giving opponents the chance to jump in and answer before you. Contestants who wish to buzz in as fast as possible must either try to guess when the buzzers will activate (risking getting locked out if they are too early) or rely on their reflexes to buzz in when they see the lights (risking having reflexes that are too slow and allowing another player to buzz in first).
Mastery of this problem is critical to “Jeopardy!” success—possibly even more so than actual trivia knowledge. As Michael G. Dupree, who won “Jeopardy!”’s 1996 Tournament of Champions (TOC) and wrote How to Get on “Jeopardy!” and Win, put it in his treatise on the subject, “although each of the TOC competitors possessed an extremely broad range of knowledge, I came out on top because of my ability on the buzzer.” And Ken Jennings, who will be competing against Watson in the coming contest, has written on his website that, “‘Jeopardy!’ victory goes not to the biggest brain—it goes to the smoothest thumb. Timing on the tricky ‘Jeopardy!’ buzzer is often what separates the winner from the, well, non-winners, and the ‘Jeopardy!’ buzzer is a cruel mistress.”
That fact seems to have been borne out by Watson’s practice run. Of the 61 questions that can come up in a full game of “Jeopardy!” there is precisely one—the Final Jeopardy question—where we see how each of the contestants answers. On every other question, we only hear a second answer if the first one given is wrong. And in the recent test match between Watson, Jennings, and “Jeopardy!”champion Brad Rutter, none of the 15 questions were answered incorrectly by any of the players. In each case, the person who buzzed in first won the points associated with that clue. For all we know, all three players knew the answers to all the questions. Watson won that round, and it could easily have been because Watson was faster to the buzzer.
Indeed, when I called Watson’s creators to ask how the supercomputer controls its buzzer, they admitted that Watson does have a strong built-in advantage. According to David Shepler, who is IBM’s Challenge Program Manager for the Watson project, “The buzzer is enabled when the clue is done being read, when Alex Trebek gets to that last syllable, and the guy off stage pushes a button. That’s when people can buzz in, and at the same time a signal is sent to Watson saying the same thing—telling Watson that it can buzz in if it so desires.” This is akin to playing against an opponent with near-perfect reflexes. “When we built the demonstration system, the first incarnation of a fully functioning game-playing Watson, Watson was buzzing in electronically,” Shepler said, but they decided this was too unfair. IBM added an electromagnetic “hand” to Watson that will depress an actual buzzer which gives the humans a little more time, but not much. Asked if Watson still has an advantage under the new arrangement, Shepler told me that experienced players will sometimes get the jump on the supercomputer because they can anticipate the end of a clue, “but it's true that on average a machine has very good reflexes. The machine is probably going to beat a human at buzzing.” And the computer will never buzz in too early and get locked out.
So, while “Jeopardy!” clues provide a good test of Watson’s ability to decipher natural human language—and this ability really is quite amazing—the competition against the world’s best “Jeopardy!” players is probably a misleading gimmick. A better comparison with human minds might be an individual version of “Jeopardy!” where Watson answers as many questions correctly as it can, as quickly as it can, and then its score and time are compared to those of other trivia whizzes. And if Watson wins the televised match next week, don’t panic: It won’t mean that humanity’s superiority is in jeopardy.
Ezra Deutsch-Feldman is a web intern at The New Republic.