At three o'clock on the afternoon of Wednesday, September 24, John McCain announced a bold move. He was suspending his campaign and rushing back to Washington to deal with the escalating financial crisis, even if it meant skipping a scheduled debate with Barack Obama that Friday night. The McCain campaign heralded the ploy as evidence that its man was a decisive leader who would put duty to his country above partisan politics.
But the move ran into instant p.r. trouble. Barely two hours after McCain's announcement, a snap poll appeared in the inboxes of reporters and political activists. A majority of Americans felt the debate should go ahead as planned, while just 10 percent thought it should be postponed. By 5:17 p.m., the poll had been posted on the popular liberal blog Talking Points Memo under the headline AMERICANS RESPOND TO MCCAIN STUNT. Before Republicans could even spin on the prime-time airwaves, the poll laid the foundation for an insta-consensus that McCain's move was a politically motivated stunt that wasn't fooling anyone.
The man who soured McCain's moment is a rotund former newspaper reporter and natty dresser named Jay Leve. He is the editor and founder of SurveyUSA, the polling firm that tested public reaction to McCain's gambit, and has conducted many dozens of other polls that have informed and infuriated political watchers this year. Although Leve has trademarked the phrase "America's Pollster," he is just one among a fast-growing and fractious cadre of American pollsters spitting out numbers faster than Sarah Palin drops non sequiturs. On a typical day this fall, RealClearPolitics, a hub of news, opinion, and polling obsessively trafficked by political junkies, might feature 30 state and national presidential surveys. At least five different firms now conduct daily "tracking" polls to detect the slightest shifts of public opinion about the race. "We've seen a huge increase" in political polling, says Nancy Mathiowetz, a former president of the American Association for Public Opinion Research (AAPOR), the polling community's establishment body. In the 1960s, perhaps half a dozen firms conducted presidential surveys. Today, there are dozens.
Leve and his ilk are proliferating because an unprecedented demand exists for the information they peddle. An anxious, politically savvy public has developed a compulsive need to know precisely where the presidential contest stands at any given moment. The profusion of poll numbers in turn fuels the public's hunger for more definitive, or more reassuring, polls--a cycle made all the more relentless by a panoply of websites, such as RealClearPolitics and Talking Points Memo, that post every last number almost in real time.
This new, frenetic age of polling has not necessarily led to more empirical certainty. The very instantaneousness of polls like Leve's threatens to shape perceptions as much as record them. And the deluge of polling data has just given partisans another opportunity to cherry-pick facts and impugn their rivals. In this besieged environment, even pollsters themselves fight bitterly over the best way to measure public opinion and whether the likes of Jay Leve have it exactly right--or very, very wrong.
Less than an hour after Leve released his snap McCain poll, Mark Blumenthal, a longtime Democratic pollster, was already attacking it. In a blog item posted at 5:53 p.m at Pollster.com, one of a handful of new websites dedicated to monitoring the booming polling biz, Blumenthal noted that many of the 1,000 adults contacted for the poll surely hadn't even heard the news of McCain's campaign suspension. (Leve's questions didn't mention McCain's gambit, asking only if the debate should continue in light of the economic crisis.) "It is an extreme stretch to treat a bunch of calls made over the span of an hour or two on a Wednesday afternoon as a representative snapshot of what 'Americans' are thinking," Blumenthal wrote.
Leve didn't appreciate the criticism. By 8 p.m., he had posted a comment under Blumenthal's item. "SurveyUSA's work in response to fast-breaking news is nothing short of remarkable," he wrote. "[I]f you are presented with data from a competing research firm that you feel does a better job of capturing this extraordinary moment in time, post that data and please delete ours." But Leve's salvo didn't address the question of the accuracy of his methodology--especially when it comes to his controversial brand of instantaneous polling.
On a recent September afternoon, with the Dow diving and McCain touting the economy's "strong" fundamentals, I drove about 20 miles west of Manhattan through the suburban sprawl of Verona, New Jersey, to visit Leve. SurveyUSA's offices are located in a small office park surrounded by trees. One level down the stairs, a receptionist buzzed me through the door and summoned Leve. The office, which was divided into a set of cubicles, was strangely empty.
Leve, 52, emerged from a back office with an extended hand and a smile. He wore a crisp white shirt and a matching orange necktie and suspenders. For added effect, his glasses had light orange frames, which matched the orangey stubble peppering his head and chin. Leve led me into his personal office. On one wall hung SurveyUSA's red and yellow logo, illuminated by a stage light, with a TV camera at the ready in case Leve was needed for a quick interview. Leve sat at his tidy desk and clicked open a computer screen to what resembled a train departures schedule. "There are four jobs running right now," he said softly, running his finger down the screen. "There's a poll in the state of Georgia--a Senate race down there. There's a poll being run in New Mexico for the presidential race. There's a poll in a congressional district in Pennsylvania." The fourth was a non-political marketingresearch poll in the Seattle area.
All these polls were being conducted in a bedroom-sized chamber just outside Leve's door called "the vault," in recognition of its actual use back when a rare-coins dealer owned the space. Leve led me inside, and pointed to a corner. "We even kept one of his safes," he said with a smile. Leve doesn't use the old steel safe, but the vault is still an apt name because it currently guards the workhorses of Leve's business: a set of black IBM calling machines, each about the size of a stereo tuner and stacked horizontally in a pair of large metal cabinets. Each machine is capable of having as many as 288 phone lines plugged into its back, creating a messy tangle of multicolored wires running from the machines up into the ceiling. On a busy day, Leve explained, his machines might place a few hundred thousand calls for 30 different polls. (For this election, he is polling in 28 states.) Since Leve began conducting surveys in 1992, his machines have completed 24 million interviews. "If we were radiologists, you would want us to read your x-ray because we see so much data every day," he said.
While most pollsters employ real people--sitting in call centers, wearing headsets--to gather data for them, Leve relies on these machines. His innovation is to get news anchors from local television affiliates in the areas he's sampling to record scripts for him. A trusted anchor's voice conveys that the call is "legitimate, authentic, civic-minded, and not a scam," Leve says, and people are less likely to hang up on the call. (In return for their anchors' services, the affiliates get to make use of Leve's findings.) A SurveyUSA poll is like an airline's automated customer-assistance system--press one if you support John McCain, press two for Barack Obama--except that you receive the call instead of placing it. With the raw results in hand, Leve will make some technical adjustments and write an analysis, which he will send to one of the more than 50 media outlets that commission his work and feature it in their print and television news stories. This sort of computerized polling is controversial--but also increasingly popular, thanks to its lightning speed and low cost. "Gallup might charge $10,000 and take four to five days," Leve boasts. "We can do that in one night, for maybe a thousand dollars."
This insight has made Leve a very successful pollster. But the St. Louis native originally set out to become a reporter, not a pollster. After getting a journalism degree from Northwestern, he even worked briefly at The Miami Herald before he was lured into a short-lived experiment with transmitting news via telephone. Leve soon made a career around interactive data, including designing more customer-friendly ATMs for Citibank. Three years after the 1987 market crash cost him his job, he started his own polling firm. At the time, automated polling was a relatively new technology, and Leve became one of its early adopters. The site of his Verona office complex was chosen for its thenunusual access to the fiber optic wiring that would support his huge volume of phone calls.
The U.S. polling industry has long been dominated by a staid old guard, one that did not welcome Leve to its ranks with open arms. Campaign polling first emerged in the 1930s, led by Literary Digest, which mailed millions of postcards to Americans asking how they intended to vote. The pollster establishment was a clubby lot, holding private gatherings where they sang and drank into the late hours. But they also prided themselves on their commitment to the idea of a social science that used slow and laborious techniques. Even today, major polling firms like Gallup can spend days collecting and refining their data, which is why the polling establishment still looks askance at automated calling. One major textbook in the field brands robo-calling as Computerized Response Automated Polls--or CRAP. Leve says an academic once called SurveyUSA "the McDonald's of polling." When Leve turns up at aapor's annual conferences, his presentations in defense of his method are met with eye- rolling. "Some of the folks at AAPOR can't stand him," says Mark Blumenthal, who says he generally admires Leve's work.
Leve, for his part, can be withering about the establishment that rejects him. He bridles at the commonly used term "robo-calling" as a label for what he does. "It could not be a more offensive term," he says. "It literally is like using the N-word." He prefers the phrase "recorded-voice polling." He can also give as good as he gets: In one 2005 interview, for instance, Leve recounted a 1998 call from one of those snooty old polling firms asking his opinion of then- New York Jets head coach Bill Parcells. "Why would an interviewer want to know if I thought Bill Parcells was honest?" Leve wondered. It turned out that the operator was botching the name of New Jersey Representative Bill Pascrell, then running for reelection in Leve's district. "How many times a day do you think something like that happens with headset operators?" Leve asked. "How many different ways can you think of for an eight-dollar-an-hour employee doing monotonous work to make a mistake? Does it matter how many Ph.D.s worked to draw the sample for that survey? ... It doesn't. The data was worthless."
But attacking Leve is a fruitless exercise these days. Whereas he may once have been ahead of the pack in the early '90s, now he is in the thick of the pack. A slew of new pollsters have emerged in recent years that follow Leve's example of defying the establishment with untrained, instantaneous, and methodologically unorthodox work. And, this election cycle, they make the original debate over recorded-voice calling seem quaint.
SHOCK POLL--blared a Drudge Report headline on December 26, 2007, just one week before the Iowa caucus. At a time when most pollsters were showing a dead heat in Iowa, this new survey found Hillary Clinton with a 15-point lead over Barack Obama. But the only shock, as it turned out, was that someone could have gotten it so wrong: Obama would beat Clinton in Iowa by eight points. The offender was the New Hampshire-based American Research Group. ARG is a black sheep of the polling world; I repeatedly heard it singled out for scorn by other pollsters. They complain that it releases little information about its sponsors and its methodology--for a time, there was even confusion about whether ARG relied on automated surveys or human operators. "ARG is a mystery," Leve says. "They release almost nothing about what they do. It's possible they don't even make phone calls." ("There's plenty of disclosure," says ARG's Dick Bennett. "I guess people are into trashing.")
It's not only ARG. These days, upstarts with authoritative sounding names like Insider Advantage and Strategic Vision crank out polls that many of the old hands consider suspect. The buttondown polling mandarins like those based in AAPOR's Lenexa, Kansas, offices have lost their grip. Nancy Mathiowetz, the former president of aapor, laments that many new polling firms nowadays fail to meet the association's formal code of ethics--specifically by refusing to release the exact wording of their polls, which can easily skew results, and the sponsors behind the surveys.
The glut of new polls--and vast spectrum of quality--has created a Darwinian environment in which pollsters and watchdogs attack one another with nerdy ferocity. (One pollster described a firm he considers disreputable to me as "a street gang with a calculator.") Mark Blumenthal and his colleagues at Pollster. com routinely flag suspect polls, calling out their authors when they don't disclose crucial information like the wording of their questions and their demographic weighting. Before the Iowa caucus, for instance, Blumenthal challenged pollsters to explain how they were screening for "likely voters" in that unusual contest; five pollsters refused to respond and others grudgingly provided incomplete answers--a fact Blumenthal publicized in an angry New York Times op-ed column.
Another influential arbiter is Nate Silver, a 30-year-old baseball statistician who has made a name for himself and his blog, FiveThirtyEight.com, by transferring his formidable number-crunching talents to political polling. On October 2, Silver (who also blogs for The New Republic) wrote a harsh online critique of RealClearPolitics. RCP's signature innovation is its practice of averaging campaign polls into one handy, pseudo-scientific figure which pundits from Karl Rove to Chris Matthews cite as gospel. Silver was outraged to find that RCP had recently begun including a conservative-tilting poll in its averages, while excluding another poll that has typically favored Obama. "Their partisan leaning is infused into their numbers," Silver wrote, accusing the site's editors of "cherry-picking those results that [are] to its liking, then coming up with post-facto rationalizations to justify its decisions." After a "contentious" call from an RCP editor, Silver withdrew his charge of bias, but still grouses that the site has no clear set of standards to explain its decisions. (Silver has also tangled with ARG, challenging the firm to an accuracy contest with a prize of $1,000 per state. ARG's Bennett recently fired back in New York magazine by likening Silver's work to palm-reading.)
In fact, almost anyone with an Internet connection and an interest in cross-tabs can become an ombudsman--much like Alan Abramowitz, a political science professor at Emory University. After the longtime Democratic pollster Celinda Lake co-produced national polling numbers showing surprising weakness for Obama earlier this month, for instance, Abramowitz sent Lake a tart e-mail, blind cc'd to several other recipients, demanding to see her raw data. Unappeased, he followed up a few days later: "Celinda--do you believe your own poll?" During this campaign, Abramowitz has badgered several other major pollsters this way, all for the benefit of the fellow academics and journalists he copies on his e- mails.
Sometimes it seems that pollsters spend nearly as much time arguing with their critics as they do actually gathering data. As a result, pollsters face the same fate as other traditional voices of political authority--not least the mainstream media. The more they bash one another in the public eye, the less the public trusts the objectivity of their work.
'Please be advised that you have been placed on our list of phony pollsters, " read an e-mail Jay Leve received earlier this month. "These are Republican controlled pollsters who put out biased, slanted polls. Our list is sent to over 1,650,000 subscribers and media outlets. Congratulations!" The e-mail was sent under a clearly bogus name from a non-existent domain-- obviously just the minor harassment of an aggrieved Obama supporter. But it is typical of the dozens of angry e-mails Leve gets every week, from both the right and the left. "Why is your garbage data so different from everyone elses [sic] putting McCain ahead of Obama???" reads another stink bomb, sent just two days later.
Partisans on the Web have created an unprecedented demand for more empirical data about the presidential campaign. And yet, when that data doesn't comport with their beliefs, they are quick to attack it. This attitude extends not just to the data itself but also to the collection of it. Leve says he was bombarded with angry mail when he stopped polling Hillary Clinton as a possible Obama running mate. Likewise, just after McCain named Sarah Palin as his running mate, the pollster Scott Rasmussen tested voters' head-to-head preferences between Palin and Hillary Clinton. The survey showed voters preferred Clinton, 52-41, and Rasmussen got "tons of mail" from Palin supporters who accused him of conducting the poll prior to her Republican convention speech in a sinister effort to measure her standing before she could strut her stuff onstage. To be a pollster is to expect this sort of abuse. "I've had people call me at home," sighs the longtime pollster Andrew Kohut.
Indeed it seems that everyone is a polling expert nowadays. "Don't trust Zogby," declared one online commenter back in August. This shot at a survey conducted by John Zogby, a marquee name in polling, didn't appear on, say, the Huffington Post or the website of National Review, but on Wild West Sports--a crude message board whose users post opinions on college football, pro basketball, and, it seems, the merits of polling professionals.
It's actually not irrational for partisans to get so exercised over polls. More than providing a cheap thrill, poll numbers drive conventional wisdom and fund-raising, which, in a Heisenberg Principle effect, can become a campaign's reality. "We seem to have become a nation of political junkies, and clearly the donor class has access to and pays attention to these public polls," says the veteran Democratic strategist Jim Jordan. "And they exert pressure and complain and worry and prematurely celebrate based on these things. The campaign decision-makers simply have to tune it all out."
But that's more easily said than done. (Jordan himself called me back a day later raging at Rasmussen's new GOP-friendly findings in two Senate races.) Clinton's sustained lead in national polling last fall had Obama backers in a state of high alarm, even though his campaign staff knew that the race would depend on his showing in Iowa, where he was always poised for a win. "The abundance of polling--and the daily tracking polls in particular--have been the worst thing to happen to political journalism in twenty years," grouses Obama communications director Dan Pfeiffer.
Similarly, the Obama team fended off another wave of Democratic panic after Palin gave McCain a late summer polling surge that Obama aides privately believed to be temporary. Their irritation was evident in a September 8 meeting between Obama campaign manager David Plouffe and campaign reporters. Plouffe complained of press "hyperventilating" about recent polls, and, when asked by a Washington Post reporter about a poll his paper conducted showing a huge swing of white female voters to McCain, Plouffe tersely snapped, "Your poll is wrong. " Likewise, when the McCain team didn't like a September Post poll that caused a stir by showing McCain nine points behind Obama, they convened a press call with their house pollster, Bill McInturff, to undermine the poll's findings.
In short, no one trusts anyone else's polls anymore. So it was perhaps fitting when a new national tracking poll joined the already-crowded scrum of such surveys this summer. This poll was sponsored not by a cable network or national newspaper, as most are--but rather by that mother ship of liberal blogs, Daily Kos. The poll, conducted by the Maryland-based firm Research 2000, purports to be just like any other: "It's not a liberal poll or a conservative poll, it's a poll," the website announced last month. As chance would have it, however, the Kos poll makes very optimistic assumptions about African American and Hispanic voter turnout. Thus, it has routinely showed Barack Obama further ahead of John McCain than most other surveys. The moral of the story seems to be that, if existing polls don't reflect the world as you would like to see it, you can always just commission your own.
Many pollsters insist that the way to restore public confidence in their profession is to re-impose methodological standards, which treat polling as a social science rather than an amateur's pursuit. But the problem is that, despite the meticulous nature of Gallup and Pew--and the quicker, less orthodox approach of Leve and some other newcomers to the field-the establishment firms don't necessarily have all the answers. Jay Leve, as it turns out, actually has a very strong track record. In March, Nate Silver ranked SurveyUSA as the most accurate of 18 major national firms, ahead of more venerable outfits like Gallup and CNN's pollster, Opinion Research.
Leve's winning track record may be the result of the technology he uses; automated polling allows him to tap into a larger pool of voters at a faster rate than live human polling--offering an instant snapshot rather than results blurred by time, as with a slow camera shutter. But Leve's technique is subject to some of the same criticisms as traditional polling. For example, some phone surveys are missing young voters with cell phones but not land lines. Overall, response rates have been declining for years. And, this season, race and gender have added tricky new variables. In short, Leve's success is hard to explain-- affirming a postmodern sense that methodology no longer ensures accuracy more than instinct and dumb luck.
The irony is that this perception, one that critics use to deride polling, is now widespread even among pollsters themselves. Last January, after all the big pollsters failed to predict Clinton's stunning New Hampshire primary victory, aapor asked them to hand over their raw data for evaluation. Leve complied immediately but says others dragged their feet. "That report has never come out," he says with a shake of his head. Mathiowetz, the former aapor president, says the report is coming soon. But, she says, a wider lack of transparency is a sign of changing times. It wasn't just the Chicago Tribune that blew the 1948 Truman-Dewey presidential election call--it was the entire polling establishment. Afterward, a panel of public opinion professionals studied what went wrong. Almost every major polling firm cooperated and submitted data about their research. The thought of this lost era clearly moves Mathiowetz: "What a lovely ..." she says, trailing off. "It just kind of brings tears to my eyes."
Leve has none of this sentimentality for his own industry. In fact, he can be downright dismissive of our national obsession with polling. When a surprising number appears in the news, he says, he often fields calls from his excitable mother. "I pick up the phone and she says, 'Hey, I just heard-- McCain's now in front!' I say, 'Mom, ignore it!' I often tell people to pay no attention whatsoever. There are an awful lot of numbers coming out that mean absolutely nothing."
Michael Crowley is a senior editor at The New Republic.