I have received a myriad of requests asking me to respond to this Seth Coulter Walls article at the Huffington Post on changes in the party ID composition of recent polls. Although the article itself is a good and thoughtful piece, some of the interpretations of it are not. So let's get a few things straight:
1. Polls that show changes in party ID composition are not "cooked", "rigged", or "biased".
Changes in party ID composition may occur for a variety of reasons,
including random chance, response bias, temporary changes in party
affiliation, and longer-term changes in party affiliation. It is
possible that, if the partisan ID composition of a poll shifts rapidly
from week to week (more so than is likely from random chance alone, and
more so than is precipitated from external events), this may
be an indictment of a pollster's methodology -- that the pollster is
having difficulty getting a good, random sample. But the notion that
credible pollsters like Gallup or SurveyUSA are deliberately rigging
their samples is patently ridiculous. Polling is a very competitive
industry, with relatively low barriers to entry; they would go out of
business in a hurry if they did this.
2. The decision a pollster faces is whether or not to weight its sample by party ID. In fact, the whole point is that pollsters like Gallup and SurveyUSA do not
weight their samples by party ID -- they just tally the results, and
let the chips fall where they may. So in some sense accusing them of
"cooking" their samples has it backward; what you're really arguing is
that they should weight their samples, presumably in a way that is more favorable to your preferred candidate.
However,
there is no one right answer as to whether to weight by party ID is the
superior choice. A couple of pollsters, notably Rasmussen, do weight
their samples by party ID. These pollsters tend to get more stable
results from week to week. However, more stable does not necessarily
mean more accurate over the long run, as these pollsters may miss real changes in party ID that occur over time.
Rasmussen,
which targets its party ID figures based on a rolling, three-month
sample of interviews with thousands and thousands of voters, has what I
consider to be a eminently reasonable approach. Even so, the true party
ID composition of the electorate is difficult and perhaps impossible to
ascertain. You can take a survey of party ID, but it is still a survey,
and therefore itself may be subject to issues like poor methodology and
response bias.
There might be some promise in targeting party ID
figures based on actual changes in voter registration -- although many
states have open (non-partisan) voter registration, and some of those
that do require registration by party do not report these figures. Some
state-level pollsters seem to flirt with this approach, although I do
not believe that any of the more prolific, national pollsters do.
3. Changes in party affiliation are tied up in the notion of the "convention bounce".
Successful party conventions usually produce some shift in the
self-reported partisan composition of surveys taken in the days and
weeks that follow them. This may be a form of response bias, if voters
of a particular partisan persuasion are fired up by their convention
and more likely to take a pollster's phone call. (It may particularly
be a form of response bias if the poll is taken during the convention itself,
as voters of a particular party may be more likely to be at home
watching their party's convention, and therefore more available to take
a phone call).
However, remember what the conventions are. They are the Republican National Convention and the Democratic National Convention -- not the McCain National Convention and the Obama
National Convention. They are big advertisements for the parties
themselves. And so, it is not unlikely that, if voters respond
favorably to a convention, they will respond favorably both to the
candidate and to the candidate's party.
In other words, while
some of these polls may have implausibly high numbers of self-reported
Republicans in their samples, this is more a symptom of the convention
bounce rather than a separate and distinct phenomenon. Moreover,
conventions sometimes do trigger longer-term shifts in party
identification. In 2004, for instance, the Republican convention appeared to cause a semi-permanent shift in party ID toward the Republicans, amounting to a couple of points. As I also argued recently, it is also plausible that unenthusiastic Republicans or Republican-leaning independents (which I call "shy Tories") were being undersampled prior to the Republican convention, until the convention renewed their enthusiasm.
The
fact that the Bush administration is in its last days, that much of the
dissatisfaction with the Republican Party had to do with George W.
Bush, and that this is an election featuring two non-incumbents, is a
salient factor here. It may be that, rather than trying to tie John
McCain to George W. Bush, the Democrats should be making more of an
effort to indict the Republican Party as a whole, of which John McCain
is indisputably a member.
4.
Responsible pollsters should always disclose the partisan ID
composition of their samples, and should notate in their press releases
when material shifts occur. Regardless of the above, there is
little doubt that the partisan ID composition of a poll is an extremely
important element in the proper interpretation of a poll. Pollsters
should always report the partisan ID composition of their sample, and
should qualify in their write-ups when substantial changes in party ID
have occurred since their last survey.
--Nate Silver