Election season can always be a particularly volatile time for polling. Different polls reporting different results with different samples at different times can all make for confusion over where the race really stands. So many factors go into trying to “predict” a race (though these results are not really “predictions,” but rather just snapshots of what the public expresses during that moment in time). Polls can often have varying results that are more than just a few points apart from one another.
The best thing to do, like we said in our last post, is to pay attention to several polls – or at least to the average of many polls, such as provided on realclearpolitics.com. But it is also beneficial to take a closer look at polls individually since it is the individual polls that create confusing headlines. And with the current governor’s race in New Jersey, in particular, the media love focusing not on who will win – every poll has Christie up by double digits – but rather by how much. So as each poll has appeared in recent weeks on this race, headlines have been obsessed with reporting how big or small Christie’s margin has become.
The most recent results come from a Quinnipiac University Poll, which shows Christie running away with a 34-point lead, 64 percent to 30 percent for Buono. This is Christie’s widest margin – with likely voters, no less – from any major poll since the race heated up with the end of summer; until this poll, Christie’s lead has appeared to be tightening slightly into the 20s, as we reported earlier this month. We would expect narrowing: races often get closer in the last few months and weeks as the challenger gains greater recognition, as the opponents ramp up efforts to vie for the vote, and as the portion of the electorate who will actually vote (likely voters) solidify in their choices. So to see this most recent spike in Christie’s margin – much higher than any likely voter poll in the past month – comes as a bit of a “September surprise.”
We should also note that Kean University released a poll immediately after Quinnipiac with Christie up only 18 points. The issue with Kean’s poll, is that, much like recent polling by Stockton, they provide little transparency about their polling – no sample characteristics and few, if any, methodological details, nor do they show what their questionnaires look like. Polls that fail to be transparent are unfortunately impossible to assess.
What has changed? Anything? How did Quinnipiac come up with a 34-point lead when we had it at 20 only two weeks before? Did Christie benefit from yet another bump due to his leadership during and after the Seaside boardwalk fire? Could Buono’s salacious Tea Party retweet have really caused that much of a stir?
Actually, if we read between Quinnipiac’s numbers a bit, however, we find some possible explanations for this larger margin and a few good lessons in general that we should always keep in mind about election polling.
Telling the fence sitters to get off the fence
It may seem like everyone in New Jersey has an opinion on the governor’s race, but our most recent Rutgers-Eagleton Poll numbers show that 8 percent of likely voters still do not know where they stand. Eight percent may seem small, but that’s actually quite a significant number of New Jerseyans, reaching into the hundreds of thousands. Democrats in our polling are even more likely to express uncertainty; 11 percent are unsure; 8 percent of independents are as well. Even if likely to vote by our measure, we count them as uncertain at this particular time and do not award them to either major candidate.
A close look at Quinnipiac, on the other hand, shows that their results are actually based on both the typical vote question and a follow-up question that encourages undecideds to pick a side. In other words, Quinnipiac’s reported results include “leaners” – respondents who initially express uncertainty but are prompted to choose the candidate that they lean toward most. As a result, Quinnipiac reports many fewer “don’t knows.” Only 5 percent of all voters, 5 percent of Democrats, and 5 percent of independents are unsure in their poll. Moreover, most of those reluctant voters move to Christie when pushed. This certainly seems to have an impact on Democrats especially: 35 percent choose Christie in Quinnipiac’s poll, while we found only 28 percent for the governor. Interestingly, both polls put Democratic support of Buono at 60 percent. Quinnipiac also shows a slightly higher percentage of independents supporting the governor than we did.
Since Quinnipiac does not report their head-to-head numbers without leaners, we can’t be sure about how great of an impact these leaners really make. But the point here is that we should always know who is included within the numbers we are talking about. Knowing leaners are included in their reported head-to-head may explain a few percentage points of the large 34-point margin, compared to the smaller margins found by Rutgers-Eagleton and other polls in past weeks.
This also makes the point that it is crucial to know exactly what the question wording is, and exactly what, if any, follow-ups are asked. Polls that do not provide this information should be taken with extreme caution since you can’t know what they mean without this vital information.
Determining who will show up on Election Day
So who will vote on Election Day, November 5? The answer is, we really don’t know. We can put together some good estimates, which is what we do when we talk about “likely voters.” But that’s all they are – estimates. While likely voters are a mystery that pollsters often think they can crack, it’s tough to do so, and we can never be sure until Election Day. This is particularly true in special elections like the upcoming October 16 U.S. Senate race, where polling is also widely divergent.
As we said the other day, likely voters are the subset of registered voters who are most likely to vote in an election based on a variety of characteristics. These characteristics may include past voting behavior, interest and attention to the race, and a direct question about likelihood of voting in the election. But there is no set formula for how to take these characteristics – or even requiring the usage of these characteristics – to determine the perfect likely voter subsample.
Polls are often vague about how they determine likely voters – much like magicians, never revealing their secret process. They hint at the types of questions they use but not how they use them. Do they weight one likely voter screening question more than another? Do they all count equally? What and how many questions make up their “likely to vote” scale? What is their cut-off between those likely to vote and those not likely to vote? Some polls even derive likely voter subsamples from listed samples of registered voters, simply counting those registered voters who vote more frequently as their likely voter subsample. The possibilities for determining such a subset are seemingly endless.
So as we look at the newest results from Quinnipiac (and any other likely voter sample) we must keep this in mind. Every polling organization has its own unique likely voter formula, and based on what questions go into that formula and how some respondents who are more or less likely to vote for Christie or Buono answer those questions, likely voter numbers can and will differ.
Looking at the samples they report (again, Quinnipiac, Monmouth, and Rutgers-Eagleton have all provided transparent information about their recent samples; Kean and Stockton have not) we see that recent polls using likely voter models have noticeable differences in their weighted sample characteristics, particularly in terms of partisanship. In the past three likely voter polls that report demographic breakdowns – Monmouth from August, Rutgers-Eagleton from September, and the most recent Quinnipiac – Quinnipiac’s likely voter sample has by far the lowest percentage of Democrats (clearly mostly Buono supporters) by at least 5 percentage points, as well as the highest shares of independents/other (a majority of whom are consistently for Christie) by 5 points or more. We at the Rutgers-Eagleton Poll, on the other hand, have more Democrats than either Quinnipiac or Monmouth, are in the middle with independents, and somewhat at the low end with Republicans.
These differing sample breakdowns must be a result of how each poll is determining likely voters through its screening. Unfortunately, none of us provide a lot of detail on this, although we at Rutgers-Eagleton do report exactly what “likely voter” characteristic questions we ask and what the response percentages are, something others do not appear to do.
The nature of the sample has a huge effect on the final numbers and the size of Christie’s lead when you consider that nearly all Republicans support Christie, as do a large majority of independents. A sample with more Republicans and independents, will give Christie a larger lead than a sample with more Democrats. Above everything else, this is probably why Quinnipiac shows such a large margin favoring Christie because it’s rooted in who they think will actually vote.
Of course, none of us know who will show up, how many voters will vote, and whether one party will be more energized than the other. We can make some guesses, but in the end only time will tell if those guesses are on or off the mark.
It is taking a closer look at times like these that teaches us just how much of an art and not just a science polling really is. Paying attention to how each poll derives its numbers – whether through adding leaning likely voters into the mix or through how polls concoct their likely voter formulas – gives us a better understanding to the differences we see in the headlines. Is Christie really up by 34 points against Buono with only a little over a month to go until Election Day? Is Buono starting to close the gap? What about Booker/Lonegan – is Booker winning by 35 as we showed three weeks ago? Probably not quite. Is he winning by only 12 as Quinnipiac shows? Most likely not that either.
The answer, in the end, is as we always say here at Rutgers-Eagleton. Polls are fun, and we can learn a lot from what they tell us, but no one should ever get overly excited about any one poll. Follow the aggregators like www.realclearpolitics.com or others (for some commentary on that, click here). Then sit back and see what actually happens in the real world.