Last week we released a report from Langer Research Associates commissioned by the Eagleton Institute of Politics examining the reasons for our mis-estimates of the U.S. Senate race in October and the gubernatorial race in November 2013. In addition to examining question order priming effects, which were found to be the primary cause of the mis-estimates, Langer Research Associates examined some general operational aspects of the Rutgers-Eagleton Poll to assess the degree to which any of them influenced the estimates. None of these items were found to be causes of the mis-estimates, but Langer’s assessment provides useful guidance for the Poll.
This week we provide a brief summary of the operational portions of the report, along with our responses, which in some cases include changes in some of our processes going forward. We do this as part of our commitment to transparency and our educational mission.
Likely Voter Modeling
Issue: Pre-election polls have to estimate who will vote, which is done through likely voter modelling. This means identifying through a series of questions which respondents are likely to turn out to vote and which are not.
Summary of Langer Analysis: The Rutgers-Eagleton Poll employed a series of questions including self-reported likelihood of voting, awareness of the election date (for the Special Senate election), reported last time voting, following news of the election, and attention to debates. The calculation of likely voter was applied independently to the Senate and Governor elections. The Poll employed a scoring methodology, which assigned points to each response. The likely voter modeling was conceptually sound. However the cutoff point used greatly overestimated actual turnout. Likely voter models that overstate turnout include non-voters in their vote-preference estimates, which can compromise the accuracy of these estimates.
Likely voter modeling, however, was not the culprit in the discrepancies in the 2013 Rutgers-Eagleton estimates. Constructing three tighter likely voter models with turnout estimates as low as 32 percent in the Senate race and 38 percent in the gubernatorial contest made no substantive difference in vote-preference estimates.
Rutgers-Eagleton Poll Response: The LV screens were relatively loose in these reports, due to the size of the original sample, which simply did not allow tightening screens since that would result in too small samples. The Rutgers-Eagleton Poll staff did test varying screens and found few differences, as also determined by the Langer analysis. So the looser screens were chosen for reporting. One potential revision for future LV screens would be to use propensity scoring rather than a cutoff approach, which would allow all cases where the likelihood of voting was greater than zero to remain in the sample, weighted to reflect their relative propensity to vote. One attempt to improve the estimates that was employed was an adjustment to reflect the greater likelihood of Republicans turning out, beyond what was appearing in the LV screens. Such an approach is not industry standard and should not be employed in the future. As it turned out, this adjustment made no significant difference in the estimates. But even if it improves an estimate, we agree with the Langer report that this approach departs from best practices and should not be employed.
The Rutgers-Eagleton Poll is currently involved in a broad research project to reassess the weighting process that we use, and we anticipate the results of that project will begin to be used in polls beginning in the 2014-2015 academic year.
Issue: Non-response generally results in variation between the sample that is completed and target population norms which are based on U.S. Census data. One potential problem could be incorrect weighting of the sample prior to reporting the results. The Rutgers-Eagleton Poll is a random digit dial (RDD) survey, requiring that respondents be asked if they are registered voters in order to determine if they should be included in the sample for the purposes of asking election-related questions. For both the October and November 2013 polls, those who responded that they were not registered to vote were immediately terminated, meaning no additional questions were asked. Thus the samples are of registered voters only and must be weighted to norms for registered voters.
Summary of Langer Analysis: The Langer Report suggests that it is standard practice to weight to demographic variables for the full population, not to the registered voter population. To do so would require not terminating non-registered voters and at least asking them a series of demographic questions. Since the Rutgers-Eagleton Polls analyzed here were of registered voters only, this option was not available. The registered voter sample was weighted to the Census Bureau Current Population Survey (CPS) from March 2012 using age, gender, race and ethnicity as target demographics. Had the Poll sampled all adults, the all adult sample would have been weighted to the Census’s American Community Survey (ACS), the standard source for weighting population surveys. It is important to note that the CPS registered voter norms were from the 2012 election, making them 11 months old in 2013, which means they do not capture voter registration changes that might occur during the election season.
Rutgers-Eagleton Poll Response: The Rutgers-Eagleton Poll will continue to generally weight samples to age, gender, race, and ethnicity targets for the population from which the sample is drawn. We will make sure that we are using the most recent available norms at all times. We may do more “all adult” samples which will include subsamples of registered voters, and will allow the full sample to be weighted to current ACS norms. However, in doing so we will necessarily have smaller samples of registered voters since to increase the overall sample size would require additional financial resources not currently available.
Sampling, including cell phones
Issue: Surveys of NJ residents must be based on a probability sample of landline and cell phone respondents in New Jersey. In this survey, respondents were asked if they were registered voters and were terminated if they were not. A second area of investigation is the relative share of cell phone calls placed as part of the sample. NJ has one of the lowest cell phone-only penetration levels, but nonetheless a significant number of residents cannot be reached without dialing cell phones.
Summary of Langer Analysis: The sampling process appears appropriate for both cell phones and landlines. However, the termination of non-registered voters means the sample must be weighted to norms for registered voters, which are generally less current than adult population norms. As noted above, the Langer report suggests that non-registered respondents should be retained for the collection of demographic data before termination.
Given the increasing use of cell phones and in particular the increasing proportion of cell-phone-only households in the United States, the inclusion of a robust sample of cell phones is a necessary practice. The Rutgers-Eagleton Poll uses an overlapping dual frame sample that includes a sample of cell phone respondents regardless of whether or not they have landlines. Estimates from the federal National Health Interview Survey (NHIS) indicate that, as of December 2011, 16.5 percent of New Jersey adults used cell phones only, and an additional 24.7 percent relied “mostly” on cell phones. These proportions surely have increased since then. The October and November Rutgers-Eagleton polls, using an overlapping dual frame design, included 17 percent and 23 percent cell phone interviews (weighted to 22 and 26 percent, respectively), with 4 and 7 percent cell phone-only respondents (weighted to 6 and 8 percent, respectively), well below available NHIS estimates.
Rutgers-Eagleton Poll Response: The Rutgers-Eagleton Poll has increased the cell phone target to 30 percent of the sample and will monitor whether this provides a reasonable share of cell phone-only households. We are also collecting additional information from respondents including the number of adults in the household (for landlines) and the number of adults sharing a cell phone (for cell respondents). These data will help with improving weighting calculations.
Question Wording & Field Dates
Issue: Field dates and question wording are other potential causes of differences in survey estimates.
Summary of Langer Analysis: The review finds no indication that either field dates or wording influenced Senate or gubernatorial vote preference estimates in these surveys. Question wording, while different in each survey, in all cases was balanced and neutral.
Rutgers-Eagleton Poll Response: Question wording was slightly different between the Senate and gubernatorial head-to-head questions. The biggest difference was that voters who responded don’t know in the gubernatorial question in October we not asked about which way they leaned. They were asked this in November, and the Senate vote asked about leaners in October.