Summer in academic polling centers like ours is often relatively quiet, as we are mostly doing our planning for the next year. We will be back in the field with new polling in late August. In the meantime, we look today at our most recent poll, at the beginning of June, where we re-asked a question on the NYPD surveillance of Muslim groups in New Jersey. But this time we embedded an experiment in the question, so we could see if it is Muslim groups in particular that trigger the significant support we found for the NYPD program.
In March, the Rutgers-Eagleton Poll asked New Jerseyans their opinion on the then-breaking news story regarding the New York City Police Department’s (NYPD) secret surveillance of certain Muslim groups in New Jersey. Just under half (47 percent) agreed that the monitoring was necessary to protect the country, and three-quarters of New Jerseyans said the secret monitoring of groups has been at least “somewhat effective” in protecting the country from terrorist threats. For the complete results, see our March 30, 2012 press release.
Always interested in the academic side of survey research, we decided to ask the question again in our most recent poll (May 31 – June 4, 2012), but this time, we included a question wording experiment to see if New Jerseyans are just as accepting with the NYPD using secret surveillance on other kinds of groups as they were on Muslims. Our theory here is that when we asked about secret surveillance in the context of “Muslim groups” New Jerseyans may have been supportive since they could be linking the specific target group – Muslims – to stereotypes they may hold that generate a sense of threat or worry. Therefore, we tested the same question with different target groups on our latest poll to see if attitudes change based on who is being monitored.
To do this experiment, we divided our sample into four random subsamples, assigning each respondent to one of four versions of our original question asking if the secret monitoring was a “violation of civil liberties” or a “necessary action.” The four targets of the surveillance named were “Muslim groups,” “activist groups,” “student groups,” and a baseline of “groups” in general with no specific reference. Each respondent got only one version of the question. In reality, it was reported that the NYPD had been secretly monitoring all of these types of groups – though their surveillance of Muslims made the most headlines.
A quick note on experiments: by randomly assigning respondents to one of the four versions of the question, we ensure that each group of respondents is generally representative of the whole population of respondents, allowing us to make comparisons between them. Also, because this is an experiment, we are interested in the differences across the groups, and we are not trying to generalize to the population. In other words, we want to know how people process these different groups, not what public opinion is on the issue. As a result, these data are reported as raw data – that is, not weighted to represent a population, and thus the overall percentages cannot be directly compared to our March poll.
We see from the frequencies for each version of the question, that respondents in the “Muslim groups” version and the “generic” groups version are less likely to say that monitoring these groups is a “violation of civil liberties” (23 percent and 24 percent, respectively) and more likely to say it is a “necessary action” (both at 63 percent) than respondents in the other two versions. Those given the “Muslim groups” version are the least likely of all – but by less than a percentage point – to say the monitoring violates civil liberties. Two-thirds of respondents said secret surveillance is necessary when the monitoring is put in the context of “Muslim groups” or “groups” in general. While those respondents given the “activist groups” version differ only slightly – 27 percent believe it is a violation versus 61 percent who think it is necessary – those asked about “student groups” give a somewhat different response, being more likely to believe secret monitoring is a violation (32 percent) and less likely to believe it is necessary (57 percent). (Click on image to see full size).
What does this tell us? Well, first of all, using the generic “groups” wording without naming the group may well have simply triggered the same thing as using the word “Muslim” to modify the “groups” term. Why? Well our question is specific about the NYPD surveillance, and the issue received a great deal of press coverage, nearly all of which was about Muslim groups. By not providing a modifier, we probably simply triggered the Muslim association just as much as we did by using the word itself. Also interestingly, respondents are more likely to think monitoring “student groups” is a violation of civil liberties than monitoring “Muslim” groups. But even so, a clear majority sees little problem there.
When we break the results down by subgroup, the partisan division between Democrats and Republicans narrows in the “student groups” version. While only 7 percent of Republicans believe secretly monitoring “Muslim groups” is a violation of civil liberties and 13 percent of Republicans think it is a violation in the generic “groups” version, for “student groups” this increases to 20 percent, nearly 3 times as many as for Muslim groups. In comparison, Democrats show virtually no differences between most of the groups: 32 percent asked about “groups” say the monitoring violates civil liberties, compared to 34 percent for “Muslim Groups”, and 38 percent for “student groups”. Most of the movement we see overall comes from the Republicans who express different attitudes depending on who is being asked about. However, there is one exception: “activist groups.” Democrats are notably outraged when the question uses this wording with almost half (45 percent) believing monitoring is a violation. And they are significantly more concerned than they are about “Muslim groups”. Only 10 percent of Republicans share this opinion for “activist groups” – not surprising, since activists are stereotypically perceived as more liberal and thus more likely to be Democrats. Overall, looking within partisanship shows an interesting pattern among the different question versions, and the trend of respondents being less accepting of surveillance in the “student groups” version can be seen across other various demographic groups.
The differences, though, are relatively small in the full sample – so are they statistically significant? That is, are they real differences or merely likely to have happened due to chance? An analysis of variance (ANOVA) allows us to test this. When we do, we find that the generic “groups” version and the “Muslim groups” version are in fact statistically different from the “student groups” version – but not from the “activist groups” version. Therefore, we can say that those given the “student groups” wording of the surveillance question are significantly more opposed to the NYPD’s secret monitoring compared to those who are given the question in the context of “Muslim groups” or simply “groups.”
Overall, this question wording experiment shows how different words and emphasizing one particular target group over another can make a difference in the attitudes that respondents express and thus cause shifts in public opinion depending on how the question is asked. What is brought to mind when New Jerseyans are given the “student groups” wording might be very different – and from the data, indeed appears to be – from how respondents interpret and relate to the same basic question about secret surveillance when the emphasis is put on “Muslim groups” instead. Respondents are significantly more disapproving of these monitoring techniques when students are specifically the target, but they are more accepting of the necessity of these actions when they are done toward Muslims and “groups” in general. Therefore, question wording is a very important part of the survey process. It influences not only what the respondent thinks about a particular topic but also how they think about it. In this case, the necessity versus violation of the NYPD’s actions depends on who the NYPD is monitoring, with actions becoming increasingly necessary for the groups that respondents may perceive as more threatening to their safety and well-being.
Note: This blog post was developed by Ashley Koning, a Rutgers political science graduate student and Research Assistant to the Eagleton Center for Public Interest Polling.