the truth of focus groups and surveys

I am deep into coding focus groups, along with my colleagues at CIRCLE. We have convened working-class, urban youth in several American cities. We listen to audio recordings of their discussions with the software package called NVivo and, in addition to making open-ended notes, we attempt to categorize individuals’ statements into one of several hundred codes that we have constructed.

Often, what you hear is not a belief, a preference, or a principle. It is the sound of someone thinking about and around a topic that he or she may never have considered before. Asked whether voting makes a difference, for example, an individual may give a short monologue that drifts between yes and no and then back again, passing by way of such ideas as “no, but you should do it anyway,” and “yes, but only if other people do it, too.”

This reminds me of Nina Eliasoph’s comments from Avoiding Politics (p. 18):

Research on inner beliefs, ideologies, and values is usually based on surveys, which ask people questions about which they may never have thought, and most likely have never discussed. … The researcher analyzing survey responses must then read political motives and understandings back into the responses, trying to reconstruct the private mental processes the interviewee ‘must have’ undergone to reach a response. That type of research would more aptly be called private opinion research, since it attempts to bypass the social nature of opinions, and tries to wrench the personally embodied, sociable display of opinions away from the opinions themselves. But in everyday life, opinions always come in a form: flippant, ironic, anxious, determined, abstractly distant, earnest, engaged, effortful. And they always come in a context–a bar, a charity group, a family, a picket–that implicitly invites or discourages debate.

That’s why the qualitative research we are doing now is interesting. And yet, there is a different way of thinking about people’s mental states and the relationship to their actions. It turns out (from a study of ethics rather than our topic, politics) that people “have a hard time offering an account of their moral reasoning that contains consistent substantive content.” They are “largely incapable of articulating their moral decision-making process in substantive, propositional terms.” Often, their responses to open-ended questions are rationalizations of what they have done, not reasons that will guide what they do.*

A cynic would conclude that people are just not very reasonable; our principles and reasons do not affect our behavior. But it turns out that individuals answer multiple-choice questions in ways that are consistent with their own responses; distinctive, when compared to other respondents; and strongly predictive of their own behavior. In other words, we are guided by something that’s in our heads, and it differs from person to person, but it is not linguistic or explicit. It is more like an unconscious network of associations. That is why fixed-response or “multiple choice” surveys often predict behavior better than open-ended questions do. They may work better for prediction because an actual decision (such as whether or not to vote) is more like checking a box than explaining a personal philosophy. So answering the forced choices on a survey resembles our ordinary decision-making process.*

Yet I remain interested in people’s explicit, verbalized, public thinking. We ought to give good reasons to justify (or criticize) our own actions. We should be interested in other people’s reasons and their reactions to ours. The act of interpreting the public thoughts of working-class urban youth thus has a moral motivation, even if those reasons are not strongly influential in their own lives. I don’t think that current psychological research precludes the hope that good arguments can change people’s implicit stances or premises, which then affect their behaviors.

In short, we should strive to understand other people’s arguments in case they are right and to decide how to respond effectively if they are not.

*Stephen Vaisey, “Motivation and Justification: A Dual-Process Model of Culture in Action,” The American Journal of Sociology, vol. 114, no 6 (may 2009), pp. 1675-1714.

This entry was posted in philosophy on by .

About Peter

Associate Dean for Research and the Lincoln Filene Professor of Citizenship and Public Affairs at Tufts University's Tisch College of Civic Life. Concerned about civic education, civic engagement, and democratic reform in the United States and elsewhere.

5 thoughts on “the truth of focus groups and surveys

  1. Rebecca Townsend

    Nice post, Peter.  I found myself nodding along for most of this, first out of recognition that we are both  using NVivo8 for analysis (in my case, it’s of of student-led discussions among  community groups in traditionally under-heard groups). And second, out of appreciation that you find similar value in explicit verbal matters.  (I stop the nodding at the word “thinking,” because reference to speaking as verbalized thought seems to lessen the status of speaking, as somehow secondary to the more important “thought.”)

    The act of speaking is necessarily social.  Are you paying attention to the interactional moves people make as they hear, listen to, and respond to what others (or themselves) are saying?  By my training, I find it difficult to look for individual moral reasoning; reasoning is social.  Even a “personal philosophy” is performed in an interaction with others; that’s all we really have to go on.  How can you ever know a single person’s inner thoughts if the only basis you have is what a person says or does?  Motives and reasons are tricky, and they often are in the process of developing while a person is interacting. 

    1. Peter Levine

      Rebecca, I agree that the interactions are fascinating and important. Focus groups thus gave advantages over individual interviews. I am noting some interactions, but I am not sure we are coding them systematically enough. – Peter

  2. Ron Lubensky

    My feeling (!) is that most people lack a vocabulary to talk about ethics, values and beliefs, especially at the meta-level. Perhaps that’s one reason why a moral code or dogma is attractive to many, as the explanations and prescriptions are laid out flat in front of them. So before we can “understand other people’s arguments”, we should ask them the questions they might ask themselves, using a range of new words and framings that could help them reveal what is behind their tacit attitudes. The research interview becomes a very reflexive, subjective exercise, rather than the objectified, standoffish affair that is more typical. I’m not so sure how helpful NVivo would be with that, though, I’ve not tried.

  3. Pingback: links for 2011-06-23 | KevinBondelli.com: Youth Vote, Technology, Politics

  4. Pingback: An example of "panning for gold" polling methodologies? » 21stCenturyFix.org.uk

Leave a Reply