the truth of focus groups and surveys

I am deep into coding focus groups, along with my colleagues at CIRCLE. We have convened working-class, urban youth in several American cities. We listen to audio recordings of their discussions with the software package called NVivo and, in addition to making open-ended notes, we attempt to categorize individuals’ statements into one of several hundred codes that we have constructed.

Often, what you hear is not a belief, a preference, or a principle. It is the sound of someone thinking about and around a topic that he or she may never have considered before. Asked whether voting makes a difference, for example, an individual may give a short monologue that drifts between yes and no and then back again, passing by way of such ideas as “no, but you should do it anyway,” and “yes, but only if other people do it, too.”

This reminds me of Nina Eliasoph’s comments from Avoiding Politics (p. 18):

Research on inner beliefs, ideologies, and values is usually based on surveys, which ask people questions about which they may never have thought, and most likely have never discussed. … The researcher analyzing survey responses must then read political motives and understandings back into the responses, trying to reconstruct the private mental processes the interviewee ‘must have’ undergone to reach a response. That type of research would more aptly be called private opinion research, since it attempts to bypass the social nature of opinions, and tries to wrench the personally embodied, sociable display of opinions away from the opinions themselves. But in everyday life, opinions always come in a form: flippant, ironic, anxious, determined, abstractly distant, earnest, engaged, effortful. And they always come in a context–a bar, a charity group, a family, a picket–that implicitly invites or discourages debate.

That’s why the qualitative research we are doing now is interesting. And yet, there is a different way of thinking about people’s mental states and the relationship to their actions. It turns out (from a study of ethics rather than our topic, politics) that people “have a hard time offering an account of their moral reasoning that contains consistent substantive content.” They are “largely incapable of articulating their moral decision-making process in substantive, propositional terms.” Often, their responses to open-ended questions are rationalizations of what they have done, not reasons that will guide what they do.*

A cynic would conclude that people are just not very reasonable; our principles and reasons do not affect our behavior. But it turns out that individuals answer multiple-choice questions in ways that are consistent with their own responses; distinctive, when compared to other respondents; and strongly predictive of their own behavior. In other words, we are guided by something that’s in our heads, and it differs from person to person, but it is not linguistic or explicit. It is more like an unconscious network of associations. That is why fixed-response or “multiple choice” surveys often predict behavior better than open-ended questions do. They may work better for prediction because an actual decision (such as whether or not to vote) is more like checking a box than explaining a personal philosophy. So answering the forced choices on a survey resembles our ordinary decision-making process.*

Yet I remain interested in people’s explicit, verbalized, public thinking. We ought to give good reasons to justify (or criticize) our own actions. We should be interested in other people’s reasons and their reactions to ours. The act of interpreting the public thoughts of working-class urban youth thus has a moral motivation, even if those reasons are not strongly influential in their own lives. I don’t think that current psychological research precludes the hope that good arguments can change people’s implicit stances or premises, which then affect their behaviors.

In short, we should strive to understand other people’s arguments in case they are right and to decide how to respond effectively if they are not.

*Stephen Vaisey, “Motivation and Justification: A Dual-Process Model of Culture in Action,” The American Journal of Sociology, vol. 114, no 6 (may 2009), pp. 1675-1714.

This entry was posted in philosophy on by .

About Peter

Associate Dean for Research and the Lincoln Filene Professor of Citizenship and Public Affairs at Tufts University's Tisch College of Civic Life. Concerned about civic education, civic engagement, and democratic reform in the United States and elsewhere.