Category Archives: 2012 election

polls as truth, polls as strategy, and what that tells us about social knowledge

I am a pollster. My organization, CIRCLE, just released a national survey of young adults that shows Obama ahead of Romney among likely voters under 30: 52% to 35%.

Pollsters much more prominent than I are under fire. Look at the comment thread on any high-traffic blog post or news article that reports a poll and you will see fervent remarks denouncing the survey for deliberate, partisan bias. Typically, the charge is that a poll showing Obama ahead has been conducted to help Obama (even though one might think that a lowball estimate would work better, by alarming his supporters into voting).

One sees blanket denunciations as well as very precise, faux-erudite critiques. For instance, the Detroit News showed a pro-Romney bias because it has a libertarian editorial board. Nate Silver is cooking the books because the Times is liberal. National polls are biased to Romney because they miss cell phone users. I was on talk radio yesterday in San Francisco, and a caller argued that Obama’s support cannot have declined because of the first presidential debate. Instead, his decline in national polls must be a deliberate distortion to set up the fraud that will occur on Election Day, when Romney will use doctored voting machines to steal the vote.

I would like to say: Who you want to win is different from who’s ahead. The former is a value-judgment; the latter is an empirical proposition. Empirical propositions are true or false. So just because you don’t like a poll, it doesn’t mean it’s wrong.

This is largely true, but won’t quite suffice.

I do believe that the average of the polls provides an accurate a picture of the horse race. Aggregating all available surveys produces a gigantic sample, and using lots of pollsters’ data reduces the error caused by their specific methods. We said that youth favored Obama by 52%-35%. We had interviewed a randomly recruited online sample from Knowledge Networks. The very same day, completely independently, and using a random-digit-dialing survey of land-lines and cell phones, the Pew Research Center pegged the youth vote at 56%-35%, well within the margin of error of our result. I take this as confirmation of our finding. I do not read it as mere coincidence, because the same thing happens every day. Separate pollsters draw modest random samples of Americans, use different questions and modes of contact, and come up with quite similar results. The method works.

On the other hand:

1. There is no truth now about how people will vote next week. If the polls are supposed to be predictive, that’s not a typical empirical truth.

2. Each poll requires a whole set of choices that affect the findings. We hired Knowledge Networks, which randomly recruits a national sample and provides people with free Internet access if they need it. We drew a random sample of their panel with large minority sub-samples that we adjusted to make the sample resemble the Census demographic profile of 18-29 year-old citizens. We asked respondents: (1) how likely they were to vote, (2) whether they were certain to vote for Obama, and (3) whether they were certain to vote for Romney. (We randomized the order of the two latter questions.) To calculate Obama’s share of the youth vote, we reported the proportion who said that they were extremely likely to vote, they preferred Obama, and they did not prefer Romney.

The alternatives are myriad. If you call people by phone, you are likely to give them a choice of the candidates and code people as undecided only if they refuse to respond. Often, the interviewer pushes back and says, “If you had to choose …?” That reduces the undecided rate, which was fairly high in our poll. The survey can just ask about Obama and Romney, or it can add two minor party candidates (Libertarian Gary Johnson and Green Jill Stein), or it can add even more options. We accepted a generic response about “someone else,” and four percent chose that.

It is typical of social science that truth is somewhat obdurate–not just made up, but stubbornly out there–yet reality is very much colored by our methods and choices.

3. None of us directly calls more than 1,000 Americans to interview them about the presidential election. We trust other people to do that for us. You trust me and my colleagues to survey youth–or I hope you do. We trusted Knowledge Networks to draw a good sample for us. I trusted my colleagues to run the numbers right. Since social knowledge is mediated, it relies on trust of strangers or of institutions, or both. This was the problem for the caller in San Francisco. He took as a premise that Bush had stolen the 2004 election, which would require a very large conspiracy. If that’s afoot, then all the surveys in Nate Silver’s model could be deliberately distorted to show Romney gaining after Oct. 4. Clearly, I do not agree with this, but then again, I would be part of the conspiracy, so why listen to me? More seriously, trust is fundamental, and excessive or automatic trust is foolish. So the questions for all of us are: whom to trust, how much, and when?

Obama at 52% and Romney at 35% among youth

CIRCLE today released a poll of young people’s views of the election. Our survey, commissioned by the Youth Education Fund, is unique in that it polled 1,695 youth (ages 18-29) in June/July and 1,109 of the same youth between October 21 and 23. Surveying the same people twice provides powerful evidence of change over time.

  • The proportion saying they are extremely likely to vote has risen 9.9 points, from 44.7% to 54.6%. Two-thirds (67.3%) of young adults are “very” or “extremely” likely to vote, up 7.1 percentage points since June/July.
  • The proportion who are paying attention to the election has also risen, from 56.1% to 71%.
  • If the election were held today, Obama would win the youth vote by 52.1% to 35.1% among those registered voters who are “extremely likely to vote.”

My quote from our press release: “The conventional wisdom holds that youth enthusiasm is down compared to 2008,” said CIRCLE Director Peter Levine. “But intent to vote is rising fast. President Obama has a majority of likely young voters behind him, but a significant proportion are open to voting for Governor Romney, who has a clear opportunity to improve over John McCain’s record-low support in 2008.”

I will be on MSNBC around 11:30 am eastern today, on WGBH-Boston’s Innovation Hub at around 1 pm today, and on KALW-San Franciso at 10 am Pacific tomorrow to discuss, and later I will post the audio or video here.

[Update: actually, MSNBC cancelled to cover Sandy 24/7. But the WGBH discussion with host Kara Miller and the Pew Research Center’s Paul Taylor is available online, here. And the KALW discussion with Ali Budner (the host) and Lee Rowland of the Brennan Center and Tova Andrea Wang of Demos is here.]

CIRCLE in the news

(Woods Hole, MA) With our new report on civic education published recently–and the election coming up–we have been in the news a lot lately. Here’s a sampling of recent coverage:

Nora Fleming, Out of School Engagement in Civic Education and the 2012 Election, Education Week, 10/10/2012 (interview format)

Education Week: And in the future? Is there more interest now in early engagement around elections, politics, and civic life than in the past?

Peter Levine: It’s a mixed picture. I think more organizations and individuals are concerned about these issues and doing their best to help. I think some of the new strategies are very innovative and promising, such as the use of computer simulations to teach politics. On the other hand, as our new study shows, states have cut back a lot on civics requirements, and social studies tests have shifted to exclusively multiple-choice. Neither No Child Left Behind nor Race to the Top did anything positive for civics. So policies have been unhelpful.

Nora Fleming,  Civic Education Found Lacking in Most States, Education Week, 10/10/2012

“The standards in most states include some high aspirations, but typically have nothing to do with assessments. The standards are miscellaneous, the assessments are lacking, and when they are high stakes, they are trivial,” Levine said. “I think in a big, deep way, civics and preparation for citizenship has been left out by policymakers, who think in terms of preparation for college and for a difficult labor market but don’t think of civics as part of this.”

Amelia Woodside, Parenting young voters: There’s still time in this election season, Christian Science Monitor, 10/12/2012

“Young people age 18-29 are a large bloc of 46 million eligible voters, larger than the senior population, and they tilted sharply in favor of Obama in 2008,” writes Peter Levine, director of The Center for Information and Research on Civic Learning and Engagement (CIRCLE) & Lincoln Filene Professor of Citizenship & Public Affairs at Tufts University in an e-mail interview. “If their turnout is much lower, or if Mitt Romney controls more of their votes than John McCain did in 2008, that could have a substantial effect on the outcome. In 2008, if young voters had not supported Obama, he would have lost Indiana, North Carolina, and Virginia.”

Steven Yacino, Colleges Take a Leap Into Voter Registration, The New York Times 10/13/2012

Roughly 11 million eligible voters ages 18 to 24 are in college, about a quarter of all eligible young voters, according to the Center for Information and Research on Civic Learning and Engagement at Tufts University.

Zoë Carpenter, The Missing Millenials, The Nation, 10/17/2012

The public discussion about millennial turnout has focused on educated voters like Amber rather than her sisters. “The media talks about college students as if they’re all young people, and all young people are college students,” CIRCLE’s Peter Levine says. “And that contributes to the fact that noncollege youth are overlooked.” Voter registration and turnout are strongly correlated with education, and about 42 percent of the current youth population has never been to college. However, studies show that when undereducated youth are registered to vote, they do so at rates similar to nearly every other group. It’s the classic chicken-or-the-egg problem: campaigns don’t target poor and uneducated voters because they’re considered “low potency,” while those populations are less likely to vote because they aren’t engaged by the campaigns. The destruction of institutional networks in poor neighborhoods has further increased their distance from the electoral process. Older generations left high school, joined unions and worked in organized workplaces such as factories. They read newspapers and went to church. That is no longer the case, according to Levine: “They’re on their own in a way that is unprecedented.”

Fawn Johnson, Growing Young Voters (Without Boring Them), National Journal (online)  10/22/2012

If the casual mention of a high school social studies class makes your eyes glaze over, you aren’t alone. The stereotype of the throw-away, easy A class taught by the football coach is there for a reason.

But you also aren’t thinking about civics the way that education scholar Peter Levine thinks you should. “In 1948, 41 percent of American kids took a class called Problems of Democracy. It was reading the newspaper and discussing the issues and writing papers about it, which is pretty much what I would want to happen. …It’s basically gone now,” said Levine, who runs the civic engagement organization CIRCLE.

Levine worries a lot about how kids learn to become citizens. He says schools aren’t teaching them about civics in any consistent or meaningful way. CIRCLE’s research on government curriculum finds that all states require some form of social studies, but most states don’t test on it and those that do use the cheapest multiple-choice tests.

Also:

what is wrong with this campaign, in a nutshell

We will field a survey immediately after the election that will assess, among other things, whether people voted knowledgeably and in synch with their own values and beliefs. We will ask them their top policy issue, followed by a policy preference about that issue, followed by some questions about the candidates’ stances on that issue. We want to know whether voters can pick an issue, know where the candidates stand on it, and voted for the one who shared their opinions.

We are having one heck of time choosing or writing questions that fairly and validly assess citizens’ knowledge of the candidates’ policy stances. On practically every issue, both candidates say inconsistent or fuzzy things or deny reasonably neutral characterizations of their positions. For example, is Romney in favor of Medicare vouchers? He says he isn’t. Even if one disagrees, it seems wrong to assess voters’ knowledge by asking them which candidate advocates Medicare vouchers. The fact that the candidate in question denies it is a pretty good excuse for not being sure.

Speaking emphatically for myself alone and not for the CIRCLE team–I think this is a lousy presidential campaign. The incumbent can’t say: “Vote for me so that I can veto Republican efforts to undo some of the unpopular but beneficial things I achieved in my first four years, but I will not be able to do much else.” And the Republican cannot say, “Vote for me so I can cut upper-income tax rates and raise the national debt even though I said I would reduce it.” So they spew a lot of Malarkey, and even if you worked for 15 years in a graduate school of public policy (as I did), you cannot write survey questions that reasonably assess their positions.

Lest I be accused of false-equivalence, I didn’t say that they spew equal quantities of malarkey and buncombe. I think the Romney campaign is substantially worse. But if you think the president is being straightforward, try writing survey questions that assess citizens’ understanding of his positions.

what does an election forecast mean?

On Oct. 4, Nate Silver “forecast” (that is the label of his graph) that Barack Obama had an 87.1% chance of winning the November election. Ten days later, he said that Obama’s chance was 62.9%. Princeton’s Sam Wang offered somewhat higher percentages at both times, but in the same pattern.

The odds changed by 25 points, way outside any reasonable margin of error. Does that invalidate the early-October forecasts? If Obama wins in November, will that show that the forecast was more correct in early October than it is now? Or will we know with hindsight that Obama actually had a 100% chance of winning, because he did win? An election doesn’t seem random in the same way that a die-roll is random. Once the result occurs, it seems that it had to. If there is an element of sheer randomness in an election, like the effects of rain on turnout, that element is small.

One excuse for the forecasters is that they could not foresee the President’s poor debate performance and its substantial impact on public opinion. I didn’t think he performed well, but, as Kevin Drum shows, the slide in the polls began before the debate and continued smoothly through the debate–suggesting that it was not the reason for the decline. So the forecasters failed to predict a trend (not a single, random-seeming event like a flubbed debate). Does that make them bad forecasters?

A forecast can move in either direction substantially and unpredictably, because who knows what crazy events may occur before Election Day? Mitt Romney could suddenly start yelling imprecations at the 47% during tonight’s debate. But if a forecast moves for a foreseeable reason, such as reversion to the mean, then the forecasters should have predicted it, and their earlier predictions were errors.

Thus, a forecast is a falsifiable hypothesis, but we don’t test it by waiting for the ultimate election result, which will either be a 100% Obama victory or a 100% Romney victory. We rather ask whether the forecast changes in a theoretically foreseeable fashion before Election Day. The only changes that are acceptable are (1) those caused by large and truly random events, and (2) a gradual movement toward certainty, which reflects the diminishing time left for random events to occur. By that standard, the predictions made in early October have now been falsified, and we’ll see how the current ones hold up.