Monthly Archives: September 2009

much better than a town meeting

If you or a group that you’re part of wants to discuss health care policy without descending into the kind of shouting matches that dominated August, the Kettering Foundation and the National Issues Forums have just the tools you need. Click through to a work book, other background materials, and a guide to holding a neutral, productive dialog in your community.

against caricaturing your opponents

If you spend a lot of time, as I do, on sites like Steve Benen’s Washington Monthly and The American Prospect, you can get at least a daily dose of outrageous remarks by Republican politicians and conservative celebrities. These comments often combine fallacy, bias, malice, and myth. (As an example from the last few days, The National Review says this week that Barack Obama “likes tyrants and dislikes America [and] is trying to get control over as much of our lives as he can.”)

Meanwhile, if you spend time reading the open comments on newspapers and other news sources, you can find plenty of similarly outrageous statements all on your own. In addition to the flaws cited above, these amateur contributions tend to be poorly spelled and ungrammatical.

Spend your time this way, and you can convince yourself–or reinforce your presumption–that you are dealing with an opposition that’s stupid and mean. Compromise then seems foolish. Unfortunately, large numbers of Americans vote for the same candidates as the shock jocks and right-wing politicians whose frightening quotes fill liberal blogs. So it’s hard to avoid the conclusion that you are dealing with a fairly benighted population. Either there is something fundamentally wrong with them, or else the media system is somehow capable of misleading the public and distorting their views.

But selective quotation is a very poor “research method.” As long as the pool of material available for quotation is large, you can get any result you want. I think it would be possible, for example, to fill a daily blog with nothing but thoughtful and conciliatory comments by elected Republicans. Likewise, conservative bloggers are able to find outrageous statements by liberals to fill their pages.

Whatever the truth may be about “conservatism,” selective quotation is no way to find it. Not only is this method intellectually lax, it yields bad strategy. It leads us to misunderstand the real reasons that substantial numbers of Americans disagree with us and thus prevents us from persuading them. For instance, relatively conservative Americans may fear the Democrats’ health plan because they are skeptical that the government can keep its promises to cut costs, not because they believe that President Obama is a socialist Muslim born in Kenya who hates America.

Although I’m sure some Republicans in Congress disagree, many leading ones seem to share Florida Gov. Charlie Crist’s view that President Obama is headed for “a massive electoral defeat.” I think they have been reading too much of their own side’s polemics (including selective quotations from liberals that seem far from mainstream views). As a result, they are not revising their own policies in ways that might help them to regain a majority. The long-term polling trend suggests that they really only have about 20-25% of the population on their side and should be seriously rethinking their positions:

But there is equal danger for Democrats. The President has but a bare majority–less on difficult issues such as health care and climate change–and faces valid doubts and criticisms. The worst way to get majority support for a Democratic legislative agenda would be to convince ourselves that we are only dealing with Michelle Bachmann, Glenn Beck, and the National Review.

being an informed consumer of polls

Nate Silver is accusing the Strategic Vision polling firm of making up its data (which would be much cheaper than collecting it!). I don’t know the truth about this case, but I’m especially interested because one of the disputed Strategic Vision polls found extremely poor knowledge of civics among Oklahoma students–and civics is my main interest.

In any case, this seems an opportune moment for some remarks about polling, in general. I’ve been involved (usually with collaborators) in commissioning nine national surveys using at least five different firms. I’ve had mixed experience, ranging from deep respect to real concerns. All the firms I’ve been involved with have been basically reliable, but polling is not a pure science. There is an important aspect of art or craft, and quality is inconsistent. If you take survey results as precise and fully reliable, you’re naive. But if you reject high quality surveys because you don’t like the results–something I’ve seen happen on many occasions–you are equally mistaken. Although polling is an art or a craft and not a pure science, a good poll is far from arbitrary.

Polling would be more of a science if probability sampling really worked. The theory suggests that you can say something about a whole population based on a small number of respondents, randomly selected. But it is never possible to achieve pure random selection. That’s partly because there is no list of all Americans from which names can be drawn; you have to use a substitute (such as randomly generated telephone numbers) which must omit some individuals. To make matters worse, most people don’t agree to be surveyed. The response rate is always far below 50% and very uneven across demographic groups. So if you randomly dialed a bunch of phone numbers and took down the answers of the first 1000 people who agreed to be interviewed, you would have a deeply biased survey.

Instead, all pollsters I’ve dealt with use some combination of demographic quotas (i.e., they contact people randomly until they have enough respondents within each particular category), lists of individuals who are likely to respond, and weighting. To “weight” a sample after it’s been collected, you adjust it to match the whole population. If, for instance, you have only half as many young white males as their prevalence in the whole community, each one’s responses must count for two.

Every survey I know of has been weighted, but it makes a great deal of difference how. To double the responses of 150 young white males is not a big problem; but sometimes three young Latino males can be counted for 100. That stretches the representativeness of the sample past the breaking point. There is also a huge question about what categories to use for weighting. You can weight a sample to match the ethnic, age, and gender composition of the whole population but still get badly biased answers to opinion questions if you don’t weight to religion or political party. But if you weight to everything, you haven’t really taken a survey.

A pollster can reduce the need for weighting by making dogged efforts to interview the first people who were randomly selected to be interviewed–but that’s expensive. It’s much cheaper to move on to someone else and then “weight” after the fact. Even the most dogged efforts never yield particularly high response rates, which is why weighting is unavoidable.

I have focused here on sampling issues, but of course the questions one chooses to ask on a poll also introduce all kinds of bias. Writing good questions is very much an art, and no amount of statistical testing for reliability and validity can ever tell you for certain whether your questions are good.

At CIRCLE, we are an unusual client because we never purchase a report from a survey firm. We purchase the data, including the unweighted dataset. We then spend a lot of time analyzing the sample and making judgment calls. Sometimes, we will refrain from talking about particular subsamples because the weights are too large. In one case, we threw out a whole national poll that had cost more than $100,000 to collect because we did not believe it was reliable.

If you’re a regular reader of polls in the news, you can’t analyze the raw data. You have to rely on intermediaries, such as reporters, editors, clients, and associations of pollsters, to separate the wheat from the chaff. I don’t believe that simple distinctions (such as random-digit-dialing versus online samples) are all that helpful, because there are tradeoffs between the various methods. You can only hope that the intermediaries you trust have looked closely at response rates, quotas, and weighting schemes. It’s also helpful to apply some common sense–for instance, I find it difficult to believe that Oklahoma students know as little as Strategic Vision claims–and to compare more than one poll if they are available.

Finally, I wouldn’t throw the baby out with the bathwater. Most respectable pollsters struggle very hard to get representative samples and to ask good questions. Their results are more reliable for certain purposes than others. (For instance, an estimate of the frequency of a behavior–like “37% of people volunteer”–is less reliable than a pattern in the data, such as “volunteers are much better educated than non-volunteers.”) But certainly it’s wise to credit their basic findings about what people like and what they believe.

memories of Broughton Castle

William Fiennes has published a memoir about growing up in Broughton Castle near Oxford. In today’s Times, he says he that he resisted “writing about growing up in such a big house” in case “people would laugh at me,” but then he decided, “‘Look at this. What a world of images.’ How could I be a writer and not sing about all this? … Here was a microcosm, a place surrounded by a ring of water, and everything a writer could want to write about was going on here–wonder and excitement and love, but also loss and difficulty and violence and fear and strangeness.”

I was at Broughton Castle in November 2007. I was attending a conference, and a visit to Broughton was an optional break. My family and I had just learned that my father might have cancer in his lungs. We were waiting for results, but it was hard to communicate with the US, and nothing would be known for hours. So I went along to Broughton for a distraction.

Lord and Lady Saye and Sele greeted us and were warm, gracious, unpretentious, and amusing hosts. They were the only people in the castle, so they collected our donations and gave us the tour. We saw both the magnificent public rooms of their castle–medieval to Georgian–and their cozy private spaces with children’s art on the refrigerator and vegetables on the counter. My fellow visitors were English, polite but at ease their their hosts. There was a joke about the difficulty of pricing a home like Broughton, which hasn’t been sold since the 13th century.

This was exactly the kind of place that I had visited throughout my childhood, guided by two parents who have professional knowledge of English cultural history. So I felt comfortable and nostalgic–also deeply sad that my parents couldn’t be with me. The Baron and Baroness are in their eighties, and I couldn’t help wondering whether my parents would also be able to enjoy that decade together.

At the same time, I experienced a less defensible emotion: competitiveness. Thanks to my upbringing, I have a detailed understanding of places like Broughton. For instance, as soon as I saw a fireplace upstairs, I knew it was an extraordinary piece, and it turned out to be an almost unique example of Italian Renaissance art from the England of Henry VIII. King Henry had a whole palace, Nonesuch, decorated by Italian Mannerist artists, but it burned to the ground. The Broughton fireplace is an extremely rare surviving work by the sculptors of the lost Nonesuch. And here I was, a Yank among Englishmen, being guided by a self-deprecating aristocrat who probably knew everything about the art in his house but pretended to be naive. I secretly wanted to lecture everyone about what they were seeing (and therefore about what I knew). I did keep my mouth shut except for a muttered aside about the fireplace.

Almost as soon as we returned from this short visit, I learned that my father’s cancer had spread and was inoperable. Six weeks later, he was dead. The last time in my life when I could have hope for him was during the ride back from Broughton along green sunken lanes. So now I have something in common with the writer William Fiennes: the castle where he was raised is also for me a place of nostalgia, fondness, and regret.