Monthly Archives: February 2014

what is generalizable knowledge?

In a group devoted to community-based research, we were discussing the tendency of academics to seek “generalizable” knowledge, while community-based groups want knowledge that has immediate significance to their own circumstances. That difference can generate conflicts over priorities and objectives. Scholars and community leaders may disagree about what projects should be funded, how time and effort should be spent, and even the ethics of research on human subjects. For example, NIH says, “The goal of clinical research is to develop generalizable knowledge that improves human health or increases understanding of human biology.” But a community group may want to find out whether their kids have a well-understood disease. They may see that as “research.”

What is generalizable knowledge? I think generalizability comes in many forms, each playing a different role in each discipline. That makes the tension between scholarly priorities and the needs of community groups complex. The situation will be very different depending on whether the scholar in question is an epidemiologist, an ethnographer, or a theologian.

Statistical generalization means applying a finding from a given population to other populations. If smoking increases the prevalence of emphysema in Somerville, MA, will that also be true in Boston or in Beijing? Widely applicable findings are more useful than narrower findings. However, people obstinately vary, and if a finding does not generalize, it can still be valuable for the population in which it was found. Thus community groups and academics only differ in the relative value they place on statistical generalizability.

Theoretical generalization means developing or contributing to theoretical frameworks that apply in other situations than the one being studied. A theory can be predominantly explanatory or predictive, like Keynes’ theory that excessive saving causes recessions. Or a theory can be predominantly moral/normative, like Keynes’ theory that the state ought to stimulate economies to increase employment. (This implies that the state has the right and the obligation to intervene.) In my view, almost all explanatory theories about people have normative elements, and vice-versa; but certainly the emphasis varies. In any case, theories must generalize: you can’t have a new theory for each case. You can, however, resist theorizing because it overgeneralizes. Or you can resist spending time on theory because you have a program to run. (Yet programs always have their own theories.)

Methodological generalization means developing or demonstrating a method that others can use in different contexts. Methodological innovations can range from highly practical new medical techniques to essays like Clifford Geertz’ “Deep Play: Notes on the Balinese Cockfight.” We do not read that piece because we are eager to understand cockfights in Bali; we read it to learn and debate the method of “thick description,” which we might apply in other settings. Academics value methodological innovation; practitioners rarely care.

Interpretive generalization. Interpreters of texts, images, events, rituals, and dreams describe the particular objects in ways that convey their form, context, and purpose. Sensitive interpretation resists generalization–but not completely. As Geertz notes:

The besetting sin of interpretive approaches to anything —literature, dreams, symptoms, culture—is that they tend to resist, or to be permitted to resist, conceptual articulation and thus to escape systematic modes of assessment. You either grasp an interpretation or you do not, see the point of it or you do not, accept it or you do not. [But] this just will not do. There is no reason why the conceptual structure of a cultural interpretation should be any less formulable, and thus less susceptible to explicit canons of appraisal, than that of, say, a biological observation or a physical experiment.

Indeed, ethnographers and humanists do generalize from their interpretations of particular objects, although, as Geertz concedes, theory based on interpretation should “stay rather closer to the ground than tends to be the case in sciences more able to give themselves over to imaginative abstraction.”

Political generalizability means using a case to achieve some kind of legal or administrative change that affects other people in other places. Some would say that policies should always reflect statistical and theoretical generalities. The law should treat like cases alike; hard cases make bad law. Those maxims suggest that we should first find general patterns through research, and then make policy fit the patterns. But I think sometimes good laws come straight from dramatic cases, which suffice to demonstrate valid points about justice.

Ellen Miller on passing the baton

I am no Ellen Miller (and I am not going to retire any time soon), but when the time comes, I will consider it a mark of an outstanding career if I can write as she did today:

When I was a young congressional staffer, I thought changing the world was going to be a sprint. In the middle of my career, when I launched the Center for Responsive Politics and then started Public Campaign, I thought it was going to be a marathon and you just had to be a long distance runner; so I trained up for that. Now, after eight years building and running the Sunlight Foundation, I now see this process as a relay race — and one where I’m glad to say there are many other great people running alongside me, including the terrific team we have built here at Sunlight. It’s truly extraordinary. And it’s time to pass the baton.

Francis Bacon on confirmation bias

Nowadays, we call it “confirmation bias:” our deep-seated tendency to prefer information that confirms our existing positions. A political controversy erupted in 2010 when the libertarian blogger Julian Sanchez accused conservatives of falling prey to “epistemic closure,” which was his preferred term for the same problem:

Reality is defined by a multimedia array of interconnected and cross promoting conservative blogs, radio programs, magazines, and of course, Fox News. Whatever conflicts with that reality can be dismissed out of hand because it comes from the liberal media, and is therefore ipso facto not to be trusted.

Edward Glaesser and Cass Sunstein call this phenomenon “asymmetric Bayesianism” and give examples from the left as well as the right. I argued earlier this week that Amy Chua’s and Jed Rubenfeld’s book The Triple Package suffers from the same problem.

But–as is often the case–Francis Bacon got there first. In his remarkable compendium of human cognitive frailties (published in 1620), he included this problem:

The human understanding when it has once adopted an opinion (either as being the received opinion or as being agreeable to itself) draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate. And therefore it was a good answer that was made by one who, when they showed him hanging in a temple a picture of those who had paid their vows as having escaped shipwreck, and would have him say whether he did not now acknowledge the power of the gods — “Aye,” asked he again, “but where are they painted that were drowned after their vows?” And such is the way of all superstition, whether in astrology, dreams, omens, divine judgments, or the like; wherein men, having a delight in such vanities, mark the events where they are fulfilled, but where they fail, though this happen much oftener, neglect and pass them by. But with far more subtlety does this mischief insinuate itself into philosophy and the sciences; in which the first conclusion colors and brings into conformity with itself all that come after, though far sounder and better. Besides, independently of that delight and vanity which I have described, it is the peculiar and perpetual error of the human intellect to be more moved and excited by affirmatives than by negatives; whereas it ought properly to hold itself indifferently disposed toward both alike. Indeed, in the establishment of any true axiom, the negative instance is the more forcible of the two.

 (Novum Organon, XLVI) 

Of course, Bacon’s goal was not lament our tendency to err but to help us fix it. Evidence has accumulated to reinforce his concerns about our blinkers and biases. Yet, as the Yale psychologist Paul Bloom recently wrote in The Atlantic, even the accumulated evidence does not show that we are fundamentally irrational. We very often make wise and deliberate choices. It all depends on the context of choice and the methods we use to deliberate:

So, yes, if you want to see people at their worst, press them on the details of those complex political issues that correspond to political identity and that cleave the country almost perfectly in half. But if this sort of irrational dogmatism reflected how our minds generally work, we wouldn’t even make it out of bed each morning.

As the heirs of Bacon’s scientific revolution, we should relentlessly investigate all systematic forms of human error–not to shake our faith in reason but to help us to reason better.

Saul Alinsky on video

I am teaching about Saul Alinsky (with due consideration to both his strengths and his weaknesses). I enjoyed showing my class actual footage of the man at work–from a 1968 Canadian documentary.

In this excerpt:

[o:00 to 0:55] A Chicago woman complains about rats the size of cats in her apartment. This is not a “one-on-one” interview in which she tells her own story and displays her agency as a human being. It is a documentary filmmaker’s invasion of her apartment to demonstrate her poverty. However, it does reveal some of the problems that would emerge in a one-on-one.

[0:55-1:42]: Two Alinksy-trained organizers plan “direct actions,” including pickets and a rent strike. In the ideal model, they would choose the topic and target only after much consultation with residents in one-on-ones and house meetings. But sooner or later, they would have to deal, as they do here, with practical issues like babysitting the moms who picket. By the way, a rent strike is a classic collective action problem: an individual renter who fails to pay will be evicted, but a whole building can obtain relief from the landlord if they act together. Even a picket line works a lot better if everyone joins it. That’s why I enjoy this exchange between the organizers at 0:58:

How many tenants in the building?

Ah, the building has, I think something like 27-30 tenants.

Tell you what–why don”t we turn out all 30 of them?

[1:43-3:40] A dignified picket line and some interaction with police.

[3:40-5:15] A debate with a neighbor (a young White man) who doesn’t like to see picketing on his street. The resident seen earlier talking about rats in her bedroom expresses herself pretty effectively.

[5:15-6:41] Alinsky himself, trying to establish his credibility as an anti-racist organizer with a group, probably from Dayton, OH, that is considering hiring him.

[6:41-8:09 ] The committee deliberates on whether to employ him.

In the next segment, he argues with a Canadian hippie about building utopian alternatives. (Alinksy is against.)

Chua and Rubenfeld, The Triple Package

The new book by Amy Chua and Jed Rubenfeld (The Triple Package: How Three Unlikely Traits Explain the Rise and Fall of Cultural Groups in America) is stirring controversy, as it was meant to do. The authors are nothing if not successful marketers.

I don’t agree with critics like Sekuta Mehta who argue that exploring the relationship between cultural traits and prosperity is racist. If a culture is a cluster of norms, preferences, habits, and values, than each culture will be more or less consistent with the culture of capitalism. Consistency with capitalism will confer advantage, and we should understand that.

Chua and Rubenfeld should not be condemned just for arguing that culture matters. But I think they fail in two other ways, and these weaknesses bother me less for ideological reasons than because I don’t like to see senior academics ignoring complexity and rigor in order to publish best-sellers that would not pass even the most ideologically open-minded peer-review process. In other words, I am not offended by the conclusion but by the methods.

First, their empirical argument is deeply problematic. They identify eight prosperous US ethnic groups and, after rejecting other explanations for their economic success, insist that the essential reasons must be three cultural traits (a sense of superiority, a sense of insecurity, and impulse control). This is the kind of method that persuaded physicians, for many centuries, to use leeches to bleed patients. Look at my eight most successful medical cases–people who recovered from mysterious illnesses. What do they have in common? I bled them all. Ergo, leeches work. You don’t have to be a fool to think this way, but the scientific method works much better.

When studying the impact of culture on prosperity, the scientific gold standard (an experiment) may not be possible. But one must at least avoid selection bias to choose large and representative samples, measure each case using independent data, and rigorously consider multiple explanations. In this case, you would look at all US ethnic groups (poor as well as rich) and treat their median income or wealth as a function of many plausible explanations that can be reliably measured. Cultural factors might or might not emerge as significant predictors in a multivariate model. You won’t find out from Chua and Rubenfeld.

The second problem is their very thin moral (or normative) framework. Assume that having a sense of cultural superiority conveys competitive advantage in a market economy–what of that? The authors do raise the point that striving for material success may not be the best way of life, but that is an aside. Surely, each of the grand ideas in their book–a sense of superiority, insecurity, delayed gratification, and wealth itself–is problematic.

More rigorous research than theirs brings into focus other ways in which fitting well with capitalism can be morally troubling. For instance, Annette Lareau finds that middle-class siblings are quarrelsome because they are raised to see each experience as an investment in their own success. If Brother has to wait during Sister’s violin lesson, that is a waste for him, and he acts out. Working-class children are much better at cooperating and more cheerful about it, because they are expected to make their own entertainment together. Sister is a potential playmate for Brother.

This is just one small example of the ways in which fitting well with capitalism can deform the soul. Capitalism may also uplift, liberate, and improve. The moral impact of capitalism is a complex and challenging topic, on which Chua and Rubenfeld have little to offer.

That said, we ought to have this conversation and not treat it as inherently offensive. Mehta, for example, decries as racist the claim that culture affects prosperity and ends with his uncle Vipinmama, “who lived every day of his life in the pursuit of happiness” and was wise enough not to fit with capitalism. But that just underlines the importance of capitalism as a culture with which a person may fit–or not. If you live like Vipinmama, you may be happy, but you will not long be rich. We ought to recognize that fact before we decide what to do about it.

Eight years ago, I proposed that we might think of three axes of debate: whether cultural norms affect success; whether material success is a worthy goal; and whether state action can enhance welfare. All combinations of answers to those three questions are tenable. I think I am with people like Pierre Bourdieu and Annette Lareau who emphasize the economic impact of culture while doubting that it is morally good to be materially successful under capitalism. They want to use the state to change the relationship between culture and wealth, thus falling at the front top of this cube (which I explain more fully here):