Yesterday’s post was Part 1 of a critique of expertise in public policy. Part 2 focuses on the issue of generalization.

Experts generalize. An important aspect of almost all professional training is the identification of general concepts or categories that trigger appropriate responses. Told a story about specific people interacting in a particular context, any professional will look for abstractions. For instance, in medical school, one learns the signs and definitions of diseases, and when a disease is present, a physician knows which treatments to offer. When more than one condition is involved, or when the diagnosis is uncertain, the decision becomes complex, and good physicians fully understand the roles of judgment and luck. Doctors could never be replaced by machines that simply took in data and spat out treatment plans. But diseases and other general health conditions remain central to physicians’ analysis. They look for the necessary and sufficient conditions that define conditions, and then apply general causal theories that say: this medicine reduces that illness.

Lawyers, meanwhile, try to apply general rules from statutes, constitutions, and court rulings. Their advice may be controversial or uncertain if no single, definitive legal rule covers the situation–and they understand that–but their professional thinking involves rules. For engineers, economists, psychologists, and virtually all other professionals, the important abstractions may be different, but the basic habits of mind are alike. Professionals have achieved monumental advances (and prestige) by discovering generalizations that apply widely. For example, the polio vaccine reliably prevents polio, and that is extremely valuable to know.

You can also hear ordinary people generalize if you listen in public spaces. They say things like, “Of course Amtrak is always late, it’s a government monopoly.” Or, “You’re getting a cold; you should take vitamin C.” Research and data disprove these assertions, and a trained professional would not make them. Even an economist who was hostile to monopolies would not draw a direct line from Amtrak’s monopoly status to the tardiness of its trains. (Other countries have monopolistic railroads that run on time.) Instead of being too quick and bold with generalizations, a good professional is fully aware of complexities and nuances.

Even so, there are drawbacks to using general concepts as the main units of analysis. A person, a situation, an institution, or a community can be apprehended as a whole object. We can assess it, judge it, and form opinions about how the entity should change. Evaluating a whole situation need not be any harder or less reliable than analyzing general categories abstracted from such situations. If we can say something valid and useful about a generality (like diabetes, tax incentives, or free speech), we can talk just as sensibly about this patient, this school, or this conversation. The particular object or situation is not just an aggregate of definable components. It has distinctive features as a whole, and we human beings are just as good at understanding those as we are at generalizing abstractly.

The form that our understanding takes is often narrative: we tell stories about particular people or institutions, and we project those stories into the future as predictions. We may find generic issues and categories embedded within a story: King Lear, for example, was a king and a father, and there are general truths to be said about both categories. But the story of King Lear is much more than an aggregate of such categories, which are not especially useful for understanding the play.

In public policy, non-professionals are often better at the assessment of whole objects than experts are. That is because ordinary members or clients of a school, a neighborhood, or a firm know its whole story better than an outsider who arrives to apply general rules.

Often, professionals have in the back of their mind an empirical finding that is valid in academic terms, but that should not tell us what to do. Even when results are statistically significant, effect sizes in the social sciences are usually small, meaning that only a small proportion of what we are interested in explaining is actually explained by the research. Statistical studies shed some light on why individuals differ, but can tell us nothing about why they are all alike. In research based on surveys or field observations, the sample may not resemble the population or situation that we face in our own communities. Experimental research is conducted with volunteers (often current undergraduate psychology majors) in artificial settings. Even if a particular finding is strong, and the sample does resemble our own, there is always a great deal of variation, and any particular case may differ from the mean. Measures are always problematic and imperfect, and some important factors are virtually impossible to measure. Unmeasured factors may be responsible for the relationships we think we see among the things we measure.

All of this is well known and may be thoughtfully presented in the “limitations” section of a published paper. When carefully and cautiously read, such a paper can be very helpful. But the professional’s temptation is focus on a statistically significant, published result even if its practical import and relevance are low. Besides, it is rarely the author of a paper who tries to influence a practical discussion. Often professionals have not even read the original paper that influences them. They rely instead on their graduate training or abstracts and second-hand summaries of more recent research. The caveats in the original studies tend to be lost.

Of course, people with professional credentials can be excellent observers and assessors of whole objects like schools, neighborhoods, or firms. In some affluent communities, practically everyone holds an advanced degree and is therefore a “professional.” But their judgments of whole objects and situations are best when they think as experienced laypeople, not as specialists. They should draw on professional expertise, but only as one source of insight (and should not rely on only one profession).

Arguments about the proper role of generalization take place within professions, not just between professionals and laypeople. Physicians, for example, are being pressed to adopt “evidence based medicine,” which deprecates doctors’ intuitions and personal experiences in favor of general scientific findings, especially those supported by randomized experiments. Some medical doctors are pushing back, arguing that experimental findings never yield reliable guidance for complex, particular cases. What matters is the whole story of the particular patient.

The same argument plays out in education. The No Child Left Behind Act of 2001 favors forms of instruction proven in “scientifically-based research,” and the gold standard is a randomized experiment. (The frequently accepted second-best is a statistical model, which can be understood as an estimate of what would be found in a randomized experiment.) Like physicians, some educators resist this pressure, on the ground that an experienced teacher can and should make decisions about individual students and classrooms that are heavily influenced by context and only marginally guided by scientific findings.

This debate will never be fully resolved, but there is a logic to the idea that if we are going to train people in expensive graduate schools and rely on their guidance to shape general policies, they should be the bearers of “scientifically-based research.” In other words, the most optimistic claims about the value of expertise rely on a notion of the expert as an abstract and general thinker. When professionals are seen instead as experienced and wise craftspeople, no one exaggerates their role in public life. The physician who is a seasoned healer is left to treat his or her patients; it is the medical researcher with general findings who is invited to influence policy. My claim is that we err when we give such research too much credence.

We need expertise to make wise public decisions. You wouldn’t ask just any fellow citizen to operate on your heart; you would find the best-trained and most experienced cardiologist. In the same way, if you want to fix public schools or the justice system, you need economists, psychologists, criminologists, and other experts to advise and perhaps decide.

Everyone finds some merit in this argument, but it can be grossly exaggerated. For example, my friend Harry Boyte often quotes a speech by Donna Shalala that can be found in full here. Shalala was president of the University of Wisconsin in the 1980s and went on to eight years as Secretary of Health and Human Services under Bill Clinton. She epitomized the policy expert who attains public influence. In her 1989 speech, she began by citing scientific discoveries that had “improved human life, prolonged human life, enriched and protected human life. The great plagues are basically behind us,” she said, thanks to “scientific research done under the sheltering arms of research universities.” She went on to defend the “idea of a distinterested technocratic elite” that arrived in America before 1900 and shaped both the modern research university and government:

It persisted because everyone–farmers and professors and business owners and politicians and homemakers and workers–basically agreed on some important ideas: That those without wealth and power must be protected. That government must be open. That there must be some social control over those with huge economic strength. And that the government ought to be used as a tool to achieve social equity–to level the playing field for everyone. All acknowledged that the university’s experts could help secure those goals. And the rightness of those goals was held to be a notion that transcended politics.

Shalala ended her speech with a call for the university’s experts to take on the pressing challenges of the late twentieth century, especially persistent poverty and educational failure, with “grand strategies” grounded in apolitical social science. Four years later, she was running a huge federal agency responsible for health care. The administration she served devised a complex health reform bill, described as the work of a “distinterested technocratic elite.” It was quickly defeated; the major trends in public health remained basically unaffected.

When Donna Shalala was studying for her doctorate in public affairs during the 1960s, the “moon-ghetto” metaphor was popular. This was the idea that engineers and other specialists had put human beings on the moon (and brought them safely home), so it should be possible to tackle the problems of the so-called “ghetto” in much the same way. It was all a matter of scientifically diagnosing the causes of poverty and efficiently deploying solutions.

Actually, the moon and the “ghetto” are very different. The moon is almost perfectly detached from all other human issues and contexts, because it is almost 240,000 miles away from our planet (although NASA’s launch facilities in Florida and Houston might have some local impact). The goal of the Apollo Program–whether you endorse it or not–was clear and easily defined. The challenges were physical; thus Newtonian physics allowed engineers to predict the impact of their tools precisely in advance. The costs were also calculable–in fact, the Apollo Program was completed under budget. The astronauts and other participants were highly motivated volunteers, who had signed up for a fully developed concept that they understood in advance. The president and other national leaders had committed enough funds to make the Apollo Program a success, because its value to them exceeded the costs.

In contrast, a low-income urban neighborhood is enmeshed with other communities. Its challenges are multi-dimensional. Its strengths and weaknesses are open to debate. Defining success is a matter of values; even how to measure the basic facts is controversial. (For example, how should “race” be defined in a survey? What are the borders of a neighborhood?) Everyone involved–from the smallest child on the block to the most powerful official downtown–has distinct interests and motivations. Outsiders may not care enough to provide adequate funds, and residents may prefer to leave than to make their area better. When social scientists and policymakers implement rewards or punishments to affect people’s behavior, the targets tend to realize what is happening and develop strategies to resist, subvert, or profit from the policies–a response that machines never manage. No wonder we could put a man on the moon but our poor urban neighborhoods persist.

Thanks to personal computers, spreadsheets, and the World Wide Web, the resources and skills necessary to analyze social data have fallen by orders of magnitude since Donna Shalala was first trained in social science. Now anyone with a computer and basic knowledge of statistics can copy columns of numbers from official websites and look for correlations or more complex statistical relationships. Yet, if anything, we feel less confident about our ability to diagnose and cure social problems than we did in 1970. Shalala’s “grand strategies” have receded from view.

Although I acknowledge the value of expertise, we can identify several important general reasons why it is never enough and we always need citizens’ participation to tackle social problems.

First, professions cannot be trusted to make decisions for the public, even when their tools and techniques are appropriate and effective. Professionals are human, and if people outside their group turn to them for guidance but do not closely scrutinize their work, they are sure to become lazy, biased, careless, or even corrupt. In 1913, George Bernard Shaw wrote about the reluctance of doctors to testify against one another in malpractice suits. “The effect of this state of things is to make the medical profession a conspiracy to hide its own shortcomings. No doubt the same may be said of all professions. They are all conspiracies against the laity.” I think Shaw was joking, but there was a kernel of truth in his aphorism. Unless professionals are forced to justify their methods, assumptions, and conclusions in frequent, detailed, open discussions with laypeople, corruption is inevitable.

Second, social issues involve inescapable questions of value. It is not enough to know that A causes B. An engineer, an economist, or a biochemist might tell us that with some reliability. But we must also know whether B is desirable, whether A is an ethically acceptable means to B, and whether the cost is worthwhile. For example, you can often cause a low-income neighborhood to vanish by building a mass-transit station that links it to the downtown business core. Rents will rise around the station and poor people will move out. Crime will fall and investment will follow. Whether those changes count as “success” depends on your values, not on the data alone.

Third, experts are trained to think in terms of categories: to classify situations and then to recommend the rules, methods, solutions, or “best practices” that apply to each classification. There is value to thinking in categories, and experts do it better within their own fields than other people do. However, there are also serious limitations to categorical thinking, and laypeople often see a particular situation better than experts do. I believe this point has great significance for how we arrange our politics and will develop it tomorrow.