Often, people who say that they have experienced specific services or opportunities also report better outcomes–such as educational success, employment, or health–compared to similarly situated people who never received those opportunities. For example, according to a paper published by CIRCLE, young adults who were required to perform community service as part of their middle school coursework are 14 percent more likely to graduate on time from college, even when one compares them to people who are similar in respect to all the other factors measured in the survey, such as test scores, parental education, race, and gender. One could conclude that community service has a 14-point positive impact on college graduation.

Indeed, that effect is possible. But service-learning has not yet been tested with a more rigorous method of evaluation. The “gold standard” is a randomized experiment, in which some people are randomly assigned to receive an experience that others (the “treatment group”) don’t get. If assignment is random, then the difference in outcomes is a measure of the impact of the experience.

Experiences that appear highly beneficial in studies of whole populations often show modest or no results in experimental tests. This is such a common pattern that it requires some general reactions.

why experiments rarely show impact

Survey-based studies and experiments may produce divergent results because the people who receive opportunities have advantages that account for their later successes and that are not measured in surveys–such as motivation or perseverance, helpful networks, enrollment at subtly better schools, or ties to motivated teachers and other helpful individuals. These advantages explain their good outcomes and account for what we originally hoped were benefits of specific programs.

(Another possible reason for the failure of programs to “work” when tested in randomized experiments is that the control groups actually find alternative programs. If that happens, an experiment will miss real benefits. But it’s my general sense that this is a relatively rare explanation.)

the injustice of testing only some programs

We treat programs and institutions with profound inconsistency. Government-funded programs for poor people are expected to show long-term impact in “gold standard” experiments, but no one ever asks whether services provided to high-income people work–even when such services are publicly subsidized. For instance, my university provides four years of elaborate educational, social, recreational, health, and housing services for its undergraduates. No one imagines that the impact of that package of services would ever be tested in a randomized experiment. Universities like mine obviously confer advantages on the individuals who graduate; our graduates are preferred in the job market. Since they benefit (relative to others), they want the opportunity to attend. And since they have political and economic clout, they get the opportunities they want. But a randomized experiment might find that the social benefit of a Tufts education is small, especially if the control group got a BA at half the cost.

what we should do

Notwithstanding this serious unfairness, I believe we should expect programs aimed at poor people to “work” under rigorous tests. Political power in unequally distributed. Those without power never get much public assistance. Given limited funds, we need to spend every dollar well. To test programs experimentally is not punitive; it’s a matter of making sure that we really do good.

In fact, I’m in favor of widespread field experimentation, with the following caveats:

1. There is an appropriate life-cycle for programs. They shouldn’t be expected to “work” in randomized studies from Day One. There should first be a fairly long process of informal experimentation and adjustment. Such experimentation should be supported.

2. When many programs fail to show impact, we shouldn’t become generally pessimistic about social interventions. We are holding them to standards that we never use in the private sector or when assessing other types of government programs (such as weapons purchases, agricultural subsidies, or macroeconomic policies).

3. Benefits need not always be long-term. Many beneficial interventions wear off, and that is an argument for follow-up, not for canceling the programs. Besides, if a program actually makes life better for 13-year-olds, that seems like an important advantage even if they are not still better off when they are 20.

4. Experiments should be used to improve programs. That is much more promising than lurching from one untested strategy to another every time results are disappointing.

5. Randomized experimentation is a rather detached, arm’s length method. But it is also an easy method to explain. Participants should influence important aspects of the research, such as decisions about what to measure and interpretations of the results.

(From Princeton, NJ) The Dallas Morning News’ Laura Isensee talked to me and several experts on civic engagement about whether all those young people who voted at high rates in 2008 will remain involved between elections. One of the best things about her article is her recognition that it’s not all about what young people feel or think; it also matters whether the political system is structured to permit their participation. The story was picked up by the AP and ran in quite a few other papers.

Today, Congress passed the GIVE Act, also known as the Edward M. Kennedy Serve America Act, which will expand AmeriCorps by 170,000 positions and direct much of the service toward three national priorities: reducing the high school dropout rate, conserving energy, and providing health care to needy people. Contrary to some rumors floating around the rightward reaches of the blogosphere,* the program is completely voluntary and will, I’m sure, have to turn away most of its eager applicants. (Also, it funds independent nonprofits that provide service opportunities; it’s not really a national corps.) My full analysis is here.

The Act represents the biggest expansion of civilian service since the New Deal. In a way, the story of “service” has been continuous since the Civilian Conservation Corps of the 1930s. Its thread passes from the CCC to the Peace Corps, VISTA, the Points of Light Foundation, AmeriCorps, and USA Service Corps.

But I think the Kennedy Act represents a major milestone for a particular movement that arose in the 1980s to promote voluntary, educational service opportunities for young people. The Campus Opportunity Outreach League or COOL was founded in 1984, Campus Compact in 1985, Youth Service America in 1986, City Year in 1988, and YouthBuild USA in 1990. These groups and their supporters helped make voluntary national service a popular issue and achieved the National and Community Service Trust Act of 1993, which launched AmeriCorps. But after that launch, AmeriCorps always had to struggle with declining resources. The Kennedy Act puts it back on a growth path. Many of the leaders of the effort to pass it are veterans of the 1992-1993 struggle.

The immediate next step for the movement is to get the Act fully funded, and then help to implement it well so that the volunteers really learn and make a difference on the three important problems. That is a tall order by itself.

Unfortunately, the task is bigger still. As I’ve argued before, no national service bill can engage most Americans in productive civic work. AmeriCorps is open to citizens of all ages, but most volunteers will be young adults who can take a year for educational service or k-12 students who will benefit from the service-learning provisions. We must treat the Kennedy Act as a positive step and a momentum-builder, but not as an adequate way to fulfill the President’s pledge to make “service and active citizenship” a “central cause” of his administration.

*A Google search will reveal lots of comments that equate the bill with the Hitler Youth or with various Communist organizations. (Conflating Nazism with Communism seems to be a standard trope on the right.) In fact, if you search for “Obama active citizenship,” most of the hits will be right-wing responses that mention Hitler. This reaction is evidence of profound distrust for Democrats and for the federal government in some quarters. Moderate Republicans tend to like voluntary national service because of its moral orientation and its suggestion that citizens–not the state–can address public problems. But if you start with the assumption that federal authorities and Democrats want to steal all your rights, then a national service program for youth must indeed sound creepy. First big rallies, then a massive crowd on the National Mall shouting “O-BAM-a,” then the government buys banks and car companies, and finally a “national service” bill passes. It’s all enough to frighten you–if you start out as profoundly disaffected and suspicious. Such a reaction would be easy to mock, but I recommend taking the underlying fear quite seriously.

I just heard an anecdote: several candidates for president of a major university were asked about the purposes of the humanities. All but one talked in terms of “art appreciation.” As a result, the committee–which included scholars from the humanities–selected the one remaining candidate, who understood how to talk like a modern humanities professor.

This anecdote reminded me of a scientist whom I used to hear holding forth to his graduate students at a take-out restaurant in College Park. Once he advised them to take an art appreciation course to meet women. Apart from other problems with this advice, the University of Maryland does not offer art appreciation. It offers art history, a discipline that sees itself as just as rigorous as the natural sciences.

Within the humanities themselves, I think the prevailing view is almost the opposite of this scientist’s. Rather than teach “appreciation,” we teach critical distance. A major goal of a class in English or art history is to help students learn how cultural products are made and how they function so that the students shed their automatic reactions and assumptions. When we understand works of art and literature, sometimes we like them more, but sometimes less. The point of a humanities course is not to raise or lower approval, but to enhance critical understanding.

My own view is that critical distance is a moral stance, and often a good one; but it is only an example of a moral position. In general terms, the purpose of the humanities is ethical thought. What ethics demands is sometimes criticism, but sometimes it is tolerance, solidarity, or even appreciation. The sciences and social sciences provide information relevant to ethical choices, but they deal with “is,” not “ought.” Only the humanities address, in various ways, the questions of how one should live and how a society should be structured.

At CIRCLE, we are primarily interested in the civic engagement of disadvantaged or marginalized young people. One simple definition of that group is people who are not on course to attend college. This is a slideshow about them and how they relate to politics and communities.