The country needs and, unless I mistake its temper, the country demands bold, persistent experimentation. It is common sense to take a method and try it: If it fails, admit it frankly and try another. But above all, try something. The millions who are in want will not stand by silently forever while the things to satisfy their needs are within easy reach. We need enthusiasm, imagination and the ability to face facts, even unpleasant ones, bravely. We need to correct, by drastic means if necessary, the faults in our economic system from which we now suffer. We need the courage of the young. Yours is not the task of making your way in the world, but the task of remaking the world which you will find before you. May every one of us be granted the courage, the faith and the vision to give the best that is in us to that remaking!

Franklin D. Roosevelt, Oglethorpe University Commencement Address (22 May 1932)

I think FDR’s words apply today–to the letter–because our problems are as serious, and our understanding of what will fix them is as tentative and preliminary, as it was in 1932. Once again, we especially need a new generation to be social innovators. But I would distinguish between two forms of “experimentation.”

1. Looking for “cures,” replicable programs and strategies that are proven to work. This is a very popular goal today (consider the Department of Education’s What Works database). The major analogy is to bench science, and the hope is to find treatments for social ills that work as reliably as chemicals that kill harmful microbes on contact. A social entrepreneur is someone who invents a new solution, proves that it works, and helps it spread through society, rather like Jonas Salk with the polio vaccine.

2. Tinkering: setting up programs that embody one’s profound ethical commitments and theories of society and then experimenting in order to improve their impact or–if they consistently fail, scrapping them and moving on. (As FDR advised, sometimes you have to “admit [failure] frankly and try another” approach.)

One of the flash-points today is the proper role of randomized experiments with control groups. (See, for example, this New York Daily News story headlined, “200 families on brink of homelessness being treated like ‘rats in lab experiment'”.) Randomized experiments do raise ethical questions about the treatment of the people in the studies. But they can be handled ethically, and they are powerful tools for determining what works. I think they have special promise as part of a “tinkering” strategy.

Experiments are actually rather simple and accessible tools for practitioners to use in program-improvement. Imagine, for example, that you are running a program that depends on regular meetings, and you need good turnout. Who shows up seems to depend on when you schedule the meeting, but you aren’t sure whether recent changes in the meeting time have helped because there are many other factors in play (the weather, the subject of the meetings, and so on). You can answer the question with a simple experiment. Randomly divide the next ten meetings into two groups: say, evenings and Saturdays. Count the total number of people who attend the two categories. If one is much better than the other, go for that. No fancy math, survey design, or other research skills are required.

If many organizations routinely applied experimental designs, they could become more effective, and that would help society. Randomization is merely a tool; they would also need a general ethic of experimentation and rigorous self-review.

But often randomized experimentation is motivated by a faith in a shorter path. The hope is to identify big cures that can be quickly “taken to scale” after experiments prove they work. This hope is likely illusory.

A randomized experiment can prove that a social intervention works; its success was not caused by other factors, such as the participants’ enthusiasm. But an experiment cannot prove that the same intervention would work in different contexts. To find real “cures,” one would have to replicate success in many contexts. But experimental results rarely replicate when human beings and communities are involved.

One reason is that the roots of problems lie in human motivations and choices. Often, the chief culprits are not the people directly in view as one experiments, such as the delinquent adolescents who are enrolled in a program. The real fault may lie with policymakers, business leaders, and other people out of view. But in any case, human intelligence and will are involved. So when one intervenes, all the affected parties adjust and seek their own goals, often frustrating the intervention. Microbes don’t respond that way. It is true that bugs can evolve to develop resistance, but human change is deliberate and immediate, not evolutionary.

Besides, many of the most important factors that determine human well-being are not programs or interventions that we can possibly offer to people or assign them to. An example of a program would be a community-service opportunity that students can be required to take or else rewarded for choosing. That is worth assessing with a randomized experiment. But consider a community service activity that kids develop completely on their own. That might be more important and valuable than a program, yet there is no point in assigning individuals to such an experience; it is essential that they created it. Or consider a community, like the city of Somerville, MA, where I am writing this paragraph. There are good things and bad things about Somerville as a context for human development, and we should understand its pros and cons and tinker with its elements as citizens. But Somerville wasn’t designed by anyone, nor can people be assigned to live there (without changing it dramatically). Somerville emerged from three centuries of choices and work by countless powerful and relatively powerless people. If you try to replicate its positive aspects elsewhere, you will be creating something entirely different.

In short, I believe in “bold, persistent experimentation.” I take “experimentation” literally and believe that techniques developed for scientific experiments, such as randomized control groups, have value. But we shouldn’t fool ourselves into thinking that there are cures waiting to be discovered. Persistent experimentation is the key: constantly refining our programs and projects in the light of the best available evidence.