assessment woes

I?m on the advisory board of a program for adolescents that?s organized by Temple University in Philadelphia, the Middlesex County Community College in New Jersey, and Georgetown University in Washington, DC. I went to Temple today to help plan the program?s evaluation.

This group faces the same problems that bedevil my colleagues and me when we try to evaluate our work with kids in Maryland. Their program is too short (at 50 total hours) to cause substantial changes in the kind of indicators that CIRCLE has collected. With so little instructional time, no one wants to spend hours on evaluation. Because it’s a fairly small group of students, any changes in their responses to a questionnaire between the start and conclusion of the program are unlikely to meet statistical tests of significance. The population in the three sites ranges from adolescents with criminal records (in DC) to 5-to-12 year olds (in New Jersey), so it makes no sense to combine all the sites? data. If students do improve, it?s impossible to tell whether the program is responsible. The best way to tell would be to recruit a larger group of students and to randomly assign some of them to participate in the program and some (the control group) to be assessed without participating. But there?s neither the money nor the will to organize a control group.

The goal of the project?s organizers is to make students more capable of sticking up for themselves politically. They want their students to become confident and to know where to go for political help. Graduates of the program should also be able to work effectively with peers in a political context. With these goals in mind, I suggested conducting the same educational exercise on the first and last day of the program, videotaping the results, and asking an outsider to reflect on any differences. Students would be asked to work in small groups to plan a response to a hypothetical local problem, such as a dangerous street corner or a lack of basketball courts. The small groups would report their plans to the whole class both in writing and orally. Between the beginning and the end of the 50-hour program, we would expect the students? political plans to improve; we would hope that they would become more optimistic about their chances of success; and we would expect them to share the planning, writing, and oral presentation more equitably within their small groups.