I’ve spent the last two days helping to advise the government on the design of background questions that will accompany the National Assessment of Education Progress (NAEP), a test-like survey of American kids. The background questions are about demographics and students’ experiences in and out of school. They help researchers to interpret and explain trends and gaps in the academic performance of our students.
The process for designing and administering the assessment is complex. It exemplifies the moral and practical complexity of government, and makes me wonder whether governance could be simpler. (I’m not sure that it could be.)
Take the example of how to categorize people by race and ethnicity. In the 1990s, people who considered themselves “multiracial” started lobbying for a separate category on the Census and other official data-collections. There were some objections, for instance, by people who didn’t want to reduce the count of African Americans because that would affect policies regarding legislative districts. Believe it or not, the Office of Management and the Budget (OMB) contracted with my colleagues and friends at the Institute for Philosophy & Public Policy to decide what to do. OMB recognized that the problem was basically moral, not empirical. My colleagues recommended that respondents be allowed to check as many categories as they liked. There would be no “multiracial” category, but researchers could construct it by counting all the respondents who checked more than one box.
That became federal policy, which means that NAEP must now use a racial/ethnic identity survey that allows respondents to “check all that apply. But that leaves many questions:
♦ Who should answer the questionnaire: students, parents, or educators?
♦ When should it be answered (e.g., while taking a test, or at some other time)?
♦ Must students who have already been categorized now be recategorized according to the new scheme, or can they be “grandfathered in”?
♦ How many separate racial and ethnic categories should be offered? What should they be called (e.g., Hispanic or Latino)?
♦ How quickly must the new scheme be implemented?
♦ How should trends in educational performance be calculated and reported after any changes in methodology? For instance, it has been proposed to separate Asians from Pacific Islanders. Historically, Asians perform much better on the NAEP (on average). Separating the groups will therefore boost Asians’ scores and create a bigger gap, especially between Asians and African Americans and Latinos. The impact of the change, however, will be small. Should the trends in performance be reported without comment, with a huge caveat to discourage the public from paying attention, or not reported at all?