face validity and value-judgments

It’s very common in psychology (and in all the disciplines influenced by psychology) to construct scales that measure mental constructs. We can’t directly observe confidence, responsibility, spatial awareness, or any type of intelligence. But such mental constructs can exist even if they can’t be observed–unless the behaviorists were right, but they seem mostly to have disappeared.

The standard psychological method is to generate a list of survey questions (or checklists to be used by observers) that seem to measure some aspect or component of the mental construct that’s being studied. These items are said to have “face validity”–on their face, they are relevant to the mental construct of interest. These questions are then asked of a sample of people. The questions that cluster together statistically are kept; the outliers are discarded. The list is reduced to a small set of questions that can explain most of the variance in the results. Often, items can be separated into different categories that do not correlate with each other. Then one concludes that the mental construct actually includes several underlying “factors.”

I see the value of doing this, in part because I do believe in empirically identifiable mental constructs that aren’t directly observable. However, as a philosopher, my instinct is to see decisions about “face validity” as basically value judgments. For example, if we listed the behaviors and attitudes that make someone an “engaged citizen,” the reason to include each item would be our belief that the particular attitude or behavior was good. Voting is on CIRCLE’s list of civic indicators, and to defend that choice, we owe an argument about why people should vote. Whether voting correlates with volunteering or protesting may be interesting, but it isn’t necessarily the point. Either voting is part of civic engagement, or it isn’t; the reasons have to do with our sense of how a society should function.

My concern, in short, is that psychological research may look more scientific than it is, and the really important questions may be value-judgments buried at an early stage of the empirical method.