A good critical thinker asks smart questions—questions that lead to useful truths and effective actions. A question whose power is underappreciated is: "Compared to what?"
Of course, you can't ask this question unthinkingly. (Do these pants make my hips look big? "Compared to what?") You have to ask it at the right time. Consider the raft of ads that use incomplete comparisons. They say some product is "better" without telling you what it's better than. This cereal is 50% crunchier! This laxative works 25% faster! These tires last three times longer! In each case, a good question to ask is: Compared to what, meatballs? (Well, you can drop the "meatballs" part.)
1. An Apple A Day
Another good time to ask "Compared to what?" is when you're pondering a general causal claim (a claim of the form X causes Y that isn't specific to a single case.) Suppose I want to figure out whether an apple a day keeps the doctor away. (Eating an apple a day causes lower levels of illness.) As evidence, suppose I identify a lot of people who eat an apple a day and it turns out they're generally pretty healthy.
I'm guessing you're not convinced. Correlation is not causation. To know whether eating an apple a day prevents illness, you need to know what difference eating an apple a day makes to getting sick. Eating an apple a day prevents illness—compared to what? And the answer is obvious: Compared to not eating an apple a day. To support the causal claim "An apple a day prevents illness" you need evidence about two conditions:
So suppose I go out and identify a bunch of people who eat an apple a day and a bunch of people who don't, and I find that the apple-eaters are, on average, healthier than the non-apple-eaters.1 Now I have good evidence for my general causal claim, right?
I'm guessing you're still not convinced. There are too many potential differences between the groups that might account for why the apple-eaters are healthier. There are, in other words, too many confounding factors. For example, the apple eaters might also tend to have healthier lifestyles (diet, exercise) or live in healthier environments than the non-apple-eaters. And so maybe eating apples has nothing to do with better health. In fact, maybe the apple-eating makes people sicker but this is more than compensated for by the confounding factors. To eliminate these possibilities, you have to control for these confounding factors.
Researchers learn to control for confounding factors in various ways (e.g., with a randomized study). But we don't need to go farther into the guts of experimental methods. Let's just stick to a simple idea: To be confident in a general causal claim "X causes Y," you typically want to know what happens to Y when X is present as opposed to when X is absent, and other relevant factors are held constant.2
2. Asking "Compared to what?"
Let's see how you might apply "Compared to what?" fruitfully to some causal claims.
You're going to want to get clear about what's causing what. Each of the above claims can be put into the form X causes Y. You need to identify the X (prayer, shaving, premarital sex, teaching sign, having an abortion, giving participation awards) and the Y (better test results, fuller and thicker hair, unhappiness, increased intelligence, long-term psychological damage, being spoiled).
You're probably going to have some concerns. For example, on #1, you might wonder: "Are you saying prayer works for anybody? Or just for those who believe in God?" And on #4, you might wonder: "What do you mean by smarter? And how would you measure that?" For the purposes of this exercise, let's shelve these (often legitimate) concerns. In each case, try to interpret the claim however you think is most reasonable.
Once you're reasonably clear about the claim, you're going to want evidence about what difference X makes to Y. So you'll want to know what happens to Y when X is present (the eXperimental condition—get it?) and when X is absent, and everything else that's relevant is held constant.
Researchers have conducted good, controlled studies for some of the above claims. If you're interested, you can search out that information. This post isn't about what you should think. It's about how you should think: In the face of a general causal claim, you should ask, "Compared to what?" If you do, I think you'll find that getting good evidence for such claims is sometimes tricky.
3. Socratic wisdom
Being a good critical thinker can be hard. Socrates taught us that wisdom sometimes means admitting, "I don't know." Perhaps you have a strong opinion about (say) the effects of participation awards on kids' well-being. If your opinion is based on anecdote and speculation rather than controlled evidence, then perhaps you need a dose of Socratic humility.
But just because causal reasoning can be difficult doesn't mean it's impossible to get good evidence about the effects of (say) participation awards. Socratic wisdom does not license a lazy and comfortable "plague on both your houses" skepticism. For many important questions, even if perfect certainty eludes us, there are better and worse answers. Wisdom recommends that we apportion our confidence to a claim in proportion to the quality of evidence we have for it.
(Part 2 is coming soon.)
1. You might try to avoid this lesson by going biochemical. Instead of comparing apple-eaters to non-apple-eaters, we could study the effects of the chemicals in apples on the human body (or on chemicals in the human body). Whatever the merits of this move, it doesn't help you avoid "Compared to what?" You're still going to need evidence about the difference that the apple-chemicals make to body chemistry.
2. This is a simplification. In some studies, you might compare the effects of the relative amount of X on Y (e.g., whether eating more trans fats rather than less is related to illness). And retrospective studies will compare the histories of populations with and without Y.
I invite you to follow my very modest activity on Twitter.