We've all heard the saying, "there are lies, there are damn lies, and then there are statistics." God knows I have heard this same quote (ad nauseum) hundreds of times when I have discussed research findings. But statistics do not lie. They can't even talk (or sing, or perform an interpretative dance, or draw, or otherwise get a lie across).
The problem isn't with the statistics. It is with people's inability to interpret stats accurately. A case in point (albeit perhaps not a serious one) I heard on ESPN yesterday. Michael Vick (the quarterback for the Philadelphia Eagles) was commenting at a press conference that he does not get the same penalties called in his favor as other NFL quarterbacks. The ESPN commentator stated that Michael Vick had received more roughing the passer penalties than any other quarterback over two years, so he does not have a case. This was reported on the network's myriad of shows throughout the day as evidence that Vick had no case.
This is a classic example of people misunderstanding even the simplest statistics. In short, the ESPN commentators were taking a total amount of penalties in Vick's favor and ignoring how many calls he should have gotten in his favor. This is not the same thing at all.
Imagine that you hear that a police officer gives 100 speeding tickets a week, and that another gives 10 speeding tickets a week. You might want to assume that the first cop is doing a more strict job delivering tickets. But what if you heard that the first cop (the 100 tickets) worked in a town of 100,000 people, and the second cop worked in a town of 300. Percent wise, the second cop is giving out more tickets. It would not be fair to conclude that the first cop is more harsh.
Likewise, it isn't accurate to say that because Vick gets more calls in his favor than any other quarterback, he is not right to "complain" that the referees are not calling the game fairly for him. A much better estimate would be the amount of times he is say, hit after the pass or in his helmet (both illegal), and how many times it is called, relative to other quarterbacks. If Vick gets 5 calls a game, and should get 9, but say, Tom Brady should get 3 and they call 2, then the referees are not being as harsh on defenses who hit Vick (illegally) as thay are for Brady. If this was the case for most NFL quarterbacks, then Vick would have an adequate point.
But because people do not seem to (often) understand statistics, you have an entire, large, television audience being mislead. (And this is ESPN, so imagine what cable news networks do with statistics to mislead people).
To emphasize, the problem isn't the statistics. It is the people reporting them and understanding those reports that are the issue. It is true that Vick gets more calls in his favor than all other QBs. But this does not mean that he is favored by refs (though it could), and he still could be getting a raw deal from the officials.
To take another example, I heard Rush Lumbaugh the other day (thanks for the sympathetic pity when you read that I had to listen to him. I appreciate it) state that "climate change is a hoax." But, the actual study he was using to argue this, and the scientist he had on, argued that it is perhaps not as strong as originally thought. That is not saying it is a hoax at all. It is taking an incremental argument based on data (it is less than thought) and making it absolute (it is a total hoax).
There are lies, damn lies, and then there are people who cannot understand statistics. But that isn't the fault of science or the statistics.
