Deadly Data: Sometimes Numbers Lie
Blindly trusting in data can be deadly.
Posted Sep 09, 2012
Data. It’s something we draw upon in science, policymaking, and any number of other settings. It’s important. But there are times when blindly trusting in data can be devastating. Let me give you an example.
One day, when I was in Junior High, shortly after I was hit by a car, my family and I drove past the location where I’d been hit. In the intersection was a car, spun at an awkward angle, windshield shattered, sides bashed in, with the doors sticking out at unnatural angles. Clearly totaled.
Someone in the car spoke up, “Wow! That’s the second bad accident we’ve seen in as many weeks. It seems like there are accidents here all the time…I wonder why nobody does anything? Especially after what happened to you!” I wondered this, too.
My father wondered more. It stuck in his brain. He couldn’t let it go. Understandably. The intersection was directly in front of a school. It was part of the main route to school for most of the kids in my neighborhood.
My cousins walked this route. Many of my classmates walked it. Worst of all, my father knew that when I could walk again, I’d have to walk it. What if it was unsafe, as our observations (and my experience) would seem to suggest?
He started asking questions. Wasn’t there some mechanism in place to identify unsafe numbers of traffic accidents in any given intersection? He was told that yes, there was. The administration reviewed all the accident reports filed, and kept tallies of the number of accidents within each intersection. Any numbers above a certain threshold would be investigated, and mitigating action would be taken.
My father obtained a copy of the report. He looked up the intersection listed on the accident report that the police had filed the day of my accident. But the numbers didn’t make any sense…he came home shaking his head. He ran the numbers by the rest of the family, and they didn’t seem to make sense to us either. They seemed vastly out of step with the number of accidents we’d personally witnessed. Were we overcounting? Or was something really wrong with the report?
So, my father asked more questions – and stumbled on an interesting little fact. The street on which my accident had occurred was called “Division.” It was a name I had never thought much about, but it was in fact called this for a reason. The street served as the boundary between two different townships. Our house was in one township, the school with is in the other.
My father’s questioning brought to light another aspect to this. It seemed that at one point, a debate had broken out regarding where on the road the official dividing line should be placed. At issue was the question of resources. It was a busy four lane road, lined with many businesses and homes…who would bear the cost of its upkeep? Who would provide the law enforcement resources to patrol it?
The dispute was finally settled, as many political disputes are, through compromise. The townships would split the cost, drawing the official dividing line right down the middle of the road. Thus, any accident or incident that occurred on one side of the yellow line would be the jurisdiction of one township, and any that occurred on the other side would be the jurisdiction of the other.
Since the traffic reports were compiled by jurisdiction, the numbers my father was given were only half the story. They only reflected accidents that occurred on the same side of the yellow line that mine did. So, my dad paid a visit to the other township’s administrative offices, and got a second report.
But there was something odd here, too. Despite what he was told, there was no entry listed for the intersection named on my accident report. After a few more questions, my father would learn why. The intersections were named according to the streets that existed within the police officer’s jurisdiction.
The intersection in question was a T intersection – so on one side of the yellow line the cross street didn’t “exist” in that township’s jurisdiction. It was on the wrong side of the yellow line. As a result, it was given a different name. Armed with this knowledge, my father was able to aggregate the full statistics for the intersection.
Individually, the statistics for each jurisdiction were under the safety threshold. But taken together, they told the story we’d been observing for months. The intersection was a death trap – and I was only one of its victims.
In the years since, I’ve wondered what true cost of this statistical failure really was. For me, there are the injuries that I sustained (which still cause me trouble today), and the missed school. For my step brother, it was the time and attention that wasn’t available to him, because it was tied up in taking care of me.
For my parents, it meant a number of things. Having to give up their room, because I could no longer take the stairs to my room. The cost of installing an emergency call button, and other household modifications. The lost sleep, when they had to get me up at night to go to the bathroom. The time spent assisting me in the everyday efforts, that were once done independently, with ease.
For my aunt and uncle, it meant the cost of building a ramp onto their home, so that I could get into their house each morning. And the effort of helping me get around during the day, when my parents were at work. For my friends, it meant bringing me home my homework. For my interim teacher, an extended family member who volunteered, it mean the time and effort to run house calls
I was just one number on that list. How many more costs, emotional, physical, and fiscal, were a result of this statistical anomaly?
Through his tenacity, my father taught me a number of important lessons. First of which, is that you don’t need to be the most educated or the most eloquent person in the room, to have something to say. He was a former record store clerk turned insurance underwriter, with only a high school education. He struggled with people, and often seemed to say the wrong thing. Groups of people were overwhelming to him. He was often dismissed as odd or strange. Sometimes worse.
Yet, when it came to the safety of the neighborhood’s children – he was the only one who stood up to the administration and asked the hard questions. The right questions. Most of the people who were impacted by it, probably never knew anything about what he had done. They may have scoffed at him in a supermarket line, or commiserated behind his back about his eccentricities. But he didn’t do it for them, and he didn’t do it for fame. He did it because it was the right thing to do.
He taught me that when something is wrong, when something or someone is hurting someone else, the right thing is to do something. Sometimes that means making making people mad, or standing up against those in authority. Often it means holding on to something long beyond the point that people say you should let it go.
Inherent in my family’s experience is a cautionary tale for anyone who works with data. When you’re the person setting up a program – whether it’s a policy, a research study, or any other data intensive project – it’s important to carefully consider the unintended consequences of the decisions you make, and the definition of your data. And if you’re the one making important decisions based on the data, make sure you ask the right questions.
Sometimes, numbers do lie – and people get hurt as a result.
- The Atlantic: Lies, Damned Lies, and Medical Science
Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice?
(Coming soon: Part two of Labels and Expectations series)