The Numbers May Add up Though the Conclusion Doesn't.
We hear and read dire warnings about the dangers lurking in the food stream, on the job, and in the home. It sometimes seems like everything is making us sick or killing us. Life itself seems dangerous, after all, none of us get out of it alive. But how much stock should we put in the daily reports of impending doom?
Our first example concerns the difference between absolute risk and relative risk. There's a difference between the two, and depending how you report a risk can make the difference between a scary sounding scenario and a relatively benign one.
Let's imagine a condition called fingernail blight. Let's say statistics show 13 in a thousand people who pick their noses get fingernail blight, while only 10 in a thousand non-nose-pickers get it. Here's what your typical headline in The Daily Gloom will read:
Picking Your Nose Increases Risk of Fingernail Blight 30%
Sounds like pretty strong evidence, sounds like nose picking is pretty dangerous. But is it? While the headline is correct, there is only a 13 in a thousand chance for nose-pickers getting fingernail blight, which means a 1.3% chance of contracting it. At the same time non-nose-pickers suffer at a 1.0% rate. Comparing those slim chances relative to each other one is 30% higher than the other. Still the absolute risk, the chances of getting fingernail blight has only increased by 0.3% from picking your nose, not the reported 30%. That's pretty slim pickings.
Besides, if ten in a thousand non-nose-pickers get it, we can't assume all 13 nose-pickers have gotten it from picking their noses. It could be random chance of whatever caused it in the non-nose-pickers and only 3 in a thousand people got fingernail blight from picking their nose. What's overlooked in all this is 99% of non-nose-pickers and 98.7% of nose-pickers don't get it at all.
After all, buying two lottery tickets rather than one doubles your chance of winning the lottery. Yeah, from one in ten million to two in ten million. Whoop-de-do.
Sifting through the evidence, some intrepid reporter at The Daily Gloom discovers something interesting in the incidence of fingernail blight relating to when people are born. Thus in a few days we have a follow-up to the previous story:
Being Born in July Increases Risk of Fingernail Blight 20%
Should you be worried if you were born in July? Not really. Again, this is a comparison of relative risk, not absolute risk. Furthermore, random distribution is not the same as even distribution. Rarely are conditions like fingernail blight distributed evenly through the population, it's almost always randomly. Below are two patterns, in the first the dots are evenly distributed, in the second they are distributed randomly.
Next we overlay a 12 section grid. Label each section with a month and you'll find some months have more dots than others on the random side, unlike the even side. There are 120 dots on the entire grid, so the average would be 10 per section. Still, July has 12 dots, a 20% increase over the average.
This is nothing more than a reflection of natural variability of randomness where there will be clusters here and there. These are not statistically meaningful until you start seeing twice as many or more in a given association. Then again, while July might be 20% higher than average, it might be only 3% higher than the number two month and even 40% higher than the month with the least number of fingernail blight cases. And if we are comparing a 12 in a thousand to a 10 in a thousand average it's a paltry 0.2% increase in absolute terms.
This is like the story of the Texas Sharpshooter. A guy shoots 60 rounds aimlessly at the side of a barn. He then finds the biggest cluster of hits and paints a bulls-eye around it, declaring himself an expert marksman. Of course, if he'd also painted his bulls-eye randomly he might turn out to be a lousy shot.
So, whenever you read scary headlines about a less than 100% increase in risk, if they are comparing small relative risks it shouldn't be alarming or even surprising. In fact, it might be statistical mumbo-jumbo that's both essentially true and essentially meaningless.
This leads us to another way news reports can be alarming because of a medical term that's largely misunderstood...
Risk Factor, an observed condition or trait which coincides with another condition, such as a disease.
The dirty little secret is a risk factor isn't necessarily a cause and effect link, it doesn't mean the thing described as a risk factor has caused anything. To date the media have reported over 1,000 risk factors for cancer. Does this mean there are 1,000 things proven to cause cancer? Not at all. Risk factors can be significant, possibly related, or meaningless.
Take this hypothetical scenario of traffic data gathered in The Republic of New Freedonia. As a percentage of miles driven: Fast drivers were in fatal accidents more often than slow drivers; Drivers who wore glasses were in fatal accidents more often than those who didn't; Drivers of green cars were in fatal accidents more often than drivers of other colors.
So in Freedonia we have three risk factors for being in a fatal car crash: driving fast, wearing glasses, driving a green car. Which seems significant, which appears possibly related, and which is likely meaningless? Are people who wear glasses getting in fatal crashes because of poor eyesight or maybe they drive faster as a group, or maybe they drink and drive more? The statistics don't really tell us. It might be another case of the Texas Sharpshooter, random chance.
Now I'll show you another way a risk factor can be misleading. Say statistics in Freedonia show drivers wearing sunglasses are as a percentage in fewer fatal accidents than those not wearing them. Would this mean sunglasses are protecting drivers? Or could it be the numbers are skewed by including fatal accidents at night when people don't wear sunglasses and often are driving home from the bar in a drunken stupor? In other words, a joint effect where both not wearing sunglasses and getting in fatal crashes are both effects of another cause, driving at night.
Joint Effect
Here's an example of how the joint effect can fool you. Take the case that there is a direct, very close correlation between how tall people are and their level of education. Taller folks are more educated than the shortest, and height and education level increase together. Here it is expressed as a graph:
Many readers may dismiss this out of hand. It seems outrageous, couldn't possibly be the case. But once filled in with the missing information everyone will agree it's true. For now, for argument's sake let's say it's true.
If we assume a cause and effect relation for this correlation, what can it mean? Are taller people smarter than shorter people? Does education make you taller? You might surmise either until you get the missing information, which is babies and toddlers are much shorter than adults and also have little or no education, while college graduates are almost certainly fully grown adults.
There is the underlying cause for both education level and being taller, growing up. Education and height are a joint effect of aging from birth to adulthood and going to school from kindergarten through college. You might say being tall is a risk factor for education level or vice-versa. All the same, even though they go hand in hand, one didn't cause the other.
When you come across one of these scary headlines or news teasers on television, there's usually more to the story. What are they really comparing? What's the actual risk? How was the study done? If it was a meta study, a study of multiple studies, what did they include or exclude and why? In the end, when there's more to the story it often means there's less to the story than meets the eye.
Related article on terrycolon.com: Misleading Indicators