Jan 19 2009
Mining for Meaning
This headline surprised me — Physical Disability Brings Marital Happiness.
The lead sentence asserted -
A new study finds that the onset of physical disability boosts marital happiness more often than not.
My first reaction was “interesting . . . who’da thunk it?!” The researchers reported that “in some cases” the disability to one partner brings more interaction between the two. I could also imagine it infusing more meaning and sensitivity into the relationship. Maybe.
But upon a closer reading, I wondered whether this preliminary finding should be viewed with even greater skepticism. Was it a product of data mining?
“Data Mining” is a concept that critical thinkers should be familiar with. It basically involves investigating and searching a whole number of variables and seeing if you can find a link between any two. It is quite commonly used in the health fields and the social sciences.
One amusing illustration was provided by a study conducted in Canada, if I remember correctly. The researchers crossed a whole number of hospital-patient variables (disease, symptoms, personal characteristics, etc.) with their astrological signs. They “discovered” some interesting things. Did you know, for instance, that Virgos are more likely to get nauseous during pregnancy? (Or some such thing.) By sorting through a deep enough pit of variables, the researchers were able to find a number of bogus statistically significant correlations — in this case between a person’s astrological sign and their health. But the study was a joke. In the first sense of the word.
How can joke results reach statistical significance? Via data mining. Cross enough variables and one or a number of your crosses will “jump out of the noise” by chance alone. Flip a coin enough times and by chance alone you’ll get “heads” ten times in a row.
Another example that comes to mind was one of the first on the power of intercessory prayer. It looked at how being prayed for might influence the recovery of cardiac patients. If I remember correctly, the researchers looked at a whole number of variables and found one that seemed to be influenced. No, the prayed-for group didn’t live longer or have a reduced risk of a second heart attack. They required less medication during their stay in the hospital. Or something. Follow-up studies failed to find the same benefit.
When encountering any study that looks at dozens of variables — be skeptical. Sure, they can suggest links and a direction for further, more pointed/controlled study. But you can’t draw any sure conclusions from them.
As for the finding that physical disability brings greater marital satisfaction, here is how the research was conducted -
The results are based on information provided by 1,217 married people randomly selected from around the country. Researchers tracked the lives of the study participants for 12 years. By study’s end, about one-fourth of participants – ranging from 36 to 75 years old – reported that either they or their spouse had permanent physical conditions that restricted activities such as dressing, bathing, eating or working around the house. The researchers zeroed in on this group, comparing their satisfaction in marriage before and after the physical disability occurred.
For the roughly three hundred couples that experienced a disability, “more often than not,” they reported greater marital satisfaction. Notice that we are given not a precise number but a “more often than not.”
Why did the researchers “zero in on it”? Was it the only variable that came up “heads” ten times in a row?
Saying “I’m skeptical,” doesn’t necessarily mean I doubt the finding. In fact, I wouldn’t be surprised if the finding holds up upon subsequent study. But I also wouldn’t be surprised if it doesn’t. What I’m waiting for is research that intentionally targets the variables in question. Then I’ll call it a finding.
If you weren’t searching for it, can you really claim you found it?
