Jan 30 2010
Study of AA Lacking Adequate Controls
Not all research is created equal. Consider the following bit of research as an illustration.
The headline: Attendance at Alcoholics Anonymous meetings may reduce depression symptoms.
Gotta give recognition and kudos for the use of the word “may.”
The methodology -
The researchers analyzed data from Project MATCH, a federally funded trial comparing three treatment approaches for alcohol use disorder in more than 1,700 participants. While participants in that study were randomly assigned to a specific treatment plan, all were able to attend AA meetings as well. Among the data gathered at several points during Project MATCH’s 15-month study period were participants’ alcohol consumption, the number of AA meetings attended, and recent symptoms of depression.
Good number of subjects. Three treatment plans with random assignment in each — good. But wait. All were able to attend AA meetings? This means AA attendance was voluntary, and, uh-oh, a giant, confounding variable has entered the building. Say hello to possible volunteer bias and the action of other, hidden variables. Because the study conclusion is about the effectiveness of AA, that is a huge problem.
The results -
Those participants who attended more AA meetings had significantly greater reductions in their depression symptoms, along with less frequent and less intensive drinking.
Positive results. But what do they mean? Can we state outright or even suggest that it was the attending of AA meetings, variable one, that caused a change to variable two, depressive symptoms? Actually, we can’t. Not with a reasonable degree of certainty.
Here’s why. This was no controlled experiment, with the attendance of AA meetings the independent variable. Attendance was voluntary, rather than manipulated. Which is a huge difference. When you manipulate a variable (group 1 attends x times a week, group 2 attends y times a week, group 3 doesn’t attend at all . . . ) you can test that variable’s effect open another variable or variables.
Furthermore, even if the research wasn’t of the experimental variety, it still could have been controlled better. Right up front we’ve got one massive control that is missing. Huge. Recall that the claim is that attending AA meetings may reduce (is at least associated with — that much we can state with confidence) depressive symptoms. But is the qualifier of AA before the meetings warranted? Do those specific type of meetings work? We don’t know. For there was no control group of subjects given the option to voluntarily attend some other sort of regularly occurring social event/meeting. It may well be that regularly participating in any social event reduces depressive symptoms, and the AA part has absolutely nothing to do it. Which is my suspicion.
My greatest critique of this research, and the vast majority of research into the alleged benefits of religious involvement, is the complete lack a secular control. Until a secular control is employed, we don’t know whether the particular ideology of a group is the actual vehicle that brings change, or if it is merely a superficial label placed over the more mundane truth.
“It may well be that regularly participating in any social event reduces depressive symptoms, and the AA part has absolutely nothing to do it.”
And it may well be that willingness to get involved with social events, or even with “help” groups, may be an early signal of a downswing in the depressive symptoms. And again, the circumstance that the event is an AA meeting may have absolutely nothing to do with it.