We have been exploring the important differences between retrospective and prospective studies, clarifying that the latter offer the better data, albeit more slowly. Most of the examples in these discussions have involved observational studies. Also called naturalistic, this design is broadly used but suffers in comparison to randomized controlled clinical trials (RCT), the current gold standard for evidence.
With all research, some is better than others and we will begin with an example of a fairly good observational study that clarifies the value of walking as exercise and its impact on life expectancy.[1] Lead authors Amy A. Hakim, MS, and Helen Petrovitch, MD, utilized a prospective registry (the Honolulu Heart Program) to observe that, after adjustment for age, the mortality rate over 12 years for men who walked less than 1 mile per day was nearly twice that of men who walked more than 2 miles per day.
The main limitation of observational rather than controlled studies is the difficulty eliminating confounding factors – causes for the outcome other than the hypothesis. When we looked at cohort studies, we saw how researchers attempt to control for these by choosing subjects to observe that have matching characteristics, except for the hypothesized difference.
Hakim, et al., attempted to limit the effects of confounding by not including men who smoked, worked, or were not physically fit at the initial evaluation, so that these factors would not muddy their conclusions. Statistical adjustments were made for age, concentrations of total and high-density lipoprotein (HDL) cholesterol, blood pressure, presence of diabetes, alcohol intake and diet. As mentioned in the previous post, eliminating these confounding variables makes it an adjusted risk estimate. Additionally, the researchers lost only 5 original participants to follow-up (0.004%), a remarkable accomplishment considering even controlled studies have a major problem with study non-completers. Even so, the authors admit that selection bias may have played a role: healthier men may have survived long enough to enter the study, whereas less fit ones would have already died, or those who walked less had already come into contact with disease factors that limited their activity. Good studies always list limitations, even if they are difficult to identify.
Now let’s examine a poorer observational study: a prospective population-based study attempting to measure the onset of dementia in patients newly prescribed benzodiazepine medications (minor tranquilizers, commonly known as antianxiety medications, such as Xanax and Valium).[2] Review the study and see if you can find the problem.
While this study is prospective and does attempt to eliminate confounders (age, sex, educational level, marital status, wine consumption, existence of diabetes mellitus or high blood pressure, cognitive decline and use of statins, platelet inhibitors and oral anticoagulants), it trips up by not distinguishing between short term cognitive side effects and permanent dementia that may persist after a medication is stopped: “In the main analysis, we did not consider subsequent exposure to benzodiazepines when estimating the association between benzodiazepine use and occurrence of dementia.” Had this been an RCT, such a distinction would have been possible to build into the design. As this effort observes a community and attempts to draw conclusions, such surgical preciseness of observation was not possible. As a result, the results are useful only to confirm what specialists already knew: benzodiazepines can affect cognition while being used. Unfortunately, the authors claim it supports that new benzodiazepine use is associated with an increase chance of dementia.
It is disappointing to see such effort go into a study, only to be invalidated by a design flaw. This can occur in any study, including an RCT, it’s just more common when controlling for the variables is difficult. While it may be tempting to study populations and draw conclusions, even prospectively, RCT, though expensive and difficult, are more likely to provide a useful and repeatable result.
[1] Hakim AA, Petrovitch H, Burchfiel CM, et al.: Effects of walking on mortality among nonsmoking retired men. New Engl J Med. 338: 94–99, 1998, DOI: 10.1056/NEJM199801083380204. [2] Billioti de Gage S, Bégaud B, Bazin F, et al. Benzodiazepine use and risk of dementia: prospective population-based study BMJ 2012; 345: e6231