The imperative for sound clinical research continues to increase in response to the demand for evidence-based, guideline-driven medicine. To increase the quality of health care and improve clinical decision-making, innumerable studies have contributed to a hierarchy of evidence for each disease state. The randomized trial often stands at the pinnacle of this hierarchy, yielding the highest level of evidence through a blinded experiment of an intervention or treatment. Preceding observational studies often generate the equipoise and hypotheses for randomized trials. In other instances, observational research can be the best vehicle for generating the evidence. The experimentation inherent in a randomized trial might be unnecessary, inappropriate, or even impossible under certain circumstances. Political, legal, and ethical barriers also can prohibit randomized trials. Increasingly, the costs of large trials are creating another barrier to generating randomized evidence, reducing incentives for funding the head-to-head trials needed to understand the real-world effectiveness of novel therapies. It is therefore important, in the context of the emerging field of comparative effectiveness research, to understand the value and appropriateness of observational research versus randomized trials in answering different clinical questions (Figure 16–1). Prior observational studies, both successes and failures, can inform future efforts in comparative effectiveness.
FIGURE 16–1. Strengths and weaknesses of clinical trials versus observational studies.
Hierarchies of evidence are not the same across disease states. In prevalent conditions with substantial morbidity and mortality, public and private resources are often devoted to generating high-quality evidence from randomized trials. However, even in cardiovascular disease, the leading cause of death in the United States, guideline recommendations are largely developed from lower-level sources of evidence or expert opinion (1,2).
In unstudied subgroups, observational data often serve as guidance for clinical decision-making. For example, age is a significant determinant of outcomes for patients with acute coronary syndromes (ACS), and almost 25% of percutaneous coronary interventions (PCIs) in contemporary practice are performed in patients older than 75 years (3,4). However, most clinical trials exclude such patients. From 1991 to 2000, patients older than 75 years accounted for only 9% of trial enrollment, despite accounting for 37% of all patients with myocardial infarction (MI) in the United States (5). In place of randomized clinical trials, large clinical registries have provided much of the information that informs treatment patterns for this growing subgroup of patients undergoing coronary revascularization.
The American Heart Association (AHA) 2007 scientific statement on acute coronary care in the elderly (6) relied on the National Registry of Myocardial Infarction (NRMI), the Global Registry of Acute Coronary Events (GRACE) (7), and the Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines? (CRUSADE) national quality improvement initiative (8) to determine in-hospital and long-term outcomes for elderly patients (Table 16–1) (6). Complementing data from randomized trials (9–13), these registries indicated that elderly patients with ACS are at high risk for death and other adverse events and therefore derive greater benefit from revascularization than do younger patients. Observational data from these registries will continue to complement trial data by monitoring the safety and efficacy of ACS therapies in the elderly to better understand their utility in this understudied subgroup.
TABLE 16-1
Data sources for the American Heart Association’s scientific statement on acute coronary care in elderly patients
The results of clinical trials often diverge from observational data. For example, prior observational studies established total plasma homocysteine level as an independent cardiovascular risk factor (14–18). These findings were disproven by subsequent randomized studies, however, which showed that lowering homocysteine levels had no effect on recurrent cardiovascular disease in patients with acute MI or on major cardiovascular events in patients with vascular disease (19,20). Similarly, observational data suggested that vitamin E supplementation was associated with a reduced risk of coronary artery disease (CAD) and progression of coronary artery lesions (21–23). Subsequent randomized data indicated that long-term vitamin E supplementation did not affect cardiovascular outcomes (24).
The discrepancy between observational research and randomized trials is well illustrated by the case of hormone replacement therapy (HRT). After first receiving approval for the treatment of menopausal symptoms, HRT subsequently was found in numerous observational studies to have potentially beneficial effects in a variety of conditions. A 1992 meta-analysis in the Annals of Internal Medicine (25) recapitulated many of the observed benefits in observational studies, concluding that HRT should probably be recommended for women who have had a hysterectomy, who have CAD, or who are at high risk for CAD. Several meta-analyses, including the 1992 report, drew from the Nurses’ Health Study, in which 12,1700 female nurses aged 30–55 years completed mailed questionnaires about postmenopausal hormone use and medical history (26). The study estimated the relative risk (RR) of CAD with the use of HRT at 0.61 overall (95% confidence interval [CI], 0.52–0.71), with an RR of 0.55 (95% CI, 0.45–0.68) for women taking estrogen alone and 0.64 (95% CI, 0.49–0.85) for women taking estrogen plus progestin.
This benefit of HRT in women with or at risk for CAD was not ultimately seen in the Women’s Health Initiative (WHI), a randomized trial conducted from 1992 to 2007 (27). A complex trial of multiple strategies for prevention of common causes of morbidity and mortality in women, it was designed to test the efficacy of a low-fat diet, HRT, and calcium and vitamin D supplementation. The study’s hypothesis was that HRT would reduce the risk of CAD and bone fractures (although potentially increasing the risk of breast cancer). Over 15 years, WHI randomized 16,608 postmenopausal women aged 50–76 years who had not undergone hysterectomy to receive either combined estrogen and progestin or placebo. In a second study, it randomized 10,739 postmenopausal women aged 50–79 years who had undergone hysterectomy to receive estrogen alone or placebo.
The first study was terminated prematurely due to a statistically significantly increased risk of breast cancer with combination HRT. The second study was also terminated prematurely because the trial showed neither cardioprotective benefits nor increased breast cancer risk, with an unacceptably high risk of stroke. Compared with previously observed benefits for CAD in nonrandomized studies, WHI showed a statistically significant increase in the unadjusted rate of CAD among women treated with estrogen plus progestin (RR versus placebo, 1.29; 95% CI, 1.02–1.63) (Table 16–2) (25–32). The WHI finding of an increased risk for breast cancer, on the other hand, was concordant with both preceding epidemiological studies and data from nonhuman primates (33).
TABLE 16–2
Results from observational studies of combined estrogen–progestin replacement therapy and the Women’s Health Initiative
Observational studies can yield either smaller or larger risk estimates compared with randomized trials. However, observational data tend to underestimate harm. As a result, those that do identify a signal for harm can be useful as hypothesis-generating studies that have implications for safety surveillance. Many observational cohort studies have benefited from a long period of surveillance, such as the Nurses’ Health Study, but few are designed to capture clinical events early during therapy. In the Nurses’ Health Study, information was collected at 2-year intervals, a study design that could not identify women who both initiated HRT and had an acute MI within a 2-year period (26). Such women were considered nonusers of HRT, leading to overestimation of the rate of CAD in healthy women and ultimately a false conclusion about the cardioprotective benefits of HRT.
Several other biases can be inherent in observational study designs. Studies such as the Nurses’ Health Study are prone to selection bias when subjects are lost to follow-up. Classification bias can occur when personnel responsible for data collection know the status of each subject. In the case of HRT, indication bias further confounded the study. Until the mid-1990s, HRT was contraindicated in women with hypertension, diabetes, and cardiovascular disease. Thus, HRT users historically were healthier, were more educated, were of a higher socioeconomic status, and had fewer cardiovascular risk factors. Blinded randomization in trials can preclude such biases and minimize the confounding caused by residual differences between cohorts.
Current trends are placing a greater importance on the appropriate use of observational data. First, technology is enabling the rapid acquisition of real-world clinical data, which can then be used for observational analyses. Electronic health records are facilitating surveillance of the efficacy and risks of established and emerging therapies. The US Food and Drug Administration, for example, is piloting an active safety surveillance system called Mini-Sentinel, which can actively monitor approved medical products using electronic health information from more than 60 million people (34).
Second, observational analyses of such real-world data will also play an important role in understanding therapeutic risks and benefits through direct comparisons of treatment options. To date, many randomized trials are studies of efficacy rather than effectiveness relative to alternative options. In outlining the national priorities for comparative effectiveness research, the Institute of Medicine has articulated the need to improve the quality of health care through not only the pragmatic design of randomized trials but also the judicious use of observational data (35). Proper recognition biases and unmeasured confounding can allow observational data to provide the best understanding of treatment options when trials are not possible.
Third, when trials are possible, observational analyses of electronic health information can be used to accelerate clinical discovery and spur future trials. Studies conducted within online communities using matching algorithms are emerging as another means of monitoring the safety and effectiveness of therapies (36). This merging of randomized trials with the environment of online communities and registries emphasizes the continuing need to learn from prior observational studies to improve the future of clinical research.
1. Lloyd-Jones D, et al. Executive summary: heart disease and stroke statistics–2010 update: a report from the American Heart Association. Circulation. 2010;121(7):948-954.
2. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301(8): 831-841.
3. Singh M, et al. Trends in the association between age and in-hospital mortality after percutaneous coronary intervention: National Cardiovascular Data Registry experience. Circ Cardiovasc Interv. 2009;2(1):20-26.
4. Bauer T, et al. Predictors of hospital mortality in the elderly undergoing percutaneous coronary intervention for acute coronary syndromes and stable angina. Int J Cardiol. 2011;151(2):164-169.
5. Lee PY, Alexander KP, Hammill BG, Pasquali SK, Peterson ED. Representation of elderly persons and women in published randomized trials of acute coronary syndromes. JAMA. 2001;286(6):708-713.
6. Alexander KP, et al. Acute coronary care in the elderly, part I: Non-ST-segment-elevation acute coronary syndromes: a scientific statement for healthcare professionals from the American Heart Association Council on Clinical Cardiology: in collaboration with the Society of Geriatric Cardiology. Circulation. 2007;115(19):2549-2569.
7. GRACE Investigators. Rationale and design of the GRACE (Global Registry of Acute Coronary Events) Project: a multinational registry of patients hospitalized with acute coronary syndromes. Am Heart J. 2001;141(2):190-199.
8. Hoekstra JW, et al. Improving the care of patients with non-ST-elevation acute coronary syndromes in the emergency department: the CRUSADE initiative. Acad Emerg Med. 2002;9(11):1146-1155.
9. Metz BK, et al. Randomized comparison of direct thrombin inhibition versus heparin in conjunction with fibrinolytic therapy for acute myocardial infarction: results from the GUSTO-IIb trial. Global use of strategies to open occluded coronary arteries in acute coronary syndromes (GUSTO-IIb) investigators. J Am Coll Cardiol. 1998;31(7):1493-1498.
10. The PARAGON Investigators. International, randomized, controlled trial of lamifiban (a platelet glycoprotein IIb/IIIa inhibitor), heparin, or both in unstable angina. Circulation. 1998;97(24):2386-2395.
11. Mukherjee D, et al. Promise of combined low-molecular-weight heparin and platelet glycoprotein IIb/IIIa inhibition: results from Platelet IIb/IIIa Antagonist for the Reduction of Acute coronary syndrome events in a Global Organization Network B (PARAGON B). Am Heart J. 2002;144(6): 995-1002.
12. The PURSUIT Trial Investigators. Inhibition of platelet glycoprotein IIb/IIIa with eptifibatide in patients with acute coronary syndromes. N Engl J Med. 1998;339(7):436-443.
13. Simoons ML. Effect of glycoprotein IIb/IIIa receptor blocker abciximab on outcome in patients with acute coronary syndromes without early coronary revascularisation: the GUSTO IV-ACS randomised trial. Lancet. 2001;357 (9272):1915-1924.
14. Homocysteine Studies Collaboration. Homocysteine and risk of ischemic heart disease and stroke: a meta-analysis. JAMA. 2002;288(16):2015-2022.
15. Arnesen H, et al. Sustained prothrombotic profile after hip replacement surgery: the influence of prolonged prophylaxis with dalteparin. J Thromb Hae-most. 2003;1(5):97-15.
16. Nygård O, et al. Plasma homocysteine levels and mortality in patients with coronary artery disease. N Engl J Med. 1997;337(4):230-236.
17. Boushey CJ, Beresford SA, Omenn GS, Motulsky AG. A quantitative assessment of plasma homocysteine as a risk factor for vascular disease. Probable benefits of increasing folic acid intakes. JAMA. 1995;274(13): 1049-1057.
18. Wald DS, Law M, Morris JK. Homocysteine and cardiovascular disease: evidence on causality from a meta-analysis. BMJ. 2002;325(7374):1202.
19. Lonn E, et al. Homocysteine lowering with folic acid and B vitamins in vascular disease. N Engl J Med. 2006;354(15):1567-1577.
20. Bønaa KH, et al. Homocysteine lowering and cardiovascular events after acute myocardial infarction. N Engl J Med. 2006;354(15):1578-1588.
21. Stampfer MJ, et al. Vitamin E consumption and the risk of coronary disease in women. N Engl J Med. 1993;328(20):1444-1449.
22. Rimm EB, et al. Vitamin E consumption and the risk of coronary heart disease in men. N Engl J Med. 1993;328(20):1450-1456.
23. Hodis HN, et al. Serial coronary angiographic evidence that antioxidant vitamin intake reduces progression of coronary artery atherosclerosis. JAMA. 1995;273(23):1849-1854.
24. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. N Engl J Med. 2000;342(3):154-160.
25. Grady D, et al. Hormone therapy to prevent disease and prolong life in postmenopausal women. Ann Intern Med. 1992;117(12):1016-1037.
26. Grodstein F, et al. A prospective, observational study of postmenopausal hormone therapy and primary prevention of cardiovascular disease. Ann Intern Med. 2000;133(12):933-941.
27. Rossouw JE, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women’s Health Initiative randomized controlled trial. JAMA. 2002;288(3):321-333.
28. Grodstein F, Newcomb PA, Stampfer MJ. Postmenopausal hormone therapy and the risk of colorectal cancer: a review and meta-analysis. Am J Med. 1999;106(5):574-582.
29. Grodstein F, et al. Prospective study of exogenous hormones and risk of pulmonary embolism in women. Lancet. 1996;348(9033):983-987.
30. Grodstein F, Stampfer MJ. The epidemiology of postmenopausal hormone therapy and cardiovascular disease. In: Goldhaber SZ, Ridker PM, eds. Thrombosis and Thromboembolism. New York: Marcel Dekker; 2002.
31. Grodstein F, Clarkson TB, Manson JE. Understanding the divergent data on postmenopausal hormone therapy. N Engl J Med. 2003;348(7):645-650.
32. Marchioli R, et al. Early protection against sudden death by n-3 polyunsaturated fatty acids after myocardial infarction: time-course analysis of the results of the Gruppo Italiano per lo Studio della Sopravvivenza nell’Infarto Miocardico (GISSI)-Prevenzione. Circulation. 2002;105(16):1897-1903.
33. Cline JM, et al. Effects of hormone replacement therapy on the mammary gland of surgically postmenopausal cynomolgus macaques. Am J Obstet Gynecol. 1996;174(1 Pt 1):93-100.
34. Behrman RE, et al. Developing the Sentinel System–a national resource for evidence development. N Engl J Med. 2011;364(6):498-499.
35. Committee on Comparative Effectiveness Research Prioritization, Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. Washington DC: National Academies Press; 2009.
36. Wicks P, Vaughan TE, Massagli MP, Heywood J. Accelerated clinical discovery using self-reported patient data collected online and a patient-matching algorithm. Nat Biotechnol. 2011;29(5):411-414.
