Clinical research can be broadly divided into two subsets: experimental research and observational research. The vast majority of new medical treatments and technologies are tested through experimental or interventional research, often in the form of randomized trials, before they are adopted into clinical use. In contrast, observational studies are primarily conducted on technologies after they have already been adopted and are being implemented in some sector of the healthcare community. Observational research occupies a critical niche within healthcare research that is complementary to experimental studies. Understanding the relative strengths, weaknesses, similarities, and differences between observational and experimental research is critical to accurately interpreting clinical research.
Observational research is a research in which the investigator cannot control the assignment of treatment to subjects because the participants or conditions are not being directly assigned by the researcher. Observational research examines predetermined treatments, interventions, and policies and their effects. In practical terms, observational comparative effectiveness research (CER) is typically conducted within one of two settings, either within registries or as subgroup analyses within randomized clinical trials. Registries are generally created with a specific disease, treatment, or population of interest, and can occur within a specific institution, network of institutions, or geographic region within which clinically relevant outcomes are recorded. Subgroup analyses within clinical trials include any subset for which patients are not randomly assigned. Because subgroups are not randomly assigned, subgroup analyses share all the strengths and weaknesses of conventional observational studies, such as confounding and multiple hypotheses testing, and provide a similar level of evidence.
In contrast to observational research, researchers in experimental studies directly manipulate or assign participants to different interventions or environments. A third type of research involves descriptive studies, which are conducted without a treatment and are neither experimental nor observational (1). This type of research is used in the initial exploration and characterization of a healthcare issue. Descriptive studies play no direct role in CER, whereas experimental and observational studies are important in both developmental and CER.
Recent focus on the importance of CER was reinvigorated with passage of the Patient Protection and Affordable Care Act (PPACA) in 2010 (2). From a practical research standpoint, this emphasis on CER makes it important to define and understand observational research within the context of CER.
The Agency for Healthcare Research and Quality (AHRQ), the lead federal agency responsible for improving healthcare quality in the United States, defines CER as research that provides “evidence on the effectiveness, benefits, and harms of different treatment options” (3). This evidence is generated via comparative studies of drugs, medical devices, tests, surgeries, or ways to deliver health care. We find such evidence in one of two ways: through experimental studies or through observational studies. Each of these approaches has different strengths and weaknesses, and each uses both overlapping and distinct methods.
In a CER context, the strength of experimental studies is that they provide the strongest evaluation or validation of a specific, well-defined intervention or treatment. In particular, experimental studies provide the highest level of evidence for evaluating new therapies, which includes the current “gold standard” in CER—the randomized controlled trial (RCT). In recent years, several examples have arisen in which observational studies have been confounded by selection bias, which have later been contradicted by RCTs (Table 10–1).
TABLE 10–1
Examples of randomized controlled trials contradicting observational data
One of the best-known examples of this is the effect of hormone replacement therapy on women’s health. Numerous observational studies had shown a cardioprotective benefit of this therapy against heart attack and stroke in women. However, a large randomized clinical trial later showed that women randomized to receive hormone therapy showed no difference in cardiovascular events, a reduced fracture risk, and significant increases in venous clot formation and embolic events (4). Another example is the concept that vitamin B12 and folate supplementation could reduce the risk of major coronary events. This hypothesis was initially supported by numerous observational data but was later refuted by the randomized Study of the Effectiveness of Additional Reductions in Cholesterol and Homocysteine (SEARCH) (5).
In both cases, the conclusion was that selection bias was likely confounding the results of observational studies. Women receiving hormone therapy appeared to differ systematically from women who were not receiving hormone therapy (prevalent user bias), and the previously observed association between hormone therapy and cardioprotection likely resulted from confounding by overall improved access to health care, patient demographics, and other unobserved phenomena associated with therapy and improved cardiovascular health. The use of hormone therapy in women and its effects on cardioprotection remain actively debated topics; nonetheless, they serve as important examples of discrepancies between observational studies and RCTs.
The strength of the experimental study—evaluation of a specific, controllable treatment, or exposure—is also responsible for its key weakness: being difficult, impractical, or even impossible to conduct in complex, poorly defined, or poorly controlled situations. This limitation becomes particularly apparent in complex or organization-level studies of healthcare systems, in unpredictable or emergency settings, and when examining factors that cannot be assigned for either ethical or practical reasons, such as patient demographics, genotypes, social factors, or disease status. A further area for caution with regard to experimental studies stems from the fact that they take place in tightly controlled clinical settings. An intervention tested by experts in tertiary-care academic medical centers might lack generalizability (i.e., not be applicable to general clinical practice or the general patient population), if the trial setting differs significantly from conditions in actual practice (Table 10–2).
TABLE 10–2
Examples of observational studies providing insight unavailable through randomized trials
For example, the Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines? (CRUSADE) registry of more 400 emergency departments and medical centers was designed to examine actual practices regarding management of patients with non-ST-segment elevation myocardial infarction (NSTEMI). The investigators observed an inpatient mortality rate of ~5% (6), roughly two to three times greater than the rates observed in three randomized clinical trials of NSTEMI treatments (7–9). This same analysis showed that in clinical practice, guideline adherence was not uniform and substantially affected patient outcomes; a 10% increase in guideline adherence was associated with a 10% relative decrease in mortality (6). In another example, implantation of cardiac defibrillators occurs more often in white men than in other demographics, with black women being about half as likely as to receive the same care (10). Finally, patients who are at highest risk of having an acute coronary event appear to be least likely in practice to undergo cardiac catheterization (11). The general conclusion to keep in mind when interpreting randomized trials, then, is that such trials often do not reflect general practice; patients seen in actual practice can be older, have more comorbid conditions, and not receive the standard of care that would be expected from academic medical centers. This is especially true for long-term therapies after discharge. Clinical trials in acute coronary syndromes typically stop at hospital discharge, even though observational studies of patient outcomes after hospitalization show that patients often stop taking their medications within 6 months after discharge (12).
Finally, experimental studies are costly in terms of both time and resources—a single trial can easily cost millions of dollars, take years to complete, and involve large numbers of clinical, administrative, and research personnel.
The observational approach to CER provides a complementary approach to randomized trials, with a different set of strengths and weaknesses. The strengths of observational research lie in the fact that it can be conveniently applied to actual practice patterns and any naturally (or unnaturally) occurring phenomenon. It is not limited to artificial, relatively limited, controlled research settings that may have questionable generalizability. Observational studies can reflect how an intervention is truly being used in general medical practice. In addition to the theoretical advantage of more generalized inferences, observational studies offer several practical advantages. They are typically much less expensive and easier to conduct with regard to practical, clinical, and ethical issues. As a result, they typically involve much larger sample sizes, which allow for increased statistical power. With the appropriate dataset, entire populations can be directly studied, such as through the use of registries or census data.
The key weakness of observational studies is that, by definition, they cannot randomly (or otherwise) assign an intervention to study participants. Because assignment of treatment is not under the control of the investigator, any observational study will always have the potential for confounding and thus drawing incorrect inferences from a study. Patients or study participants who receive an intervention can be and most often are systematically different from patients who do not receive the intervention. New investigational interventions are typically applied to patients for whom standard care has failed, often selecting for sick patients. On the other hand, new investigational tools are often applied only to patients who are healthier and believed to stand a chance of tolerating a new therapy. Unfortunately, there is always a potential for confounding in any observational study, making correct study design and accurate interpretation of paramount importance.
The two main benefits of observational studies over experimental studies, feasibility and generalizability, are balanced by their critical weakness, the potential for confounding. As a result, observational studies are generally considered a lower standard of evidence with regard to CER.
In general, CER is used to determine the relative effectiveness of one intervention versus another. When evaluating the efficacy of potential treatments, the randomized, double-blind, placebo-controlled trial is considered the strongest or highest level of evidence that an intervention is better than either a placebo or an alternative therapy (Figure 10–1) (13). There are several reasons for this, the most important of which is the fact that randomization removes the potential for confounding of the perceived effects of the intervention. Other benefits of the randomized, controlled, double-blind trial come from blinding of the study personnel to avoid evaluator bias and using a placebo to provide an accurate and fair control reference. Sometimes blinding is not used because it is inconvenient or impossible to blind either subjects or investigators—for example, when one treatment group is easily identified because its patients receive surgery or drugs that carry significant side effects. Blinding is important when outcomes are subjective, but it is less of an issue when “hard” end points are being studied (death, organ failure, etc.). Such outcomes are unlikely to be affected by the investigator’s opinion or mindset.
FIGURE 10–1. Hierarchy of clinical evidence. Adapted, with permission, from SUNY Downstate Medical Center, Medical Library of Brooklyn. Evidence Based Medicine Course. A Guide to Research Methods: The Evidence Pyramid. 2004. http://library.downstate.edu/EBM2/2100.htm. Accessed May 4, 2012.
Evidence hierarchies such as the one depicted in Figure 10–1 often refer to the evaluation of well-defined interventions in which the goal is to evaluate whether an intervention improves a specific outcome. Some researchers have suggested that different hierarchies should be used depending on the research goal; for example, when exploring complex, new, or relatively unknown phenomena, observational or even anecdotal studies can serve as initial steps in gaining knowledge and directing future study. Some investigators and researchers are split between the experimental and observational mindsets, but regardless of one’s background, it is important to appreciate the strengths and weaknesses of each. In the end, it may be best to consider all available evidence from different approaches. In CER, both experimental and observational studies are trying to assess the same thing—causality of an intervention (14).
In practice, observational studies of health often fall into one of three areas within healthcare research: epidemiology, health economics, and health services research. Distinctions among these areas have developed over time, largely because different research questions require different types of data, methods, and background knowledge to be successfully answered. Epidemiology has evolved around naturally occurring exposures, interventions, or differences that are distributed by forces outside the healthcare system, whereas health services research tends to focus on those directly or indirectly caused by a healthcare system. Health services researchers often further define their interests in quality, access, or costs. Health economics focuses on this last topic, examining the economics of healthcare costs, resource allocation, and business models of health care. Although each of these areas has developed a subject-specific set of knowledge, methods, data, and perspectives, these types of studies can overlap substantially. As with all observational research, they are interested in inferring causality so that we can learn something about health.
Several formal definitions are used to describe epidemiology. One of the most useful ways to define epidemiology functionally is by examining its word origin. “Epi” and “dem” come from the Greek for “upon” and “people,” respectively, to yield the definition “upon the people.” This probably best describes what makes epidemiology unique as a field within health research: epidemiology is the study of exposures or traits placed “upon the people” by nature or outside forces, usually in contrast to exposures chosen by or given to the people via policy, healthcare providers, or other direct intention.
Epidemiological work in health services research tends to focus on exposures or other interventions that cannot be controlled or for which there is not sufficient money, interest, time, or effort to control in a formal trial. No randomized trial has been conducted to show that smoking causes heart or lung disease, yet the association of smoking with these diseases, particularly lung cancer, makes smoking the most potent modifiable risk factor in health care to date. Effects of lead poisoning on child development, identification of carcinogens, the interplay between the environment and asthma, and the association of diet and lifestyle choices with health are all important areas of health knowledge that have been advanced through epidemiological studies.
Health services research as defined by Rosenbaum (1) is an “applied multidisciplinary field that uses scientific inquiry to produce knowledge about the resources, provisions, organizing, financing, and policies of health services at the population level.” Within the “iron triangle of health care”— quality, access, and costs—health services research tends to focus more on quality and access than on economics, but this is largely an artificial distinction caused by differences in methods. Health services research finds its use in areas of health care where the intervention of interest cannot feasibly be manipulated, such as demographics or distance to care, and it is critical for examining and guiding healthcare policy.
Examples of health services research include analyses of large healthcare systems’ practices or health policy changes. Health services research can be used to analyze the effects of national, state, or local government policies, differences between health systems, and effects of sociodemographics on care, health disparities, or practice patterns. These types of studies are numerous; they have been used to investigate how Medicare changes in reimbursement affect quality of care, to assess the adoption of new medical technology in general use, and to identify consistent associations between race and disparate health outcomes.
The economics of healthcare focus on costs, where costs can be measured in terms of money, time, resource use, quality of life, or some combination of these. Health economics research can be conducted on a global, national, regional, or institutional scale, depending on the stakeholders involved and the question being investigated. These studies are often observational; for example, they can be conducted after implementing a change in medical practice to evaluate its financial implications and long-term sustainability.
The most readily available examples of health economics research are typically those that examine the costs of health care and related expenses. Such research can focus within a healthcare entity to examine work-time-flow efficiency, billing practices, revenue generation, and effects of intramural policies. More societally focused examples of health economics research include population-level budgeting and cost-effectiveness analyses. For example, in April 2012, Medicare released its annual Trustees Report, which uses actuarial spending trends to calculate when Medicare will no longer be solvent under current practices (15). Cost-effectiveness analyses serve to assess and provide a common denominator regarding how much a group is willing to pay for a certain amount of benefit. For example, in the United States, treatments are generally considered to be reasonably cost-effective at a cost of $50,000 per quality-adjusted year of life (QALY) (16). Treatments or interventions that are the most cost-effective will, in theory, be adopted most readily and widely.
Although the focuses of specific fields differ within observational studies of health, these disciplines also have many similarities and overlapping features (17,18). All observational research seeks to establish causality (14) and inform practice. The same basic conceptual approach is used—a treatment, intervention, or exposure of interest is selected; the relevant outcome is identified; the outcome of the study is assessed by statistical analysis; and the magnitude of effect is used to assess implications. Similar challenges are faced in all observational research, including a potential for misinterpretation and poor study design. Again, most studies use the same basic concepts, even if they use different language (19). For example, the primary outcome of a study might be referred to as the dependent variable in some studies versus the primary end point in others. Observational research, regardless of the field, still generally uses the same statistical methods, with the only real difference being a focused appreciation for practical concerns (20).
Observational research is research in which the assignment of treatment to subjects is not directly assigned. This typically occurs within the context of disease or treatment registries or in nonrandomized subgroup analyses within randomized clinical trials. The key strength of observational studies is that they can assess care as it is delivered in clinical practice, not in the carefully controlled settings of randomized controlled trials. The caution in using observational studies comes from their inability to remove treatment bias and confounders, which can lead to spurious results. In the end, observational research forms a key component of health outcomes research that must be incorporated into CER, to fully understand how treatments studied under ideal circumstances will translate when applied to general practice.
1. Rosenbaum PR. Observational Studies. 2nd ed. New York, NY: SpringerVerlag; 2010.
2. U.S. Congress. Patient Protection and Affordable Care Act. 2010. http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/content-detail.html. Accessed May 4, 2012.
3. Agency for Healthcare Research and Quality. What Is Comparative Effectiveness Research. http://effectivehealthcare.ahrq.gov/index.cfm/what-is-comparative-effectiveness-research1/. Accessed May 4, 2012.
4. Hulley S, et al. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. JAMA. 1998;280(7):605-613.
5. Armitage JM, et al. Effects of homocysteine-lowering with folic acid plus vitamin B12 vs placebo on mortality and major morbidity in myocardial infarction survivors: a randomized trial. JAMA. 2010;303(24):2486-2494.
6. Peterson ED, et al. Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA. 2006;295 (16):1912-1920.
7. The PURSUIT Trial Investigators. Inhibition of platelet glycoprotein IIb/IIIa with eptifibatide in patients with acute coronary syndromes. N Engl J Med. 1998;339(7):436-443.
8. Yusuf S, et al. Effects of clopidogrel in addition to aspirin in patients with acute coronary syndromes without ST-segment elevation. N Engl J Med. 2001;345(7):494-502.
9. Ferguson JJ, et al. Enoxaparin vs unfractionated heparin in high-risk patients with non-ST-segment elevation acute coronary syndromes managed with an intended early invasive strategy: primary results of the SYNERGY randomized trial. JAMA. 2004;292(1):45-54.
10. Hernandez AF, et al. Sex and racial differences in the use of implantable cardioverter-defibrillators among patients hospitalized with heart failure. JAMA. 2007;298(13):1525-1532.
11. Tricoci P, et al. Temporal trends in the use of early cardiac catheterization in patients with non-ST-segment elevation acute coronary syndromes (results from CRUSADE). Am J Cardiol. 2006;98(9):1172-1176.
12. Eagle KA, et al. Adherence to evidence-based therapies after discharge for acute coronary syndromes: an ongoing prospective, observational study. Am J Med. 2004;117(2):73-81.
13. SUNY Downstate Medical Center, Medical Library of Brooklyn. Evidence Based Medicine Course. A Guide to Research Methods: The Evidence Pyramid. 2004. http://library.downstate.edu/EBM2/2100.htm. Accessed May 4, 2012.
14. Dowd BE. Separated at birth: statisticians, social scientists, and causality in health services research. Health Serv Res. 2011;46(2):397-420.
15. Boards of Trustees, Federal Hospital Insurance and Federal Supplementary Medical Insurance Trust Funds. 2012 Annual report of the Boards of Trustees of the Federal Hospital Insurance and Federal Supplementary Medical Insurance Trust Funds. 2012. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/ReportsTrustFunds/Downloads/TR2012.pdf. Accessed May 4, 2012.
16. Ubel PA, Hirth RA, Chernew ME, Fendrick AM. What is the price of life and why doesn’t it increase at the rate of inflation? Arch Intern Med. 2003; 163(14):1637-1641.
17. Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Med. 2008;5(3):e67.
18. Berwick DM. The science of improvement. JAMA. 2008;299(10):1182-1184.
19. Maciejewski ML, Weaver EM, Hebert PL. Synonyms in health services research methodology. Med Care Res Rev. 2011;68(2):156-176.
20. Kennedy PE. A Guide to Econometrics. 5th ed. Cambridge, MA: MIT Press; 2003.
