CHAPTER 14
How to design a study that everyone will believe: Retrospective reviews

Jonathan Auten1 and Paul Ishimine2

1 F Hebert School of Medicine, Uniformed Services University of Health Sciences, Bethesda, MD, USA

2 Department of Emergency Medicine, UC San Diego Health System, San Diego, CA, USA

I can still recall the overwhelming sense of frustration generated by my first research project, a retrospective review of a poison control center database. The senior researcher carefully reviewed the protocol guidelines for abstraction, went over the data forms, and gave the junior researchers practice charts for review. I went into the project feeling well prepared, but, several days later, after reviewing hundreds of charts, I felt utterly defeated. Despite my best efforts and attention to detail, many abstraction sheets did not have full sets of variables and many charts were limited by incomplete follow-up. My mentor noticed my frustration and handed me an article by RS Hoffman titled, “Understanding the limitations of retrospective analyses of poison center data” [1]. In this article, the author explained, in an almost accepting and even-handed tone, that the barriers that I had experienced were common difficulties with conducting a retrospective review. The ability to understand these limitations offers a certain degree of confidence as you take a structured approach to designing and eventually interpreting a retrospective study. It is our hope that this chapter provides clarity on the strengths of retrospective reviews and the importance of understanding and accepting the limitations before moving forward.

Why are retrospective reviews held in a negative light?

You may have read a meta-analysis, participated in a journal club or been part of an academic discussion in which the merits of retrospective chart reviews are marginalized and dismissed. Someone will cite poor methodology, researcher bias and incomplete databases as reasons to avoid retrospective studies. However, those same clinicians will recommend HIV postexposure prophylaxis for a high-risk needle stick exposure, even though the initial evidence for these recommendations came from a case–control study [2]. Similarly, you may also find those same colleagues counseling a patient against smoking tobacco, even though the initial research regarding the deleterious health effects of smoking came from retrospective studies [3]. A quick glance at the venerable research pyramid shows that, while not at the apex, retrospective case control and cohort studies are part of the foundation of the current paradigm (Figure 14.1). A thoughtfully conceived and designed retrospective review can not only be believable, but may add significant value to the acute care and emergency medicine literature.

c14-fig-0001

Figure 14.1 Research pyramid.

(Source: Courtesy of Dylan Auten, 4 years old)

The starting point for designing a good retrospective study is to first understand the pitfalls of chart review. A 1996 study found close to one quarter of all emergency medicine research to be retrospective [4]. The authors of this manuscript reviewed 244 articles published in three major emergency medicine journals for eight key criteria of high-quality chart reviews and found inconsistent methods throughout these articles. They found little standardization of abstractor training, monitoring, or blinding. Inter-rater reliability was rarely mentioned and tested less than 1% of the time [4]. Changes to chart review methodology was suggested and adopted by at least one major emergency medicine journal [5]. Nearly a decade later, two separate articles followed up this sentinel review to ascertain the effect that greater awareness had on improvement of methodological flaws of chart review studies. Both articles found that within the international emergency medicine literature, only marginal improvements had been made [6, 7]. Abstractor training, meetings, monitoring, and blinding were still poor. Inter-rater reliability testing occurred in less than 15% of papers and descriptions of sampling methods and missing data management were often omitted.

With the existing bias against chart reviews, why spend the time?

Retrospective studies offer unique advantages to the researcher. Every researcher should go through the “FINER” (Feasible, Interesting, Novel, Ethical, Relevant) mnemonic when generating a research question (Chapter 3). A working knowledge of the strengths of retrospective design can assist you in picking the right project, as certain types of research questions are particularly well suited to this type of study (Box 14.1) [8]. Chart reviews do not require patients to be followed into the future, as events have already taken place. Retrospective studies also offer the distinct advantage of requiring less infrastructure and time to complete than prospective studies. You generally only need time, a well-designed concept, solid patient database, and fundamental understanding of the limitations and advantages of such research. A retrospective study may be the only ethical method for studying harmful treatments or bad outcomes. These types of studies are alluring to some, as they are generally less likely to be biased by outside sources like the pharmaceutical or medical device industry (Box 14.1).

There are several key advantages to retrospective design that can assist you in balancing the limitations of design and deciding whether your investment of time can generate believable results (Box 14.2). If the outcome of interest is uncommon, a prospective investigation would typically be too large to be feasible and favored measures of association, such as absolute risk reduction or number needed to treat, cannot be generated. However, if proper methodology is applied, retrospective research can be reported in terms of odds ratios or an estimate of relative risk [9].

What are the limitations to retrospective review?

Medical record reviews will rarely provide “the answer” or “the best treatment,” but they can offer unique perspectives that are limited in prospective studies. Furthermore, they can lead to prospective validation studies that generate more impactful answers; it is in this manner that retrospective research can add to the medical literature, by generating ideas or creating steps up the research pyramid.

The main question you should ask yourself is the same that your readership should ask: “Do I believe in the methodology of this study?” A study does not have to be perfect, but the researcher must understand the limitations and common errors associated with retrospective design (Boxes 14.3 and 14.4; Chapter 10).

The single largest limitation to medical record reviews is that documentation contained within the record is frequently incomplete or missing for the needs of record review for research [10]. This chapter began with a discussion on a retrospective poison center chart review. It was not uncommon during review to encounter ambiguous entries, such as, “the patient was hypertensive,” “CXR normal, “not sure of method of exposure”or “patient is altered.” There is no systematic method for recording chief complaints, capturing a full set of vitals or laboratory studies, conducting diagnostic workups, administering treatments, or assessing outcomes. This may be adequate for real-time patient care, but may be inadequate for study data collection. This problem is not limited to reviews in the emergency medicine and acute care literature, and may be further compounded by the advent of the electronic medical record, where greater room of transcription error is present [11]. The first limitation that you must accept and report is that almost all variables in a medical record (histories, physical findings, test results, discharge diagnoses, and outcomes) depend on how physicians chart their observations and synthesize findings.

The second most significant limitation is the degree of bias in a study. This directly affects the validity or generalizability of the study and occurs in addition to random error or chance [9, 12]. The best way to avoid bias is to understand how it can affect the outcome of a study. One of the best historical lessons on selection bias was presented in an article, titled “Coffee and Cancer of the Pancreas,” in 1981 in the New England Journal of Medicine [13]. The authors performed a case–control study of histologically confirmed pancreatic cancer from 11 Boston and Rhode Island hospitals. Study controls were selected from a group of patients without pancreatic cancer that were admitted by the same doctor at the same hospital as the study cases. However, these controls were selected from the gastroenterology doctor’s clinic and carried pre-existing conditions such as peptic ulcers that preselected them to be non-coffee drinkers. This selection bias created a control group that was not representative of the general population. The apparent association between coffee and pancreatic cancer has failed to be proven in subsequent prospective studies. In addition to selection bias, the absence of rules for abstraction, coding, review, and oversight creates a large degree of bias that influences the abstraction itself [4, 14] Creation and adherence to protocols, training, performance monitoring and performance review are the best ways to minimize this bias. The single greatest flaw in most retrospective research is the failure to control for this form of bias. Left uncontrolled, the results are difficult or impossible to interpret.

Missing variables are another source of error in these studies. Deletions of the cases or categories introduce additional bias into the study and need to be clearly addressed in the results and limitations sections of your paper. Sampling and measuring reliability are remaining sources of bias. If possible, chart abstractions should be performed between two and four people and inter-rater reliability should be measured (e.g., calculation of a kappa statistic). You should also clearly state within your methods how you arrived at your sample size. Was it a convenience sample or derived with the assistance of a statistician targeting a specific confidence interval?

Designing a believable retrospective study

Once you have generated your idea, performed a thorough literature search and further refined your research question, you will begin to consider your methodology. Carefully weighing the pro’s and con’s, you have decided to move forward with you research idea. For some of the reasons highlighted above you have chosen a retrospective study design. Now what? Firstly, consider what type of retrospective study fits your available data set. Retrospective studies are categorized into two types: non-analytical and analytical. Non-analytic retrospective studies can broadly be classified as descriptive studies. These studies do not try to determine any quantitative relationships but try to give us a picture of what is happening in a population. Types of non-analytic studies include case reports and case series. Some measures of association can be determined from these types of studies. Analytic studies utilize a comparison group as well as the intervention or exposed group [15]. These include case–control, matched case–control, and retrospective observational cohort studies. The difference between a case–control and a matched case–control study design is that the latter control subjects are selected so they resemble cases (Boxes 14.5 and 14.6). A retrospective cohort study can evaluate two groups, with and without a high risk exposure, to find associated disease or outcomes of interest (Figure 14.2). The opposite is true of case–control studies (Figure 14.3), where you start with the disease or outcome and look backwards for the exposure of interest. A case–control study will allow you to calculate an odds ratio to describe the observed measure of association. Retrospective cohort studies allow us to calculate further measures of association, like the relative risk.

c14-fig-0002

Figure 14.2 Retrospective cohort (start with exposure to find disease)

(Source: Courtesy of Lucas Auten, 3 years old).

c14-fig-0003

Figure 14.3 Case–control (start with disease to find exposure)

(Source: Courtesy of Lucas Auten, 3 years old).

Once you have decided between a non-analytical or analytical design you will need to design a method for record abstraction. There are clear published recommendations that assist in controlling for bias (Box 14.7) [4, 16]. In addition to these two separate guidelines, the STROBE statement represents an international initiative of epidemiologists, methodologists, statisticians, researchers and journal editors aiming to strengthening observational studies in the medical literature. The resources which can be found on their web site (http://www.strobe-statement.org/index.php?id=available-checklists) contains checklists for various types of retrospective studies [17–20]. As you move forward, acknowledge imperfections and limitations, control for obvious sources of bias external to the data set, and adhere to one of these guidelines. Your study may not be featured on your favorite podcast, but perhaps it can serve as the building block for some yet unproven advance in the practice of medicine.

References

  1. 1 Hoffman, R.S. (2007) Understanding the limitations of retrospective analyses of poison center data. Clin Toxicology, 45(8):943–945.
  2. 2 Cardo, D.M., Culver, D.H., Ciesielski, C.A., et al. (1997) A case-control study of HIV seroconversion in health care workers after percutaneous exposure. N Engl J Med, 337(21):1485–1490.
  3. 3 Doll, R. and Hill, A.B. (1950) Smoking and carcinoma of the lung. Br Med J, 2(4682):739–748.
  4. 4 Gilbert, E.H., Lowenstein, S.R., Kozoil-McLain, J., et al. (1996) Chart reviews in emergency medicine research: where are the methods? Ann Emerg Med, 27:305–308.
  5. 5 Anon (nd) Instructions for Authors: Chart Reviews. Ann Emerg Med. http://www.annemergmed.com/content/designs#chart; last accessed 7 May 2015.
  6. 6 Badcock, D.B., Kelly, A., Kerr, D., and Reade, T. (2005) The quality of medical record review studies in the international emergency medicine literature. Ann Emerg Med, 45(4):444–447.
  7. 7 Worster, A., Bledsoe, R.D., Cleve, P., et al. (2005) Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med, 45(4):448–451.
  8. 8 Lowenstein, S.R. (2005) Medical record reviews in emergency medicine: The blessing and the curse. Ann Emerg Med, 45(4). 452–455.
  9. 9 Guyatt, G., Jaeschke, R., and Meade, M. (2008) Why study results mislead: Bias and random error. In: G. Guyatt, D. Rennie, M. Meade, and D. Cook (eds) User’s Guide To The Medical Literature: Essentials of Evidence Based Clinical Practice, 2nd edn. McGraw-Hill Professional, pp. 77–84.
  10. 10 Schwartz, R.J., Boiseneau, D., and Jacobs, L.M. (1995) The quantity of cause-of-injury information documented on the medicaI record: An appeal for injury prevention. Acad Emerg Med, 2:98–103.
  11. 11 Burnum, J.F. (1989) The misinformation era: The fall of the medical record. Ann Intern Med, 110:482–484.
  12. 12 Worster, A. and Haines, T. (2004) Advanced statistics: understanding medical record review studies. Acad Emerg Med, 11:187–192.
  13. 13 MacMahon, B., Yen, S., Trichopoulos, D., et al. (1981) Coffee and cancer of the pancreas. N Engl J Med, 304(11):630–633.
  14. 14 Hess, D.R. (2004) Retrospective studies and chart reviews. Respir Care, 49(10):1171–1174.
  15. 15 Mann, C.J. (2003) Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J, 20:54–60.
  16. 16 Gearing, R., Mian, I., Barber, J., and Ickowicz, A. (2006) A methodology for conducting retrospective chart review research in child and adolescent psychiatry. J Can Acad Child Adolesc Psychiatry, 15(3):126–134.
  17. 17 Gregory, B.H., VanHorn, C., and Kaprielian, V. (2008) 8 steps to a chart audit for quality. Fam Pract Manag, 15(7):A3–A8.
  18. 18 Lerner, E.B., Zachariah, B.S., and White, L.J. (2002) Conducting retrospective emergency medical services research. Prehosp Emerg Care, 6(2 Suppl):S48–S51.
  19. 19 Rangel, S.J., Kelsey, J., Colby, C.E., et al. (2003) Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. J Pediatr Surg, 38:390–396.
  20. 20 STROBE Statement (2009) STROBE Checklists. http://www.strobe-statement.org/index.php?id=available-checklists; last accessed 7 May 2015.