Chapter 9

Interpreting Information

It is more of a job to interpret the interpretations than to interpret the things.

—Michel de Montaigne

The public … demands certainties… but there are no certainties.

—Henry Mencken

How certain are we of the correct interpretation of the news? Fake and inaccurate news often provide interpretations that assert “proof” or “disproof” of a cause or result. These interpretations fail to acknowledge the uncertainty or probability of events and other threats or potential explanations that contribute to false conclusions.

We live in a world where we want certainty. Do we become upset when the weather forecaster cannot tell us, with certainty, if it will rain tomorrow? How do we plan for tomorrow if we cannot be certain?

Checklist “look for” indicators of appropriate methods of interpretation:

• ____Scores should indicate a measurement error (standard error of measurement) or how much a score could vary if measured again or was rated by others with a smaller measurement error suggesting more confidence that the score is accurate.
• ____Predicted or projected scores and outcomes should clearly indicate the method of estimation and the assumptions made.
• ____Predicted scores should include a measurement error in prediction (standard error of estimate) or how much the predicted score could vary with a smaller error suggesting more confidence that the predicted score is accurate.
• ____Interpretations are made indicating a level of probability or certainty rather than absolute “proof” or “disproof” with most commonly used statistics requiring at least 90% (.90 level of certainty) so that the probability of chance or another explanation is less than 10% (.10 for statistical significance).
• ____Current information is analyzed and interpreted rather than counterfactual or hypothetical “would have happened if.”
• ____Past events and documents are accurately and completely cited if used to interpret current events.
• ____Accurate information is provided for why other potential threats or alternative explanations to the accuracy of the information are irrelevant.
• ____Accurate information is provided for why results are relevant and can be generalized to a larger group, other situations, or other measures.
• ____Group statistics are not used to make a conclusion about each individual in the group.
• ____Statistics from one or a few individuals in a group are not used to make or deny a conclusion about the entire group with any sufficient level of certainty.
• ____Accurate information is provided for irrelevance or probability of other potential adverse results or “collateral damage.”
• ____Positive evidence is required so that burden for supporting a causal statement is on the person making a positive claim rather than the burden of proof to negate a claim of causation.
• ____Multiple potential causes, rather than a single cause, for an outcome are noted with varying probabilities for each.
• ____Interprets nonstatistical claims of “causation” by checking a timeline to determine a cause that comes BEFORE an outcome.
• ____Interprets potential nonstatistical “causation” for an outcome or statement consistent with the Federal Rules of Evidence where “probable cause” is when a sufficient probability of a cause exists based upon information from a reliable and trustworthy source.
• ____Interprets information regarding accuracy of potential nonstatistical claims of “causation” according to the following levels in the Federal Rules of Evidence:
  1. (1) “Preponderance of evidence” where a statement is more likely true than false,
  2. (2) “Clear and convincing evidence” where a statement is highly and substantially more probably true than false,
  3. (3) “Beyond a reasonable doubt” where a statement has no other logically plausible reason to not be true.

Checklist “look for” indicators of fake news in interpretation:

• ____Only a single dependent variable or outcome measure is reported to support a conclusion with no reasons provided for exclusion of also reporting other outcome measures that might provide contradictory results or conclusions.
• ____Scores do not indicate a measurement error and/or the error is large and contributes to difficulty determining a reliable or consistent interpretation of achievement or performance.
• ____Predicted or projected scores and outcomes do not indicate the methods and assumptions and/or the methods and assumptions may not be realistic.

Examples:

1. (1) Predicting the risk or probability of “recidivism”—or repeating a crime—is now being considered as a factor to set jail sentences for those convicted of a crime. However, the statistical methods used relied upon biased and faulty assumptions since race (black and Latino) contributes to increased probability of arrest and conviction. The consequence is that black and Latino defendants will be rated as riskier than white defendants and more likely to be given a longer jail sentence.

The researchers who developed this risk-assessment model have not released computational details to allow other researchers to determine the accuracy of the predictions (Eckhouse, 2017). Who says that statistics and methods of analysis are not important? These predictions or estimates are used to make decisions about liberty and how long a person remains in jail. It is important to determine the accuracy of any method of prediction.

(2) The lack of consistent assumptions supporting the notions of “dynamic scoring” for tax cuts increasing economic growth has already been presented. The state of Kansas has seen increased budget deficits from such tax cuts and is now having to consider raising taxes because the predictions for income and revenue increase have not been obtained.

Despite the lack of positive economic growth, Ohio and other states continue to follow this concept. The state of Ohio cut the state income tax rate during the past few years. The evidence, however, indicates that per capita personal income of Ohioans is unchanged at approximately 90% of the U.S. average, as it was before the earlier tax cuts (The Plain Dealer, 2017).

• ____Predicted scores do not include and/or have a large error of estimate for the predicted score.
• ____Interpretations are made indicating a “proof” or “disproof” as a certainty rather than as a level of probability.

Example:

An example is the statement about whether Saddam Hussain had weapons of mass destruction in Iraq. George Tenet, CIA director, was certain the information “proved” the weapons were in Iraq and stated, “Don’t worry, it’s a slam dunk” (Leibovich, 2004). No weapons of mass destruction were found, and thousands of military were killed or wounded and billions of dollars were spent on a war.

• ____Results from statistical analysis indicate a level of certainty less than 90% (.90) so that the level of probability due to chance or another cause is greater than 10% (.10).
• ____Current results are either ignored or interpreted with a counterfactual or hypothetical approach that estimates “would have happened if.”

Examples:

1. (1) How many times have we heard our friends, relatives, and even call-in radio programs claim that a result would or would not have happened “if” something else would have been done? This is often combined with the statement from our friends of “I just knew that was going to happen!” or “I told you so.”
2. (2) A concept of the “Butterfly Effect” is used to state that “if only” something had not happened in the past, then our present would be different. For example: “if only” something did not happen to the butterfly, then the bird would have eaten the butterfly and not have starved, then the cat would have eaten the bird and not have starved, then continuing up the food chain. Here is another example that we might see in sports: “if only” the team would have called a different play, then they would have won.

The “Butterfly Effect” is sometimes also called the “future of the past” where we state what our current situation would be “if only” something else had not happened. The difficulty with this concept is that many other potential events “could have happened” that would have changed the current situation.

• ____Past events and documents are inaccurately cited or reconstructed and used to interpret current events.
• ____Inaccurate or no information is provided for why other potential threats or alternative explanations to the accuracy of the information are irrelevant.
• ____Inaccurate or no information is provided for relevance to a larger group, other situations, or other measures.
• ____Inaccurate or no information is provided for irrelevance or probability of other potential adverse results or “collateral damage.”

Example:

A recently signed executive order changed the metric to calculate potential damage of carbon emissions upon climate change. Is there any potential collateral damage from change of consideration of carbon emissions upon the climate?

• ____Group statistics are used to make a false conclusion about every individual in the group (ecological fallacy).
• ____Statistics from one or a few events or individuals are used as “proof” or “disproof” to make a conclusion about all events or individuals.

Examples:

Here are two statements that we can hear illustrating the misinterpretation from a just single person or event:

1. (1) One cold day of weather will make some say: “Guess that proves there is no such thing as global warming.”
2. (2) One scientist might dispute or disagree with the consensus of over 97% other scientists and we might conclude: “Not all scientists agree so it must not be true.”
   • ____ Negative evidence where the burden for supporting the causation or claim is upon skeptics to disprove, so that if proof cannot explain or disprove, then the claim must be true.

Examples:

This interpretation error is common in fake news and conspiracy theories. Here is a typical comment: “You can’t prove it didn’t happen.”

We can add to that statement almost any fake news assertion or any conspiracy theory that anyone wants to develop. Let’s try a few and see how easy it is to have fake news so that the fake news story becomes difficult to debunk.

1. (1) “You can’t prove it didn’t happen”—that there was massive voter fraud.
2. (2) “You can’t prove it didn’t happen”—of a conspiracy to fix the election.
3. (3) “You can’t prove it didn’t happen”—[FILL IN THIS BLANK WITH ANY OTHER CLAIM OR CONSPIRACY THEORY FROM FAKE NEWS—Go ahead as this is what fake news does.]

The danger with this thinking is the difficulty of disproving the story so we then think the story is not “disproven” so it must be correct. Fake news extensively uses this technique as a basis for conspiracy theories.

• ____Attributes result to a single “cause” rather than to multiple causes with varying degrees of probability.

Example:

Multiple factors influence a child’s education including, for example, socioeconomic status, neighborhood, and complexity of language used in the home (Berliner & Glass, 2014). These outside-of-school factors appear to be the most significant contributors of “causes” of student achievement.

The relation between student achievement and socioeconomic status was reported from a study of 11,200 school districts across all fifty states. Approximately 200 million reading and mathematics test scores were examined over a five-year time frame. A strong positive correlation was obtained between student achievement and socioeconomic status of students in the district. In general, the higher the socioeconomic status of the school district, the higher the students’ achievement. Socioeconomic factors accounted for approximately three-quarters of the variation among achievement between school districts (Reardon, Kalogrides, & Shores, 2017).

The fake news is the assumption that the teacher is the single most important “cause” of student achievement. This assumption is a fundamental flaw and error in value-added models with various methods proposed for teacher accountability. Helping students to attain higher achievement may require more assistance and support from outside-of-school factors.

• ____Does not interpret potential non statistical claims of causation by examining a timeline so that a potential cause came after an event or outcome,
• ____Fake news interprets nonstatistical information as indicating “certainty” for the cause regardless of the source of information, rather than “probable cause” as consistent with the Federal Rules of Evidence procedures,
• ____Fake news can present either very large or small numbers to create a different emotional response and interpretation:

Example:

A national debt represented by a huge number of approximately $20 trillion creates a much different response than a small number that indicates interest on the national debt of 1.45% of the gross national product.

Importance of issue:

Different methods of interpreting the information can produce different cognitive and emotional responses that affect our decisions. We may believe or interpret information in many ways; but the obvious fact is that our decisions can have significant consequences.

We need to understand and apply appropriate methods of interpretation of the news. The unfortunate observation is how often the “look for” indicators of misinterpretations in fake news are in our daily news.

The reality is that studies cannot control for all other possible influences upon results and measurement and estimation errors in statistical analysis. These uncontrolled influences and measurement errors mean that we need to interpret any results or claim with a degree of uncertainty or probability rather than as absolute “proof” or “disproof.”

We may become more anxious making decisions based upon probability rather than certainty. This is uncomfortable for us, and we often become susceptible to wanting to understand, with certainty, the cause for an event. We often accept the reality of things not seen and accept undocumented or inappropriate information as evidence, if the information promises certainty of understanding. This may predispose us to believe in conspiracy theories—or other false stories—that provide a “definite” or “certain” explanation.

Certainty is an elusive goal. The paradox is that those who believe they are more certain of a cause may have less understanding of other more potentially significant causes.

Inaccurate and fake news often promises certainty—but at what price? The question remains as to how that promise can be true in relation to the many “look for” concerns noted so far in this book. The hope is that each of us will reduce the influence of fake news by interpreting information objectively, rather than subjectively.

Here are some of the most common ways fake news misinterprets information to make a statement with certainty:

• Biased sources of information,
• Contradictory results are not reported,
• Conclusions are only supported with partial information or information from flawed methods or statistical analysis,
• Conclusions based upon a limited or nonrandom sample of persons or events,
• Biased information collection instruments,
• Inappropriate generalization to other groups or to all individuals in a group,
• Lack of control for other possible causes for an event,
• Use of negative evidence argument where burden is upon others to disprove rather than prove a claim (we can’t prove it didn’t happen),
• Counterfactual argument which explains past events as what “would have happened if,”
• Not reporting a measurement error of scores or an estimation error in predicting scores,
• Constructing a claim to align with personal biases of an intended audience.

Discussion questions:

1. (1) Have students identify a fake news story that they read. Ask students to identify the specific ways that fake news misinterprets the information in the story.
2. (2) Ask students to compare the different ways that fake news misinterprets the information and evaluate which way is the most harmful.