import pandas as pd
import numpy as np
from sklearn import tree, preprocessingWe’ve just imported several libraries that we’ll use for our analysis. We use an alias, pd, for pandas, and the alias np for numpy. We now have access to all of the functions in pandas and numpy. We can choose to import all of the functions or just a few. From scikit-learn, we’ll import only two functions. One is named tree and the other is named preprocessing.
Next, let’s import the data from a comma-separated values (CSV) file that is also sitting somewhere on the Internet. Specifically, this CSV file is sitting on a server owned by Amazon Web Services (AWS). We can tell because the base URL of the file (the first part after http://) is s3.amazonaws.com. A CSV file is a file of structured data in which each column is separated by a comma. We’re going to import two different Titanic data files from AWS. One is a training data set, another is a test data set. Both data sets are in CSV format. Let’s import the data:
train_url =
"http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/train.csv"
train = pd.read_csv(train_url)
test_url = "http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/test.csv"
test = pd.read_csv(test_url)pd.read_csv() means “please invoke the read_csv() function, which lives in the pd (pandas) library.” Technically, we created a DataFrame object and called one of its built-in methods. Regardless, the data is now imported into two variables: train and test. We’ll use the data in the train variable to create the model, and then we’ll use the data in the test variable to test our model’s accuracy.
Let’s see what’s in the head, or the first few lines, of the training data:
print(train.head())
It looks like the data is twelve columns. The columns are labeled PassengerId, Survived, Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, and Embarked. What do these column headings mean?
To answer this, we need a data dictionary, which is provided with most datasets. The data dictionary reveals the following:
Pclass = Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
Survived = Survival (0 = No; 1 = Yes)
Name = Name
Sex = Sex
Age = Age (in years; fractional if age less than one (1). If the age is estimated, it is in the form xx.5)
Sibsp = Number of Siblings/Spouses Aboard
Parch = Number of Parents/Children Aboard
Ticket = Ticket Number
Fare = Passenger Fare (pre-1970 British pound)
Cabin = Cabin number
Embarked = Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)For most of the columns, we have data. For some column values, we do not have data. For PassengerId 1, Mr. Owen Harris Braund, the value for Cabin is NaN. This means “not a number.” NaN is different than zero; zero is a number. NaN means that there is no value for this variable. This distinction might seem unimportant for everyday life, but it’s crucially important in computer science. Remember that mathematical language is precise. For example, NULL indicates an empty set, which is also different than NaN or zero.
Let’s see what’s in the first few lines of the test dataset:
print(test.head())
As you can see, test has the same type of data as train, minus the Survived column. Great! Our goal is to create a Survived column in the test data that contains a prediction for each passenger. (Of course, someone already knows which passengers in the test data set survived—but it wouldn’t be much of a tutorial if the data set already contained the answers.)
Next, we’re going to run some basic summary statistics on the training dataset in order to get to know it a little better. When data journalists do this, we call it interviewing the data. We interview data just like we might interview a human source. A human has a name, an age, a background; a dataset has a size and a number of columns. Asking a column of data about its average value is a bit like asking someone to spell their last name.
We can get to know our data a bit by running a function called describe that assembles some basic summary statistics and puts them into a handy table, as follows:
train.describe()
The training dataset has 891 records. Of these, only 714 records show the age of the passenger. For the data we have available, the average age of the passengers is 29.699118; normal people would say that the average age is thirty.
A few of these statistics require interpretation: Survived has a min of 0 and a max of 1. In other words, it is a Boolean value. Either someone survived (1), or they didn’t (0). We can calculate an average, which turns out to be 0.38. Similarly, we can calculate an average for Pclass, or passenger class. Passengers’ tickets were for first, second, or third class. The average doesn’t literally mean that someone traveled 2.308 class.
Now that we’ve gotten to know our data a little bit, it’s time to do some analysis. Let’s first look at the number of passengers. We can use a function called value_counts to do this. Value_counts will show how many values there are for each distinct category in a column. In other words, how many passengers are traveling in each passenger class? Let’s find out:
train["Pclass"].value_counts()
1 216
2 184
3 491
Name: Pclass, dtype: int64The training data shows 491 passengers traveling third class, 184 passengers traveling second class, and 216 passengers traveling first class.
Let’s look at the numbers for survival:
train["Survived"].value_counts()
0 549
1 342
Name: Survived, dtype: int64The training data shows that 549 people perished and 342 survived.
Let’s see those numbers normalized:
print(train["Survived"].value_counts(normalize = True))
0 0.616162
1 0.383838
Name: Survived, dtype: float64Sixty-two percent of passengers perished, and 38 percent survived. In other words, most people died in the disaster. If we were to make a prediction about whether a random passenger survived, we’d likely predict that they did not survive.
We could stop here if we wanted. We just drew a conclusion that would allow us to make a reasonable prediction. We can do better, however, so let’s keep going. Are there any factors that might help improve the prediction? In addition to survival, we have some other columns in the data: Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, and Embarked.
Pclass is a proxy for the socioeconomic class of the passengers. That might be useful as a predictor. We could guess that first-class passengers got on the boats before third-class passengers. Sex is also a reasonable guess for a predictor. We know that “women and children first” was a principle used during maritime disasters. This principle dates to 1852, when the British HMS Birkenhead, a troop ship, ran aground off the coast of South Africa. It’s not a uniformly applied principle, but it’s recurrent enough to use for social analysis.
Now, let’s do some comparisons to see if we can find variables that seem predictive:
# Passengers that survived vs passengers that passed away
print(train["Survived"].value_counts())
0 549
1 342
Name: Survived, dtype: int64
# As proportions
print(train["Survived"].value_counts(normalize = True))
0 0.616162
1 0.383838
Name: Survived, dtype: float64
# Males that survived vs males that passed away
print(train["Survived"][train["Sex"] == 'male'].value_counts())
0 468
1 109
Name: Survived, dtype: int64
# Females that survived vs females that passed away
print(train["Survived"][train["Sex"] == 'female'].value_counts())
1 233
0 81
Name: Survived, dtype: int64
# Normalized male survival
print(train["Survived"][train["Sex"] == 'male'].value_counts
(normalize=True))
0 0.811092
1 0.188908
Name: Survived, dtype: float64
# Normalized female survival
print(train["Survived"][train["Sex"] == 'female'].value_counts
(normalize=True))
1 0.742038
0 0.257962
Name: Survived, dtype: float64We can see that 74 percent of females survived, and only 18 percent of males survived. Therefore, for a random person, we might adjust our guess to say that they survived if they were female, but not if they were male.
Remember that the goal at the beginning of this section was to create a Survived column in the test data that contains a prediction for each passenger. At this point, we could create a Survived column and fill in “1” (meaning “yes, this passenger survived”) for 74 percent of the females and “0” (meaning “no, this passenger did not survive”) for the remaining females. We could fill in “1” for 18 percent of the male passengers and “0” for 81 percent of the remaining males.
But we won’t, because that would mean assigning probable outcomes randomly based only on gender. We know there are other factors in the data that influence the outcome. (If you’re truly curious to see the nitty-gritty of how this is determined, I encourage you to look at the DataCamp tutorial or something similar online.) What about women traveling third class? Women traveling first class? Women traveling with spouses? Women traveling with children? This quickly becomes tedious to calculate manually, so we’re going to train a model to do the guessing for us based on the factors that we know.
To construct the model, we’re going to use a decision tree, a type of algorithm. Remember, there are a handful of algorithms that are standard in machine learning. They have names like decision tree, or random forest, or artificial neural network, or naive Bayes, or k-nearest neighbor, or deep learning. Wikipedia’s list of machine-learning algorithms is quite comprehensive.
These algorithms come packaged into software like pandas. Very few people write their own algorithms for machine learning; it’s much easier to use one that already exists. Writing a new algorithm is like writing a new programming language. You really have to care a lot and you have to devote a lot of time to doing it. I’m going to wave my hands and say “math” to explain what happens inside the model. Sorry. If you really want to know, I encourage you to read more about it. It’s very interesting, but it’s beyond the scope of the current discussion.
Now, let’s train the model on the training data. We know from our exploratory analysis that the features that matter are fare class and sex. We want to create a guess for survival. We already know whether the passengers in the training data survived or not. We’re going to make the model guess, then compare the guesses to reality. Whatever the percentage is that we get right is our accuracy number.
Here’s an open secret of the big data world: all data is dirty. All of it. Data is made by people going around and counting things or made by sensors that are made by people. In every seemingly orderly column of numbers, there is noise. There is mess. There is incompleteness. This is life. The problem is, dirty data doesn’t compute. Therefore, in machine learning, sometimes we have to make things up to make the functions run smoothly.
Are you horrified yet? I was, the first time I realized this. As a journalist, I don’t get to make anything up. I need to fact-check each line and justify it for a fact-checker or an editor or my audience—but in machine learning, people often make stuff up when it’s convenient.
Now, in physics you can do this. If you want to find the temperature at point A inside a closed container, you take the temperature at two other equidistant points (B and C) and assume that the temperature at point A is halfway between the B and C temperatures. In statistics … well, this is how it works, and the missing-ness contributes to the inherent uncertainty of the whole endeavor. We’ll use a function called fillna to fill in all of the missing values:
train["Age"] = train["Age"].fillna(train["Age"].median())The algorithm can’t run with missing values. Thus, we need to make up the missing values. Here, DataCamp recommends using the median.
Let’s take a look at the data to see what’s in there:
# Print the train data to see the available features
print(train)
If you read all of those hundreds of lines, bravo—but if you skipped ahead, I’m not surprised. I printed many rows of data here, instead of using a small subset, in order to illustrate what it feels like to be a data scientist. Working with columns of numbers feels value-neutral and occasionally tedious. There’s a certain amount of dehumanization that occurs when you deal only with numbers. It’s not easy to remember that each row in a dataset represents a real person with hopes, dreams, a family, and a history.
Now that we’ve looked at the raw data, it’s time to start working with it. Let’s turn it into arrays, which are structures that the computer can manipulate:
# Create the target and features numpy arrays: target, features_one
target = train["Survived"].values
# Preprocess
encoded_sex = preprocessing.LabelEncoder()
# Convert into numbers
train.Sex = encoded_sex.fit_transform(train.Sex)
features_one = train[["Pclass," "Sex," "Age," "Fare"]].values
# Fit the first decision tree: my_tree_one
my_tree_one = tree.DecisionTreeClassifier()
my_tree_one = my_tree_one.fit(features_one, target)What we’re doing is running a function called fit on the decision tree classifier called my_tree_one. The features we want to consider are Pclass, Sex, Age, and Fare. We’re instructing the algorithm to figure out what relationship among these four predicts the value in the target field, which is Survived:
# Look at the importance and score of the included features
print(my_tree_one.feature_importances_)
[ 0.12315342 0.31274009 0.22675108 0.3373554 ]The feature_importances attribute shows the statistical significance of each predictor.
The largest number in this group of values is the considered the most important:
Pclass = 0.1269655
Sex = 0.31274009
Age = 0.23914906
Fare = 0.32114535Fare is the largest number. We can conclude that passenger fare is the most important factor in determining whether a passenger survived the Titanic disaster.
At this point in our data analysis, we can run a function to show exactly how accurate our calculation is within the mathematical constraints of the universe represented by this data. Let’s use the score function to find the mean accuracy:
print(my_tree_one.score(features_one, target))
0.977553310887Wow, 97 percent! That feels great. If I got a 97 percent on an exam, I’d be perfectly content. We could call this model 97 percent accurate. The machine just “learned” in that it constructed a mathematical model. The model is stored in the object called my_tree_one.
Next, we’ll take this model and apply it to the set of test data. Remember: the test data doesn’t have a Survived column. Our job is to use the model to predict whether each passenger in the test data survived or perished. We know that fare is the most important predictor according to this model, but age and sex and passenger class matter mathematically too. Let’s apply the model to the test data and see what happens:
# Fill any missing fare values with the median fare
test["Fare"] = test["Fare"].fillna(test["Fare"].median())
# Fill any missing age values with the median age
test["Age"] = test["Age"].fillna(test["Age"].median())
# Preprocess
test_encoded_sex = preprocessing.LabelEncoder()
test.Sex = test_encoded_sex.fit_transform(test.Sex)
# Extract important features from the test set: Pclass, Sex, Age, and Fare
test_features = test[["Pclass," "Sex," "Age," "Fare"]].values
print('These are the features:\n')
print(test_features)
# Make a prediction using the test set and print
my_prediction = my_tree_one.predict(test_features)
print('This is the prediction:\n')
print(my_prediction)
# Create a data frame with two columns: PassengerId & Survived
# Survived contains the model’s prediction
PassengerId =np.array(test["PassengerId"]).astype(int)
my_solution = pd.DataFrame(my_prediction, PassengerId, columns = ["Survived"])
print('This is the solution in toto:\n')
print(my_solution)
# Check that the data frame has 418 entries
print('This is the solution shape:\n')
print(my_solution.shape)
# Write the solution to a CSV file with the name my_solution.csv
my_solution.to_csv("my_solution_one.csv," index_label = ["PassengerId"])Here’s the output:
These are the features:
[[ 3. 1. 34.5 7.8292]
[ 3. 0. 47. 7. ]
[ 2. 1. 62. 9.6875] …,
[ 3. 1. 38.5 7.25 ]
[ 3. 1. 27. 8.05 ]
[ 3. 1. 27. 22.3583]]
This is the prediction:
[0 0 1 1 1 0 0 0 1 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0 1 0 1 1 1 0 0 0 1 0 1 0 0 0 0 1 0 1 0 1 1 0 0 0 1 1 1 0 1 1 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 1 0 1 1 0 1 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 1 0 1 1 1 1 1 0 1 1 0 0 0 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 0 1 0 0 1 0 1 0 1 0 1 1 1 0 0 1 0 0 0 1 0 0 1 0 0 1 1 1 1 1 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 1 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 1 1 0 0 0 0 0 0 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 1 1 0 0 0 1 0 1 0 0 1 0 1 1 1 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 1 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 1 1 1 1 0 0 1 0 0 0]
This is the solution in toto:
Survived
892 0
893 0
894 1
895 1
896 1
897 0
898 0
899 0
900 1
901 0
902 0
903 0
904 1
905 1
906 1
907 1
908 0
909 1
910 1
911 0
912 0
913 1
914 1
915 0
916 1
917 0
918 1
919 1
920 1
921 0
… …
1280 0
1281 0
1282 0
1283 1
1284 1
1285 0
1286 0
1287 1
1288 0
1289 1
1290 0
1291 0
1292 1
1293 0
1294 1
1295 0
1296 0
1297 0
1298 0
1299 0
1300 1
1301 1
1302 1
1303 1
1304 0
1305 0
1306 1
1307 0
1308 0
1309 0
[418 rows x 1 columns]
This is the solution shape:
(418, 1)That new column, Survived, contains a prediction for each of the 418 passengers listed in the test data set. We can write the predictions to a CSV file called my_solution_one.csv, upload the file to DataCamp, and verify that our predictions were 97 percent accurate. Ta-da! We just did machine learning. It was entry level, but it was machine learning nonetheless. When someone says they have “used artificial intelligence to make a decision,” usually they mean “used machine learning,” and usually they went through a process similar to the one we just worked through.
We created the Survived column and got a number that we can call 97 percent accurate. We learned that fare is the most influential factor in a mathematical analysis of Titanic survivor data. This was narrow artificial intelligence. It was not anything to be scared of, nor was it leading us toward a global takeover by superintelligent computers. “These are just statistical models, the same as those that Google uses to play board games or that your phone uses to make predictions about what word you’re saying in order to transcribe your messages,” Carnegie Mellon professor and machine learning researcher Zachary Lipton told the Register about AI. “They are no more sentient than a bowl of noodles.”17
For a programmer, writing an algorithm is that easy. It gets made, it gets deployed, it seems to work. Nobody follows up. You maybe try turning the dials differently the next time to see if the accuracy seems to go up any. You try to get the highest number you can. Then, you move on to the next thing.
Meanwhile, out in the world, these numbers have consequences. It would be unwise to conclude from this data that people who pay more have a greater chance of surviving a maritime disaster. Nevertheless, a corporate executive could easily argue that it would be statistically legitimate to conclude this. If we were calculating insurance rates, we could say that people who pay higher ticket prices are less likely to die in iceberg accidents and thus represent a lower risk of early payout. People who pay more for tickets are wealthier than people who don’t. This would allow us to charge rich people less for insurance. That’s bad! The point of insurance is that risk is distributed evenly across a large pool of people. We’ve made more money for the insurance company, but we’ve not promoted the greatest good.
These types of computational techniques are used for price optimization, or grouping customers into very small segments to offer different prices to different groups. Price optimization is used in industries from insurance to travel—and it often results in price discrimination. A 2017 analysis by ProPublica and Consumer Reports found that in California, Illinois, Texas, and Missouri, some major insurers charged people who lived in minority neighborhoods as much as 30 percent more than people who lived in other areas with similar accident costs.18 A 2014 analysis by the Wall Street Journal found that customers were being charged different prices for the same ordinary stapler on Staples.com. The price was higher or lower based on the customer’s estimated zip code.19 Christo Wilson, David Lazer, and a team of other Northeastern University researchers found different prices were offered to customers on Homedepot.com and on travel sites depending on whether the users viewed the sites on mobile devices or desktops.20 Amazon admitted to experimenting with differential pricing in 2000. CEO Jeff Bezos apologized, calling it “a mistake.”21
In an unequal world, if we make pricing algorithms based on what the world looks like, women and poor and minority customers inevitably get charged more. Math people are often surprised by this; women and poor and minority people are not surprised by this. Race, gender, and class influence pricing in a variety of obvious and devious ways. Women are charged more than men for haircuts, dry cleaning, razors, and even deodorant. Asian-Americans are twice as likely to be charged more for SAT prep courses.22 African American restaurant servers make less in tips than white colleagues.23 Being poor often means paying more for necessities. Furniture on installment plans costs more than outright purchase. Payday loans have a far higher interest rate than bank loans. Housing is considered affordable if it takes 30 percent or less of a household’s monthly income, but poor renters are often stuck paying more for housing because of a variety of factors related to economic instability. “In Milwaukee, the majority of poor renters devote at least half their income to rent, and a third pay at least 80 percent,” sociologist Pat Sharkey writes in a review of two ethnographies, Matthew Desmond’s Evicted: Poverty and Profit in the American City and Mitchell Dunier’s Ghetto: The Invention of a Place, the History of an Idea.24 Inequality is unfair, but it’s not uncommon. If machine-learning models simply replicate the world as it is now, we won’t move toward a more just society. “The allure of the technology is clear—the ancient aspiration to predict the future, tempered with a modern twist of statistical sobriety,” law professor and AI ethics expert Frank Pasquale writes in The Black Box Society. “Yet in a climate of secrecy, bad information is as likely to endure as good, and to result in unfair and even disastrous predictions.”25
Part of the reason we run into problems when making social decisions with machine learning is that the numbers camouflage important social context. In the Titanic example, we picked a classifier, survival. We used features to predict our classifier, but there are other possible factors. For example, our Titanic dataset includes only age, sex, and the other factors. We built our predictor based on the information we had. However, because this was a human and not a mathematical event, there were other factors at work.
Let’s look at the night of the Titanic disaster. The Titanic received multiple warnings of ice from nearby ships over the course of the day on April 14, 1912. At 11:40 p.m., the ship hit an iceberg. Just after midnight, the Titanic’s captain, Edward John Smith, mustered the passengers and began to evacuate the ship. Smith issued an order: “Put the women and children in and lower away.” First Officer William Murdoch was in charge of the lifeboats on the starboard side. Second Officer Charles Lightoller was in charge of the boats on the port side. Each man interpreted the captain’s command differently. Murdoch thought the captain meant women and children first. Lightoller thought the captain meant women and children only. Murdoch let men onto the boats if all the nearby women and children had been loaded. Lightoller loaded all the women and children nearby, then lowered the boat even if it had empty seats. Both men let the boats down into the water even if the full capacity of sixty-five people had not been reached. There were not enough lifeboats for the people on board: Titanic carried only twenty boats for a ship rated to carry 3,547 people. The best records show that the ship carried a light load of 892 crew members and 1,320 passengers.
There is a potentially interesting test to be done on lifeboat numbers. Murdoch’s boats on the starboard side had odd numbers; Lightoller’s boats had even numbers. Men probably survived at a different rate according to their lifeboat number, because Lightoller, who was in charge of even-numbered boats, didn’t load men. However, the lifeboat number isn’t in the data. This is a profound and insurmountable problem. Unless a factor is loaded into the model and represented in a manner a computer can calculate, it won’t count. Not everything that counts is counted. The computer can’t reach out and find out the extra information that might matter. A human can.
There’s also the problem of false causality. If we did have the lifeboat numbers, from a computational perspective it might look like men in odd-numbered lifeboats had a better chance of surviving the Titanic disaster. If we made decisions based on data, we might decide that all lifeboats should be odd-numbered so that we could save more men in case of emergency. Of course, this is ridiculous; it was the officer, not the number of the boat, that made the difference.
Two young men also confound the pure mathematical explanation. Walter Lord’s A Night to Remember, a bestselling nonfiction account of the Titanic disaster, is a moving account of the ship’s last hours.26 Lord tells the story of Jack Thayer, a seventeen-year-old who boarded the Titanic in Cherbourg, France, after a long European holiday with his parents. Thayer made a friend on the ship, Milton Long, another young man his age traveling in first class. As the crisis on the ship intensified, both young men helped to get other passengers to safety. By 2:00 a.m., almost all the lifeboats had launched, with Long and Thayer handing women and children into the lifeboats. By 2:15 a.m., the last lifeboats had washed away in the swells. The ship was listing to port. There was an explosion; a wave crashed over the boat deck. Chef John Collins was standing on the boat deck holding a baby, helping a steward and a woman from steerage who was traveling with two children. He and the others were swept out to sea. The baby was torn out of his arms by the force of the wave.
Thayer and Long saw the chaos on the decks. Suddenly, the lights winked out; the water had reached the fireboxes in boiler room two. The only light came from the moon and the stars and the lanterns on the lifeboats slowly rowing away from the sinking ship. The second funnel collapsed with a crash. Thayer and Long looked around: the lifeboats were gone and no rescue ship was in sight. They realized the moment had come to jump. They shook hands. They wished each other good luck. Lord writes:
Long put his legs over the rail, while Thayer straddled it and began unbuttoning his overcoat. Long, hanging over the side and holding the rail with his hands, looked up at Thayer and asked, “You’re coming, boy?”
“Go ahead, I’ll be right with you,” Thayer reassured him.
Long slid down, facing the ship. Ten seconds later Thayer swung his other leg over the rail and sat facing out. He was about ten feet above the water. Then with a push he jumped as far out as he could.
Of these two techniques for abandoning ship, Thayer’s was the only one that worked.
Thayer survived by swimming to a nearby overturned lifeboat and clinging to it with forty others. He watched as the Titanic cracked in half, the bow and stern slipping under the water amid a field of debris. Thayer heard people crying in the water. It sounded like locusts, he thought. Eventually, lifeboat twelve picked up Thayer and the others from the icy water. Help arrived hours later. Thayer shivered in the lifeboat until 8:30 the next morning, when the passengers were rescued by the Carpathia.
Thayer and Long were young men of the same age, same physical ability, same social status, and absolutely the same opportunity to survive the disaster. The difference came down to a jump. Thayer leaped out as far as he could away from the ship; Long dropped down next to the ship. Long was sucked into the abyss; Thayer wasn’t. What I find unsettling is that whatever the computer predicts for Thayer or Long, it will be wrong. The prediction is based only on fare class, age, and sex—but what really happened was a difference of jumps. The computer just fundamentally misunderstands. Long’s death, the randomness of it, is why our statistical prediction of who survived and who died on the Titanic will never be 100 percent accurate—no statistical prediction can or will ever be 100 percent accurate—because human beings are not and never will be statistics.
This speaks to a principle called the unreasonable effectiveness of data. Unless you are alert to the possibilities of discrimination and disarray, AI seems like it works beautifully. One of my favorite explanations of the search to explain the world through computer science comes from a paper by Google researchers Alon Halevy, Peter Norvig, and Fernando Pereira. They write:
Eugene Wigner’s article “The Unreasonable Effectiveness of Mathematics in the Natural Sciences” examines why so much of physics can be neatly explained with simple mathematical formulas such as f=ma or e=mc2. Meanwhile, sciences that involve human beings rather than elementary particles have proven more resistant to elegant mathematics. Economists suffer from physics envy over their inability to neatly model human behavior. An informal, incomplete grammar of the English language runs over 1,700 pages. Perhaps when it comes to natural language processing and related fields, we’re doomed to complex theories that will never have the elegance of physics equations. But if that’s so, we should stop acting as if or goal is to author extremely elegant theories, and instead embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data.27
Data is unreasonably effective—seductively so, even. This explains why we can build a classifier that seems to predict with 97 percent accuracy whether a passenger survives the Titanic disaster and why a computer can defeat a human Go champion. It also explains why, when we look closely at what happens during the machine-learning process, the machine doesn’t take into account any of the flukes that humans know happen in real-life disaster situations. Data is very effective. However, the data-driven approach ignores a number of factors that humans think matter a great deal.
Law and society are set up to accommodate all of the things that humans think matter. Data-driven decisions rarely fit with these complex sets of rules. The same unreasonable effectiveness of data appears in translation, voice-controlled smart home gadgets, and handwriting recognition. Words and word combinations are not understood by machines the way that humans understand them. Instead, statistical methods for speech recognition and machine translation rely on vast databases full of short word sequences, or n-grams, and probabilities. Google has been working on these problems for decades and has the best scientific minds on these topics, and they have more data than anyone has ever before assembled. The Google Books corpus, the New York Times corpus, the corpus of everything everyone has ever searched for using Google: it turns out that when you load all of this in and assemble a massive database of how often words occur near each other, it’s unreasonably effective. Let’s take something simple. In n-grams, the word boat usually occurs near water, so the two are probably related. The probability is higher that boat is closer to water than to electorate or stink bug, so a search pulls up terms or documents related to boats and water rather than to boats and stink bugs. People generally talk about the same types of things and search for the same types of things, and common knowledge is really quite common. The machine is not really learning; the search process is just inspired by human learning. If you read the math, which is all posted online, it’s very clear that these calculations are not magic and are just math. The computer will get enough things right enough of the time that we may be tempted to call it mostly correct—but it will get things right for exactly the wrong reasons.
Because social decisions are about more than just calculations, problems will always ensue if we use data alone to make decisions that involve social and value judgments. Traveling first class on the Titanic meant someone was more likely to survive—but it would be wrong to deploy a model that suggests first-class travelers deserve to survive disasters more than people who travel second or third class. Nor should we do other things that derive from a flawed model like the one we created. Our Titanic model could be used to justify charging first-class passengers less for travel insurance, but that’s absurd: we shouldn’t penalize people for not being rich enough to travel first class. Most of all, we should know by now that there are some things machines will never learn and that human judgment, reinforcement, and interpretation is always necessary.