Consider trying to learn the next stock price from several past stock prices. Suppose you have 20 years of daily stock data. Discuss the effects of various ways of turning your data into training and testing data sets. What are the advantages and disadvantages of the following approaches?
Take the even-numbered points as your training set and the odd-numbered points as your test set.
Randomly select points into training and test sets.
Divide the data in two, where the first half is for training and the second half for testing.
Divide the data into many small windows of several past points and one prediction point.
Figure 13-17 depicts a distribution of "false" and "true" classes. The figure also shows several potential places (a, b, c, d, e, f, g) where a threshold could be set.
Draw the points a–g on an ROC curve.
If the "true" class is poisonous mushrooms, at which letter would you set the threshold?
How would a decision tree split this data?
Refer to Figure 13-1.
Draw how a decision tree would approximate the true curve (the dashed line) with three splits (here we seek a regression, not a classification model).
Draw how a decision tree would fit the true data in seven splits.
Draw how a decision tree would fit the noisy data in seven splits.
Discuss the difference between (b) and (c) in terms of overfitting.
Why do the splitting measures (e.g., Gini) still work when we want to learn multiple classes in a single decision tree?
Review Figure 13-4, which depicts a two-dimensional space with unequal variance at left and equalized variance at right. Let's say that these are feature values related to a classification problem. That is, data near one "blob" belongs to one of two classes while data near another blob belongs to the same or another of two classes. Would the variable importance be different between the left or the right space for:
decision trees?
K-nearest neighbors?
naïve Bayes?
Modify the sample code for data generation in Example 13-1—near the top of the outer
for{}loop in the K-means section—to produce a randomly generated labeled data set. We'll use a single normal distribution of 10,000 points centered at pixel (63, 63) in a 128-by-128 image with standard deviation (img->width/6, img->height/6). To label these data, we divide the space into four quadrants centered at pixel (63, 63). To derive the labeling probabilities, we use the following scheme. If x < 64 we use a 20% probability for class A; else if x ≥ 64 we use a 90% factor for class A. If y < 64 we use a 40% probability for class A; else if y ≥ 64 we use a 60% factor for class A. Multiplying the x and y probabilities together yields the total probability for class A by quadrant with values listed in the 2-by-2 matrix shown. If a point isn't labeled A, then it is labeled B by default. For example, if x < 64 and y < 64, we would have an 8% chance of a point being labeled class A and a 92% chance of that point being labeled class B. The four-quadrant matrix for the probability of a point being labeled class A (and if not, it's class B) is:0.2 x 0.6 = 0.12
0.9 x 0.6 = 0.54
0.2 x 0.4 = 0.08
0.9 x 0.4 = 0.36
Use these quadrant odds to label the data points. For each data point, determine its quadrant. Then generate a random number from 0 to 1. If this is less than or equal to the quadrant odds, label that data point as class A; else label it class B. We will then have a list of labeled data points together with x and y as the features. The reader will note that the x-axis is more informative than the y-axis as to which class the data might be. Train random forests on this data and calculate the variable importance to show x is indeed more important than y.
Using the same data set as in exercise 6, use discrete AdaBoost to learn two models: one with
weak_countset to 20 trees and one set to 500 trees. Randomly select a training and a test set from the 10,000 data points. Train the algorithm and report test results when the training set contains:150 data points;
500 data points;
1,200 data points;
5,000 data points.
Explain your results. What is happening?
Repeat exercise 7 but use the random trees classifier with 50 and 500 trees.
Repeat exercise 7, but this time use 60 trees and compare random trees versus SVM.
In what ways is the random tree algorithm more robust against overfitting than decision trees?
Refer to Figure 13-2. Can you imagine conditions under which the test set error would be lower than the training set error?
Figure 13-2 was drawn for a regression problem. Label the first point on the graph A, the second point B, the third point A, the forth point B and so on. Draw a separation line for these two classes (A and B) that shows:
bias;
variance.
Refer to Figure 13-3.
Draw the generic best-possible ROC curve.
Draw the generic worst-possible ROC curve.
Draw a curve for a classifier that performs randomly on its test data.
The "no free lunch" theorem states that no classifier is optimal over all distributions of labeled data. Describe a labeled data distribution over which no classifier described in this chapter would work well.
What distribution would be hard for naïve Bayes to learn?
What distribution would be hard for decision trees to learn?
How would you preprocess the distributions in parts a and b so that the classifiers could learn from the data more easily?
Set up and run the Haar classifier to detect your face in a web camera.
How much scale change can it work with?
How much blur?
Through what angles of head tilt will it work?
Through what angles of chin down and up will it work?
Through what angles of head yaw (motion left and right) will it work?
Explore how tolerant it is of 3D head poses. Report on your findings.
Use blue or green screening to collect a flat hand gesture (static pose). Collect examples of other hand poses and of random backgrounds. Collect several hundred images and then train the Haar classifier to detect this gesture. Test the classifier in real time and estimate its detection rate.
Using your knowledge and what you've learned from exercise 16, improve the results you obtained in that exercise.
