WHEN PART IS BETTER THAN THE WHOLE
To Karl Pearson, probability distributions could be examined by collecting data. He thought that if he collected enough data, he could expect them to be representative of all such data. The correspondents for Biometrika would select hundreds of skulls from ancient graveyards, pour shot into them to measure the cranial capacity, and send Pearson these hundreds of numbers. A correspondent would travel to the jungles of Central America and measure the lengths of arm bones on hundreds of natives, sending these measurements to Pearson’s biometrical laboratory.
There was a basic flaw in Pearson’s methods, however. He was collecting what is now called an “opportunity sample.” The data were those that were most easily available. They did not have to be truly representative of the entire distribution. The graves that were opened to find the cranial capacity of skulls were those that happened to have been found. The unfound ones could differ in some unknown way.
A specific example of this failure of opportunity sampling was discovered in India in the early 1930s. Bales of jute were assembled on the dock in Bombay for shipment to Europe. To determine the value of the jute, each bale was sampled and the quality of the jute determined from the sample. The sampling was done by plunging a hollow circular blade into the bale and pulling out a small amount in the core of the blade. In the packing and shipment of the bales, the outside of the bales tended to deteriorate and the inner parts tended to become more and more compacted, often including some frozen parts in the wintertime. The sampler would push the hollow knife into the bale, but it would be deflected from the denser part of the bale, and the sample would tend to consist almost entirely of the outer and damaged region. The opportunity sample was biased in favor of finding inferior jute when the quality of the bale was much higher.
Professor Prasanta Chandra Mahalanobis, head of the Department of Physics in the Presidency College, Calcutta, often used this example (which he had discovered when working for the railroad that shipped the jute to the dock) to show why opportunity samples were not to be trusted. Mahalanobis came from a wealthy family of Calcutta merchants and could afford a period of graduate and postgraduate study as he pursued his interests in science and mathematics. During the 1920s, he traveled to England and studied under both Pearson and Fisher. Students like F. N. David had to subsist on scholarship aid, but Mahalanobis lived the life of the grand seigneur while he pursued his studies. He returned to head the Department of Physics at Presidency College. Soon afterward, in 1931, he used his own funds to set up the Indian Statistical Institute on the grounds of one of his family estates.
At the Indian Statistical Institute, he trained a group of brilliant Indian mathematicians and statisticians, many of whom went on to make important contributions to the field—people like S. N. Roy, C. R. Rao, R. C. Bose, P. K. Sen, and Madan Puri, among others. One of Mahalanobis’s interests lay in this question of how to produce
an appropriately representative sample of data. It was clear that, in many situations, it was almost impossible to get all the measurements in a set. For instance, the population of India is so great that, for years, no attempt was made to get a complete census on a single day—as is tried in the United States. Instead, the complete Indian census takes over a year, as different regions of the country are counted in different months. Because of this, the Indian census can never be accurate. There are births and deaths, migrations, and changes of status that occur during the time of the census. No one will ever know exactly how many people there are in India on a given day.19
Mahalanobis reasoned that it might be possible to estimate the characteristics of the larger population if one could gather a small sample that was adequately representative of the larger. At this point, we run into two possible approaches. One is to construct what is known as a “judgment sample.” In a judgment sample, whatever is known about the population is used to select a small group of individuals who are chosen to represent different groups in the larger population. The Nielsen ratings to determine how many people are watching TV shows are created from a judgment sample. Nielsen Media Research selects families based on their socioeconomic status and the region of the country in which they live.
A judgment sample seems, at first glance, to be a good way to get a representative sample of the larger population. But it has two major faults. The first is that the sample is representative only if we are absolutely sure that we know enough about the larger population to be able to find specific subclasses that can be represented. If we knew that much about the larger population, we would
probably not have to sample, since the questions we ask of the sample are those that are needed to divide the larger population into homogeneous groups. The second problem is more troublesome. If the results of the judgment sample are wrong, we have no way of knowing how far from the truth they are. In the summer of 2000, Nielsen Media Research was criticized for not having enough Hispanic families in their sample and underestimating the number of families watching Spanish-language TV.
Mahalanobis’s answer was the random sample. We use a randomizing mechanism to pick individuals out of the larger population. The numbers we get from this random sample are most likely wrong, but we can use the theorems of mathematical statistics to determine how to sample and measure in an optimum way, making sure that, in the long run, our numbers will be closer to the truth than any others. Furthermore, we know the mathematical form of the probability distribution of random samples, and we can calculate confidence bounds on the true values of the things we wish to estimate.
Thus, the random sample is better than either the opportunity , sample or the judgment sample, not because it guarantees correct answers but because we can calculate a range of answers that will contain the correct answer with high probability.
The mathematics of sampling theory developed rapidly during the 1930s, some of it at the Indian Statistical Institute under Mahalanobis, some of it from two papers by Neyman in the late 1930s, and some of it by a group of eager young university graduates who gathered in Washington, D.C., during the early days of the New Deal. Many of the practical problems of how to sample from a large population of people were addressed and solved by these young New Dealers at the Commerce and Labor Departments of the federal government.
A young man or woman who received a bachelor’s degree in the years 1932 to 1939 often walked out of the university into a world where there were no jobs. The Great Depression saw to that. Margaret Martin, who grew up in Yonkers, New York, and went to Barnard College, and who eventually became an official of the U.S. Bureau of the Budget, wrote:
When I graduated in June of 1933 I couldn’t find any job … . A friend of mine who graduated one year later in 1934 felt very fortunate. She’d gotten a job selling at the B. Altman department store; she worked 48 hours a week and earned $15. But, even those jobs were relatively scarce. We had an occupation officer, Miss Florence Doty, at Barnard, and I went to her to talk about the possibilities of going to Katherine Gibbs, a secretarial school. I didn’t know where I’d find the money, but I thought that would be a skill that could at least earn something. Miss Doty … was not an easy person to get along with, and a lot of the students stood very much in awe of her … . She just turned on me, “I would never recommend that you take a secretarial course! If you learn to use a typewriter, and show that you can use a typewriter, you will never do anything else but use a typewriter … . You should be looking for a professional position.”
Martin eventually found her first job in Albany, as a junior economist in the office of research and statistics of the New York State Division of Placement and Unemployment, and used that job as a springboard to graduate studies.
Other newly graduated young people went directly to Washington. Morris Hansen went to the Census Bureau in 1933, with an undergraduate degree in economics from the University of Wyoming. He used his undergraduate mathematics and a hasty
reading of Neyman’s papers to design the first comprehensive survey of unemployment. Nathan Mantel took his new City College of New York (CCNY) degree in biology and went to the National Cancer Institute. Jerome Cornfield, a history major at CCNY, took a job as an analyst at the Department of Labor.
It was an exciting period to be in government. The nation lay prostrate, with much normal economic activity idle, and the new government in Washington was looking for ideas on how to get things started again. First they had to know how bad things were throughout the country. Surveys of employment and economic activity were begun. For the first time in the nation’s history, an attempt was being made to determine exactly what was happening in the country. It was an obvious place for sample surveys.
These eager young workers initially had to overcome the objections of those who did not understand the mathematics. When one of the earlier surveys at the Department of Labor showed that less than 10 percent of the population received almost 40 percent of the income, it was denounced by the U.S. Chamber of Commerce. How could this be true? The survey had contacted less than one-half of 1 percent of the working population, and these people were chosen by random means! The Chamber of Commerce had its own surveys, taken from the opinions of its own members about what was happening. This new survey was dismissed by the chamber as inaccurate, since it was only a random collection of data.
In 1937, the government attempted to get a full count of the unemployment rate, and Congress authorized the Unemployment Census of 1937. The bill, as passed by Congress, called for everyone who was unemployed to fill out a registration card and deliver it to the local post office. At that time, estimates of the number of unemployed ranged from three to fifteen million, and the only good counts were a few random surveys that had been taken in New York. A group of young sociologists, led by Cal Dedrick and Fred Stephan in the Census Bureau, realized that there would be many unemployed who would not respond and the result would yield numbers filled with unknown errors. It was decided that the first serious random
survey across the entire country should be run. With young Morris Hansen designing the survey, the bureau chose 2 percent of all the postal routes at random. The postal carriers on those routes handed out questionnaires to every family on their routes.
Even with a 2 percent sample, the Census Bureau was overwhelmed with the huge number of questionnaires. The U.S. Postal Service attempted to organize them and make initial tabulations. The questionnaire had been designed to pick up detailed information about the demographics and the work history of the respondents, and no one knew how to examine such large amounts of detailed information. Recall that this was before the computer, and the only aids to pencil-and-paper tabulations were hand-operated mechanical calculators. Hansen contacted Jerzy Neyman, whose papers had formed the basis of the survey’s design. In Hansen’s words, Neyman pointed out that “we didn’t have to know and match all the cases and understand all the relationships” to find answers to the most important questions. Using Neyman’s advice, Hansen and his coworkers set aside most of the complicated and confusing details of the questionnaires and counted the numbers of unemployed.
It took a series of careful studies in the Census Bureau, under Hansen, to prove that these small random surveys were much more accurate than the judgment samples that had been used before. Eventually, the U.S. Bureau of Labor Statistics and the Census Bureau led the way into a new world of random sampling. George Gallup and Louis Bean took these methods into the region of political polling.20 For the 1940 census, the Census Bureau initiated elaborate plans for sample surveys within the overall census. There was a young, newly hired statistician at the bureau named William Hurwitz. Hansen and Hurwitz became close collaborators and
friends; they issued a series of important and influential papers, culminating in the 1953 textbook Sample Survey Methods and Theory (written with a third author, William Madow). The Hansen and Hurwitz papers and text became so important to the field of sample surveys, and were quoted so often, that many workers in the field came to believe that there was one person named Hansen Hurwitz.
Many of the new young workers who arrived in Washington during the New Deal went on to become major figures in government and in academia. Some of them were too busy creating new mathematics and statistical methods to go for graduate degrees. A prime example is Jerome Cornfield. Cornfield participated in some of these early surveys at the Bureau of Labor Statistics and then moved to the National Institutes of Health. He published papers jointly with some of the leading figures in academia. He solved the mathematical problems involved in case-control studies. His scientific papers range from work on random sample theory to the economics of employment patterns, the investigation of tumors in chickens, problems in photosynthesis, and effects of environmental toxins on human health. He created many of the statistical methods that have now become standard in the fields of medicine, toxicology, pharmacology, and economics.
One of Cornfield’s most important achievements was in the design and initial analysis of the Framingham Study, begun in 1948. The idea was to take Framingham, Massachusetts, as a “typical town,” measure a large number of health variables on everyone in the town, and then follow these people for a number of years. The study has now been running for over fifty years. It has had a “Perils of Pauline” existence as, from time to time, attempts have been made to cut its funding in the interests of budget reduction in government. It remains a major source of information on the long-term effects of diet and lifestyles on heart disease and cancer.
To analyze the first five years of data from the Framingham Study, Cornfield ran into fundamental problems that had not been addressed in the theoretical literature. Working with faculty members at Princeton University, he solved these problems. Others went on to produce papers on the theoretical development he started, but Cornfield was satisfied to have found a method. In 1967, he was a coauthor of the first medical article to emerge from the study, the first article to show the effects of elevated cholesterol on the probability of heart disease.
I was on a committee with Jerry Cornfield, convened in 1973 as part of a set of hearings before a Congressional committee. During a break in our work, Cornfield was called to the phone. It was Wassily Leontief, an economist at Columbia University, calling to say that he had just been awarded the Nobel Prize in economics and wanted to thank Cornfield for the role Jerry had played in their work, which led to this prize. This work had originated in the late 1940s when Leontief had come to the Bureau of Labor Statistics for help.
Leontief believed that the economy could be broken down into sectors, like farming, steel manufacturing, retailing, and so forth. Each sector uses material and services from the other sectors to produce material or a service, which it supplies to those other sectors. This interrelationship can be described in the form of a mathematical matrix. It is often called an “input—output analysis.” When he first began investigating this model at the end of World War II, Leontief went to the Bureau of Labor Statistics to help gather the data he needed. To assist him, the bureau assigned a young analyst who was working there at the time, Jerome Cornfield.
Leontief could break the economy down into a few broad sectors, such as putting all manufacturing in one sector, or he could subdivide the sectors into more specific ones. The mathematical theory of input–output analysis requires that the matrix that describes the economy have a unique inverse. That meant that the matrix, once assembled, had to be subjected to a mathematical
procedure called “inverting the matrix.” At that time, before the widespread availability of computers, inverting a matrix was a difficult and tedious procedure on a calculator. When I was in graduate school, each of us had to invert a matrix—I suspect as a kind of rite of passage “for the good of our souls.” I remember trying to invert a 5 X 5 matrix and taking several days, most of which I spent locating my mistakes and redoing what I had done wrong.
Leontief’s initial set of sectors led to a 12 x 12 matrix, and Jerry Cornfield proceeded to invert that 12 x 12 matrix to see if there was a unique solution. It took him about a week, and the end result was the conclusion that the number of sectors had to be expanded. So, with trepidation, Cornfield and Leontief began subdividing the sectors until they ended with the simplest matrix they thought would be feasible, a 24 x 24 matrix. They both knew this was beyond the capacity of a single human being. Cornfield estimated that it would take him several hundred years of seven-day workweeks to invert a 24 x 24 matrix.
During World War II, Harvard University had developed one of the first, very primitive computers. It used mechanical relay switches and would often jam. There was no longer any war work for it, and Harvard was looking for applications for its monstrous machine. Cornfield and Leontief decided to send their 24 X 24 matrix to Harvard where its Mark I computer would go through the tedious calculations and compute the inverse. When they sought to pay for this project, the process was stopped by the accounting office of the Bureau of Labor Statistics. The government had a policy at that time; it would pay for goods but not for services. The theory was that the government had all kinds of experts working for it. If something had to be done, there should be someone in government who could do it.
They explained to the government accountant that, while this was theoretically something that a person could do, no one would be able to live long enough to do it. The accountant was sympathetic, but he could not see a way around the regulation. Cornfield
then made a suggestion. As a result, the bureau issued a purchase order for capital goods. What capital goods? The invoice called for the bureau to purchase from Harvard “one matrix, inverted.”
The work of these young men and women who rushed into government during the early days of the New Deal continues to be of fundamental importance to the nation. This work led to the regular series of economic indicators that are now used to fine-tune the economy. These indicators include the Consumer Price Index (for inflation), the Current Population Survey (for unemployment rates), the Census of Manufacturing, the intermediate adjustments of Census Bureau estimates of the nation’s population between decennial censuses, and many other less well-known surveys that have been copied and are used by every industrial nation in the world.
In India, P. C. Mahalanobis became a personal friend of Prime Minister Jawaharlal Nehru in the early days of the new government of India. Under his influence, Nehru’s attempts to imitate the central planning of the Soviet Union were often modified by carefully conducted sample surveys, which showed what really was happening to the new nation’s economy. In Russia, the bureaucrats produced false figures of production and economic activity to flatter the rulers, which encouraged the more foolish excesses of their central economic plans. In India, good estimates of the truth were always available. Nehru and his successors may not have liked it, but they had to deal with it.
In 1962, R. A. Fisher went to India. He had been there many times before at Mahalanobis’s invitation. This was a special occasion. There was a great meeting of the world’s leading statisticians to commemorate the thirtieth anniversary of the founding of the Indian Statistical Institute. Fisher, Neyman, Egon Pearson, Hansen, Cornfield, and others from the United States and Europe were
there. The sessions were lively, for the field of mathematical statistics was still in ferment, with many unsolved problems. The methods of statistical analysis were penetrating to every field of science. New techniques of analysis were constantly being proposed and examined. There were four scientific societies devoted to the subject and at least eight major journals (one of which had been founded by Mahalanobis).
When the conference closed, the attendees went their separate ways. As they arrived home, they heard the news. R. A. Fisher had died of a heart attack on the boat returning him to Australia. He was seventy-two years old. His collected scientific papers fill five volumes, and his seven books continue to influence all that is being done in statistics. His brilliant original accomplishments had come to an end.