“STUDIES IN CROP VARIATION”
Early in my career as a biostatistician, and on one of my trips to the University of Connecticut at Storrs to talk over my problems, Professor Hugh Smith had a present for me. It was a copy of a paper entitled “Studies in Crop Variation. III. The Influence of Rainfall on the Yield of Wheat at Rothamsted.” The paper was fifty-three pages long. It is the third in a series of remarkable mathematical articles, the first of which had appeared in the Journal of Agricultural Science, vol. XI, in 1921. Variation in output is the bane of the experimental scientist but the basic material of statistical methods. The word variation is seldom used in the modern scientific literature. It has been replaced by other terms, like “variance,” which refer to specific parameters of distributions.
Variation is too vague a word for ordinary scientific use, but it was appropriate in this series of papers, because the author used the variation in crop outputs from year to year and from field to field as a starting point from which to derive new methods of analysis.
Most scientific papers have long lists of references at the end, identifying the previous papers that had addressed the problems discussed. “Studies in Crop Variation. I,” the first in this series, had only three references: one indicated a failed attempt made in 1907 to correlate rainfall and wheat growth; a second, in German, from 1909, described a method for computing the minimum of a complicated mathematical formula; and the third was a set of tables published by Karl Pearson. There was no previous paper dealing with most of the topics covered in this remarkable series. The “Studies in Crop Variation” were sui generis. Their author is described as R. A. Fisher, M.A., Statistical Laboratory, Rothamsted Experimental Station, Harpenden.
In 1950, the publisher John Wiley asked Fisher if he would cull through his published papers and provide a selection of the most important, to form a single volume. The volume, entitled Contributions to Mathematical Statistics, opens with a contemporaneous photograph of a white-haired R. A. Fisher, his lips tightly pressed together, his tie slightly askew, his white beard not too well trimmed. He is identified as “R. A. Fisher, Department of Genetics, University of Cambridge.” “Studies in Crop Variation. I” is the third article in this book. It is preceded by a short note from the author identifying its importance and its place in his works:
In the author’s early work at Rothamsted much attention was given to the massive records of weather, crop yields, crop analyses, etc., which had been accumulated during the long history of that research station. The material was obviously of unique value for such problems as that of ascertaining to what extent meteorological readings were capable of supplying a prediction of the crop yields to follow. The present paper is the first of a series devoted to this end … .
There were, at most, six articles in the “series devoted to this end.” “Studies in Crop Variation. II” appeared in 1923. There is the
paper Professor Smith gave me, entitled “III. The Influence of Rainfall on the Yield of Wheat at Rothamsted,” from 1924. “Studies in Crop Variation. IV” appeared in 1927 and “Studies in Crop Variation. VI” was published in 1929. Study number V does not appear in Fisher’s collected works. Seldom in the history of science has a set of titles been such a poor descriptor of the importance of the material they contain. In these papers, Fisher developed original tools for the analysis of data, derived the mathematical foundations of those tools, described their extensions into other fields, and applied them to the “muck” he found at Rothamsted. These papers show a brilliant originality and are filled with fascinating implications that kept theoreticians busy for the rest of the twentieth century, and will probably continue to inspire more work in the years that follow.
There were additional authors for two of the later papers in Fisher’s series. For “Studies in Crop Variation. I,” he worked alone. It required prodigious amounts of calculation. His only aid was a calculating machine named the Millionaire. It was a primitive, hand-cranked mechanical calculator. If one wanted to multiply, for instance, 3,342 by 27, one put the platen on the units position, set in the number 3,342, and cranked seven times. Then one put the platen on the tens position, set in the number 3,342, and cranked two times. It was a Millionaire because the platen was big enough to accommodate numbers in the millions.
To get some idea of the physical effort involved, consider Table VII that appears on page 123 of “Studies in Crop Variation. I.” If it took about one minute to complete a single large-digit multiplication, I estimate that Fisher needed about 185 hours of work to generate that table. There are fifteen tables of similar complexity and four large complicated graphs in the article. In terms of physical labor alone, it must have taken at least eight months of 12-hour days to prepare the tables for this article. This does not include the
hours needed to work out the theoretical mathematics, to organize the data, to plan the analysis, and to correct the inevitable mistakes.
Recall Galton’s discovery of regression to the mean and his attempt to find a mathematical formula that linked random events to each other. Fisher took Galton’s word, regression, and established a general mathematical relationship between the year and the wheat output of a given field. Pearson’s idea of probability distribution now became a formula connecting year to output. The parameters of this more complicated distribution described different aspects of the change in wheat output. To go through Fisher’s mathematics one needs a solid knowledge of calculus, a good sense of the theory of probability distributions, and a feeling for multidimensional geometry. But it is not too difficult to understand his conclusions.
He pulled the time trend of wheat output into several pieces. One was a steady overall diminution of output due to deterioration of the soil. Another was a long-term, slow change that took several years for each phase. The third was a set of faster moving changes that took into account the variations in climate from year to year. Since Fisher’s first pioneering attempts, the statistical analysis of time series has built on his ideas and methods. We now have computers that can do the immense calculations with clever algorithms, but the basic idea and methods remain. Given a set of numbers fluctuating over time, we can pull them apart into effects that are due to different sources. Time series analyses have been used to examine the lapping of waves on the Pacific shores of the United States and thereby identify storms in the Indian Ocean. These methods have enabled researchers to distinguish between underground nuclear explosions and earthquakes, pinpoint pathological
aspects of heartbeats, quantify the effect of environmental regulations on air quality; and the uses continue to multiply.
Fisher was puzzled by his analysis of grain harvested from a field named Broadbalk. Only natural animal dung had been used on this field, so the variation in yield from year to year was not the result of experimental fertilizers. The long-term deterioration made sense as the soil was depleted of nutrients missing from the dung, and he could identify the effects of differing rainfall patterns in the year-to-year changes. What was the source of the slow changes? The slow-change pattern suggested that in 1876, the output had begun to deteriorate more than would be expected from the other two effects, the deterioration getting more rapid after 1880. There was an improvement starting in 1894 and continuing until 1901, with a drop thereafter.
Fisher found another record with the same slow change but with the pattern reversed. This was the infestation of weeds in the wheat field. After 1876, the weeds became ever heavier, with new varieties of perennials establishing themselves. Then in 1894 the weeds suddenly began to diminish, only to start flourishing again in 1901.
It turned out that it had been the practice prior to 1876 to hire small boys to go into the fields and pull weeds. It was common at that time to see weary children in the fields of England on an afternoon combing through the wheat and other grains, constantly pulling weeds. In 1876, the Education Act made attendance at school compulsory, and the legions of young boys began disappearing from the fields. In 1880, a second Education Act provided penalties for families that kept their children out of school, and the last of the young boys left the fields. Without the little fingers to pull them out, the weeds began to flourish.
What happened in 1894 to reverse this trend? There was a boarding school for girls in the vicinity of Rothamsted. The new schoolmaster, Sir John Lawes, believed in vigorous outdoor activity
to build up the health of his young charges. He arranged with the director of the experimental station to bring his young girls out into the fields to pull weeds on Saturdays and evenings. After Sir John died in 1901, the little girls went back to sedentary and indoor activities, and the weeds came back to Broadbalk.
The second study on crop variation also appeared in the Journal of Agricultural Science in 1923. This paper does not deal with accumulated data from the past experiments at Rothamsted. Instead, it describes a set of experiments on the effects of different fertilizer mixes on different varieties of potatoes. Something remarkable had happened to the experiments at Rothamsted since the arrival of R. A. Fisher. No longer were they applying a single experimental fertilizer across an entire field. Now, they were cutting the field up into small plots. Each plot was further divided into rows of plants and each row within a plot was given a different treatment.
The basic idea was a simple one—simple, that is, once it was proposed by Fisher. No one had ever thought of it before. It is obvious to anyone looking at a field of grain that some parts of the field are better than others. In some corners, the plants grow tall and heavy with grain. In other corners, the plants are thin and scraggly. It may be due to the way in which water drains, or to the changes in soil type, or the presence of unknown nutrients, or blocks of perennial weeds, or some other, unforeseen, effect. If the agricultural scientist wants to test the difference between two fertilizer components, he can put one fertilizer on one part of the field and another elsewhere. This will confound the effects of the fertilizer with effects due to the soil or drainage properties. If the tests are run on the same fields but in different years, the effects of fertilizer are confounded with changes in weather from year to year.
If the fertilizers are compared right next to one another and in the same year, then the differences in soil will be minimized. They will still be there, since the treated plants are not on exactly the same soil. If we use many such pairs, the differences due to soil will average out in some sense. Suppose we want to compare two fertilizers, one with twice as much phosphorus as the other. We break the field into tiny plots, each with two rows of plants. We always put the extra phosphorus on the northern row of plants and treat the southern row with the other mixture. Now, I can hear someone saying, then they won’t “average” out if the fertility gradient in the soil runs north and south, for the northern row in each block will have slightly better soil than the southern row.
We’ll just alternate. In the first block, the extra phosphorus will be in the northern row. In the second block, it will be in the southern row, and so forth. One of my readers has now drawn a rough map of the field and put X’s to indicate the rows with extra phosphorus. He points out that if the fertility gradient runs from northwest to southeast, then the rows treated with extra phosphorus will all have better soil than the others. Someone else points out that if the gradient runs from northeast to southwest, the opposite holds. Well, demands another reader, which is it? How does the fertility gradient run? To which we reply, no one knows. The concept of a fertility gradient is an abstract one. The real pattern of fertility may run up and down in some complicated way as we go from north to south or east to west.
I can imagine these discussions among the scientists at Rothamsted once Fisher pointed out that setting treatments within small blocks would allow for more careful experimentation. I can imagine the discussions over how to determine the fertility gradient, while R. A. Fisher sits back and smiles, letting them get more and more involved in complicated constructs. He has already considered these questions and has a simple answer. He removes the pipe from his mouth. Those who knew him describe Fisher as sitting, quietly puffing on his pipe, while arguments raged
about him, waiting for the moment when he could insert his answer. “Randomize,” he says.
It is simple. The scientist assigns the treatments to different rows within a block at random. Since the random ordering follows no fixed pattern, any possible structure of fertility gradient will cancel out, on the average. Fisher springs up and begins to write furiously on the blackboard, filling it with mathematical symbols, sweeping his arms across the columns of math, crossing out factors that cancel on either side of the equations, and emerging with what was to become probably the single most important tool of biological science. This is a method for separating out the effects of different treatments in a well-designed scientific experiment, which Fisher called the “analysis of variance.” In “Studies in Crop Variation. II” the analysis of variance shows up for the first time.
The formulas for some examples of analysis of variance appear in Statistical Methods for Research Workers, but in this paper they are derived mathematically. They are not worked out in sufficient detail for an academic mathematician to be satisfied. The algebra displayed is made specific to the situation of comparing three types of fertilizer (manure), ten varieties of potatoes, and four blocks of soil. It takes a few hours of painstaking work to figure out how the algebra might be adapted for two fertilizers and five varieties, or for six fertilizers and only one variety. It takes even more mathematical sweat of the brow to figure out the general formulas that would work in all cases. Fisher, of course, knew the general formulas. They were so obvious to him that he did not see the need to produce them.
No wonder his contemporaries were mystified by young Fisher’s work!
“Studies in Crop Variation. IV” introduces what Fisher called “analysis of covariance.” This is a method for factoring out the
effects of conditions that are not part of the experimental design but which are there and can be measured. When an article in a medical journal describes a treatment effect that has been “adjusted for sex and weight,” it is using the methods pioneered by Fisher in this paper. Study VI produces refinements in the theory of design of experiments. Study III, to which Professor Smith introduced me, will be discussed later in this chapter.
In 1922, Fisher finally got his first article published in the Journal of the Royal Statistical Society. It is a short note that modestly proves that one of Karl Pearson’s formulas was wrong. Writing about this paper many years later, Fisher said:
This short paper, with all its juvenile inadequacies, yet did something to break the ice. Any reader who feels exasperated by its tentative and piecemeal character should remember that it had to find its way to publication past critics who, in the first place, could not believe that Pearson’s work stood in need of correction, and who, if this had to be admitted, were sure that they themselves had corrected it.
In 1924, he was able to publish a longer and more general paper in the Journal of the Royal Statistical Society. He later comments on this and a related paper in an economics journal: “[These papers] are attempts to reconcile, with the aid of the new concept of degrees of freedom, the discrepant and anomalous results observed by different authors … .”
The “new concept of degrees of freedom” was Fisher’s discovery and was directly related to his geometric insights and his ability to cast the mathematical problems in terms of multidimensional geometry. The “anomalous results” occurred in an obscure
book published by somebody named T. L. Kelley in New York; Kelley had found data wherein some of Pearson’s formulas did not seem to produce correct answers. Only Fisher, it would seem, had looked at Kelley’s book. Kelley’s anomalous results were only used as a springboard from which Fisher utterly demolished another of Pearson’s most proud achievements.
The third of the studies in crop variation appeared in 1924 in the Philosophical Transactions of the Royal Society of London. It begins:
At the present time very little can be claimed to be known as to the effects of weather upon farm crops. The obscurity of the subject, in spite of its immense importance to a great national industry, may be ascribed partly to the inherent complexity of the problem … and … to the lack of quantitative data obtained either upon experimental or under industrial conditions … .
Then follows a masterful article of fifty-three pages. In it lie the foundations of modern statistical methods used in economics, medicine, chemistry, computer science, sociology, astronomy, pharmacology—any field where one needs to establish the relative effects of a large number of interconnected causes. It contains highly ingenious methods of calculation (recall that Fisher had only the manual Millionaire to work with), and many wise suggestions on how to organize data for statistical analysis. I am forever grateful to Professor Smith for introducing me to this paper. With each reading, I learn something new.
The first volume (of five) of the Collected Papers of R. A. Fisher ends with the papers he published in 1924. There is a photograph of Fisher, at this point thirty-four years old, near the end of the volume. His arms are folded. His beard is well trimmed. His glasses do
not seem as thick as they did in earlier photos. He has a confident and secure look. In the previous five years, he has built a remarkable statistics department at Rothamsted. He has hired coworkers like Frank Yates, who would go on, under Fisher’s encouragement, to make major contributions to the theory and practice of statistical analysis. With a few exceptions, Karl Pearson’s students have disappeared. While they worked at the biometrical laboratory, they aided Pearson and were no more than extensions of Pearson. With few exceptions, Fisher’s students responded to his encouragement and plowed brilliant and original paths themselves.
In 1947, Fisher was invited to give a series of talks on the BBC radio network about the nature of science and scientific investigation. Early in one talk, he said:
A scientific career is peculiar in some ways. Its raison d’être is the increase of natural knowledge. Occasionally, therefore, an increase of natural knowledge occurs. But this is tactless, and feelings are hurt. For in some small degree it is inevitable that views previously expounded are shown to be either obsolete or false. Most people, I think, can recognize this and take it in good part if what they have been teaching for ten years or so comes to need a little revision; but some undoubtedly take it hard, as a blow to their amour propre, or even as an invasion of the territory they have come to think of as exclusively their own, and they must react with the same ferocity as we can see in the robins and chaffinches these spring days when they resent an intrusion into their little territories. I do not think anything can be done about it. It is inherent in the nature of our profession; but a young scientist may be warned and advised that when he has a jewel to offer for the enrichment of mankind some certainly will wish to turn and rend him.