Ever since the earliest civilisations, rulers have known that you can’t tax something without knowing how much of it there is. Among the earliest surviving written records, pressed into wet clay 4,000 years ago in cuneiform symbols, are Mesopotamian tax receipts.
Even before we could write, our ancestors used clay tokens to record the quantities of oxen, grain and whatever else could be traded and taxed. For all we know, that wolf bone with 57 notches in groups of five was a prehistoric tax return, showing that 57 mammoths had been eaten, and somebody was due to pay their share of cave upkeep, spear maintenance and dung removal charges.
But it’s much easier to send a tax demand when you also have a written record of somebody’s name and where they live.
No pig left behind
In 1085, King William I of England sent researchers out to record the assets of his kingdom. Another war against Denmark looming, William, better known as William the Conqueror, wanted to know how much tax he could raise, and what military resources he could draft. He had no standing army, so men, horses and weaponry from all over the country were a tax-in-kind as vital as hard cash.
The inspection was thorough. The Anglo-Saxon Chronicle recounts that not a yard of land, ‘nor one ox nor one cow nor one pig’ was left out. But if you know the King’s inspectors are coming to count and tax your livestock, you have a strong incentive to hide them. There may not have been any accountants, or schemes to put your money in an offshore bank account on a Caribbean island,1 but if you could hide a pig somewhere, or pretend it belonged to somebody else, you might not have to pay tax on it. Conversely, if you could convince the inspectors that a piece of land belonged to you, you could claim it as yours forever.
So a complete census was very labour intensive. Each draft report was checked at a special sitting of the county court, where a jury was asked to verify the area of land, the number of fisheries, plough teams and so on. Only then were the local reports written up neatly in Latin,2 and collected up to be transcribed into the Domesday Book.
Or strictly, Little Domesday covering the eastern counties of Essex, Norfolk and Suffolk, and Great Domesday covering most of the other counties, though some of them are a bit sketchy because it was never fully completed. The bound volumes, written on parchment with quill pens, leave out the pig-by-pig detail of the original reports and would already have been out of date by the time the last scribe put down his quill and called it a day, after William’s death in 1087.
No wonder nobody attempted another national UK census for more than 700 years.
Yes, take a moment to be thankful you live in the era of the internet, with all the world’s information a few clicks away from wherever you are right now.
OK, that’s long enough.
No, you can’t just watch the end of that amusing cat video. I want to talk to you about death.
Births, deaths and marriages
Like most people in seventeenth century England, John Graunt was a man of faith. In his case, the Roman Catholic faith, which in those days was politically risky. Nevertheless, somehow he survived both being on the unpopular side of the religious divide and fighting on the losing Royalist side in the English Civil War. So when he buried his young daughter Frances, he must have asked: Why?
He’d already buried his parents in the last 12 months, which was bad enough, but in the natural order of things. Why, when he had lived through wars, political upheavals and repeated outbreaks of plague, had his daughter succumbed to consumption? Perhaps this was part of his motivation for poring over hundreds of death records in search of order and meaning. Or perhaps he just wanted a time-consuming hobby to take his mind off the grief.
The first organised death records in the UK were kept in London when the city was ravaged by plague in the sixteenth century. Individual deaths were already recorded locally, in each church parish, but for the first time that information was collected together in bills of mortality, which also noted the age and cause of death.
The bills of mortality counted deaths in each parish, but not the names of the deceased, making them an early example of anonymous data.3 As well as providing vital information to the city authorities week by week, the bills were made available to individual citizens, for a price, so they could decide whether to stay in the city or flee the latest outbreak of plague.4
As a haberdasher in the City of London, John Graunt would have been active in civic life through the guild system, and wouldn’t have found it too hard to get to the bills, which went back over 50 years.
Looking for patterns in the hundreds of lists, Graunt combined the figures into his own tables, organised by age and cause of death. In 1662, he published Natural and Political Observations Mentioned in a Following Index and Made upon the Bills of Mortality, which held the record for Least Snappy Book Title until 1929, when Henryk Grossmann published The Law of Accumulation and Breakdown of the Capitalist System, being also a theory of crises.5
By putting together a lot of data, Graunt spotted a number of things that nobody had noticed before. He saw that some diseases tended to kill roughly the same number of people every year and others, including plague, come and go. Even accidental deaths such as drowning happened at a stable rate every year. Not exactly the same, but varying around a fairly steady number. In fact, Graunt was probably the first to identify this stable number by adding together the total and dividing it by the number of years, what today we’d call the average.6
Graunt set down the horrifying rate at which young children were dying, one in three dead by the age of five. He was also the first to note that more boys than girls were born, a ratio of 14 to 13. And he confirmed that plague was a frequent visitor to the city, not a permanent resident, and that the pattern of its victims suggested that it passed from person to person.
King Charles II was impressed by this work, especially the new understanding it brought of the much-feared plague. John Graunt became a Fellow of the Royal Society, a new and already prestigious organisation of scientists. But, although he laid the foundations of modern statistics, taking raw records and turning them into a resource that answers your questions, his glory was short-lived.
Plague returned to London in 1665, followed in 1666 by the Great Fire of London, which destroyed Graunt’s haberdashery business. His lowly background and Catholic beliefs meant he’d never been fully accepted in the Royal Society, and he was pushed out. For a long time after his death in poverty, even his book was attributed to another man, William Petty.
The next man to take an interest in the sex ratio of infants was a Scot, John Arbuthnot, also a Catholic and a Fellow of the Royal Society. Arbuthnot worked with Jonathan Swift to create the satirical character John Bull, the archetypal Englishman, and his brother was killed fighting for the Jacobites on the side of Catholic King James II, but he was still given the job of physician to Queen Anne in 1709.
The Royal Society published Arbuthnot’s paper, An Argument for Divine Providence, taken from the constant regularity observ’d in the births of both sexes in 1710. Using the London birth records for the preceding 82 years, Arbuthnot noted that male births exceeded female ones in each of those years.
Having previously published Of The Laws of Chance, his translation of a work by Huygens and the first English-language text on probability, he naturally thought in terms of gambling. Arbuthnot imagined that the sex of each baby has a 50:50 chance of being male or female. In any given year, there may be fewer boys or more boys.7
Suppose you assume a coin is fair, but it comes down tails five times in six. Are you just unlucky, or have you been cheated with an unfair coin? If you can calculate how likely a fair coin is to give you such a disappointing result8, you can also work out how likely your disappointment is to have been caused by an unfair coin.
Taking the probability of a ‘more boys’ scenario as half,9 he calculated the probability of 82 successive years of more boys, if the underlying ratio is 50:50. It’s the same probability as tossing a coin 82 times and throwing 82 heads in a row.
The chance of getting heads 82 times out of 82 is one in 4,835,703,278,458,516,698,824,704. Very unlikely indeed. Probably not a fair coin.
So Arbuthnot rejects the hypothesis that the difference he observed is down to pure chance, and asserts that it must be divine providence at work.10 This approach, to test a null hypothesis of no underlying difference, by calculating how likely you would be to see your results if the null hypothesis were true, is widely used in science today.
Another Scotsman introduced the word statistics to the English language. Sir John Sinclair, 1st Baronet of Ulbster, wrote his Statistical Account of Scotland in 21 volumes between 1790 and 1799. Sir John heard the word statistics while travelling in Germany,
Though I apply a different meaning to that word – for by ‘statistical’ is meant in Germany an inquiry for the purposes of ascertaining the political strength of a country or questions respecting matters of state – whereas the idea I annex to the term is an inquiry into the state of a country, for the purpose of ascertaining the quantum of happiness enjoyed by its inhabitants, and the means of its future improvement.
Quantum Of Happiness. Great name for a spy thriller – I should write that next. Anyway, Sinclair’s desire to use his information for the future happiness of the Scots is evident in the 160-question form he asked all 938 parish clergymen to complete:
159. Do the people, on the whole, enjoy, in a reasonable degree, the comforts and advantages of society? And are they contented with their situation and circumstances?
160. Are there any means by which their condition could be ameliorated?
These busy men were not prompt to complete Sinclair’s extensive questionnaire, covering the population’s common illnesses, religion, occupation and age at death, but also geography, climate, agricultural production, provision for the poor and the price of fish. It took nine years to gather all the information, so it’s not a census in the modern sense of the word.
Sinclair went on to become the oldest founder member of the Statistical Society of London (now the Royal Statistical Society) in 1834.
For a more successful early attempt to gather data on a population, let’s take a look at Sweden, where astronomer Pehr Elvius also took an interest in the number of babies being born, though for different reasons.
Elvius – the early numbers
Pehr Elvius was appointed to the Swedish Royal Academy of Science in 1744, and in the same year he published an article in their journal under the title, Catalogue of the annual number of children that are born in U—— town during the last 50 years, with reasons for remarks upon it.11 The coy initial disguising Uppsala town was to baffle Sweden’s foreign enemies, from whom Elvius was keen to disguise the size and health of the population.
By the early eighteenth century, although each diocese had records of births and deaths from all its parishes, these records were not always kept in the same format, or compiled centrally. But in the 1730s there was rising interest in using the data. Bishop Erik Benzelius presented some of the figures to parliament in 1734, a national health board with responsibility for development of the population was set up in 1737, and the Royal Academy of Science12 was founded in 1739.
Sweden was, at the time, a largely agrarian country, and had recently lost a war, and with it some provinces in northern Europe. So they had reason to worry about the size and vigour of their population.
The first Swedish census, or Tabellverket, was taken in 1749 by parish priests, completing the forms designed by Pehr Elvius and three other members of the Royal Academy of Sciences. However, reality proved messier than the forms. Parish priests were used to recording births, marriages and deaths, but less happy with age and occupation.
The annual census was reduced to being a three-yearly and then a five-yearly event. Pehr Wargentin, another astronomer who succeeded Elvius as Secretary of the Tabular Commission, noted that teenagers were implausibly likely to be recorded as under 15, and older citizens to be over 60. Only those aged 15–60 were liable to pay tax.
The urge to count the people of Sweden, to monitor their marital habits and to predict their future childbearing, survival and productive work came from the desire to shape the country’s future wealth. Borrowing the term ‘political arithmetic’ from Englishman William Petty,13 researchers quantified everything from people to pigs in specifics that William I of England would have envied. Not only did they calculate that a woman could count as three-quarters of a man, in terms of work, but they also predicted how large a population each parish could support.
This period of Swedish history is known as the Age of Freedom, but by today’s standards that freedom was limited for most of the population. The constitutional monarchy, a parliament dominated by aristocrats and wealthy merchants, and the strong social and moral power of the church ran the lives of the masses. The vision towards which they were herding the people was one of prosperity, peace and an expanding population. Wargentin even suggested that emigration should be made a criminal offence.
Sweden led the way in statistics, in a state counting and measuring its own population. But many of the underlying ideas came from wider Europe, where the sense was growing that, by applying science and reason, society and people themselves could be made better. And it is to Enlightenment France that we go next.
Laplace’s Demon
The different numbers of boy and girl babies also attracted the interest of Pierre Simon Marquis de Laplace, a Frenchman who was smart enough to combine astronomy, physics, mathematics and staying alive during the French Revolution.14
As the title of his first book, Exposition du Système du Monde (an exposition of the system of the world) suggests, Laplace was confident that science and mathematics could explain everything. And by everything he meant not only planets, light and heat, but also human population and even the likelihood that a jury will reach the correct verdict.
Laplace believed firmly that:
We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at any given moment knew all of the forces that animate nature and the mutual positions of the beings that compose it, if this intellect were vast enough to submit the data to analysis, could condense into a single formula the movement of the greatest bodies of the universe and that of the lightest atom; for such an intellect nothing could be uncertain and the future just like the past would be present before its eyes.
This theoretical, all-knowing intellect has become known as Laplace’s Demon. The deterministic view of the world it expresses, a secular version of the orderly universe controlled by an omniscient deity, gained popularity after the Enlightenment. It still convinces some people today, though it’s hard to reconcile with the idea of humans having free will. If everything in the future is determined by what happened in the past, that leaves no room for us to make choices.
Laplace did think people are determined by their past, so his theoretical Demon would know what we will do in the future. But he was also a sensible man with enough worldly wisdom to keep working throughout three major regime changes and evade the guillotine. Laplace recognised that a mere human, even one as clever as him, couldn’t achieve this kind of objective, all-knowing certainty about the future. The best he could hope for was to measure his own ignorance, and calculate the odds that different versions of the future would turn out to be true.
He wasn’t particularly interested in babies – it’s just that both France and England were keeping good records of births, which provided masses of raw material. What really interested Laplace was how to start with observations from nature and work back to the causes of things. Specifically, to work out what causes the movements of stars and planets, by observing the heavens. How could he use recorded measurements to understand why things happen?
When Laplace observed that more boys than girls were born he asked himself, like Arbuthnot: How likely is it that I’m just seeing natural variation at work?
Laplace used an approach first invented by English nonconformist clergyman Thomas Bayes in 1764, though he didn’t find out about Bayes’s work for some years.15 Starting with the same question as Arbuthnot, Laplace asked: If I assume that, in the long term, exactly equal numbers of boys and girls are born, what are the odds that I would get the results these birth records are showing me?
Like Bayes, Laplace read The Doctrine of Chances, a contemporary book on gambling by Abraham de Moivre. Both Bayes and Laplace made the same leap of imagination.
You can put a number on how likely you think you are to win in the future: one in six for a fair dice, for example, or a 50-50 chance a coin will come down heads. Can you also put a number on what happened in the past, based on what you know about the results? Not a definite number, perhaps, but a range of likely numbers, with some idea of just how likely that range is to contain the true answer.
It’s a tricky idea, and Laplace wrestled for a long time with turning his hunch into useful rules and methods. His fundamental approach, starting with a prior probability – 50:50, for example – and using the data you observe to change that to the most likely posterior probability, is still in use today, bearing Bayes’s name.
Using these methods, Laplace showed not only that the underlying birth ratio in Paris was extremely unlikely to be anything except ‘more boys’, but also that the ratio in London was very probably even more male-weighted than in Paris.
But what Laplace really wanted was a numerical description of how far wrong his results were likely to be, bringing together the idea of probability, or chance, with observations of the real world.
His Analytical Theory of Probability, published in 1812, developed the idea that the observed birth rates, for example, will vary from the underlying average according to a predictable pattern. Though the difference between real life and his theoretical model would look random to the human eye, even if the deterministic demon knew how every last atom of the universe must fall into position.
Randomness, against everything we naturally expect, follows predictable patterns. Surprisingly, for large enough numbers of observations the emerging data will follow a similar pattern to what you’d get by tossing a coin.
If you repeatedly toss it 10 times, the expected value, five heads, will be the commonest result. Less likely results appear less often. Even if you had no idea beforehand that a coin is equally likely to come down heads or tails, you would eventually infer that it must be so, by looking at the overall spread of results.
And the more often you repeat an experiment, or observe the world, the more dramatically the results converge on the true underlying average value.
Laplace’s star pupil, Siméon-Denis Poisson, developed this work further, publishing Poisson’s Law Of Large Numbers in 1835. Their assertion that human events such as suicides can be studied in the same way as the movements of planets was controversial at the time, but their results did correspond to what was recorded in official statistics.
Average man – are you normal?
While Laplace and Poisson16 were working in Paris, they met a young Belgian, visiting the city to study astronomy.
Adolphe Quetelet had shown early interest in painting and sculpture before becoming a mathematics teacher in Brussels. He had published poetry and translated Romantic writers Byron and Schiller into French, but meeting the French mathematicians set him on a new course in understanding mankind.
Quetelet continued his career in astronomy, but his use of statistics to study humanity went much further than Laplace or Poisson. He collected figures on everything from height and weight to age at marriage or penchant for committing crime.
If you’ve ever calculated your Body Mass Index (BMI), you’re using Quetelet’s work. Known until 1972 as the Quetelet Index, it is taken directly from his observation of the average relationship between height and weight.
Quetelet studied variations between individuals and across time, finding that both physical and social characteristics often fall into the same pattern around a characteristic value, if the numbers are large enough. This shape, known as the normal distribution or bell curve, is the same one used by Laplace and Poisson to describe how measurements vary around an underlying average.17
In 1831, Quetelet published two pamphlets, one on variations in bodily measurements, and the other on crime. In the second, he spelled out the implications of studying populations instead of individuals:
The greater the number of individuals, the more the influence of the individual will is effaced, being replaced by the series of general facts that depend on the general causes according to which society exists and maintains itself.
What would Quetelet’s erstwhile hero, ‘mad, bad and dangerous to know’ poet Byron, author of Don Juan, have made of this usurping of individual will by general facts and general causes?
Quetelet’s major work, published in English as A Treatise on Man in 1842, introduced the term ‘social physics’. It was a provocative parallel between the mass of humanity and the deterministic behaviour of heavenly bodies and forces that he studied in his Brussels observatory. He also coined the term ‘L’homme moyen’: average man.
The idea that crime, for example, is a product not of individual moral failings but of environmental conditions and influences, was central to his work. ‘Society prepares the crime, and the guilty person is only the instrument by which it is executed.’
One consequence of this is the idea that, by changing the circumstances in which people live, you can reduce the chance they will commit crime. Social reformers have worked on this basis for centuries. British Prime Minister Tony Blair promised to be ‘tough on crime, tough on the causes of crime’, seeking to balance the pledge of justice – punishing those who have broken the law – with the idea that crime has causes beyond the individual.
Quetelet himself tried to distinguish between different types of cause: an individual’s ‘tendency to marry’, for example, and the opportunity he has, or doesn’t have, to fulfil that desire. He was careful to say that a person’s predispositions, their environment and chance all have an influence on their actions.
For Quetelet, the Average Man was no figure of speech, but some kind of ideal that Nature aimed to produce with every individual. Deviations from this ideal were down to the influence of chance elements, which explained the fact that real people vary from the average in a similar pattern to such truly chance events as tossing the same 10 coins over and over again.
Whatever the cause, it remains true that many human characteristics do vary in this predictable pattern around an average value. Deviations from the normal distribution may genuinely be a sign that something interesting is going on.
Quetelet himself looked at the height records of 100,000 conscripts to the French army in 1817. As expected, he found that their heights varied, with over a quarter falling between 1.597–1.651m (5ft 3in–5ft 5in), and18 fewer in the other height categories, with numbers of men decreasing as the heights get further away from the mean.Except, that is, for the lowest category, those shorter than 1.57m (5ft 2in). There were 28,620 of these short men, far more than Quetelet expected from his mathematical model.
However, he didn’t spend long wondering about the cause: men in this category were too short to be eligible for military service. And sure enough, there were fewer men than he expected in the two height categories above the cut-off measurement. In short, the French army had lost a couple of thousand men to strategic slumping, hunching and bending of the knees.
Florence Nightingale was very influenced by Quetelet and his book, corresponding with him and calling him the founder of ‘the most important science in the world’, statistics. As tutor to Albert of Saxe-Coburg, Quetelet had an enduring influence on the man who later became Prince Albert, the consort of the British Queen Victoria. Albert introduced Quetelet to the pioneer of mechanical calculation, Charles Babbage, whom we’ll meet in the next chapter, and to the Reverend Thomas Malthus.
Common census
In 1799, the Reverend Malthus travelled through Sweden and observed that people were padding out their bread with tree-bark. In a revised 1803 edition of his Essay on the Principle of Population, he linked this starvation to the increase in Sweden’s population from 2,229,661 in 1751 to 3,043,731 in 1799.
He took his figures from the comprehensive records in the Tabellverket, but drew the opposite conclusion from the Swedish statisticians: that a growing population is a bad thing, and leads to less wealth, not more.
Malthus sought to show that an increasing population must inevitably lead to disease and famine, a pessimistic view of the human future that persists to this day, in spite of a world population that has vastly outstripped every doom-laden prediction from Malthus onwards.
The first proposed census in Britain was rejected in 1753, partly on grounds of superstition. In the Bible, King David orders a census, which is followed by a plague.
But there were also political reasons. The population was unwilling to be scrutinised, fearing it would lead to more taxes and military conscription, with one member of parliament declaring the project
To be totally subversive of the last remains of human liberty … The addition of a very few words will make it the most effectual engine of rapacity and oppression that was ever used against an injured people … Moreover, an annual register of our people will acquaint our enemies abroad with our weakness.
It may have been the last point that clinched the argument, and the project was dropped.
But when Malthus’s essay was first published in 1798, it fed growing fears in Britain that the population was outstripping its capacity to produce food. A bad harvest in 1800 stoked fear of starvation, or perhaps of hungry rioting masses.
The first UK census was authorised in the 1800 Population Act, and taken in March 1801. Parish officers and other worthies counted the number of houses, men and women, and their general area of employment: agriculture or manufacturing. The clergy reported the number of baptisms, marriages and deaths.
Ten years later, the exercise was repeated, with a few extra instructions, such as recording people’s ages in five-year groups, and using blotting paper when taking records in ink. With one exception, Britain’s population has had a census every 10 years since 1801.
People weren’t always happy about being thus surveyed.
In 1911, women campaigning to be given the vote organised a mass boycott, declaring, ‘women do not count, neither shall they be counted’. Some of them stayed away from home on the night of the census, listening to an Ibsen play in Portsmouth or staying in horse-drawn caravans on Wimbledon Common. By hiding in a Westminster broom cupboard, Emily Wilding Davison recorded her address as ‘the Houses of Parliament’, an act now commemorated with a plaque inside that cupboard.
As in Sweden, the parish registers kept by the priests were not entirely satisfactory. Since 1753, marriages were only legally binding if conducted within the Church of England. Many Nonconformists, such as Baptists, got married in their local Anglican Church, but Roman Catholics sometimes chose to marry illegally within their own church and thus evaded the register. Baptisms or circumcisions, and burials, were also conducted in other congregations. The Parish Register Act of 1812 tried to tighten things up, but a growing, industrialising nation needed more.
Thomas Henry Lister’s chief contribution to the world of literature is the 1826 romantic novel Granby, in which the eponymous hero’s love for Miss Jermyn overcomes her parents’ opposition when he is revealed as the heir to Lord Malton. Which seems scant qualification for his being the first Registrar-General of Births, Marriages and Deaths of England and Wales. Nevertheless, Lister took up this post in 1836, and set up the General Register Office.19
From 1 July 1837, all births were to be registered, though some parents failed to do so, either from negligence, or to avoid compulsory smallpox vaccinations. Deaths were to be reported by next of kin, or whoever was present at the death or found the body, and marriages by the officiating minister of whatever religion.
Next, Lister turned his attention to the census, next due in 1841. For the first time, every household received a form, and was asked to report not only the number of people in the house on a specified date but also their names, occupations, and birth parish. Soldiers and sailors were now included, along with 5,016 persons travelling on trains at the time of the census.
This was also the last time that the parish registers were included in the report: the General Register Office was now a more reliable and comprehensive source of information.20
Death by worms
The job of turning the census returns into a useful document fell to William Farr, who classified occupations and related them to the death records, an enormous task at a time when paper records had to be compared and transcribed by hand. He noted that miners ‘die in undue proportions’, and that tailors were dying in surprising numbers between the ages of 25 and 45.
Farr was a qualified doctor with a keen interest in public health, and saw the importance of a standardised classification of causes of death based on logical principles, not the prevailing alphabetic system running from ‘abortives’ to ‘worms’. However, it took years of argument before the International Statistical Institute agreed the first International List of Causes of Death in 1893.
The list in use today is the tenth revision, and includes entries such as ‘X24 – contact with centipedes and venomous millipedes (tropical)’ and ‘N48.3 – Priapism’.
Farr was not only concerned with death. His annual reports as Compiler of Abstracts, later Superintendent of Statistics, at the General Register Office also covered topics such as the different factors that affect one’s likelihood of getting married, such as war, abundance and wage levels. He developed life tables to predict the life expectancy of different groups, and designed the Post Office insurance scheme. He worked with Florence Nightingale on army hospital reform, on the Indian Sanitary Commission and the Royal Commission on Mines.
However, Farr did spend a lot of time analysing the causes of death. Not only the direct, proximate reason why an individual has departed the world, but also the factors that might have hastened that departure, and could possibly be prevented, or at least reduced.
One focus of his keen analytical eye was cholera, which killed thousands in Britain during the nineteenth century after arriving from India in 1831. After a second outbreak ravaged London in 1848–49, Farr collected data on where the deaths happened, and on factors that might explain the difference in death rates between districts.
South London was worst hit, with nearly eight deaths per thousand residents, compared to little over one death per thousand in north London. Farr gathered information on the density of the population in each district, measures of poverty and the underlying annual death rate. But the measure that most interested him was elevation above the high-water line of the tidal river Thames.
The prevailing theory of the time for how cholera spread was miasma: bad air, rising from the river, and carrying the illness with it. And the river Thames in the early nineteenth century was truly foul. The growing city’s human and animal sewage, and other rubbish, all ended up in the Thames, untreated, where it festered until the tide took it out to sea. The river was wide, with shallow, sloping banks, so the raw sewage probably had a few days in which to achieve maximum ripeness under the noses of the unfortunate Londoners.
Farr analysed his data for the 38 registration districts of London, combining the districts according to elevation above high water, in 3m (10ft) categories. The results were striking. The mortality rate for cholera in 1849 was highest within 6m (20ft) of river level, with over 10 people per thousand of the population dying of cholera that year. Move up to a district 9–12m (30–40ft) above the stinking Thames, and the death rate falls to around six per thousand, and so on, with the improvement becoming more gradual in higher districts. Above 104m (340ft), less than one person in a thousand died of cholera.
Conclusive proof, as far as Farr was concerned, that the miasma theory was correct, and that eliminating the noxious gases would reduce the dreadful death toll. And if you saw the results as an infographic in a modern newspaper, you’d probably agree.
The relationship between elevation and your odds of dying of cholera was large, consistent, and showed a greater effect for a greater exposure to the risk factor: in this case, being low down where the foul air lay. It also fitted current scientific theory.
However, another doctor had a conflicting theory. John Snow, a physician living and working in Soho, published On The Mode of Communication of Cholera in 1849. Snow had first encountered the disease in Newcastle when, apprenticed to a surgeon, he treated patients there in the 1831–32 epidemic.
His accounts of how whole families were wiped out within days are terrifying, and call to mind the Ebola outbreak ripping through parts of Africa as I began writing this book, with the same factors of poverty, people living in close quarters with inadequate water supplies, and failing to dispose of contaminated belongings.
Closely observing how the disease appeared to spread from patient to patient within families, or via clothes and bedding belonging to somebody who had died of cholera, or to people who washed and laid out a body, Snow developed the theory that it was spread via the alimentary canal. To put it bluntly, the diarrhoea that killed one cholera patient got into the mouth of the next victim, one way or another.
Patterns in the way the cholera epidemic spread suggested to John Snow that just washing the bedlinen used by one patient could be enough to pollute a water source, and that everyone who subsequently drank from that source was at risk of catching the disease. Observations in overcrowded courts around London, where waste water leaked into the well or spring from which people drank, reinforced his hypothesis that water transmitted cholera.
In 1854, Snow was living at 54 Frith Street, Soho Square. On 3 September, he heard of a sudden upsurge in cholera cases in nearby Broad Street – 83 deaths registered within three days. His suspicion fell on the water pump from which many local homes, pubs and coffee shops got their drinking water, but when he looked at the water, it appeared clean.
Nevertheless, seeing no other likely cause for such a violent outbreak he continued his investigations. Taking copies of the death registrations for the period, he found that most of the victims lived in houses for which the suspected pump was the nearest water supply. Of the 10 victims who had an alternative source of water closer at hand, eight were known to drink from the Broad Street pump.
The map on which Snow marked deaths by cholera with black bars has become one of the classic documents of epidemiology. The darkening of the map around the guilty pump is striking. However, the relationship between closeness to the pump and odds of dying is not entirely simple. For example, the brewery near the pump in Broad Street employed over 70 workmen, and not one of them died of cholera. The workhouse in Poland Street, whose crowded conditions and underfed inmates were surrounded on all sides by infected houses, lost only five inmates out of 535. If the workhouse had reflected the death rates of the surrounding streets, over 100 of them would have been dead.
Snow’s careful work revealed that both brewery and workhouse had their own well, strengthening his case that water supply, not mere proximity, was the main risk factor. The brewery proprietor, Mr Huggins, told Snow that his men got an allowance of malt liquor, and he did not believe they drank water at all.
To add more strength to his argument, Snow found specific cases where individuals who passed through the area just long enough to eat and drink subsequently died of cholera. He even found a Hampstead lady who died after drinking Broad Street pump water, which she had delivered daily because she preferred the taste. Her niece, visiting from Islington, went home and died from cholera. Neither Hampstead nor Islington had any other cases at the time.
On 7 September, Snow presented his evidence to the local authority of its day, the Board of Guardians of the Parish of St James, and requested that the pump handle be removed to prevent any further deaths caused by drinking the contaminated water. The guardians, while not entirely convinced by Snow’s theory, must have thought it was worth a try, because they agreed.
Since so many had already died, or fled the area, the outbreak may already have been coming to an end, but the removal of the pump handle is commemorated every year with a Pumphandle Lecture, followed by a drink in the John Snow pub in Soho.
However, neither William Farr nor general scientific opinion was convinced by Snow’s theory, in spite of the fact that Farr’s data on the 1848–49 cholera epidemic in London, which had apparently confirmed the miasma theory, also included information about the sources of water in the different districts he studied.
Water supplies in London were provided by private companies, and if you weren’t lucky enough to have your own well or pump, your drinking water would be piped in from the Thames or one of its tributaries. This meant that most of south London was drinking water drawn from the Thames between Battersea Bridge and Waterloo Bridge, downstream of the places where sewage was flowing into the river. Coincidentally, most of these districts were also low-lying.
So the clear link that Farr saw between low elevation alone and high risk was erroneous; low elevation did not cause the disease, or mean that one was at a greater risk of catching it.
If, instead of sorting the districts by elevation, he had sorted them by main water supply, he would have seen that the difference between those categories was stark. In districts getting their water from the Thames above Hammersmith, cholera killed between 11 and 19 per 10,000. Among those taking their water from the Thames below Battersea Bridge, between 77 and 168 people per 10,000 died from cholera.
In the long run, the story has a happy ending. The stench of the Thames became unbearable. The Houses of Parliament, lying alongside the river, were so filled with the Great Stink in the summer of 1858 that MPs were driven out of the building, and swiftly approved a bill to create a new sewer system for London. Joseph Bazalgette’s ambitious scheme gathered all the city’s sewage and piped it far downstream, separating drinking water from human waste.
By ending the stink, wrongly blamed for spreading disease, the Victorians inadvertently solved the real problem: contaminated drinking water.
An unhealthy legacy
‘The word Eugenics,’ begins a Jewish Chronicle article from 1910, ‘will be for ever associated with the name of Sir Francis Galton, who has devoted a long life to the pursuance of a high ideal – that of improving the fitness of the human race …’
That may have been true then, when Francis Galton was still alive, aged 89. Today, the word eugenics has more chilling associations.
Back in 1910, eugenics was a fashionable and popular idea with people who considered themselves progressive. Socialist writer George Bernard Shaw, while strongly against state-enforced eugenics, believed that social reforms would eventually lead to selective breeding of better human beings. Some campaigners for the availability of birth control were motivated not only by the desire to enhance women’s reproductive freedom, but also to reduce the population among certain sectors of society.
Charles Davenport wrote to Francis Galton from America in October 1910, telling him ‘the seed sown by you is still sprouting in distant countries’. Davenport had opened the Eugenics Record Office (ERO) at Cold Spring Harbor in Long Island, New York, and published Eugenics: The Science of Human Improvement by Better Breeding that year. Davenport’s successor Harry Laughlin used data collected by the ERO to campaign for public policy including compulsory sterilisation and restrictive immigration laws, some of which were used as models by the Nazi regime in Germany.
The Eugenics Record Office closed in 1939, but several states had already passed laws based on Laughlin’s recommendations. Sir Winston Churchill was strongly in favour of ‘the improvement of the British breed’ by segregation or sterilisation of the ‘feeble-minded’, and Canada established the Alberta Eugenics Board in 1928 to sterilise ‘mentally deficient’ individuals against their will, under an act not repealed until 1972. Australia, Iceland, Norway, Sweden and Switzerland are among other countries that have sterilised the mentally ill or handicapped.
Today, alongside its educational work in genetics, the Cold Spring Harbor Laboratory sells books, including Murderous Science: Elimination by Scientific Selection of Jews, Gypsies and Others in Germany 1933–1945 and The Unfit: A History of a Bad Idea.
But when Francis Galton, a cousin of Charles Darwin, coined the term in 1883, he described it as ‘the study of the agencies under social control that may improve or impair the racial qualities of future generations, either physically or mentally’. He would have seen himself very much as a scientific, rational man of his time, looking, like Quetelet, for ways of harnessing science to improve the future of humanity.
Galton was no great mathematician. He qualified as a doctor, but then an inheritance allowed him to give up medical practice and explore whatever interested him. Initially, that meant Africa. Next, he turned to the study of the weather, collecting meteorological data from across Europe for December 1861, and creating visual charts that enabled him to look at many different variables at once. By comparing wind direction, temperature and pressure, he discovered the anticyclone.
When his results were published in 1863 as ‘Meteorographica’, Galton included a note of warning to those about to be dazzled by what today we’d call an infographic: ‘it is truly absurd to see how plastic a limited number of observations become, in the hands of men with preconceived ideas.’
With his illustrious cousin Charles, Galton shared an illustrious grandfather: Erasmus Darwin, poet and scientist. In fact, the family produced an impressive number of prominent men, which may have led Galton to wonder whether illustriousness was an inherited trait like height. Galton made this comparison explicit in his book, Hereditary Genius, in which he developed some of Quetelet’s ideas about average man to look at individuals instead of populations.
Although an understanding of how genetics worked was still years away, people had begun to observe that inheritance could be mathematically predictable. Not only individual traits such as colour blindness, but qualities that are continuously variable, such as height, follow mathematical patterns in the population. And observations made about animals or even plants could also apply to human beings.
Galton was interested in how qualities such as height were passed down from parents to children. Bribing his experimental subjects with the chance to win money, he collected lots of data on the heights of parents and children, adjusted for the fact that women tend to be shorter, and averaged the parents’ height to get a theoretical ‘midparent’ against which to compare the children.21
Galton noticed that, although the average heights of children followed their midparent’s height quite closely, the children of very tall or short parents were not as extremely tall or short themselves.
Having read Quetelet’s work in 1863, Galton was familiar with the idea of the normal distribution, and made his own illustration of the Law of Deviation from an Average, showing how the heights of a million men would be grouped, mostly within a few inches of the average, with fewer and fewer individuals at the extreme ends of the scale.
Now Galton had discovered a principle that’s so deceptively simple it’s surprising how often we forget about it. Things slide back towards the middle, like people sharing a hammock rolling into the centre.22 Today we’d call it regression to the mean, or going back to the average.
One current example is speed cameras, which are put up on dangerous stretches of road after a run of accidents, to slow down traffic and prevent future collisions. Accident rates do tend to fall after a speed camera is installed. The problem is, you’d expect the number of accidents to go down anyway, whether or not a speed camera goes up.
Don’t believe me? Let’s think about earthquakes instead. According to the US Geological Survey (USGS), there are 16 earthquakes per year of magnitude 7 or above, worldwide. And with two offices in California, I assume they’re paying very close attention. In 2010, however, the USGS recorded 24 of them. A worrying rise! Was Max Zorin carrying out his evil plot to destroy Silicon Valley by triggering the San Andreas Fault?
Of course, they don’t really have an earthquake-prevention device. Max Zorin is James Bond’s nemesis from 1985 film A View to a Kill, and his fictional plan is as gloriously silly as any other Bond villain. Nothing sinister is going on, just natural variation around an average. The 10-year high in 2010, a year I chose because it was a 10-year high, was followed by lower counts, just as you’d expect.
And in the same way, a four-accident high on our fictional road would usually be followed by a fall back towards or below the average value. Research suggests that speed cameras can have some effect in reducing accidents, but less than you might think by looking at the figures from just before and after their installation.
What do speed cameras and earthquakes have to do with Galton’s children and mid-parents I hear you ask? If the underlying tendency is to be the same height as your midparent, but a series of chance factors all contribute to making you much taller, it’s unlikely that your own children will be dealt exactly the same hand of genetic and environmental cards. If you think of very tall, or very short, parents as extreme variations from the average, then you’d expect them to be followed by a return towards more common values. Which is what Galton observed.
However, this ‘regression towards the level of mediocrity’ was only the first step towards what Galton was really looking for. He wanted to know whether heredity could be expressed mathematically. As he put it himself:
Given a man of known stature, and ignoring every other fact, what will be the probably average height of his brothers, sons, nephews, grandchildren, &c., respectively, and what proportion of them will probably range between any two heights we please to specify?
Applying the patterns from a population to predictions about individuals is one of the oldest problems in data, or statistics. As Galton put it: ‘Whatever is statistically certain in a large number is the most probable occurrence in a small one.’
If you know nothing about an individual except the population from which you’ve picked them, there’s a simple way to calculate the probability of their height falling within a specific range, using the normal distribution, or bell-curve. If you know the mean value, and a couple of other things about how the other values spread out from it, you can calculate the chance that one value will fall within a given range.
What Galton had developed was a way to use both this data about the underlying population and the specific height of the midparent, or uncle, or brother to predict the height of the child, or nephew, or brother. Or rather, to predict how likely that height is to fall within a certain range. He called this the ‘ratio of regression’, based on:
a compromise between two conflicting probabilities: the one that the unknown brother should differ little from the known man, the other that he should differ little from the mean of his race. The result can be mathematically shown to be a ratio of regression that is constant for all statures.
Galton calculated the probability that a son would be at least the same height as his father at 50 per cent for a father 1.73m (5ft 8in) tall, but at less than 1 per cent for a father 1.96m (6ft 5in) tall. Francis Galton called this measurable link between two sets of data ‘correlation’.
Height wasn’t his main interest, however. Having done all this work, he turned his interest swiftly back to studying how genius could be passed on through families. In 1874, he had conducted a survey of 150 prominent British scientists, from which he concluded that key factors were energy, general good health, independence of mind and an interest in science, but only when combined with discipline and focus. He then called for a wider application of the statistical principles he was applying to physical characteristics in a population to character traits. ‘The habit should therefore be encouraged in biographies, of ranking a man among his contemporaries, in respect of every quality that is discussed, and to give ample data in justification of the rank assigned to him.’
He even entertained himself, during a tedious talk being given by somebody else, by measuring the frequency, amplitude of physical movement and duration of fidgeting among the audience. It was an average one fidget per person per minute, in case you want to carry out a similar study next time you’re at a boring meeting.
Not everybody was readily convinced that the ideas of correlation and regression could be applied to social questions. ‘Personally, I ought to say that there is, in my opinion, considerable danger in applying the methods of exact science to problems in descriptive science, whether they be problems of heredity or of political economy.’ These sceptical words were spoken by Karl Pearson, in his lecture on Galton’s 1889 book, Natural Inheritance, at the Men and Women Club.23
Pearson’s interests stretched to law, philosophy, the history of science, German language and literature. Then a professor of Applied Mathematics at University College London (UCL), in his spare time he gave talks on Marx and Martin Luther. So he was already seeking the reasons behind political and economic events. But was it reasonable to expect sociological theories to be as exact and logical as mathematical ones?
Pearson was impressed by Galton’s work on correlation and regression. His initial ambivalence about transferring the methods of physics or astronomy to human beings was evidently resolved, as he became the first Galton Professor of Eugenics at UCL in 1911, after leading Galton’s Eugenics Record Office at UCL since 1906. Combining the urge to improve human life with excitement about the emerging science of evolution, Pearson developed Galton’s methods to study more complex situations, in which numerous different causes might be at work.24
Speaking at a dinner in his honour in 1934, Pearson speaks of the ‘culmination’ of eugenics lying:
In the future, perhaps with Reichskanzler Hitler and his proposals to regenerate the German people. In Germany a vast experiment is in hand, and some of you may live to see its results. If it fails it will not be for want of enthusiasm, but rather because the Germans are only just starting the study of mathematical statistics in the modern sense!
Words worth remembering whenever anybody suggests that some social problem could be solved, if only everybody understood mathematics better. Pearson himself died in 1936, so thankfully he never had to see the horrific outcome of the German ‘experiment’.
Today, any suggestion that human beings should be selectively bred like farm animals is generally regarded as eugenics, a word with chilling echoes of the holocaust.
Even parents who want to use medical techniques to avoid passing on hereditary diseases to their children have to overcome this fear that creating ‘designer babies’ is the first step towards wiping out the rest. It’s important to remember, I think, that there’s a world of difference between loving parents wishing good health for their future child and others deciding they know what’s best for your children, or for the human race as a whole, and imposing it upon you.
But don’t let’s end this chapter on a bleak note. Statistics, which I like to regard as big data’s early, acoustic stuff before they were famous, has given us many gifts. Better medicine, better food crops, even better Guinness, all owe a lot to statisticians. They could probably tell us how much, within a 95 per cent confidence interval.25
Applying mathematics to understanding the real world is an art as well as a science, and finer minds than mine continue to grapple with statistics. But today, whether or not they accept the label big data, they tend to use computers to do the heavy lifting. And that’s where our little historical tour goes next: the Industrial Revolution of statistics.
Notes
1 Nobody in England was aware that Caribbean Islands even existed in the eleventh century, let alone that they could avoid tax by having a bank account on one. Banks were pretty rudimentary, come to that.
2 Latin was the most universal language of the day, especially important when ‘English’ people spoke Anglo-Saxon or French. Just as English today is the international language of science, computing and tourist menus, in the Middle Ages being able to read and converse in Latin meant you could communicate with educated people and order gruel in overpriced inns across the known world. Known to Europe, that is.
3 Though, as some parishes only had one death in a given week, it would have been easy to find out who it was by comparing the bill of mortality with the parish register of births, marriages and deaths. So it’s also an early example of data that looks more anonymous than it really is.
4 And an early example of data as a valuable commodity.
5 First published in German as Das Akkumulations-Zusammenbruchgesetz des kapitalistichen Systems (Zugleich eine Krisentheorie) – arguably even less catchy.
6 Or the mean, to be exact. Sometimes ‘average’ is used for the median, or middle value. For example, ‘average earnings’ usually means the earnings of the middle person, if you imagined lining up all the people in the country in order of earnings, and counting to the exact halfway person.
Which would be quite impractical, even assuming we were all honest about how much we earn. But I think you get the idea.
Median = middle. Mean = added up and split equally, like a restaurant bill when there are no penniless students saying, ‘but I didn’t have a starter’.
7 Or, in theory, exactly equal numbers, but that’s very unlikely.
8 Throwing a coin six times gives 64 different possible results (2 × 2 × 2 × 2 × 2 × 2 or 26), if we care about the order of heads and tails. Of those, only six fulfil our condition of one head and five tails. So the probability of getting such an unlucky result is 6 in 64, or 3 in 32 if you prefer. Which is unlucky, but not so very unlikely.
If you spent an entire evening tossing the same coin in groups of six throws, you’d expect to get that result nearly one time in 10. And for your friends to say you need to get out more.
9 Just under, including the chance of exactly equal numbers.
10 By the time the babies reach an age to marry, the numbers have evened up, which Arbuthnot took as evidence that God weights the dice to compensate for more boys dying early.
11 Still not that snappy.
12 Yes, the same one that gives out Nobel Prizes for science today. In spite of its name, the Royal Academy of Sciences was an independent body, set up by scientists, merchants, civil servants and politician Count Anders Johan von Höpken, a founder of the Hat Party.
13 The same one who got the credit for Graunt’s book.
14 He wasn’t a marquis until long after the revolution, when he’d also survived the rule of Napoleon Bonaparte and the Restoration of King Louis XVIII. He was, however, a member of the Royal Academy of Science and a teacher at the Royal Military School, and plenty of other scientists lost their heads at this time.
15 When Laplace did find out, he gave Thomas Bayes credit for the discovery, and this kind of approach is still known as Bayesian. Expressed in mathematical form, it’s called Bayes Theorem. But Laplace did much more work to turn it into a usable method.
16 Poisson also gave his name to the Poisson Distribution, which describes how many rare events occur in a particular period of time, if we know the overall rate at which they happen, though each individual event is random.
For example, statistician Ladislaus Bortkiewicz counted the number of Prussian cavalrymen killed by the kick of a horse over 20 years. In each army corps the number of such deaths varied between none and four each year. In any given year, over half the corps had no deaths at all.
Independently of Poisson, Bortkiewicz spotted the same pattern of variation, and published a paper on it, confusingly called The Law Of Small Numbers. Some people claim that we should be talking about the Bortkiewicz Distribution, not the Poisson Distribution. I predict that if we did so, the variations in spelling would take on a much wider and more random distribution.
17 It’s also sometimes called the Gaussian distribution, after mathematician Gauss. Not to be confused with the Poisson Distribution, which looks less like a bell and more like a ski slope.
18 Records suggest that Frenchmen at the time were around 7.5cm (3in) shorter than their English counterparts. This may have been due to worse nutrition or general health. Intriguingly, studies of Englishmen recruited to the East India Company found that literate recruits were around 6.35mm (1/4in) taller than illiterate recruits. So perhaps reading this book will make you grow taller?
19 Scotland, though included in the UK census, did not get civil registration of births, marriages and deaths until 1855, when William Pitt-Dundas was appointed as the first Registrar-General for Scotland. He is not recorded as having written any romantic fiction whatsoever.
20 Meanwhile, in the US, the first state law requiring registration of deaths was passed in Massachusetts in 1842. In spite of the American Medical Association urging other authorities to follow suit, national coverage wasn’t attained until 1933.
21 Obviously the children were grown up. Comparing eight-year-olds with their parents wouldn’t be much help.
22 We refer to extreme values, far from the mean, as outliers, so we can imagine them as sleepers balanced precariously on the edge of the hammock, or fallen out and lying on the ground.
23 Not a singles club, but a progressive political gathering that Pearson helped found, and which discussed social issues such as the roles of the sexes. Though he did also meet his wife there.
24 He gives his name to the Pearson correlation coefficient, a measure of how closely two variables, such as height and weight, are related.
25 Which is the way statisticians describe a range of values that probably include the true answer, and how confident they are that the true value is within that range. The bigger the percentage, the more confident they are, as you’d expect.