4

The Government Enemy

In the early 1950s John von Neumann and John Nash were not the only geniuses associated with the RAND Corporation. RAND was the incubator for another intellectual revolution, as significant as game theory but completely independent of it. And this time the genius behind it was a lowly intern.

The earliest and most enthusiastic adopters of game theory had been the military analysts at RAND, who wanted to use its powerful mathematical tools to outwit the Soviets in Cold War nuclear strategizing. But logical rigour was everything to the RAND thinkers, and by 1948 a possible flaw had been spotted in the logic of analysing nuclear conflict in game-theoretic terms. Whether the game is Scrabble or Armageddon, game theory views the players in just the same way, as rational individuals. But was the dense black web of Stalin’s sprawling Soviet state best seen as an individual? In other words, who exactly was playing the nuclear-war game against the US? Surely not Stalin himself. The opponent should be thought of not as an individual but as a group, a collective. However, this interpretation poses a problem for game theory, because its players are assumed to have clear preferences between the alternatives they face. This assumption seems plausible for a hyper-rational individual but mysterious for a group. It raises some difficult questions: what does it mean to speak of the preferences of a collective, a group of people? Where do these preferences come from? And does it make any sense to talk about a group being rational?

Some things never change in office life. If you have some really tricky, abstract questions which can’t be avoided, ask someone who can’t say no, who hopefully won’t mind looking stupid. Ask the intern.

In the summer of 1948 a graduate student from New York named Ken Arrow was an intern at RAND. His answer to these questions was contained in a RAND report he completed the following year called ‘Social Choice and Individual Values’. By 1951 it was developed into a short book of the same name, whose influence was such that, two decades later, it helped Arrow win the Nobel Prize for economics at the age of fifty-one – younger than any other winner before or since.

It is hard to convey Arrow’s status among economists. After the Second World War, most academic economics changed – first in America and then worldwide – from a discipline that was recognizably like politics and history to something which looked more like a branch of applied mathematics. Ken Arrow was probably the single most influential figure in that radical repositioning of economics. Arrow just did economics differently from almost everyone who went before him, and more successfully than the few who had tried. Arrow established the benchmark for how the most prestigious research in economics – economic theory – should be done. He is frequently mentioned on lists of the ‘greatest’ Nobel laureates in economics, while his contributions arguably qualified him for three Nobel prizes, because of his work in effectively establishing three major fields of economic theory. Five of his students would also win the Nobel Prize.

Like many of his generation, Arrow’s studies had been disrupted by the war. Serving as a weather officer in the US Air Corps from 1942 to 1946, his work in the Corps established a pattern which would be repeated over the following twenty years: using mathematics for military ends, with his first published research, ‘On the Optimal Use of Winds for Flight Planning’ being from this period. Arrow and his statistician colleagues were required to forecast the number of rainy days a month in advance. They sent a memo to the General of the Air Corps, noting that their forecasts were unsurprisingly very poor and arguing that their group should be disbanded. Six months later the general’s secretary replied: ‘The general is well aware that your forecasts are no good. However, they are required for planning purposes.’ The forecasting continued.1

Arrow’s life through the 1950s and ’60s seems unremarkable, but then much about his life during this period remains classified: his central role in US military thinking gave him a ‘top secret’ level security clearance from 1949 to 1971.

The central mathematical result at the heart of Social Choice and Individual Values is what became known as the Impossibility Theorem. While the mathematical proof appeals only to specialists, the implications of the Impossibility Theorem have reached – and astonished – a wide audience. The easiest, shortest summary – the one which proved the most popular – can even fit nicely on a bumper sticker. It goes like this: DEMOCRACY IS IMPOSSIBLE. But this was a bumper sticker which could be worn with pride on the cars of an entire generation of academics studying politics who, after reading Arrow’s work, were more likely to call themselves political scientists.

How could this nihilistic view of democracy be taken seriously and presented not as a politico-philosophical argument but as an unimpeachable fact of mathematical logic? Arrow did not need to get bogged down in millennia-old debates about the feasibility of democracy, nor could he. His ideas were too different, too new. Indeed, he was able to sidestep most of the intellectual history of democracy altogether.

Social Choice and Individual Values invented a field of economics called ‘social choice theory’. Before Arrow came along, the field did not exist; it had no name, just a few dabblers. Arrow’s intellectual ancestor was less Aristotle, more Alice in Wonderland. The Reverend Charles Lutwidge Dodgson – better known as Lewis Carroll, the author of Alice’s Adventures in Wonderland – was a mathematician at Oxford in the second half of the nineteenth century. As well as maths and Alice, Dodgson did some pioneering thinking about voting systems. Before Arrow, Dodgson’s was by far the most comprehensive analysis of voting systems, but it was hidden away in unpublished pamphlets, so Arrow did not know about it when he began work on the seemingly obscure questions posed by the RAND analysts in the summer of 1948.fn1

Arrow certainly did not know that his answers, far from being a footnote to RAND’s military analysis, would change the way we understand democracy: in the light of Arrow’s work, democracy would come to seem fundamentally flawed, at best embodying a series of unconvincing compromises.

THE IMPOSSIBILITY THEOREM

In order to use game theory to construct Cold War nuclear strategy, RAND needed to assume that the Soviet opponent was rational. Since rationality in game theory essentially just means being consistent, it is a relatively easy test for an individual, even a paranoid megalomaniac like Stalin, to pass. But if a ‘player’ in the nuclear game is really a group of individuals, it is far from clear that groups are consistent in their views or choices. Arrow’s analysis of this problem began by assuming that if a group of people – whether Soviet military strategists or a bunch of friends – can be said to have a ‘collective preference’ about something, then that collective preference must be derived from the preferences of the individuals in the group. So Arrow’s attention turned immediately to voting systems: the various ways of combining the preferences of individual voters to determine a collective choice or preference.

In particular, Arrow wanted to know if the collective preference emerging from any sensible voting system would be consistent. Defining what he meant by ‘sensible’, he spelt out a set of principles, desirable features which he thought any sensible voting system should have. With these in hand, there seems an obvious way to proceed: study a variety of voting systems which possess the desirable features, assessing for each system whether the collective decisions which emerge from them are consistent. This is the approach researchers before Arrow would have taken. But Arrow did something profoundly different.

First, he saw that a voting system can be understood in a general mathematical way. It is like a computer program in which each voter enters their preferences, preferences that the computer amalgamates using a set of rules, and then announces the collective preference. Next, Arrow expressed in mathematical terms both the desirable features of voting systems and his idea about what it means for collective preferences to be consistent (we will return to this later). Putting all this together generated a completely unexpected incompatibility: there is, Arrow concluded, no voting system – including systems as yet unimagined – which both possesses the desirable features and produces a consistent collective preference. This was the Impossibility Theorem.

The essentials of the theorem were already there in the memo Arrow wrote as a RAND intern, which took him five days in September 1948.2 Arrow was a modest man. He recalled that he thought of the Impossibility Theorem only when all his other attempts to answer RAND’s questions had failed. Even genius interns have to sweat.

Arrow was lucky too. He could never have developed the Impossibility Theorem without knowledge of a then little-known mathematical language, the logic of relations, which he became aware of only through a series of unexpected events. First, because his father lost everything in the Great Depression, family poverty gave Arrow no choice but to attend City College in New York for his undergraduate education rather than a more prestigious university. Yet at that time City College had some impressive lecturers: the leading philosopher of logic Bertrand Russell was due to take up a Chair (professorship) there, and Arrow, already interested in mathematical logic, signed up to take Russell’s course. But second, Russell was fired before he even arrived, for ‘moral indecency’. The mother of a potential student (more potential than actual: she was only twelve) brought a lawsuit against City College on the grounds that Russell’s well-publicised philosophy of ‘free love’ would endanger the morals of female students. The trial judge thundered that the college was creating a ‘Chair of Indecency’.fn2 Then came the third twist. The college was unexpectedly able to find a distinguished replacement for Russell. Alfred Tarski, one of the twentieth century’s greatest philosophers of logic, left his native Poland in August 1939 to lecture at a conference at Harvard. As it turned out, he was on the last ship to leave for the US before the German and Soviet invasions of Poland and the outbreak of war. As a Jew, Tarski could not return to Poland; lacking an income, he was glad to be offered a job at City College. In spring 1940 he lectured Arrow on the logic of relations. Tarski was not just an expert on this new kind of logic; he was its pioneer. Arrow’s knowledge was cemented when Tarski, recognizing Arrow as a gifted student, asked this eighteen-year-old undergraduate to be the main proofreader for his definitive text on logic, published in 1941.3

Social Choice and Individual Values immediately garnered wide acclaim. But outside a narrow circle few could follow Arrow’s mathematics, and even fewer were interested. And to begin with, it was a very narrow circle. There was, for instance, a missing step in the proof of the Impossibility Theorem, a gap not spotted by reviewers before publication, probably because the mathematical language used was so new and unfamiliar that it obscured the omission. (The problem was spotted and easily fixed a few years later.) This mathematical barrier has helped both supporters and critics of democracy in subsequent years to ignore the details of Arrow’s work and to associate it instead with cruder, bumper-sticker messages. To go beyond that, we need to delve deeper, to get at least a flavour of what it means to speak of ‘consistent’ collective choices and how they can be mathematically impossible.

Since it is couched in purely mathematical terms, the Impossibility Theorem claims universal applicability. The mathematics does not place any restrictions on who is voting, how they are voting and what they are voting about (as long as there are at least three alternatives to choose between). It could be the Soviet military planners deciding which US city should be their primary nuclear target – or it could be some teenagers deciding whether to go out for pizzas, burgers or sushi. So when introducing a ‘paradox of voting’ on the second page of Social Choice and Individual Values, Arrow could have illustrated the paradox in a wide range of contexts. But of course, he used an example that reflected his RAND work, a choice between the three alternatives ‘disarmament, cold war or hot war’. Suppose the decision-makers are three RAND analysts, Tom, Dick and Harry, with the following preferences: Tom prefers disarmament to cold war, and cold war to hot war; Dick prefers cold war to hot war, and hot war to disarmament; Harry prefers hot war to disarmament, and disarmament to cold war.fn3 Bear with me. Now, if Tom proposes disarmament, then Dick and Harry could propose hot war, because they both prefer it (in a vote between hot war and disarmament, hot war wins). Similarly, if Harry proposes his first choice, hot war, Tom and Dick can form a coalition to outvote it, since they both prefer cold war to hot war. And again, if Dick proposes his first choice, cold war, Tom and Harry can form a coalition to outvote it, since they both prefer disarmament to cold war.

There is no overall winner because any proposal can be defeated by a counter-proposal from two out of the three voters. In other words, the group’s preference is inconsistent: disarmament beats cold war, cold war beats hot war, but hot war beats disarmament. Arrow’s Impossibility Theorem showed that problems akin to this paradox arise for any number of voters, with any voting method, in any context: the problems follow from the pattern of voter preferences, not what the preferences are about. In this sense, the Impossibility Theorem has universal applicability.

The Impossibility Theorem was misunderstood and misrepresented from the outset. Readers of the theorem more or less split into two groups: those who didn’t understand the maths but found the impossibility result astonishing; and those who grasped the maths and were not astonished by the apparent impossibility. Many in the latter group, meanwhile, sniffed that the paradox discussed by Arrow was not new. It was known as the Condorcet Paradox, discovered by the French philosopher-mathematician Marquis de Condorcet back in 1785 (Arrow, though, did not know this – blame his ‘wanting in diligence’ research). More to the point, mathematical economists quickly saw ways to avoid the problem posed by the theorem, describing voting arrangements which ensure consistent collective choices after all. But it leaves a puzzle. If the problem posed by the Impossibility Theorem can readily be solved, why did Social Choice and Individual Values have such a huge impact? Perhaps Arrow was just in the right place at the right time. Someone else might not have been so lucky.

AMERICA 1, SCOTLAND 0

Duncan Black was born poor but clever in Motherwell, Scotland, in 1908. After studying mathematics and economics at Glasgow University he became a lecturer at the new Dundee School of Economics, where he befriended another young new lecturer there, Ronald Coase. Black was already interested in voting systems, in particular how committees might vote to reach a collective decision. Like Arrow, his career was put on hold by war duties. Black was a night-time ‘firewatcher’ at Warwick Castle – ready to sound the alarm if German bombers appeared – and it was on one lonely night in February 1942 that he ‘saw in a flash’ a key idea about voting. This turned into Black’s median voter theorem – the idea that if voters have straightforward preferences in an electoral competition between two candidates, the winner will be the one who gains the support of the centrist (median) voters. In 1946 Black had another important insight, but his ‘stomach revolted in something akin to physical sickness’ as he realized the negative implications of his insight for finding reliable voting systems.4 Black only later learned that his insight was not new – it was the Condorcet Paradox again (which had also been re-discovered by Dodgson in nineteenth-century Oxford, then forgotten again). Black’s first major publication on the subject was in 1948, three years before Arrow’s book. Then, in November 1949, Black submitted another important paper to the leading journal of mathematical economics, Econometrica. Being the first to publish new ideas is highly regarded in academia, so Black’s status in the field seemed secure.

But Black was not so lucky. He always worked alone: Ronald Coase described him as ‘unworldly, modest, diffident’; another academic colleague quipped ‘he was an expert on committees but I never saw him sit on one’.5 Unlike Arrow at RAND, Black was not surrounded by mathematicians and none of his colleagues knew anything about his research. Moreover, Black heard nothing from Econometrica about his paper until he chased the editor for news eighteen months later, in May 1951. The editor replied that Black’s paper could be published – but only on condition that it was completely revised to acknowledge that Arrow was the originator of these ideas, because Arrow’s book had been published a month or so earlier. Black was furious, because Arrow’s prior publication was due to Econometrica’s slowness. And his fury increased when a possible explanation for that slowness emerged: the editor was biased, being a research director at the Cowles Commission in Chicago, where Arrow had just written most of his book.

Still, this could have been merely a temporary setback for Black. Instead, his defeat was long term: his work on social choice theory both before and after Arrow’s book received only a fraction of the attention and praise lavished on Arrow. Arrow had been in the right place at the right time. It wasn’t just that Arrow’s links with RAND and Cowles brought him contacts and an influence unimaginable in isolated Dundee; his intellectual world was equally far removed from Black’s. Black saw himself as a disinterested scholar developing a pure science of politics: when, in 1948, a RAND researcher wrote to Black asking for advice on suitable reading for RAND staff, Black, not wanting to assist a secretive, militaristic organization like RAND, chose not to reply. Arrow, in contrast, had a subtle but clear philosophical and political outlook which fitted well with the self-image of RAND as a defender of freedom. It was this broader philosophical and political vision, woven through Social Choice and Individual Values, which ultimately had an impact far beyond the Impossibility Theorem alone. Arrow inadvertently set out a framework which would be adopted by many cheerleaders for ‘free markets’ in future years.

In the aftermath of the Second World War scepticism about the wisdom of decisions made by more or less democratic processes was not hard to find. Hitler had come to power through a democratic vote. In The Road to Serfdom, Hayek emphasized how talk of the ‘common good’, ‘public interest’ and ‘social purpose’ could be used as convenient cover to take us down the road to totalitarianism. Yet he also argued that none of these concepts exist, because a society of free individuals can never reach agreement on common goals. No wonder Arrow’s Impossibility Theorem was popular with Hayek’s supporters: Arrow seemed to have provided a mathematical proof of Hayek’s argument.

For the free marketeers, Arrow apparently had more to offer. If democracy was impossible, Arrow seemed to hint at a replacement. In the first few pages of Social Choice and Individual Values, he stated that there were two types of ‘social choice’: voting, to make political choices; and markets, to make economic ones. For Arrow, voting and markets were so ‘analogous’ that the distinction between them should be ‘disregarded’. So as far as the Hayek–Chicago axis was concerned, the obvious response to the Impossibility Theorem was to replace politics and voting with economics and markets. This was a big new step beyond Adam Smith, nineteenth-century laissez-faire and other traditional forms of conservatism – a vision not just of freedom from state interference in the market but as far as possible replacing the state altogether. And this vision looms large over twenty-first-century life, in every presumption that markets are the default way of making decisions that were formerly handled by politics.

Yet there is a basic problem with this market euphoria. If markets are perfectly analogous to voting, then the Impossibility Theorem applies to markets too, and hence, as Arrow put it just after first stating his theorem, ‘the market mechanism does not create a rational social choice.’6

But this part of Arrow’s message was ignored.

MATHEMATICAL PHILOSOPHY

No serious student of Arrow’s work could conclude that he obviously favours markets over democracy as a means of giving people what they want. It bears repeating: Arrow argued that the Impossibility Theorem poses a challenge to the legitimacy of both markets and voting. More importantly, as those who grasped Arrow’s mathematics realized almost immediately, the logical ‘impossibility’ of democracy could be avoided in a number of ways, by weakening or tweaking one or more of Arrow’s assumptions. Each of these tweaks opened the door to various possibilities for reliable voting systems.

Did Arrow feel embarrassed by the ease with which his Impossibility Theorem could be sidestepped? Not at all. Avoiding the impossibility was the whole point: although almost everyone called it ‘Arrow’s Impossibility Theorem’, the name Arrow gave it was the General Possibility Theorem. Friends joked that Arrow was an incurable optimist, but it was more than that. Arrow knew from the outset that his theorem would not hold if any of the assumptions were weakened. His purpose had never been to prove that meaningful democracy is impossible. Instead, he aimed to map out the terrain of compromises: the theorem showed that voting arrangements involving more of one desirable feature are only possible if another desirable feature is to some extent sacrificed. It would be up to others to take Arrow’s map and argue for a particular destination in terms of voting arrangements.

For instance, Arrow had asserted that collective preferences should be consistent, in the sense that if A is preferred to B and B is preferred to C, then A must be preferred to C. This consistency requirement was not met in the Cold War voting example described earlier: a majority preferred disarmament to cold war, a majority preferred cold war to hot war, but a majority preferred hot war to disarmament. Arrow’s consistency requirement is appealing because it rules out voting arrangements which lead to situations like this: it rules out voting arrangements which fail to yield a clear winner in all circumstances. But there are other ways of ensuring that voting arrangements always yield a winner. In 1998 Amartya Sen won the Nobel Prize for economics partly for his work in this area. Sen argued that Arrow was asking too much of voting systems: often, Sen pointed out, all we need to know is the winner. But Arrow’s approach looks for voting systems which give us more information – the ‘collective preference’, a complete ranking of all the alternatives from top to bottom, not just the selection of a winner. Sen proposed a mathematical framework for voting systems which just identify winners: this is much less demanding than Arrow’s approach and many plausible voting systems are possible. In many circumstances, this seems worth the sacrifice of not knowing which alternatives come second, third, and so on.

To explore another important escape route from the Impossibility Theorem, we need to spell out two of the desirable features of voting systems which Arrow had specified. They came to be known as ‘Universal Domain’ and ‘Independence of Irrelevant Alternatives’. Universal Domain asserts that a voting system must be able to handle any kind of opinion or preference expressed by individual voters. Independence of Irrelevant Alternatives requires that the voting system’s ranking of any two alternatives should depend only on how individuals rank these two alternatives, and nothing else. At first glance, these principles seem to capture the spirit of democracy, ensuring that the opinions of individual voters take priority in determining the collective decision. However, when Universal Domain says ‘any’, it truly means any, in a mathematical sense: any logically possible preference over the alternatives is allowed. Again, Arrow may be needlessly setting the bar too high here: in a democracy, we need not accommodate any kind of preference, as though allowing for people to choose their rankings over the alternatives entirely randomly, perhaps by tossing a coin. There will be some logically possible preference patterns which just make no sense and can be ruled out. Moreover, probably every democratic country or society throughout history has set limits on the preferences which citizens can express and the choices available to them at the ballot box. The limits are often themselves the outcome of some formal or informal democratic procedures, ranging from the agenda agreed by citizens’ elected representatives to the adoption of explicit constitutional prohibitions, such as the ban on ‘anti-democratic’ political parties in Germany. If such limits are legitimate, then Arrow’s Universal Domain assumption can be dropped and so the Impossibility Theorem does not apply.

Arguments such as these, about the trade-offs involved in weakening one of Arrow’s assumptions for the sake of saving the others, raged on for decades, generating a vast academic literature – but one largely ignored by outsiders, who were happy to stick with the ‘democracy is impossible’ bumper-sticker version.

Yet Arrow’s influence went far beyond the Impossibility Theorem. He laid out a framework for thinking about politics and economics which he presented as scientific, universally true and capturing the essence of democracy, which he took to be twofold: individual freedom to express any kind of opinion or preference about any subject, and collective decisions based on these individual preferences alone. But this vision of democracy was largely hidden in the mathematical statement of ‘Universal Domain’ and ‘Independence of Irrelevant Alternatives’. (Or, as mathematically inclined readers soon labelled them, ‘Axiom U’ and ‘Axiom I’). In other words, Arrow set out his personal philosophical views – because that is what they were – as though they were universal laws of algebra, expressing them in mathematical language, and unfamiliar mathematical language at that.

Taken together, Arrow’s two axioms define democracy in terms of a strong form of individualism and moral relativism, ruling out any kind of social philosophy that might legitimately override individual opinion on occasion. Arrow’s principal target was totalitarian philosophies such as Soviet Communism (which pleased readers from Hayek to RAND to Chicago economists), but his axioms U and I also banished potentially more benign philosophies, from Rousseau to Kant. In practical terms, Arrow’s seemingly inoffensive abstract axioms mean that individual preferences, no matter how mean, selfish, racist or nasty, can never be overruled, even if they conflict with law, the constitution or human rights.

Arrow’s approach fundamentally misunderstands the meaning of democracy. Democracy is about more than voting systems which add up citizens’ sacrosanct fixed preferences. In a democracy, preferences are not sacrosanct – not just because they might be nasty but because they may be the product of propaganda or deceptive marketing: in all these cases, we try to persuade citizens to change their minds. More generally, democracy involves public deliberation, debate and persuasion, hopefully bringing about some reconciliation of differing points of view. Even if these attempted reconciliations are only fractionally successful, these processes will lead to a narrower range of voter preferences than Universal Domain demands. Without any kind of reconciliation, the impossibility of establishing a consistent collective preference seems hardly surprising.

No one – apparently including Arrow himself – seemed immediately aware of the full implications of the political philosophy hidden in his maths. He seems to have got away with it because philosophically inclined critics struggled to follow the argument, woven as it was into layers of mathematical definitions and deductions. Arrow inspired a generation of academics to search for a science of politics, which they took to be expressible in purely mathematical terms, based on premises assumed to be universally true. When contemporary Western governments attempt to impose electoral democracy on countries which have never experienced it, their presumption that there are universal truths about democracy, valid in all times and places, reflects the extraordinary reach and influence of Arrow’s kind of thinking.

But before we get to these contemporary effects, we need to follow the story of this new science of politics from the well-ordered world of Arrow in the 1950s through the political and economic turbulence of the 1970s: Watergate, the quadrupling of the oil price, rampant inflation, rising unemployment and ballooning government debt. In 1978, at the height of these crises, Arrow was asked to contribute to a debate on whether American democracy and capitalism had a future. The thrust of Arrow’s essay is clear from its title – ‘A Cautious Case for Socialism’.7 If there was to be an intellectual revolution overthrowing politics in favour of markets, Arrow was not the man to lead it. It would take a very different kind of person, more prophet than mathematician.

POLITICS, HEAD LICE AND ROOT CANALS

James McGill Buchanan was born in 1919 in Murfreesboro, Tennessee, where he grew up on the family farm. Every morning before going to school he milked the cows. His poor Southern background strongly shaped his attitudes in later life. Remembering stories told by his grandparents about the occupation of the defeated Confederacy, Buchanan always retained a deep distrust of national government: he described himself as always ‘anti-state, anti-government, anti-establishment … The robber barons were very real to me’.8 And ‘for me, government has always been something to be protected from rather than to be the provider of assistance’.9 Buchanan was one of those people who grow up with firm beliefs and spend the rest of their life repeatedly discovering that they were right all along.

Within six weeks of arriving at the University of Chicago Buchanan had become a ‘zealous advocate’ of free markets, under the influence of teachers including a young Milton Friedman. He never wavered thereafter. But his ideology, rather than straightforward free-market orthodoxy, was of his own making. Buchanan was a moralist at heart, with the fire of a preacher from the Bible Belt – a secular, ascetic puritan. He did not believe parents should bequeath their wealth to their children (although Buchanan had none), and he never had any formal affiliation to a political party.

Buchanan found his distrust of what he called the ‘East Coast establishment’ confirmed first in the Navy, and then, when applying for academic jobs, when he felt repeatedly passed over in favour of less well-qualified applicants from Ivy League universities. He complained that he had been ‘subjected to overt discrimination’10 – although he could hardly worry about a glass ceiling after winning the Nobel Prize for economics in 1986. Buchanan’s career was spent mostly at universities in Virginia rather than Ivy League establishments. This physical separation from US academic elites was reinforced by a political separation: Buchanan and his colleagues saw the mainstream faith in ‘neutral’ bureaucrats who serve the public interest as leading down the road to communism. They called their own approach public choice theory, a bizarre name, given their belief that the public cannot make coherent collective choices, and indeed that there are no such things as the public interest, the public good, public service or public servants.

Unlike Arrow’s work, public choice theory is easy to summarize. As Buchanan put it in a letter to Hayek, public choice theory is ‘politics without the romance’: for Buchanan, everyone concerned with politics – including politicians, bureaucrats and voters – is motivated solely by the pursuit of narrow, selfish ends. Public choice theory is almost entirely devoted to tracing out the implications of this single, simple idea.

Nowadays there is a wide consensus that government is bloated, incompetent, inefficient, vulnerable to capture by special interests and that its interference in ordinary life knows no bounds. Put another way, as several surveys have found, many in the US compare Washington politics unfavourably to head lice and painful dental work on your root canals. Public choice theory has done more than help shape this consensus view: it has become the consensus about government and politics. Beginning in the shadow of Arrow’s work in the mid-1950s, public choice theory was built up over the following twenty years into a comprehensive analysis of every aspect of politics. However, this intellectual revolution had little immediate impact on the real world of politics beyond academia: it was like a timebomb waiting to explode.

The detonator was the political and economic crises of the 1970s. Buchanan and his followers were ready with an explanation for how governments across the West had got into this mess. They called it ‘political overload’. According to public choice theory, the selfishness of politicians makes them focus solely on getting elected and re-elected. So they will try to adopt the policies which will win them the most votes. Good for democracy, surely? No, because a policy which commands majority support among the voters may not best serve the interests of all citizens, since many citizens do not bother to vote. And there is a bigger problem: selfish voters are not wise voters.

On the contrary, public choice theory argues, selfish voters are ‘rationally ignorant’. Each voter’s single vote is highly unlikely to make any difference to the outcome, so, according to public choice theory, it is not worth the time and effort involved in obtaining information about candidates and/or policies. Instead voters will choose to remain ignorant to some degree and ignore altogether those policy impacts which have little effect on them in isolation. This wilful ignorance combines with selfishness in a dangerous way. If a politician promises a big new project for your community – perhaps a new school or hospital – but the costs of the project are borne by taxpayers generally, then public choice theory suggests the promise might win your vote, partly because the extra tax falling on you personally is likely to be small. The psychological power of the politician’s promise is clear: the benefit is local, soon, tangible and explicit, whereas the tax cost is general, later, vague and unacknowledged. As an election approaches, all politicians and their parties make promises like these, each promise bidding to win the votes of a different group or constituency. The bidding war is unconstrained, ultimately leading to political overload – a situation of exploding government debt with tax revenue continually falling short of public expenditure.

This story was new in the 1970s. Today we take it for granted, one reason why ‘politics’ has become a dirty word. And yet as an explanation for political and economic crises – in the twenty-first century as well as in the 1970s – it is incomplete, self-contradictory and ignores the facts.

IT’S THE VOTERS, STUPID

Public choice theorists were among the first economists explicitly to equate rational behaviour with selfishness. An influential early statement of public choice theory is Anthony Downs’s An Economic Theory of Democracy, published in 1957. Downs flatly asserted: ‘whenever we speak of rational behavior, we always mean rational behavior directed primarily to selfish ends’.11 In sharp contrast, Arrow never assumed people are selfish. He merely took people to be rational – meaning consistent – in pursuing their goals, whether those goals are selfish or selfless, noble or base. Remarkable, then, that Arrow was Downs’s doctoral supervisor – and An Economic Theory of Democracy was Downs’s doctoral thesis, published without any changes. Perhaps Downs’s divided loyalties explain his complete volte-face just five years later, when he acknowledged the mixed motives behind political behaviour, not all of them selfish. But it is the original Economic Theory of Democracy which was handed down to generations of students, without Downs’s later retractions, and became one of the most cited books on American politics.

If we dig into the details of the ‘political overload’ argument, it does not withstand scrutiny. There is something paradoxical at the heart of the argument and public choice theory more generally: although voters are ‘rational’, they are easily fooled. The overload argument portrayed voters as suckers, seduced by promises of public spending and oblivious to the cumulative effect of the overall spending package on government debt and future tax levels. Yet robust evidence rapidly emerged that electorates do not vote solely on the basis of the packages of public spending promises dangled before them. They judge politicians on their competence in managing the economy, too, even if this is poorly defined and hard to assess.12 For Buchanan, the problem was more one of voters’ moral degeneracy than their stupidity. He blamed the Keynesian fiscal permissiveness (government ‘tax and spend’) beginning in the 1960s for the growing moral permissiveness of the time, accusing Keynesianism of helping cause ‘increasingly liberalized attitudes toward sexual activities’ and ‘a declining vitality of the Puritan work ethic’, among other sins.13 Whatever the exact limitations of voters in public choice theory, the theory is distinctive in stressing such limitations: it is unlike most other traditions in economics, including ones concerned with reducing the size of the state.

Emerging at roughly the same time as public choice theory was a new brand of macroeconomics associated with reducing the size of the state – monetarism. Monetarists, led by Milton Friedman, were as hostile to Keynesianism as Buchanan was, but for different reasons: monetarists believe that public spending to stimulate the economy is futile. Any increase in public spending must be paid for in the future through higher taxes. Everyone knows this, and their response is to reduce their own spending now – belt-tightening to help pay for future tax rises – and so the stimulus effect of public spending is immediately cancelled out by reduced private spending. In this story, the voters don’t forget about future tax, they fixate on it. Both monetarists and public choice theorists argue for reduced public spending then, but their stories directly contradict each other.

Even within public choice theory, the foolish voter assumption is applied in a partial and self-contradictory way. If voters are consistently fooled by public spending promises, then there would be no point in asking voters to support a government committed to spending cuts and fiscal austerity. But that is what many public choice theorists did: Buchanan was involved directly or indirectly in all the American ‘tax revolt’ campaigns of the 1970s, and the administration of Ronald Reagan, elected in 1980, contained many of Buchanan’s students, including one of Reagan’s economic advisers. And again, if electoral competition forces selfish politicians to outbid each other in their public spending commitments, then there is little hope of electing governments committed to breaking the cycle. So the election of Reagan, and Thatcher in the UK the year before, both outwardly committed to reducing public expenditure, was itself powerful evidence that voters and politicians are not always foolish and selfish in the way public choice theory assumes. The contradictions here are hard to avoid. Among economists aiming to shrink the state, even close colleagues took different sides. Milton Friedman thought that voters could be persuaded to support policies and politicians committed to reducing the size of the state, while his Chicago friend and colleague George Stigler was very doubtful. Stigler dodged the nihilistic implications of his own view: ‘Milton wants to change the world; I only want to understand it.’14

There are other ironies too. First, if we should assume that politicians, bureaucrats and voters are all selfish because everyone is, then academics are, too, including public choice theorists. The public choice perspective tells us not to trust public choice theorists as objective scientists or neutral observers. Instead we should expect them to write and say whatever it takes to advance their own careers. Although Arrow and Buchanan were clearly motivated solely by their academic values and interests, it is not hard to find evidence of economists with more selfish concerns (such as some financial economists in the years leading up to the global financial crisis, as shown in the 2010 documentary Inside Job).15 More generally, evidence of self-serving behaviour in government goes beyond politicians and bureaucrats to include the external consultants and analysts who advise, for instance, moving public services to the private sector because public-sector workers cannot be trusted. The lesson here is that if we take public choice theory seriously, we should apply it comprehensively and even-handedly, to include all those involved in the policy-making process, including academics, consultants, lobbyists and advisers from the private sector.

Second, public choice theory predicts that many people will not bother to vote, because the costs of voting, albeit small, are outweighed by the expected benefits, which are even smaller, because the probability of a single vote swinging the result is almost zero. And yet of course people do vote in large numbers. Downs was sufficiently puzzled by the fact of significant voter turnout that he labelled it a ‘paradox’, and public choice theorists remain worried that this ‘paradox of voter turnout’ suggests voters are not rational calculators after all.

Wait a moment. It is so easy to turn public choice theory against itself – a public choice critique of public choice theory. It seems too easy, too much of a self-referential academic game, because there is clearly a basic truth in the public choice theorists’ perspective on politics: some politicians, voters and bureaucrats do seem largely motivated by selfish concerns. And this is not a new idea. Over 500 years ago the Italian diplomat Niccoló Machiavelli described in detail the elaborate strategies adopted by selfish, cynical politicians and bureaucrats. Public choice theory began as a reaction against a naïve and simplistic view widespread after the Second World War: that once some desirable policy change has been identified by economists or other ‘experts’, benign politicians and bureaucrats will always do their best to bring it about. Buchanan and his colleagues saw themselves – rightly – as outsiders intent on subverting this fantasy. But by the 1980s they were definitely insiders.

Beginning with Thatcher in the UK and Reagan in the US, public choice theory conquered mainstream politics. At least on the surface, it seemed to mesh neatly with monetarist macroeconomics, pessimism about democracy based on Arrow’s theorem and Hayek’s ideas about freedom, to create a powerful case for more markets, less government. And some key facts seemed to be tilting in favour of the theory. Despite occasional upticks, the trend in voter turnout seemed inexorably downwards in most developed democracies. Public expenditure continued to rise, even under governments claiming to be determined to reduce it. Why? If not the public choice story of selfish and short-sighted politicians, bureaucrats and voters, then what?

In fact, a compelling alternative explanation for the growth of public expenditure has been familiar to economists since the 1960s, but it’s an explanation that remains completely hidden from political debate.

SMARTPHONES AND STRING QUARTETS

Some things are getting cheaper, and others more expensive.

Well, of course. But this is true in a more fundamental sense too: over time, some things are getting cheaper, and others more expensive in terms of average incomes. Mass-produced manufactured goods are becoming ever cheaper. The proportion of an average worker’s pay needed to buy a TV or a washing machine today is lower than it was ten years ago and much lower than thirty years ago (and the product being bought is much better). But the reverse is true for labour-intensive services: childcare costs, retirement-home bills, university fees, luxury meals and live theatre tickets are all relatively more expensive as a proportion of average income. When the first smartphones were released in the late 1990s, their price was roughly the same as thirty classical concert tickets. In the late 2010s the price of a smartphone is equivalent to about two tickets.16 The reason for this divergence is that while productivity (output per hour worked) has been rising for decades in manufacturing – due to economies of scale, innovation and automation – it has risen only modestly in labour-intensive services. Productivity growth in manufacturing and the economy in general allows wages to rise without pushing up the prices paid by consumers. But wages in labour-intensive services will not simply stagnate because productivity growth in these sectors is weak. If pay in these service sectors falls behind the rest of the economy, employers will be unable to attract and retain workers. Instead, wages in labour-intensive services must rise roughly in line with average wages across the economy. Since there is little compensating increase in productivity – it’s not as though those better-paid workers are producing more stuff – firms are forced to raise prices. Over time, then, the prices of labour-intensive services rise inexorably relative to the prices of all other goods and services. Put differently, the proportion of national income spent on labour-intensive services will continually increase over time.

William Baumol was the economist who first noticed this phenomenon, which soon became known as Baumol’s cost disease.17 (Coincidentally, Baumol studied at City College, New York, at the same time as Ken Arrow, although Baumol majored in economics and art rather than Arrow’s major in maths.) Baumol’s cost disease is especially virulent in services for which the number of hours provided by workers is part of the definition of the service being offered. In these services, productivity cannot be increased by reducing hours worked, because that would mean a different service. As Baumol observed, musicians’ productivity in ‘producing’ a live performance of a Mozart string quartet has remained unchanged since Mozart’s day: it still requires four musicians and takes as long to play. Similarly, a one-hour consultation with a doctor is just that: the service provided would not be the same if it were thirty minutes with the doctor and thirty completing an online diagnostic questionnaire.

The implications of Baumol’s cost disease are shocking. The proportion of US GDP spent on healthcare is growing by around 1.4 per cent per year. Baumol recently predicted that this trend will broadly continue, implying that US spending on healthcare will rise from its current level, around 18 per cent of GDP, to about 60 per cent by 2100.18 Data from other countries shows the cost disease too, albeit usually less pronounced than in the US: in Britain the trend in healthcare spending takes it from about 10 per cent of GDP now to 50 per cent by 2100.

Unbelievable? Baumol’s forecasts are worth taking more seriously than many economists’ predictions, not least because Baumol first described the cost disease in the 1960s and made some empirical ‘extrapolations’ at that time which have largely been proven correct.fn4 More generally, the data supports the cost disease phenomenon. From 1978 to 2012, average price and wage increases in the US were 110 per cent and 150 per cent respectively. Over the same period healthcare costs rose by around 250 per cent, while the price of a university education rose by about 440 per cent. In many countries, a substantial part of healthcare provision is from the public sector. But the US data show the cost disease can arise in healthcare dominated by private provision too. The cost disease – both the data and Baumol’s explanation – suggests that the growth in public expenditure due to increased healthcare spending is not due to lazy or inefficient public-sector workers: it is inherent in the service being provided.

No political genius is required to realize why mainstream politics of all shades remains silent about Baumol’s cost disease. It seems to point to a frightening future in which key services such as healthcare and education become completely unaffordable. There is no cure. The unaffordability remains, irrespective of whether the services are publicly or privately provided.

Yet, thankfully, the cost disease is much less unpleasant than it sounds. While healthcare, education and other labour-intensive services are getting relatively more expensive, they are becoming more affordable.

Something can become relatively more expensive simply because other things have become cheaper – it can still be more affordable than in the past. So it is with labour-intensive services. The most important measure of affordability for society is the labour input, the number of hours worked to make something. This is the real cost incurred by humanity in producing things. Since productivity is rising almost everywhere in the economy, the labour required to produce the same output is falling in almost all sectors: things are becoming more affordable in terms of labour input. Put another way, productivity growth across the economy allows average wages to rise faster than prices so, overall, most goods and services are becoming more affordable as our purchasing power increases. If Baumol is right and US spending on health is 60 per cent of GDP by 2100, there will still be enough left to spend on everything else – because, as a proportion of GDP, many things will be cheaper.

This is a subtle story, which is one reason why it is rarely told. And, once we look more closely, it gets more subtle still. Clearly, there is room for productivity improvement in labour-intensive services over time. Even performing Mozart quartets requires less labour than it did back in the eighteenth century. When four Viennese musicians travel to Frankfurt to perform a Mozart quartet, it takes them several hours to get there. When Mozart made the same journey in 1790, he recorded that it took six days in great discomfort – and he was pleasantly surprised that it did not take longer.19 Today, some software packages make more accurate diagnoses than doctors, while university lectures can be recorded and made available to millions online. Yet there are limits to productivity growth in services such as healthcare and education. When the service provided is bespoke, tailored to the requirements of the person receiving it – physiotherapy, assessing a dissertation – it is essentially unique. There is little scope for economies of scale. And good-quality bespoke services are often defined in terms of low productivity. If a school increases class sizes, this is usually seen as a decline in the quality of the education provided, not an increase in teacher productivity.

Another complication. It might seem that, as the relative price of labour-intensive services rises, we will respond as we often do when the price of something rises: we will buy less. But in richer societies we are demanding more, not less, of key services such as healthcare and education, with greater numbers entering higher education and receiving treatment for unpleasant but minor health problems.

Finally, while the rising cost of services such as healthcare and education may be due to Baumol’s cost disease, that leaves open the possibility that costs are higher still when the services are publicly rather than privately provided. We have seen that the evidence for this possibility is not promising, because leading examples of private provision, such as US healthcare, show costs rising at least as fast as their public counterparts. Still, the idea of public bad, private good has become so deeply embedded that it is worth briefly assessing its greatest policy triumph – privatization.

The clearest test of this idea is the massive privatization programme launched in Britain immediately after Margaret Thatcher’s election as prime minister in 1979. The most comprehensive long-term study of these British privatizations shows that the results did not live up to the rhetoric.20 Since the privatized firms in question were mostly monopolies or oligopolies facing little competition, they made substantial profits. And management salaries rose significantly (with no evidence of firing poor managers in pursuit of better ones: there was minimal management turnover). Advocates of privatization had claimed that the profit motive would drive privatized firms to cut costs. But in reality, rising salary costs and shareholder dividends essentially cancelled out any cost savings: overall, privatization made little difference to long-term trends in productivity and service prices.

In sum, what does Baumol’s cost disease tell us? It provides a more convincing explanation than public choice theory of much of the rise in public expenditure in many countries in recent years. The cost disease goes to the heart of what is distinctive about services like health and education. These services have inherently low productivity (because we value small class sizes and time with our doctor). And there is no theory or evidence to suggest that goods and services are usually provided at lower cost or more efficiently by the private sector. Yet there is no unaffordability crisis here. True, the price of services such as health and education will carry on rising relative to other goods and services, but not as fast as the growth in our purchasing power. Nevertheless, big political problems lie ahead. If these services are publicly provided, then taxes must inevitably rise over time, just to maintain current service standards; privatization, meanwhile, will achieve nothing except the probable exclusion of the poorest from access to these services.

THE POLITICS WE DESERVE?

The cost disease implies tax rises to maintain existing public services like health and education. It’s easy to be pessimistic about the unwillingness of twenty-first-century democracies to accept such tax rises. And this pessimism is at least partly due to the influence of public choice theory, which suggests that politicians would never propose tax rises and voters would never agree to them. Of course, we have had pessimism about the possibilities of politics since we have had politics, but even Machiavelli believed that most politicians have honourable motives as well as base ones. In comparison, public choice theory offers just a crude, one-dimensional caricature of selfish politicians, bureaucrats and voters. Not unrelated, the fashionable dismissal of almost any role for the state is unprecedented in modern times.21 Even Friedrich Hayek’s The Road to Serfdom accepted a wider role for government, including some degree of social insurance or ‘welfare state’.

As recently as the 1960s surveys regularly showed that ordinary people more or less accepted a view of politics in which politicians and bureaucrats mostly tried to pursue the public interest, and voters ‘held power to account’. It is no coincidence that public choice theory emerged from academia at broadly the same time – the late 1970s – as modern cynicism about politics took root. This is not to suggest that we have consciously embraced public choice theory: Buchanan, Downs and co. have not become household names. The influence of public choice arguments has been more subtle: many of them are akin to self-fulfilling prophecies. Take the claim that bureaucrats and other employees of the state – in many countries this includes most doctors and teachers – have solely selfish motives. Against this, there is clear evidence that many of these employees embrace an ethos of public service and strive to meet standards of professionalism appropriate to their work, such as the Hippocratic oath taken by doctors. Policies inspired by public choice theory will corrode these ethical codes and standards: if you assume the worst of people, they will live down to your dismal expectations. The result will be a decline rather than an improvement in public-service quality and efficiency.

Another self-fulfilling prophecy concerns voter turnout. By suggesting that voting is pointless, Downs’s Economic Theory of Democracy sowed the seeds for a decline in voter turnout. Inheriting from Arrow the concept of voters as no more than bundles of fixed preferences, Downs added the assumption that these preferences are purely selfish. Downs saw voters as just like selfish consumers, and the competition between parties for their votes as like that between rival businesses for market share. This kind of competition is not won through attempts to persuade voters in a process of public debate and argument but instead through modern marketing techniques. And assuming voters are narrowly selfish implies that their concerns are one-dimensional – economic well-being – so the terrain on which elections are won and lost will be equally one-dimensional. The winning strategy here is clear: adopt the policies favoured by centrist voters. This is Duncan Black’s median voter theorem (but Black saw it as an exercise in pure theory rather than a realistic model of how voters behave).

From the 1960s onwards this analysis filtered through to influence mainstream political parties, which relied increasingly on marketing rather than argument. Their policy packages became more alike as they hugged the centre, seeking to appeal to the median voter. Predictably, the result was declining voter turnout: there is less point in voting if all parties with a chance of forming the government have similar policies. And when marketing replaces political argument, the language becomes less emotive, less impassioned and voters lose interest.

Voter engagement has been further undermined by criticism of the political process from an unexpected source – political insiders themselves. The anti-political attitude of public choice theory has been seemingly embraced by the politicians and bureaucrats it portrays as so cynical, selfish and short-termist. The bureaucrats at the United Nations, the World Bank and the European Commission have been arguing for decades that policy-making should be ‘depoliticized’. As an influential minister in the UK’s Blair government explained, ‘what governs our approach is a clear desire to place power where it should be: increasingly not with politicians, but with those best fitted in different ways to employ it … This depoliticization of key decision-making is a vital element in bringing power closer to the people.’22 The same insider cited the setting of interest rates by the Bank of England and the setting of a national minimum wage by an independent commission. If politicians themselves have come to believe that they should not be trusted to make decisions (for example because of their temptation to manipulate interest rates for electoral reasons), and that unelected technocrats should be handed much of the political power, no wonder voters come to believe these things too. Traditionally, politicians might regard handing over power to unelected officials as a failure to fulfil their democratic responsibilities, a repudiation of their own electoral mandate. That politicians and organizations such as the World Bank are proud of this activity shows that they have a rather more pessimistic view of democracy. Ironically, the fact that depoliticization seems driven by a sincere desire by political insiders to bring about better government suggests that they are not so selfish after all. They need to believe in themselves.

We get the politics we deserve. If each of us assumes politicians, bureaucrats and voters are all selfish, and each of us acts accordingly, then we cannot expect much from politics. Our assumptions about human nature are not, as public choice theory would have it, above and beyond politics: they make our politics. So does that mean, to get better politics, as the song goes, all you need is love?

We cannot pretend to trust politicians and politics more than we really do. But we should be aware that every decision not to do so comes with a cost. There is a cumulative drip-drip effect of seemingly innocuous assumptions slowly dissolving the democratic glue that holds us together. Who imagined that comparing competition between political parties to that between firms for market share would lead to most voters deciding that mainstream parties are ‘all the same’, and alarming levels of disaffection with democracy?

If we don’t like the way contemporary politics works, then any attempt to change it must begin by assessing all these assumptions case by case, on the best evidence available. Trust is not easy to rebuild, but we do have a choice.

We can’t trace the origins of our contemporary political malaise to a single event. But if we are looking for a decisive Big Bang in the history of ideas which set us on our current path, Arrow’s Social Choice and Individual Values is a convincing candidate. The impossibility theorem is a remarkable intellectual achievement. Yet even if society’s goals cannot be derived from individual voter preferences using Arrow’s questionable ideal of a voting system, it does not follow that democracy is impossible or that ideas of the common good or public interest are mythical. Rather, Arrow’s impossibility theorem reminds us that voting systems involve ethical compromises. And the solution of many public choice theorists – contracting out politics to the market – does not escape the need for these compromises. It just ignores them.

Arrow, by turns an apolitical mathematician and a cautious socialist who left meagre materials for biographers, remains an inscrutable figure in the history of economics. He was an unwitting revolutionary in the assault on politics. And he was only there by chance. How different would modern politics look if Arrow had not learned the maths he needed – if Bertrand Russell had not been accused of ‘moral indecency’?