In 1979, Dave Woody and Paul Richards of the University of California at Berkeley used a 120-metre-diameter balloon to hoist a bolometer experiment 43 kilometres up into the air. Their instrument, dangling 650 metres below the balloon, looked at the cosmic background radiation for three hours before the strong winds at such a high altitude blew it out over the Gulf of Mexico and the balloon had to be recovered. But, in those three hours, Woody and Richards’ instrument made the best measurement of the spectrum of the Big Bang radiation that anyone had so far achieved.
The experiment incorporated all the most up-to-date features. Light was collected from the sky by a trumpet-shaped horn, specially designed to keep out stray radiation from nearby warm objects. This was supplemented by an ‘Earthshine shield’ to keep out radiation from the Earth below. The horn funnelled the light down into a Michelson interferometer complete with sensitive bolometer detectors cooled by helium-3 to just 0.3 degrees above absolute zero.1 These compared the temperature of the sky with an artificial black body cooled by liquid helium.
To reduce unwanted radiation from the apparatus itself, Woody and Richards immersed their entire instrument, including the light-collecting horn, in a vacuum flask of liquid helium. It hung 650 metres beneath their balloon in order to minimise the chance of picking up unwanted radiation from the balloon itself.
The two astronomers found that the fireball radiation was approximated by a black body at a temperature of 2.96 degrees above absolute zero. ‘That was the first spectrum I really believed,’ says Dave Wilkinson.
But there was an important difference between the spectrum Woody and Richards measured and the theoretical 2.96-degree curve. Although at long wavelengths Woody and Richards’ observations hugged the 2.96-degree black body curve very closely, at the shorter wavelengths – beyond the peak in the spectrum – the agreement was not nearly so good. There was too much radiation at short wavelengths. The spectrum of the cosmic background radiation appeared to have a bump in it.
‘It was a pattern we were to see repeated several times in the field,’ says Bruce Partridge. ‘Whenever people measured the spectrum of the Big Bang radiation, the measurements at short wavelengths – less than a millimetre – always showed a puzzling excess.’
While some persevered with balloon-borne experiments, others tried firing instruments high into the atmosphere on the tip of sounding rockets. One person who specialised in this sort of experiment was Herb Gush at the University of British Columbia in Vancouver. During the 1970s, he fired a number of such experiments up to a height of several hundred kilometres and he, too, measured an excess of Big Bang radiation at short wavelengths. But Gush’s experiments were plagued by problems: for instance, glowing gases from his rocket exhaust had a habit of obscuring his instruments’ view of the Universe, making his measurements questionable.
But, in 1988, Paul Richards at Berkeley joined forces with a Japanese team led by Toshio Matsumoto at Nagoya University. Together they launched a rocket experiment which obtained a spectrum of the cosmic background that most people in the field believed. Since it, too, had a bump at a wavelength of about a millimetre, there was intense interest from the theorists. They came up with a flurry of possible explanations.
Because the hot Big Bang naturally produced fireball radiation with the spectrum of a perfect black body, the bump in the spectrum had to mean that some process since the Big Bang had injected an immense amount of energy into the Universe. There were all sorts of possibilities. For instance, there might be large amounts of warm dust suspended in galaxies or else hanging between them, and the glow of this dust might be responsible for the bump in the cosmic background spectrum. But the problem here was to find a way of heating up the dust so that it glowed brightly at around a millimetre in wavelength. The dust had to be pretty much everywhere in the Universe, since astronomers knew that the background radiation came equally from all directions. Clearly, a prodigious amount of energy would have been needed to warm it up.
The theorists thought of many possibilities. For instance, a large number of stars might have formed shortly after the Big Bang. They could have raced through their life cycles and exploded, giving out an enormous quantity of energy. Another possibility was that the early Universe contained ‘exotic’ microscopic particles, as yet unknown to science, and these had decayed since the Big Bang, releasing a lot of energy.
But all these schemes failed ultimately. The energy required to heat up such a large quantity of dust in the Universe was simply too enormous. The theorists could imagine no plausible physical process that would work.
But some considered an alternative. They wondered whether the bump in the spectrum could be explained if some process had simply redistributed the energy in the fireball radiation, sapping it from long wavelengths and redepositing it at a wavelength of around a millimetre to create the observed bump in the spectrum.
In this scheme, the photons of the background radiation had not flown unhindered across space for 13.7 billion years after all but had instead passed through clouds of extremely hot gas floating between the galaxies. Such hot gas would be permeated by fast-moving electrons, stripped from the gas atoms. If the photons of the cosmic background collided with these electrons, they would rob them of energy, boosting their own energy and shortening their wavelength. The net effect of countless collisions would be to take energy from the background radiation at longer wavelengths and deposit it at wavelengths shorter than the peak. It would create precisely the bump seen in the Berkeley–Nagoya experiment.
But there were problems with this idea, too. For a start, no one knew whether such a hot gas existed throughout the Universe. But the biggest problem with the hot-gas scheme was that it ran into the same difficulties as the dust idea. Some way would have to be found to heat the hot gas between the galaxies, and no one could think of where such a prodigious amount of energy could come from. The theorists were flummoxed. The experiments to measure the cosmic background radiation had thrown up an apparently insoluble problem.
But it was not only the experiments to measure the spectrum of the cosmic background radiation that were puzzling theorists. The experiments to measure the smoothness of the radiation were also beginning to baffle them.
However, one discovery, made in the late 1970s, had been entirely expected, and that was that the radiation was slightly hotter in one direction in the sky than in the opposite direction because of the Earth’s motion through the background radiation.
‘The discovery was made incrementally,’ says Wilkinson. ‘A series of ground-based experiments, including ours, saw something marginally, then, finally, the Berkeley group saw it from a high-flying U2 spy plane.’
What Phil Lubin and George Smoot of Berkeley found in 1977 was that the sky was about 0.1 per cent hotter in the direction of the constellation of Leo than it was in the opposite direction. This amounted to a difference of just three thousandths of a degree. No wonder it had taken more than a decade to find it.
The temperature difference could be explained if our Milky Way galaxy were flying through the cosmic background radiation at a speed of 370 kilometres per second in the direction of the constellation of Leo. The radiation in the direction we were moving would naturally be blue-shifted by the Doppler effect, boosting it in energy and temperature. The radiation behind, on the other hand, would be redshifted and reduced in temperature.
‘Jim Peebles had told me there would be such an effect even before Penzias and Wilson discovered the Big Bang radiation,’ says Wilkinson. Peebles had realised that the cosmic background radiation was a sort of universal ‘frame of reference’ against which the speed of every object in the Universe could be measured. Peebles had even predicted roughly how big the effect should be since he knew how fast a typical galaxy like the Milky Way was moving.
Astronomers referred to this observation that one half of the sky was hotter than the other as the ‘dipole effect’. ‘Seeing the dipole was a major relief,’ says Wilkinson. ‘If it hadn’t been there, it would have been a major embarrassment to everyone.’
But what was destined to become an embarrassment was the fact that apart from the ‘dipole’ variation, the cosmic background radiation seemed utterly smooth across the sky. ‘We knew the radiation had to be smooth,’ says Jim Peebles, ‘but we knew it could not be dead smooth because today’s Universe is lumpy.’
At some point the smoothly distributed matter in the early Universe had to start clumping to form galaxies and clusters of galaxies, and this should make itself visible as an unevenness in the cosmic background radiation.
Back in 1965, when Bob Dicke had first introduced him to the idea of a hot Big Bang, Peebles had realised that the fireball radiation was linked with the origin of galaxies such as the Milky Way. ‘It was pretty evident that the radiation would have an important effect on how galaxies form,’ says Peebles.
The fireball radiation completely dominated the Universe during the first 380,000 years after the Big Bang. For every particle of matter, there were about 10 billion photons, a ratio which has remained constant in the Universe to this day. But though today’s background photons have been cooled and diluted by the expansion of the Universe, in the early Universe the photons were immensely hot and packed closely together. This meant that in any cubic centimetre of the early Universe the total energy of the photons was enormously greater than the energy of the particles of matter. Matter was only a minor contaminant. In the early Universe, radiation called the shots.
The implication of all this for galaxy formation is that the process could not begin earlier than 380,000 years after the Big Bang. Any particles that came together would simply be blasted apart by photons of the fireball radiation. But after 380,000 years, atoms formed and mopped up all the free electrons through which the photons of the fireball radiation were influencing matter. The Universe became transparent to photons, and from that moment on matter and radiation went their separate ways.
Coincidentally, this was also roughly the time when the energy density of radiation in the Universe dropped below that of matter. This happened because the energy of photons was diluted as their wavelength was stretched by the expansion of the Universe. But the energy density of particles of matter cannot be diluted indefinitely because each particle has a floor – a so-called rest energy – below which it cannot go.2
So, about 380,000 years after the Big Bang, the Universe became dominated by matter. Freed from the tyranny of radiation, matter could begin to clump under the force of gravity. Gravity, not the pressure of radiation, was now the dominant force in the Universe.
Because the photons of the cosmic background radiation last interacted with matter around this time, they ought to reveal how matter was spread throughout the Universe back then. As early as 1968, the theorist Joseph Silk had pointed out that mapping the temperature of the cosmic background radiation would allow us to ‘see’ clumps of matter 380,000 years after the Big Bang, just as the process of galaxy formation was beginning.
‘The lumps would be of exceedingly great interest – that was obvious right away at the beginning,’ says Peebles.
Where the matter of the early Universe was ever so slightly denser than its surroundings, photons would have to climb out of the slightly stronger gravity. They would lose energy, becoming red-shifted. The gravitational effect, predicted by Einstein in 1915, would create a ‘cold spot’, a region of the sky where the cosmic background radiation was marginally cooler than elsewhere. Similarly, ‘hot spots’ would mark regions of the early Universe that were ever so slightly less dense than average. In effect, the radiation would carry with it an imprint of the Universe as it was soon after the Big Bang itself.
‘It took a long time for experiments to measure the smoothness of the Big Bang radiation to catch on,’ says Wilkinson. ‘For a good ten years, nobody was doing anything – except Bruce Partridge and I.’
But when other astronomers did get involved, they searched in vain for any slight departure from complete smoothness. There were false alarms. An Italian team led by Francisco Melchiorri announced finding hot spots in the cosmic background radiation. So, too, did a team led by Rod Davis of Jodrell Bank in England. It was running an experiment on top of a mountain in the Canary Islands. Both teams had to retract their findings after admitting they had made a mistake.
Even Dave Wilkinson was not immune from such errors. ‘We reported seeing something at the same time as the Italians,’ he says. ‘But we were fooled by radio emission from our Galaxy.’
By 1989, after more than two decades of painstaking observations, astronomers had still not detected any variation in the temperature of the Big Bang radiation across the sky, apart from that due to the relative motion of the Earth. The uniformity seemed to be indicating that when the radiation was produced, about 380,000 years after the Big Bang, the matter of the Universe was spread out completely evenly. This posed a very awkward question, because the distribution of matter in the Universe today is anything but uniform. How, then, did the galaxies and clusters of galaxies in today’s Universe form?
By the late 1980s, this question was beginning to give theorists serious headaches. It was not only that the experimenters were finding that the cosmic background radiation and therefore the matter of the early Universe was spread remarkably smoothly. Simultaneously, astronomers mapping how galaxies are spread throughout space were finding that the matter of today’s Universe is spread out a lot more unevenly than anyone had suspected.
These astronomers were making use of sensitive electronic light detectors known as charge-coupled devices, or CCDs. Their introduction during the 1970s had brought about a major revolution in astronomy which had never hit the headlines. CCDs were far superior to the photographic plates which astronomers had traditionally used with their telescopes to probe the Universe. Instead of picking up about 1 per cent of all the photons of light collected by the mirror of a telescope – which was typical of photographic plates – CCDs could pick up nearly 100 per cent of all photons. Just by swapping photographic plates for CCDs, any telescope could instantly be made about 100 times more sensitive than it was before. And this meant it was possible to study galaxies that were much fainter and therefore further away than any seen until now.
Using big telescopes equipped with CCDs, some astronomers mapped how galaxies were spread throughout a large volume of the Universe. What they found was that the Universe was full of complex structures. Galaxies are clustered in great chains and sheets which surround great voids of empty space where there are no galaxies to speak of – a structure remarkably similar to Swiss cheese.
The origin of these clusters and voids was one of the greatest problems of cosmology. And it was extremely difficult to square with the evidence of the cosmic microwave background, which was telling us the early Universe was astonishingly smooth. How could such complexity have come out of such simplicity? The evidence of the cosmic background radiation was saying that by rights our Milky Way should not exist.
Other astronomers probing the depths of space with CCDs discovered objects at greater and greater distances. These were quasars, the ferociously bright cores of newborn galaxies. Powered by matter being sucked down onto ‘supermassive’ black holes, quasars can be spotted at immense distances. By the early 1990s, quasars were being found that were so far away that their light had been travelling to us for most of the history of the Universe. In fact, we were seeing some of them as they were within a billion or so years after the Big Bang.
Again, these observations were extremely difficult to square with the evidence of the cosmic background radiation. How could quasars have condensed out of the cooling fireball within a billion years or so when the fireball radiation was showing no sign whatsoever of any lumpiness?
The cosmic background radiation had thrown up two baffling puzzles: it seemed to be too smooth by far and its spectrum had a peculiar bump in it which no theorist could explain. It had taken nearly 25 years to reach this point, and progress was painfully slow. If the twin puzzles were ever to be solved, it would be necessary to get above the glowing atmosphere. It was only a thin layer, comparable to the thickness of the skin on an apple, yet it was standing between the astronomers and the greatest prize in cosmology. It was clear to everyone that what was needed to solve the puzzles of the fireball radiation was an eye above the atmosphere. What was needed was an experiment in outer space.