TOGETHER AND ALONE, CLOSING THE PRIME GAP
Erica Klarreich
In the months after Yitang Zhang settled a long-standing open question about prime numbers in 2013—showing that even though primes get increasingly rare as you go further out along the number line, you will never stop finding pairs of primes separated by at most 70 million—he found himself caught up in a whirlwind of activity and excitement. He lectured on his work at many of the nation’s preeminent universities, received job offers from top institutions in China and Taiwan and a visiting position at the Institute for Advanced Study in Princeton, New Jersey, and was promoted to full professor at the University of New Hampshire.
Meanwhile, Zhang’s work raised a question: Why 70 million? There is nothing magical about that number—it served Zhang’s purposes and simplified his proof. Other mathematicians quickly realized that it should be possible to push this separation bound quite a bit lower, although not all the way down to two.
By the end of May 2013, mathematicians had uncovered simple tweaks to Zhang’s argument that brought the bound below 60 million. A blog post by Scott Morrison of the Australian National University in Canberra ignited a firestorm of activity, as mathematicians vied to improve on this number, setting one record after another. By June 4 that year, Terence Tao of the University of California, Los Angeles, a winner of the prestigious Fields Medal had created an open, online “Polymath project” to improve the bound that attracted dozens of participants.
For weeks, the project moved forward at a breathless pace. “At times, the bound was going down every thirty minutes,” Tao recalled. By July 27, 2013, the team had succeeded in reducing the proven bound on prime gaps from 70 million to 4,680.
Then, a preprint posted to arxiv.org that November by James Maynard, who was then a postdoctoral researcher working on his own at the University of Montreal, upped the ante.1 Just months after Zhang announced his result, Maynard presented an independent proof that pushed the gap down to 600. Another Polymath project combined the collaboration’s techniques with Maynard’s approach to push this bound down to 246.
“The community is very excited by this new progress,” Tao said.
Maynard’s approach applies not just to pairs of primes, but to triples, quadruples and larger collections of primes. He showed that you can find bounded clusters of any chosen number of primes infinitely often as you go out along the number line. (Tao said he independently arrived at this result at about the same time as Maynard.)
Zhang’s work and, to a lesser degree, Maynard’s fits the archetype of the solitary mathematical genius, working for years in the proverbial garret until he is ready to dazzle the world with a great discovery. The Polymath project couldn’t be more different—fast and furious, massively collaborative, fueled by the instant gratification of setting a new world record.
For Zhang, working alone and nearly obsessively on a single hard problem brought a huge payoff. Would he recommend that approach to other mathematicians? “It’s hard to say,” he said. “I choose my own way, but it’s only my way.”
Tao actively discourages young mathematicians from heading down such a path, which he has called “a particularly dangerous occupational hazard” that has seldom worked well, except for established mathematicians with a secure career and a proven track record. However, he said in an interview, the solitary and collaborative approaches each have something to offer mathematics.
“It’s important to have people who are willing to work in isolation and buck the conventional wisdom,” Tao said. Polymath, by contrast, is “entirely groupthink.” Not every math problem would lend itself to such collaboration, but this one did.
COMBING THE NUMBER LINE
Zhang proved his result by going fishing for prime numbers using a mathematical tool called a k-tuple, which you can visualize as a comb with some of its teeth snapped off. If you position such a comb along the number line starting at any chosen spot, the remaining teeth will point to some collection of numbers.
Zhang focused on snapped combs whose remaining teeth satisfy a divisibility property called “admissibility.” He showed that if you go fishing for primes using any admissible comb with at least 3.5 million teeth, there are infinitely many positions along the number line where the comb will catch at least two prime numbers. Next, he showed how to make an admissible comb with at least 3.5 million remaining teeth by starting with a 70-million-tooth comb and snapping off all but its prime teeth. Such a comb must catch two primes again and again, he concluded, and the primes it catches are separated by at most 70 million.
The finding is “a fantastic breakthrough,” said Andrew Granville of the University of Montreal. “It’s a historic result.”
Zhang’s work involved three separate steps, each of which offered potential room for improvement on his 70 million bound. First, Zhang invoked some very deep mathematics to figure out where prime fish are likely to be hiding. Next, he used this result to figure out how many teeth his comb would need in order to guarantee that it would catch at least two prime fish infinitely often. Finally, he calculated how large a comb he had to start with so that enough teeth would be left after it had been snapped down to admissibility.
The fact that these three steps could be separated made improving Zhang’s bound an ideal project for a crowd-sourced collaboration, Tao said. “His proof is very modular, so we could parallelize the project, and people with different skills squeezed out what improvements they could.”
The Polymath project quickly attracted people with the right skills, perhaps more efficiently than if the project had been organized from the top down. “A Polymath project brings together people who wouldn’t have thought of coming together,” Tao said.
PRIME FISHING GROUNDS
Of Zhang’s three steps, the first to admit improvement was the last one, in which he found an admissible comb with at least 3.5 million teeth. Zhang had shown that a comb of length 70 million would do the trick, but he hadn’t tried particularly hard to make his comb as small as possible. There was plenty of room for improvement, and researchers who were good at computational mathematics soon started a friendly race to find small admissible combs with a given number of teeth.
Andrew Sutherland, of the Massachusetts Institute of Technology, quickly became a sort of de facto admissible-comb czar. Sutherland, who focuses on computational number theory, had been traveling during Zhang’s announcement and hadn’t paid particular attention to it. But when he checked in at a Chicago hotel and mentioned to the clerk that he was there for a mathematics conference, the clerk replied, “Wow, 70 million, huh?”
“I was floored that he knew about it,” Sutherland said. He soon discovered that there was plenty of scope for someone with his computational skills to help improve Zhang’s bound. “I had lots of plans for the summer, but they went by the wayside.”
For the mathematicians working on this step, the ground kept shifting underfoot. Their task changed every time the mathematicians working on the other two steps managed to reduce the number of teeth the comb would require. “The rules of the game were changing on a day-to-day basis,” Sutherland said. “While I was sleeping, people in Europe would post new bounds. Sometimes, I would run downstairs at 2 a.m. with an idea to post.”
The team eventually came up with the Polymath project’s record-holder—a 632-tooth comb whose width is 4,680—using a genetic algorithm that “mates” admissible combs with each other to produce new, potentially better combs.2
Maynard’s finding, which involves a 105-tooth comb whose width is 600, rendered these giant computations obsolete. But the team’s effort was not a wasted one: Finding small admissible combs plays a part in many number theory problems, Sutherland said. In particular, the team’s computational tools will likely prove useful when it comes to refining Maynard’s results about triples, quadruples and larger collections of primes, Maynard said.
The Polymath researchers focusing on step two of Zhang’s proof looked for places to position the comb along the number line that had the greatest likelihood of catching pairs of primes, to figure out the number of teeth required. Prime numbers become very sparse as you go out along the number line, so if you just plunk your comb down somewhere randomly, you probably won’t catch any primes, let alone two. Finding the richest fishing grounds for prime numbers ended up being a problem in “calculus of variations,” a generalization of calculus.
This step involved perhaps some of the least novel developments in the project, and the ones that were most directly superseded by Maynard’s work. At the time, though, this advance was one of the most fruitful ones. When the team filled in this piece of the puzzle on June 5, 2013, the bound on prime gaps dropped from about 4.6 million to 389,922.
The researchers focusing on step one of Zhang’s proof, which deals with how prime numbers are distributed, had perhaps the hardest job. Mathematicians have been familiar with a collection of distribution laws for primes for more than a century. One such law says that if you divide all prime numbers by the number three, half of the primes will produce a remainder of one and half will produce a remainder of two. This kind of law is exactly what’s needed to figure out whether an admissible comb is likely to find pairs of primes or miss them, since it suggests that “[prime] fish can’t all hide behind the same rock, but are spread out everywhere,” Sutherland said. But to use such distribution laws in his proof, Zhang—and, later, the Polymath project—had to grapple with some of the deepest mathematics around: a collection of theorems from the 1970s by Pierre Deligne, now an emeritus professor at the Institute for Advanced Study, concerning when certain error terms are likely to cancel each other out in gigantic sums. Morrison described Deligne’s work as “a big and terrifying piece of 20th-century mathematics.”
“We were very fortunate that several of the participants were well-versed in the difficult machinery that Deligne developed,” Tao said. “I myself did not know much about this area until this project.”
The project didn’t just figure out how to refine this part of the proof to improve the bound. It also came up with an alternative approach that eliminates the need for Deligne’s theorems entirely, although at some cost to the bound: Without Deligne’s theorems, the best bound the project has come up with is 14,950.
This simplification of the proof is, if anything, more exciting to mathematicians than the final number the project came up with, since mathematicians care not only about whether a proof is correct but also about how much new insight it gives them.
“What we’re in the market for are ideas,” said Granville.
As the Polymath project progressed, Zhang himself was conspicuously, though perhaps not surprisingly, absent. He has not followed the project closely, he said. “I didn’t contact them at all. I prefer to keep quiet and alone. It gives me the opportunity to concentrate.”
Also absent, though less conspicuously, was Maynard. As the Polymath participants worked feverishly to improve the bound between prime pairs, Maynard was working on his own to develop a different approach—one foreshadowed by a forgotten paper that was written, and then retracted, ten years ago.
A SECRET WEAPON
Zhang’s work was grounded in a 2005 paper known as GPY, after its authors, Daniel Goldston, János Pintz and Cem Yıldırım.3 The GPY paper developed a scoring system to gauge how close a given number is to being prime. Even numbers get a very low score, odd numbers divisible by 3 are only slightly higher and so on. Such scoring formulas, called sieves, can also be used to score the collection of numbers an admissible comb points to, and they are a crucial tool when it comes to figuring out where to place the comb on the number line so that it has a good chance of catching prime fish. Constructing an effective sieve is something of an art: The formula must provide good estimates of different numbers’ prime potential, but it must also be simple enough to analyze.
Two years before GPY was published, two of its authors, Goldston and Yıldırım, had circulated a paper describing what they asserted was a powerful scoring method. Within months, however, mathematicians discovered a flaw in that paper. Once Goldston, Yıldırım and Pintz adjusted the formula to repair this flaw, most mathematicians turned their focus to this adjusted scoring system, the GPY version, and didn’t consider whether there might be even better ways to tweak the original, flawed formula.
“Those of us looking at GPY thought we had the bases covered, and it didn’t cross our mind to go back and redo the earlier analysis,” said Granville, who was Maynard’s postdoctoral adviser.
In 2012, however, Maynard decided to go back and take a second look at the earlier paper. A newly minted Ph.D. who had studied sieving theory, he spotted a new way to adjust the paper’s scoring system. GPY’s approach to scoring an admissible comb had been to multiply together all the numbers the comb pointed to and then score the product in one fell swoop. Maynard figured out a way to score each number separately, thereby deriving much more nuanced information from the scoring system.
Maynard’s sieving method “turns out to be surprisingly easy,” Granville said. “It’s the sort of thing where people like me slap their foreheads and say, ‘We could have done this seven years ago if we hadn’t been so sure we couldn’t do it!’”
With this refined scoring system, Maynard was able to bring the prime gap down to 600 and also prove a corresponding result about bounded gaps between larger collections of primes.
The fact that Zhang and Maynard managed, within months of each other, to prove that prime gaps are bounded is “a complete coincidence,” Maynard said. “I found Zhang’s result very exciting when I heard about it.”
Tao was similarly philosophical about Maynard’s scoop of the Polymath project’s headline number. “You expect the record to be beaten—that’s progress,” he said.
It is likely, Tao and Maynard said, that Maynard’s sieve can be combined with the deep technical work by Zhang and the Polymath project about the distribution of primes to bring the prime gap even lower.
This time, Maynard joined in. “I’m looking forward to trying to get the bound as small as possible,” he said.
It remains to be seen how much more can be wrung out of Zhang’s and Maynard’s methods. Prior to Maynard’s work, the best-case scenario seemed to be that the bound on prime gaps could be pushed down to 16, the theoretical limit of the GPY approach. Maynard’s refinements push this theoretical limit down to 12. Conceivably, Maynard said, someone with a clever sieve idea could push this limit as low as 6. But it’s unlikely, he said, that anyone could use these ideas to get all the way down to a prime gap of 2 to prove the twin primes conjecture.
“I feel that we still need some very large conceptual breakthrough to handle the twin primes case,” Maynard said.
Tao, Maynard and the Polymath participants may eventually get an influx of new ideas from Zhang himself. It has taken the jet-setting mathematician a while to master the art of thinking about mathematics on airplanes, but he has now started working on a new problem, about which he declined to say more than that it is “important.” While he isn’t currently working on the twin primes problem, he said, he has a “secret weapon” in reserve—a technique to reduce the bound that he developed before his result went public. He omitted this technique from his paper because it is so technical and difficult, he said.
“It’s my own original idea,” he said. “It should be a completely new thing.”