When: 1993
Where: USA
Why: Search engines revolutionised our ability to navigate the ubiquitous and ever-expanding internet
How: The exponential growth of the internet meant a succession of innovators saw a need to help people find what they wanted
Who: A range of PhD graduates and computer scientists, not least David Filo, Jerry Yang, Larry Page and Sergey Brin
Fact: Google receives about a billion search requests every day
These days, we take search engines for granted. The issue of how to find the information that users really want is one that challenged computer programmers from the internet’s inception. With more than 600 billion pages of information on the internet, and rising fast every day, the chances of people finding what they need without something like a search engine would be tiny. And the internet would not have grown into what it is now.
We would not be able to use the internet in the way we do, reaching for it from desktops, laptops and increasingly mobile devices multiple times every day. Our lives would be different. And of course Google, now one of the world’s most valuable and best-known companies, would not exist.
Back in the 1950s, the various groups involved in building the internet were faced with a problem: as the amount of information available grew, how would people find what they were looking for? Unless they knew the specific location of the file they wanted, it would be like searching for a needle in a haystack.
The solution came in the form of information retrieval (IR) systems, which searched documents for pre-specified keywords, known as ‘metadata’, building up a database of the results that users could then search through. One of the first systems to use IR was the appropriately monikered ‘System for the Mechanical Analysis and Retrieval of Text’ (SMART), developed at Cornell University during the 1960s.
It wasn’t until late 1993 that someone had the bright idea of including a form, so that even people without extensive knowledge of programming languages could search for information.
By 1993, things were getting more advanced. Matthew Gray, a student at the Massachusetts Institute of Technology (MIT), came up with the idea of a ‘robot’ that would automatically trawl the web, looking for information. The World Wide Web Wanderer would save information from web pages, such as titles and a few keywords from the first few lines of text, putting them into a database, known as an ‘index’.
While the World Wide Web Wanderer was by no means the first search engine, it wasn’t until late 1993 that someone had the bright idea of including a form, so that even people without extensive knowledge of programming languages could search for information. JumpStation, which launched in December 1993, not only crawled and indexed the internet but allowed people to search it too, making it the first search engine of the type we recognise today.
Created by a young computer scientist called Jonathon Fletcher, JumpStation never had the capacity to create a lasting impression on the world – in fact, it overloaded Stirling University’s servers on its first day. Another problem that limited the impact of JumpStation was its sheer novelty – no one really knew how to use it, let alone harness and develop it, beyond the confines of Stirling’s computer science department. Although JumpStation was nominated for a ‘Best of the Web’ award in 1994, Fletcher was unable to secure investment for his venture, and discontinued it in the same year.
However, helping people to find information online was by now becoming a popular pursuit. Many of the early search-based resources, such as Lycos, Yahoo! and InfoSeek, were portals – all-in-one sites that combined search functionality with an email service, news, sport and weather updates, and myriad additional services. Of all the popular sites around at this time, AltaVista was the only genuine search engine; although it enjoyed considerable early success, reaching a peak of 80 million hits per day by 1997, it was eventually overshadowed by newer players on the market, and after various takeovers ended up under the Yahoo! umbrella.
Yahoo! had originally been given the decidedly less catchy title of ‘David and Jerry’s Guide to the World Wide Web’, after founders David Filo and Jerry Yang. The pair started the company from a trailer while they were studying for their PhDs at Stanford University. The idea was to ‘keep track of their personal interests on the internet’, but the site swiftly gained popularity among Silicon Valley’s close-knit internet community, and Filo and Yang incorporated it as a business in March 1995.
What set Yahoo! apart from its competitors was that it was more of a ‘web directory’ than a search engine or portal. Yahoo! relied on a team of human researchers to organise all the information into categories, which would then be used to help people find what they were looking for. That made finding information easier, but the results presented when someone performed a search weren’t necessarily the most relevant.
While Yahoo! grew rapidly, the most game-changing search engine launch came in 1998, when two Stanford PhD students, Larry Page and Sergey Brin, unveiled a project they had been working on for a couple of years. Unlike its competitors’ all-singing, all-dancing sites, the search page had a simple, minimalist design (Page and Brin later admitted that this was because they weren’t very familiar with the process involved in building flashy websites). They named their site ‘Google’, a play on the word ‘googol’, a mathematical term for the number one followed by a hundred zeros, which they said reflected the seemingly infinite amount of information they were planning to organise.
Aesthetics aside, the main difference between Google and its competitors was how it ‘ranked’ the information. Other search tools based their rankings on how often a search term appeared on each site: the more times the search word appeared, the higher the site would feature in the eventual ranking list. This was playing into the hands of spam-fuelled sites that simply repeated the same words over and over again, so Brin and Page came up with a new formula – whereby each site would be ranked according to how many other sites linked to it. They developed a complex algorithm that took into account the number of ‘backlinks’, as well as how and where keywords were mentioned. That cut out the spam, because people were far less inclined to link to dubious websites, which meant that the pages nearest the top were more relevant.
Brin and Page called the new system ‘PageRank’ (in honour of the latter), and although it took a while for the engine to become established, experts were soon impressed. PC Magazine credited Google with ‘an uncanny knack for returning extremely relevant results’, just three months after its launch. The creation of multiple language sites helped Google to establish control of the search engine market at the start of the new millennium. In June 2000, Brin and Page signed a deal to provide default search listings for Yahoo!, cementing Google’s position at the summit of the search engine league.
Google refused to rest on its laurels. The same year as the Yahoo! deal saw the introduction of Google Toolbar, a browser that allowed users to trawl the net without visiting the Google homepage first. Google also launched Chinese, Japanese and Korean language services and, perhaps most crucially, rolled out a self-service advertising program known as AdWords. The venture was conceived around the time of the dotcom bust: Google’s founders realised, in the run-up to the crisis in March 2000, that the site needed to find a way to make money. While Brin and Page had, until then, shunned the idea of advertising (thinking in horror about the eye-watering, flashing banner ads that dominated their competitors’ homepages), they realised that they needed to come up with a way to sell ads that wouldn’t ruin the minimalist aesthetic they had created for the site.
Google AdWords was a revolution. Users could activate their account with just a credit card, specify their own keywords, and tell Google how much they were prepared to pay for each click their ad received. If a user wanted feedback, they could download a tailored report or speak to a Google expert. Customers were wowed by this simple, personal approach to advertising – and, because their ads were being displayed against related search results, they knew that the people viewing the ads already had a genuine interest in what they were selling.
Since establishing a control over the search engine market a decade ago, Google has become a global financial powerhouse; indeed the company’s advertising revenues totalled $28bn in 2010. Furthermore, its ranking system has reshaped the advertising landscape.
In addition to the pay-per-click advertising sector, Google’s PageRank innovation has given birth to a whole new industry: search engine optimisation (SEO). Brin and Page’s PageRank algorithm remains a closely-guarded secret, which means that website managers don’t really know what puts one page higher up the search results than another. But that doesn’t stop them from trying to guess. These days, the quest to get websites to the top of Google’s search results is big business; in fact, the SEO industry is now thought to be worth around $16bn.
Companies now spend millions on their websites, in the hope of getting a high Google ranking. Those that rank highly can realise huge benefits from the increased traffic and sales that their position brings. High-profile businesses, such as cheapflights.co.uk, have built their success on a high search engine placing, and whole industries are shaped by Google’s ranking order. For example, it is estimated that 85% of internet users rely on search engines to access hotel information on the web, so well-designed, search engine-optimised websites are essential for hoteliers of all sizes.
The search engine revolution is likely to continue apace over the coming years. A report from Zenith Optimedia, released in July 2011, predicted that website advertising will rise 31.5% between 2011 and 2013, and much of this growth will focus on SEO and pay-per-click advertising as increasing numbers of companies come to see search engine marketing as a cost-effective alternative to traditional channels.
Companies now spend millions on their websites, in the hope of getting a high Google ranking.
As the rise of Google has continued, other search engines struggle to keep up. Yahoo! still has the second-largest share of search engine traffic, but in a recent interview with the New York Times, even CEO Carol Bartz admitted that the company has stopped trying to keep up. ‘We were never a search company. It was more, “I’m on Yahoo!, I’ll do a search”,’ she explained. Like many of its competitors, Yahoo! has branched into other technologies, transforming itself into a ‘portal’ that includes news, email and shopping channels.
Who knows what the search engine space will look like in 2021?
However, new challengers have emerged in recent years. Bing, the search engine rolled out by Microsoft in 2009, has grown steadily since its launch. In July 2011, it overtook Yahoo! as the UK’s second most popular search engine. Microsoft has also inked a deal with Beijing-based search engine Baidu, which currently dominates the Chinese search engine sector; in fact it currently controls 74% of the Chinese market by ad revenue, with Google commanding just 23%. And in Russia, Yandex boasts a 64% market share of search engine traffic. As the most popular Russian website it attracted 38.3 million unique users in March 2011, and raised $1.3bn when it went public on NASDAQ in May 2011. The initial public offering was the largest for a dotcom in America since Google listed in 2004.
These figures show that Google, for all its dominance, still has plenty of competition, and must keep evolving to remain the market leader. But no matter what Google does, it’s still far from certain that the company will be as dominant in 10 years’ time as it is now. At the start of the millennium few of us used Google. Who knows what the search engine space will look like in 2021?