Paying for Distribution
In addition to having strong technology and a strong brand Google also pays for a significant portion of their search market share.
On December 20, 2005 Google invested $1 billion in AOL to continue their partnership and buy a 5% stake in AOL. In February 2006 Google agreed to pay Dell up to $1 billion for 3 years of toolbar distribution. On August 7, 2006, Google signed a 3 year deal to provide search on MySpace for $900 million. On October 9, 2006 Google bought YouTube, a leading video site, for $1.65 billion in stock.
Google also pays Mozilla and Opera hundreds of millions of dollars to be the default search provider in their browsers, bundles their Google Toolbar with software from Adobe and Sun Microsystems, and pays AdSense ad publishers $1 for Firefox + Google Toolbar installs, or up to $2 for Google Pack installs.
Google also builds brand exposure by placing Ads by Google on their AdSense ads and providing Google Checkout to commercial websites.
Google Pack is a package of useful software including a Google Toolbar and software from many other companies. At the same time Google helps ensure its toolbar is considered good and its competitors don't use sleazy distribution techniques by sponsoring StopBadware.org.
Google's distribution, vertical search products, and other portal elements give it a key advantage in best understanding our needs and wants by giving them the largest Database of Intentions.
Editorial Partnerships
They have moved away from a pure algorithmic approach to a hybrid editorial approach. In April of 2007, Google started mixing recent news results in their organic search results. After Google bought YouTube they started mixing videos directly in Google search results.
Webmaster Communication
Since the Florida update in 2003 Google has looked much deeper into linguistics and link filtering. Google's search results are generally the hardest search results for the average webmaster to manipulate.
Matt Cutts, Google's lead engineer in charge of search quality, regularly blogs about SEO and search. Google also has an official blog and has blogs specific to many of their vertical search products.
On November 10, 2004, Google opened up their Google Advertising Professional program.
Google also helps webmasters understand how Google is indexing their site via Google Webmaster Central. Google continues to add features and data to their webmaster console for registered webmasters while obfuscating publicly available data.
For an informal look at what working at Google looked like from the inside from 1999 to 2005 you might want to try Xooglers, a blog by former Google brand manager Doug Edwards.
Information Retrieval as a Game of Mind Control
In October of 2007 Google attempted to manipulate the public perception of people buying and selling links by announcing that they were going to penalize known link sellers, and then manually editing the toolbar PageRank scores of some well known blogs and other large sites. These PageRank edits did not change search engine rankings or traffic flows, as the PageRank update was entirely aesthetic.
Yahoo!
Getting Into Search
Yahoo! was founded in 1994 by David Filo and Jerry Yang as a directory of websites. For many years they outsourced their search service to other providers, considering it secondary to their directory and other content features, but by the end of 2002 they realized the importance and value of search and started aggressively acquiring search companies.
Overture purchased AllTheWeb and AltaVista in 2003. Yahoo! purchased Inktomi in December, 2002, and then consumed Overture in July, 2003, and combined the technologies from the various search companies they bought to make a new search engine. Yahoo! dumped Google in favor of their own in house technology on February 17, 2004.
Getting Social
In addition to building out their core algorithmic search product, Yahoo! has largely favored the concept of social search.
On March 20, 2005 Yahoo! purchased Flickr, a popular photo sharing site. On December 9, 2005, Yahoo! purchased Del.icio.us, a social bookmarking site. Yahoo! has also made a strong push to promote Yahoo! Answers, a popular free community driven question answering service.
Yahoo! has a cool Netrospective of their first 10 years, a brief overview of their corporate history here, and Bill Slawski posted a list of many of the companies Yahoo! consumed since Overture.
On July 2, 2007, Yahoo! launched their behaviorally targeted SmartAds product.
Microsoft
In 1998 MSN Search was launched, but Microsoft did not get serious about search until after Google proved the business model. Until Microsoft saw the light they primarily relied on partners like Overture, Looksmart, and Inktomi to power their search service.
They launched their technology preview of their search engine around July 1st of 2004. They formally switched from Yahoo! organic search results to their own in house technology on January 31st, 2005. MSN announced they dumped Yahoo!'s search ad program on May 4th, 2006.
On September 11, 2006, Microsoft announced they were launching their Live Search product.
Other Engines
One would be foolish to think that there is not a better way to index the web, and a new creative idea is probably just under our noses. The fact that Microsoft is making a large investment into developing a new search technology should be some cause for concern for other major search engines.
Through this course of history many smaller search engines have came and went, as the search industry has struggled to find a balance between profitability and relevancy. Some of the newer search engine concepts are web site clustering, semantics, and having industry specific smaller search engines / portals, but search may get attacked from entirely different angles.
On October 5, 2004 Bill Gross ( the founder of Overture and pioneer of paid search) relaunched Snap as a search engine with a completely transparent business model (showing search volumes, revenues, and advertisers). Snap has many advanced sorting features but it may be a bit more than what most searchers were looking for. People tend to like search for the perceived simplicity, even if the behind the scenes process is quite complex.
Outside of technology there are four other frontiers search is being attacked / commoditized from
- Browser & Software Distribution: Search companies are paying computer manufacturers or software companies an aggregated value of hundreds of millions or billions of dollars each year to bundle their search toolbar with their products.
- Social Search: Large social networks have significant reach and a ton of page views. Yahoo! is rumored to be entertaining buying social network Facebook nearly a billion dollars. Yahoo! has already bought social picture site Flickr and social bookmarking site Del.icio.us. In August of 2006 Google signed a 3 year $900 million contract to provide search and advertising on MySpace.
In addition some companies, like Eurekster, are trying to create products which allow groups of webmasters to make topic or community specific search services. - Content Providers: Some content providers are trying to publish content on their own domains and build off their brand. Some are refusing to be included in search indexes. Some are requiring a kickback to be indexed. Some are unsure of what they want and are choosing to sue search engines, either for further brand exposure, or to gain further negotiation leverage.
- Content Aggregators: Search is just one way of finding information. Via RSS feeds and various other technologies many sites are offering what some people consider persistent search, or a way to access any information about a specific topic as it becomes available. Google also bought YouTube for $1.65 in stock. YouTube consists largely of pirated content which Google can organize and publish ads against based on usage data and other forms of ad targeting.
Search & Legal Issues
Privacy
In 2005 the DoJ obtained search data from AOL, MSN, and Yahoo!. Google denied the request, and was sued for search data in January of 2006. Google beat the lawsuit and was only required to hand over a small sample of data.
In August of 2006 AOL Research released over 3 months worth of personal search data by 650,000 AOL users. A NYT article identified one of the searchers by name. In 2007 the European Union aggressively probed search companies aiming to limit data retention and maintain searcher privacy rights.
Publishing & Copyright Lawsuits
As more people create content attention is becoming more scarce. Due to The Tragedy of the Commons many publishing businesses and business models will die. Many traditional publishing companies enjoyed the profits enabled by running what was essentially regionally based monopolies. Search, and other forms of online media, allow for better targeting and less wasteful / more efficient business models. Due to growing irrelevancy, a fear of change, and a fear of disintermediation, many traditional publishing companies have fought search.
In an interview by Danny Sullivan, Eric Schmidt stated he thought many of the lawsuits Google face are business deals done in a court room.
Newspapers
In September of 2006 some Belgian newspaper companies won a copyright lawsuit against Google News which makes Belgium judges look like they do not understand how search or the internet work. Some publisher groups are trying to create an arbitrary information access protocol, Agence France Presse (AFP) sued Google to get them to drop their news coverage, and Google paid the AP a licensing fee. Books
In September of 2005 the Authors Guild sued Google. In October of 2005 major book publishing houses also sued Google over Google Print.
Photos
Perfect 10, a pornography company, sued Google for including cached copies of stolen content in their image index, and for allowing publishers to collect income on stolen copyright content via Google AdSense.
Access to Hate Information
In May of 2000 a French judge required Yahoo! to stop providing access to auctions selling Nazi memorabilia.
Many requests for information removal are covered on Chilling Effects and by the EFF. Eric Goldman tracks these cases as well.
Pay Per Click & Ad Targeting Lawsuits
In 1999 Playboy sued Excite and Netscape for selling banner impressions sold for searches for Playboy.
Overture sued Google for patent infringement. Just prior to Google's IPO they settled with Yahoo! (who by then bought out Overture) by giving them 2.7 million shares of class A Google stock.
Geico took Google to court in the US for trademark violation because Google allowed Geico to be a keyword trigger to target competing ads. Geico lost this case on December 15, 2004. Around the same time Google lost a similar French trademark case filed by Louis Vuitton.
Lane's Gifts sued Google for click fraud, but did not have a strong well put together case. Google's lawyers pushed them into a class wide out of court settlement of up to $90 million in AdWords credits. The March 2006 settlement aimed to absolve Google of any clickfraud related liabilities back through 2002, when Google launched their pay per click model.
Search User Information
The US government requested that major search companies turned over a significant amount of search related data. Yahoo!, MSN, and AOL gave up search data. The Google blog announced that Google fought the subpoena
In August, Google was served with a subpoena from the U. S. Department of Justice demanding disclosure of two full months’ worth of search queries that Google received from its users, as well as all the URLs in Google’s index.
A judge stated that Google did not have to turn over search usage data.
AOL not only shared information with the government, but AOL research also accidentally made search records public record.
Search as a Commoditizer
Each search company has its own business objectives and technologies to define relevancy. The three biggest issues search engines are fighting are
- Publishing Rights: All search engines are fighting trying to gain the rights to index quality content. Some of the highest quality content is so expensive to create and market that there is not a business model for openly sharing it on the web. Worse yet, as more and more people get into web publishing the businesses that delay to get their content indexed will have lost authority and distribution the whole time they delayed. This, and the fear of disintermediation, are part of the reason there are so many lawsuits.
- Distribution: The more distribution you have the more profit you can use to leverage the ability to buy more content or make better content partnerships. Also more distribution means that you can potentially send more visitors (and thus profit) to a person who lets you index their content. More usage data may also help engines improve their relevancy algorithms.
- Ad Network Size & Efficiency: Efficient ad networks can afford to pay for more distribution, and thus help the search company gain more content and distribution.
In order to try to lock users in search engines offer things like free email, news search, blogging platform, content hosting, office software, calendars, and feature rich toolbars. In some cases the software or service is not only free, but it is expensive to provide. For example, Google does not profit from Google news, but they had to pay the AP content licensing fees, and hosting Google Video can't be cheap.
In an attempt to collect more data, better target ads, and improve conversion rates Google offers
- a free analytics product
- free cross platform tracking
- free Wifi internet access in San Francisco and Mountainview
- a free wallet product which makes it quick and easy to buy products
The end goal of search is to commoditize the value of as many brands and markets as possible to keep adding value to the search box. They want to commoditize the value of creating content and increase the value of spreading ideas, the value of attention, and the importance of conversion.
As they make the network more and more efficient they can eat more and more of the profits, which was a large part of the reasoning behind Jakob Nielson's Search Engines as Leeches on the Web.
Selling Search as an Ecosystem
Because search aims to gain distribution by virtually any means possible the search engines that can do the best job of branding and get people to believe most in their goals / ideals / ecosystem win. Search engines are fighting many ways on this front, but not all of them are even on the web. For example, search engines are trying to attract the smartest minds by sharing research. Google goes so far as offering free pizza!
Google hires people to track webmaster feedback across the web. Matt Cutts frequently blogs about search and SEO because to him it is important for others to see search, SEO, and Google from his perspective. He offers free tips on Google Video in no small part because it was important for Google Video to beat out YouTube for Google to become the default video platform on the web. Once it was clear that Google lost the video battle to YouTube Google decided to buy them.
Beyond just selling their company beliefs and ideology to get people excited about their field, acquire new workers, and get others to act in a way that benefits their business model search engines also provide APIs to make portions of their system open enough that they can leverage the free work of other smart, creative, and passionate people.
Selling search as an ecosystem goes so far that Google puts out endless betas, allowing users to become unpaid testers and advocates of their products. Even if the other search engines matched Google on relevancy they still are losing the search war due to Google's willingness to take big risks, Google's brand strength, and how much better Google sells search as an ecosystem.
Extending Search
Google wants to make content ad supported and freely accessible. On October 9, 2006, Google announced they were acquiring YouTube for $1.65 billion in stock. In March, 2007,Viacom sued Google / YouTube for $1 billion for copyright infringement. In 2007 Microsoft pushed against Google's market position calling Google a copyright infringer (for scanning books) and doing research stating that many of Google's blogspot hosted blogs are spam.
Social Search
In 2006 and 2007 numerous social bookmarking and decentralized news sites became popular. Del.icio.us, a popular social bookmarking site, was bought out by Yahoo. Digg.com features fresh news and other items of interest on their home page based on user votes.
Text REtrieval Conference (TREC):
In 1992 TREC was launched to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. In addition to helping support the evolution of search they also create special tracks for vertical search and popular publishing models. For example, in 2006 they created a blog track. Past TREC publications are posted here.
Other Search Conferences:
There are a number of other popular conferences covering information retrieval.
Search Science lists a number of conferences on the right side of the Search Science blog.
There are also a number of conferences which talk about search primarily from a marketer's perspective. The three most well known conferences for are
Sources and Further Reading:
Many of the following have not been updated in years, or only cover a partial timeline of the search space, but as a collection they helped me out a lot. SearchEngineWatch is amazingly comprehensive if you piece together all of the articles Danny Sullivan has published.
- Archive.org - The Internet Archive Wayback Machine (especially useful to view old content if any resource links break).
- SearchEngineWatch - Danny Sullivan's site about search. Here are some important events from his first decade of writing about search.
- Where Are They Now? Search Engines We've Known & Loved - Danny Sullivan reflects on some of the search engines that have passed away.
- Google Corporate History - The history of Google, from 1995 to today.
- Shmula - timeline of Google, Microsoft, and Yahoo! acquisitions.
- SEO Consultants - Offers a timeline of important events associated with the history of search and the internet. Up to date through 2006.
- Story about the foundation and history of Ask - by Kelli Anderson, through 2006.
- John Wiley & Sons - by Wes Sonnenreich, a history of search through 1997.
- Google Blogoscope - by Philipp Lessen, a history of search through 2003.
- CBS MarketWatch [PDF] - colored timeline of search through 2004.
- Wikipedia: Search Engines & Timeline of Hypertext
- Search Engines: Evolution & Revolution - by Glen Farrelly, covers 1990 through 1999.
- Enquiro Eye Tracking Study - report of how humans interact with search results.
- History of the Internet, Internet for Historians (and just about everyone else) - Richard T. Griffith's article about the history of the web, from 1991 to 2001.
- Searching on the Web - Link to a page that is no longer active, but which content still exists on the wonderful Archive.org. Contents are from 1960's to 2000.
- W3C: A Little History of the World Wide Web - by Robert Cailliau, from 1945 to 1995 .
- Search History Page from Sympatico - Link to a page that is no longer active, but which content still exists on the wonderful Archive.org. Contents are from 1990 to 2003.
- Hobbes' Internet Timeline - by Robert H'obbes' Zakon, from 1957 through 2004.
- How Yahoo Blew It - Wired article about how Google beat out Yahoo to win the search market race
- A Brief History of the Internet - by Walt Howe, from 1960 through 2006.
- The Sergey Brin Story
- Search Industry Explained - Search and Go discusses search history and SEO history
- ISOC Internet Histories - list of various histories of the Internet.
Books About Search
- The Search - John Battelle's book about the history of search and how search intersects with media and culture.
- The Google Story - David Vise's book about Google.
- The Google Legacy - Stephen E. Arnold's book about why he believes Google is in a better market position than its competitors.
~~~~~~~~~~~~~~~~~~~~~~~~~
I'll check to make sure it all posted this time! I guess Ryze does have a limit after all for LOOONNNNGGG posts. Ha!
EB :D
