Defining the Web: The politics of search engines

Although the Web itself might truthfully claim a sovereign disinterested and unbiased attitude toward the people who use it, the authors of this article claim that search engines, the tools that navigate the astronomical number of pages (800 million and counting), favor popular, wealthy, and powerful sites at the expense of others. Some researchers have estimated that, taken individually, none of the Web search engines studied indexes more than 16 percent of the total indexable Web. Combined, the results from all search engines they studied covered only about 42 percent of the Web. But what about those portions of the Web that remain hidden from view? This article looks at how search engine developers, designers, and producers grapple with the technical limits that restrict what their engines can find. The authors also examine influences that may determine systematic inclusion and extematic inclusion and exclusion of certain sites, and the factors that dictate systematic prominence for some sites while relegating others to systematic invisibility.