A three-year study on the freshness of web search engine databases

This paper deals with one aspect of the index quality of search engines: index freshness. The purpose is to analyse the update strategies of the major web search engines Google, Yahoo, and MSN/Live. com. We conducted a test of the updates of 40 daily updated pages and 30 irregularly updated pages. We used data from a time span of six weeks in the years 2005, 2006 and 2007. We found that the best search engine in terms of up-to-dateness changes over the years and that none of the engines has an ideal solution for index freshness. Indexing patterns are often irregular, and there seems to be no clear policy regarding when to revisit Web pages. A major problem identified in our research is the delay in making crawled pages available for searching, which differs from one engine to another.