Weblog on the Internet and public policy, journalism, virtual community, and more from David Brake, a Canadian academic, consultant and journalist

Archive forNovember 22nd, 2004 | back to home

22 November 2004
Filed under:Search Engines at12:40 am

Google announced a few days ago going from 4bn to 8bn pages.

Which made me wonder what proportion of the web it covers now and how much I was missing before. The last major survey of search engine coverage was in 1999 (Lawrence, Steve, and C. Lee Giles. 1999. “Accessibility and Distribution of Information on the Web”:http://www.wwwmetrics.com/ Nature, 1999, 107-109.) and concluded that no one search engine covered more than 16% of the visible web (it’s probably better now). And what of the (much larger) invisible web? See Bergman, M. K. (2001) “The Deep Web: Surfacing Hidden Value”:http://www.press.umich.edu/jep/07-01/bergman.html, The Journal of Electronic Publishing, 7 (1) for more on that…