Invisible web

"There are parts of the Internet that search engines simply can't see. The search engine's robotic "crawlers" either miss or are locked out of these areas on the Internet. Behind the barriers lie treasure troves of quality information. Collectively this informatin is called the Invisible Web.
http://21cif.imsa.edu/tutorials/micro/mmpdf/invisible.pdf

• webpages that have been deliberately excluded by owners - privacy issues
• webpages that are dynamically assembled from online databse content (airline)
• password protected services, databases and webpages (Groliers, Proquest)
• non-HTML resources like some image, audio, animation and PDF files

"The Invisible Web is a vast part of the Internet that the search engines can NOT—or simply don't—include in their indexes of the Web (Gary Price)

Information and records only available from specialized searchable databases or directories that can be searched via the web.
(http://www.kn.pacbell.com/wired/21stcent/lsearch2.html)

You can get to the database, but you can't get into it - just with a search engine.

Are you aware that when you use a regular search engine, you are searching less than 10% of the actual web? that the inaccessible part of the web is 500 times larger than what is reached by regular search engines? and that the web is now estimated to be 4 billion pages? and is growing all the time?

Illustration of the content levels of the World Wide Web, including the Invisible web - About.com

Visible web versus Invisible web - organizational chart

Invisible web information - databases -

Fee based Databases @ St. Joseph's - paid subscriptions

book catalog - only available @ St. Joe's

encyclopedia databases

• Grolier's online encyclopedia
  • including
  • Lands and People
• World Geography

magazine databases

• E Library
• Proquest - examples
  • Dihydrogen monoxide article not found on web
  • Wikipedia prank - NY Times article no longer on web

databases @ Seattle Public Library - paid subscriptions / free to you

databases @ King County Library - paid subscriptions / free to you

Free databases - but they have a search system of their own - with ? in URL

• http://search.biography.com/print_record.pl?id=12594
• http://www.factmonster.com/search?fr=fmtn&query=cigarette+cards

• Biography.com
• CIA World FactBook - information in tables, charts
• FactMonster.com - and other almanacs online
• Merriam-Webster Dictionaries online - and other dictionaries online

"split level searching" For the first level, search for the database site. For the second level, go to the site and search the database itself for the information you want.

How to find information on the Invisible Web? Educational Cyber Playground

Subject directories - with sites chosen by people/experts - for broad topics

• Librarian's Index to the Internet
• Kid's Click

email listservs that list new sites

• NetHappenings - Educational Cyber Playground

Search engines designed specifically to reach the invisible web
(or search with Google, but add the word database to your search terms)

• CompletePlanet - the Deep web directory
• IncyWincy - search engine to the invisible web
• Infomine- scholarly internet resource collections
• WebLens

Subject Directories/lists of links by subjects

• Invisible web - Kid links
• Invisible web gateways - list of links - about.com
• Invisible web search sites lists of links - http://home.gwi.net/brhs/invsearch.html
• LibrarySpot.com - list of links by subjects

More information on the Invisible/Deep web

• What is the invisible web - U Berkeley article
• Invisible web revealed - Rider University article
• Deep web - Wikipedia article (wikipedia prank)
• The Deep Web - University of Albany
• Deep web FAQ - BrightPlanet.com