HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Search for Related Content

TRAWLING THE DEEP WEB
What’s the first thing you do when you want to research a subject using the Internet? You probably go to a search engine such as Yahoo! or Google and type a word or two about the subject at hand in the search box.

You might be surprised, though, to know how little of the information that exists on the Web is cataloged by these common tools, according to the authors of the newly released book, "The Invisible Web: Uncovering Information Sources Search Engines Can’t See" (CyberAge Books, 2001).

The "invisible Web" or "deep Web" refers to online information that is open to the public but essentially invisible to the queries of most general-purpose search engines. The authors contend that the invisible Web is from two to 50 times larger than the visible Web. Bright-Planet, an Internet search software company, estimates there are as many as 550 billion pieces of information hidden from the view of most search engines.

Why is all this information so elusive? The format in which the data exist is one big reason. General-purpose search engines are relatively proficient at indexing regular old Web pages written in hypertext markup language, or HTML. These days, though, online information comes in a rich variety of formats: portable document format, or PDF; database delivery; streaming audio and video; "real-time" data; and dynamically built pages. These formats often are inaccessible by automated search engine indexing software.

Another reason is that most general-purpose search engines intentionally ignore large amounts of data. Locating Web resources and updating indexes cost money, and extracting data from the complex formats mentioned above only adds to the expense. Search engines also have to cope with millions of bogus or fraudulent Web pages and may inadvertently ignore legitimate information in the process.

There are ways, though, to gain access to the wide array of information that may be overlooked by common search tools. Here are a few suggestions:

– Use specialty search engines in addition to general-purpose tools. The Search Engine Watch site, at "www.searchenginewatch.com", provides a useful list of specialty search engines organized by subject area. These tools can provide a more comprehensive catalog of resources than general-purpose search engines by focusing on just one subject. Follow the "Search Engine Listings" link on the site’s home page.

– Search data stored in online databases that common search engines cannot access. The Invisible Web Directory, at "www.invisible-web.net", catalogs more than 1,000 databases whose contents are invisible to most search engines. Each of the subject categories on the home page is divided into detailed subcategories.

– Harness the power of multiple search engines using a metacrawler. Metacrawling software employs several search engines at once and synthesizes the results from all of the searches into one set of returns. Check out online metacrawlers such as "www.metacrawler.com" and "www.search.com", which can be accessed using any Web browser.

Reported by Joe Hoyle, electronic media editor, "hoylej{at}ada.org".

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Search for Related Content