RELIGIOUS STUDIES 377

Researching, Writing, and Thinking about Religion


UNIT 1
UNIT 2
UNIT 3
UNIT 4
UNIT DESCRIPTION
WHAT IS THE WEB?
SEARCH ENGINES
DIRECTORIES
THE INVISIBLE WEB
CHOOSING A TOOL
EVALUATING WEB SOURCES
REFERENCE LIST


THE INVISIBLE WEB


Now that you know how to find information on the Web using search engines and directories, the bad news is that much of the Web is what is called invisible. Search engines, in particular, are limited from finding a large portion of the Web, a portion often figured to be twice as large as the visible Web. While directories do not have the same limitations with the invisible Web, since they are created by humans, they are still primarily accessing the visible Web.

Why are Parts of the Web Invisible?

There are many reasons that Web pages can be invisible to search engines:

UNKNOWN PAGE If a Web page is not linked to in already indexed pages then a search engine’s spiders cannot find it.
PRIVATE WEB Some sites are blocked intentionally in order to remain confidential or to ensure that users pay to access them. All password protected sites are invisible to search engines. Unless you have access rights (such as being a University of Calgary student for University of Calgary Library resources on the Internet) you will not be able to use private Web sites.
UNREADABLE FORMAT Some information is invisible because it is in a file format that particular search engines cannot read, or choose not to include in their databases, such as word files or files in portable document format (pdf). Some search engines include a greater number file types while some are limited to fewer file formats (such as Google for pdf and image files).
DYNAMIC PAGES When accessing a database through the Web (such as the Library catalogue) it is often easier for the site to create pages as they are needed, based on each individual search, rather than store all the possible responses. These pages that are created on demand are called dynamic pages and are not stored anywhere and so cannot be searched by search engines. Search engines can only search stored documents, called static documents or pages.
HIDDEN PAGE Some Web sites are enormous and many search engines will index only the first or first few pages of each site. This means that pages that are buried in a site are missed. There are some search engines that index much further, doing what is called a deep crawl, though some sites remain too large to be indexed comprehensively. The number of pages that are catalogued per site varies from engine to engine which means the ‘depth’ of your search will vary with each search engine.
URLS's CONTAINING "?" A ‘?’ indicates a script, a type of program being used in the Web page that search engine spiders are programmed to avoid. Therefore, if the site you are searching for happens to have a ‘?’ in its URL no search engine will be able to locate it.

There is a wealth of information in the invisible Web, primarily hidden in databases and file types that search engines exclude. Though a search engine can find a database for you, it often cannot search that database, so all the information that it contains remains invisible unless you search it directly.

To make full use of the Web, being able to access the invisible Web is essential!

Finding Information on the Invisible Web

When searching for information on the invisible Web you are primarily searching for useful databases or resources other than HTML files such as word files and graphics. Some directories exist whose main purpose is directing users to invisible Web resources. They can be a good place to start your invisible Web search:

Remember, you can also use subject-specific directories to locate databases and other invisible Web resources. Search engines which specialize in the type of file you are searching for are also very helpful, such as Google’s image search (click on the image tab in http://www.google.ca).

If you find sites that you will want to use again and again, bookmark them in your Web browser to save yourself the time it takes to find them every time you use them.

Click here to continue to the next section


[ Home | Department of Religious Studies | Last Modified: 29 August 2007 | University of Calgary Library | Site Map ]