Spiders, Worms, Wanderers, Robots and CrawlersSpiders, Worms, Wanderers, Robots and CrawlersYou've ventured out onto the World Wide Web. Perhaps you've even found something useful. You've probably noticed that finding the information you want on the Web is nowhere near as easy as using the Yale Libraries - why is that? The Web is growing at an astounding pace and there is no central place to register Internet sites, nor a cataloging system adequate to describe these resources. There are, however, many different search engines that try to assist people in their search for relevant information.
Most Web-based search engines use forms. Fill out the relevant boxes, click on the "Search" button and wait for the results. Some World Wide Web search engines are better than others, either in their comprehensiveness, ease of use, or server stability.
In this month's On The Web column I'm going to talk about some of the better search engines available and how to evaluate other searchers you come across. The hypertext environment of the World Wide Web will lead you to an infinite number of other related web sites from a few good starting points.
Simple search engines scan the Web for documents meeting a specified set of parameters, and generates a resulting Web page that contains links to relevant Web sites. These searchers vary in the number of pages they index, the number of words indexed, and inclusion of subject categories.
Yahoo! is probably the best known search site on the Web. Yahoo! is a terrific place to begin exploring the Internet, because it strives to bring a subject organization to the World Wide Web. It is very easy to browse Yahoo!'s web space by subject and do basic keyword searches for indexed sites. One of Yahoo!'s greatest strengths are the brief annotations it attaches to each entry. It doesn't index a very large percentage of the World Wide Web, but it makes up for this lack of coverage by providing easy access to Internet sites by subject classification.
If you don't find what you're looking for on Yahoo!, try the Open Text Web Index. Open Text is a superior search engineÑit's easy to use and it searches every word on the pages it indexes. Other search enginesÑlike Lycos, Infoseek, or WebCrawlerÑindex selected keywords from pages. I usually use the Power Search mode, where one can specify the field in which the word appears (e.g.,URL, Summary, Anywhere) and how this word relates to other words you're searching (e.g., and, or, adjacent, near). There is also the option of conducting a Weighted Search, where words are ranked by their occurrence or their presence/absence. These options make a search engine incredibly valuable, giving you greater control of the results.
A newcomer to the simple search engine field is Alta Vista, the powerful web searcher from Digital Corporation. Alta Vista indexes over 16 million web pages and 13,000 newsgroups (in real time, it claims). You can do a lot in an Alta Vista search, although is not very intuitive. You can specify many parameters, but you need to format your search with obscure symbols and tags. It's essential to read the Help file here.
All-In-One search pages are a recent development on the web front. These sites collect a large number of search engines in a single place. As with the simple search engines, some of these All-In-One collections function better than others.
The experimental version of SavvySearch will search a number of simple search engines in parallel. For instance, it first runs your search simultaneously through WebCrawler, InfoSeek, PointSearch, Inktomi, and DejaNews, and then it returns the collected results in a single file. You have the option at the bottom of the page of running your search through other sets of search engines. SavvySearch includes the following search engines: Aliweb, CSTR, Deja News, EINET Galaxy, FTPSearch95, InfoSeek, Inktomi, Internet Movie Database, Lycos, NIKOS, Open Text, Pathfinder, Point Search, SIFT - Stanford Information Filtering Tool, Tribal Voice, Virtual Software Library, WebCrawler, Yahoo!, Yellow Pages. SavvySearch also offers the search form in a wide variety of languages, from Finnish to Korean to Esperanto.
At the highly developed-end of All-In-One searchers lies The Internet Sleuth. The Sleuth provides access to over 800 web-based databases. There are direct links to a number of simple search engines : Lycos, Infoseek, Alta Vista, Open Text, Yahoo!, DejaNews, APL Quote Server, and CNN News. Unlike SavvySearch, the Sleuth does not offer the option of parallel searching. Its real strength is the "Browse the Sleuth by Category" section, which includes links to commercial databases, full-text literary databases, usenet news groups, and much more. The Sleuth offers a simple search form for each database. You can also link to the original database site, where you will find more sophisticated search forms, explanations of its content, and other valuable information. The Sleuth is incredibly easy to browse, and it is a valuable tool for anybody trying to track down information on the Web.
The most powerful All-In-One search engine is MetaCrawler. MetaCrawler is a Multi-Threaded Web Search Service, which means that it doesn't maintain its own index of Web pages. It sends out queries to other simple search engines, including Open Text, Lycos, WebCrawler, InfoSeek, Excite, Inktomi, Yahoo!, and Galaxy. As you can imagine, this is a terrific time-saver if you wish to do a fairly comprehensive sweep of Webspace. MetaCrawler is extremely simple to use, and you can control some unique parameters. It allows you to specify the search region (from The World to Your Domain) and to limit the search sites by domain types (educational, military, government, etc.). This is useful if, for example, you are looking for organizations (.org) or companies (.com) in the United States that deal with academic grants and scholarships.
There are many other search engines available on the Web, but these are the ones I find most useful. The value of a search engine depends on what you are trying to do with it. Yahoo! and The Internet Sleuth are good places to begin browsing for information by subject. If you know what you want or need to do a more comprehensive search, tools like Open Text, SavvySearch or MetaCrawler will better suit your needs. With a bit of experimentation and patience, you will soon have the riches of the World Wide Web at your fingertips.
Sarah Prown, Reference Librarian, Sterling Memorial Library. <sarah.prown@yale.edu>