Top Search Engines

Major Search Engines

Which of the many of search engines really matter? Usually, it's the search engines that are well-known and well-used. This is true whether your are a webmaster or a searcher.

For webmasters, a good listing in a search engine that promotes itself well, or has strong strategic alliances, is more likely to bring traffic than a lesser-known search engine. For example, a search engine listed on the Netscape Net Search page is guaranteed to receive much use. That translates into more traffic for sites that are ranked well by these search engines.

For searchers, well known, commercially backed search engines generally mean more dependable results. These search engines are more likely to be well-maintained and upgraded when necessary, to keep pace with the growing web.

Search Engines, Directories and Hybrids

Before naming names, it's important to explain the difference between search engines and directories. They are often confused. A more detailed explanation can also be found on the How Search Engines Work page.

Search Engines: Also called "spiders" or "crawlers," search engines constantly visit web sites on the Internet in order to create catalogs of web pages. Because they run automatically and index so many web pages, search engines may often find information not listed in directories.

Directories: Unlike search engines, directories are created by humans. Sites must be submitted, then they are assigned to an appropriate category or categories. Because of the human role, directories can often provide better results than search engines. Yahoo is an example of a directory.

Hybrid Search Engines: To further confuse matters, some search engines also have an associated directory. These are sites that have been reviewed or rated. For the most part, these reviewed sites do not appear as the "default" when a query is made to a hybrid search engine. Instead, a user must consciously choose to see the reviews.

The Major Players

Below are the current major players in the search engine game. 

Yahoo, though not a search engine, is listed below because it is such an important directory.

Does the world come to an end if your site can't be found easily in any of these "major" search engines? Not necessarily. If you want to reach apple farmers, then getting a link to your site from an obscure apple-farming web site may bring in much more meaningful traffic than by being indexed by all the general search engines in the world.

AltaVista
http://altavista.digital.com/

Opened in Dec. 1995, AltaVista is run by Digital and has gained wide-spread name recognition. It also partnered with Yahoo in June 1996, becoming the "preferred" search engine as explained below in the Yahoo section.

AOL NetFind
http://www.aol.com/netfind/

Launched in March 1997, AOL NetFind is a branded-version of the Excite search engine. It has a different name and a different look, but it is basically Excite underneath.

Excite
http://www.excite.com/

Launched in late 1995, Excite grew quickly in prominence and has eaten two of its competitors. In July 1996, Excite purchased Magellan. In November 1996, it acquired WebCrawler. These continue to run as separate services.

Excite lists sites in one of three ways: Excite Search, Channels By Excite and Excite NewsTracker.

Excite Search taps into the traditional search engine listings, created from crawling the web. Channels By Excite lists sites by topics. These sites have been approved by editors, and sometimes also have reviews. There is also much associated subject information, discussions areas and more. Excite NewsTracker allows you so search only listings generated by crawling specialty news sites.

HotBot
http://www.hotbot.com/

Launched in May 1996, HotBot is Wired's entry into the search engine market. The site is powered by the Inktomi search engine. That doesn't mean it is the same as the UC Berkeley Inktomi catalog, only that it uses the same technology that created that catalog.

Infoseek
http://www.infoseek.com/

Around since early 1995, Infoseek is well-known, well-reviewed and well-connected. The old "Infoseek Guide" index only had about 1 to 2 million URLs cataloged. In Fall 1996, the new service with 50 million URLs was introduced.

Infoseek runs a directory separate from its search engine. Sites are listed by topic, which are automatically generated using categorization software. Some sites also are listed with a red check marks. These are Infoseek Select Sites, which have been reviewed and are recommended by Infoseek.

LookSmart
http://www.looksmart.com/

Reader's Digest-backed LookSmart launched in Oct. 1996. It is the closest rival Yahoo has, in terms of a human-compiled directory. It listed about 300,000 web sites as of Feb. 1998.

LookSmart is positioned on the Netscape Net Search page and also provides browsable listings for both AltaVista and HotBot.

AltaVista provides LookSmart with search results when a search fails to find a match from among LookSmart's reviews.

LookSmart featured a unique Java-interface when it first launched, but this was toned down and eventually completely removed in order to speed up load times, when the site underwent a redesign in June 1997.

Lycos
http://www.lycos.com/

Around since May 1994, Lycos is one of the oldest of the major search engines. It began as a project at Carnegie Mellon University. The name Lycos comes from the Latin for "wolf spider."

Lycos lists sites in two main ways. There are search engine listings, and there is an associated directory called "Web Guides."

Each guide is oriented around a subject area, such as News or Shopping. They list sites, along with links to other relevant material.

The Web Guides were originally called A2Z, when Lycos added a directory to its service in Feb. 1996.

Lycos also runs a rating service called Top 5%. These are reviewer picks of what's best on the web. Reviews can be browsed or searched, and they can be sorted by review date, content, design, overall experience or alphabetical order.

Top 5% was originally called Point. Point was an independent company and one of the oldest web rating services. It launched in 1995 and aimed to list what its editors considered to be the top 5% of all web sites. Lycos acquired the service and later changed the name.

Microsoft / MSN

Microsoft announced in Oct. 1997 that it was partnering with Inktomi (which also produces HotBot for Wired Digital) to create a search engine for its MSN service and other web properties. It's expected to quickly become a major player after launch in early to mid-1998, so I mention it here. For more details, see the article below.

Microsoft To Launch Search Engine
c|net story

Northern Light
http://www.northernlight.com/
or http://www.nlsearch.com/

Northern Light opened to general use on Aug. 12, 1997. While it doesn't have the strategic alliances of the other major search engines, it is seriously crawling significant portions of the web and thus qualifies to be among the major search engines.

Northern Light features an ability to classify documents by topic, which it hopes distinguish it from some of the other search services, even those with their own topical listings.

Northern Light also has a set of "special collection" documents that are not readily accessible to search engine spiders. There are documents from about 1,800 sources, including newswires, magazines and databases. Searching these documents is free, but there is a charge of up to $4 to view them.

There is no charge to clickthrough to documents on the public web -- only for those within the special collection.

Search.com
http://www.search.com/

Launched March 1996, CNET's Search.com underwent a major change in May 1997. It became a branded-version of the Infoseek search engine, as opposed to an interface to several different search engines, as it had been in the past. See the Strategic Alliance page for more history of the changes.

There is one significant difference between Search.com and Infoseek. Search.com taps into the Infoseek database for general searching, but it uses its own database of listings for subject searches. These are generated by having the Infoseek search engine crawl select sites on a daily basis.

WebCrawler
http://www.webcrawler.com/

WebCrawler opened to the public on April 20, 1994. It was started as a research project at the University of Washington. America Online purchased it in March 1995 and was the online service's preferred search engine until Nov. 1996. That was when Excite, a WebCrawler competitor, acquired the service. Excite runs WebCrawler as an independent search engine. WebCrawler has an associated directory of reviewed sites, WebCrawler Select.

Yahoo
http://www.yahoo.com/

Around since late 1994, Yahoo is the oldest major web site directory. Yahoo is well-known, well-used and well-respected. It is also the largest directory (as opposed to search engine), listing 500,000+ web sites, as of mid-1997.

Because Yahoo is a directory based on user submissions, it may not have some sites in its catalog that a crawler might find from searching the web each day. If a search of Yahoo's catalog doesn't turn up any useful links, users may then wish to consult with a search engine.

Yahoo makes this easy to do. With a click, the query originally sent to the Yahoo catalog is "piped" or forwarded to any of the major search engines. Because so many people use Yahoo, search engines listed first on Yahoo pages have a strategic advantage over others.