Grading the Search Engines
Grading the Search Engines
Think your search engine is the best one around? You may be in for some
surprises. FSB Browser columnist Maggie Overfelt offers her take on the best
and the worst sites where you can troll for information.
Fortune Small Business
Thursday, August 29, 2002
By Maggie Overfelt
If there’s one thing web surfers are always looking for – aside from a
naked newscaster they can trust – it’s the perfect search engine. So it’s
hardly a surprise that Google and its rivals each claim to have one kind of
superiority or another. But how’s a casual surfer to know?
That’s where we come in. To test the top search engines, we ran dozens of
queries in a multitude of subjects on each search engine at the same time
(i.e., seven windows). We used a variety of techniques to reveal how each
engine works and where it stumbles: We misspelled, hyphenated, and used
multiple words; we asked questions (Why is it called Doppler radar?) and
used words with multiple meanings (“vet”). We evaluated each engine by
analyzing the quality of the top ten results for each search, and clicked
through to see just how relevant the pages actually were. We checked for any
links that were outdated, dead, or duplicated. We also considered any
additional material on search return pages, such as paid-for listings
(so-called sponsored links). Our findings? Read on.
Quality of results: Most relevant listings turn up in the top ten, but
usually aren’t the first or second item (No. 1 is always a sponsored link).
Fewer duplicate and dead links than other engines. The site tries to
anticipate searchers’ needs in two ways: First it lists what it thinks are
relevant categories (or Fast Topics) to match your query, but it weeds out
unrelated pages only about half the time. The second feature, Narrow Your
Search, lets a query for Yorkshire terriers, for instance, become more
targeted by letting users choose “Yorkshire Terriers for Sale” or “Yorkshire
Terriers UK” before getting results.
Organization: doesn’t automatically divide results into categories (unlike
Yahoo). Results come in more formats than from any other search engine: Web
pages, news, pictures, videos, MP3s, or FTP files. Automatically rewrites
queries to perform searches only with keywords (“history of the zoom lens”
became “zoom lens”), which usually boosts results. Runs three sponsored
sites before its own returns.
Advanced search: Six options, most of which are too technical to be of use
(filtering by IP address or document size?).
Notable extras: Offensive Content Reduction Filter.
Overall grade: B
Quality of results: Returns largest number of relevant results right up top.
Returns too many duplicate links. Suggests new searches if it thinks query
is misspelled (unlike MSN, which automatically corrects errors).
Organization: Most-relevant sites are first. Returns appear very unorganized
because only very general searches “senators”) yield matching categories
(“Regional>North America>.>Government.>Senate”), and it doesn’t give site
descriptions for all results, making the best link harder to see. Relevant
ads are marked as sponsored links on the right-hand side, and are equipped
with an interest bar, which tells you approximately how many other people
thought the link was relevant enough to click through.
Advanced search: Can narrow search by date, domain (e.g., fsb.com), file
format (e.g., MP3), or language. Hitting “similar page” on a relevant page
yields better results: A query of “technology terminology” resulted in only
one computer-term dictionary, but hitting “similar page” yielded eight.
Notable extras: Can translate from English to five other languages; can
search the Usenet public bulletin board system.
Overall grade: B+
Quality of results: Almost always lists relevant results on first attempt,
though best results can be hard to find. Many duplicate results (though not
as many as Google) make the list seem cluttered. If a query is somehow
related to an MSN partner (a search for “CD players” suggests MSN’s eShop
first), results can be too numerous. Many dead links, but most do redirect
to the site’s home page.
Organization: Actual search engine results are buried beneath whatever links
MSN deems relevant (“top ten most popular sites,” “featured sites,” “sponsor
sites,” and “news clips”). At bottom, “shopping results” links directly to a
site to buy, if pertinent.
Advanced search “Stemming” feature lets you broaden results to include both
word roots and word derivations, returning “hacking” and “hackers” when you
search for “hacker.” No place to add in extra search terms. Doesn’t support
quotation marks to specify an exact phrase inside the query.
Notable extras: Automatically corrects spelling.
Overall grade: B-
Quality of results: Results are highly relevant, but only if you’re looking
to buy a specific product or brand. Search a topic for which Overture
doesn’t have a sponsor and results can be as good as Google’s.
Organization: Very clean; fewer results than any other search engine (“Ford
Mustang” pulled up 79 responses; on Google, 372,000). Category links
unrelated to queries: Click on “Computing” and get 40 links to 40 sites of
paid advertisers (Westwood College’s computer learning program came before
Advanced search: None.
Notable extras: Notes how much the advertiser paid to be listed.
Overall grade: C
Quality of results: Frequently returns dead or irrelevant links. (This –
along with the site’s other quirks – will presumably be addressed soon when
Teoma starts to use Google’s listings, reflecting a deal the two recently
Organization: Divides results into three “R’s:” Results (main list of
uncategorized pages), Refine (related categories), and Resources (links to
“experts”), the latter two being frequently wrong (a search for a
technology- term dictionary pulled up a French-Dutch linguist).
Advanced search: Can search for an “exact phrase.”
Notable extras: Specific “search tips” help narrow searches.
Overall grade: C-
Quality of results: Results for multiworded queries are unrelated – as if
it searched each word separately. Pulls up many outdated links. Supports
only up to seven words per query.
Organization: Summaries don’t accompany links, so users can’t determine
relevance quickly. The category list, or “WiseGuide,” differentiates between
terms that could obviously fall into the same category (e.g., “Vietnam vets”
and “Vietnam veterans”).
Advanced search: “WiseSearch” lets you expand the search by providing space
to enter in up to seven extra search terms.
Notable extras: “Sneak-a-Peek” lets you view the site in an in-page window
if you use Internet Explorer (very cool). But service failed, complaining of
Overall grade: D-
Quality of results: Always dead-on, Yahoo divides its results into relevant
categories and gives quick, pertinent descriptions. Always offers “related
searches” for the overall query and “similar results” for individual
listings to accurately refine a search on the fly. If it doesn’t have what
you need in its own directory, it helpfully uses Google’s Web page listings.
Organization :Results are broken down into neat rows of related categories,
making relevant links easy to locate. Website listings come with a precise,
one-sentence summary. Search terms are highlighted in results, like Google.
Ads are small and usually related.
Advanced search: Not needed, because Yahoo offers up all necessary search
methods on the first pass. Otherwise, the “advanced search” is weak.
Notable extras: Click on “Research Documents” to pull up current news
stories relating to the topic.
Overall grade: A
Do you agree with our assessments? Share your opinions by writing to