Curiosity Trumps E-popularity

Kurt Kleiner
Scientific American Mind - December2006/January 2007

What stops a few popular websites from dominating the global exchange of ideas on the internet? Human curiosity powered by search engines, researches say.
Visitors view and link to a few websites such as Yahoo, eBay and MySpace often, whereas most sites are hardly noticed. Some experts worry that search engines such as Google exaggerate this trend, because they rank search results partly by how popular a website already is. As a result, big sites could get ever more popular and small sites could grow relatively more obscure-an increasingly undemocratic result that that critics deride as a "Googlearchy".

But Filippo Menczer, who teaches cognitive and computer science at Indiana University found that this is not the way it works in the real world. His team studied database of search terms and webpage traffic and then created a mathematical model to explain the observed patterns. It turns out that even though search engines reward pages for being popular, they also boost traffic to remote sites.
Menczer's model suggests that this effect occurs because people use search engines when they are looking for very specific information. So a small site that closely matches their individual interests will beat out a match more popular site that does not.
"That's a piece of behavioral data that the previous model did not consider. If you do no consider it, you assume everybody thinks the same way; everybody's interested in the same things, but that's not the case," Menczer says.

For example, if you search Google for "windows", the first hit will be Microsoft Corporation, maker of the Windows operating system and one of the world's most popular websites. But if you search "double-hung windows", you will come up with the little known website for Iowa-Based Pella Corporation, which makes actual windows.

Menczer's model suggests that search engines introduce people to 20 percent of more websites than they would find if they were forced to simply to surf from site to site—as web users did in the old days before search engines.