Wikipedia Search to compete with Google

Jimmy Wales, founder of Wikipedia, has announced that his for-profit company, Wikia Inc, will be launching an internet search engine to compete with Google and Yahoo!.

According to the Times of London  :

Mr Wales has begun working on a search engine that exploits the same user-based technology as his open-access encyclopaedia, which was launched in 2003.
The project has been dubbed Wikiasari — a combination of wiki, the Hawaiian word for quick, and asari, which is Japanese for “rummaging search”.

The Times article goes on to say:

Mr Wales, a 40-year-old former options trader, believes that, as the popularity of Google has grown, obvious flaws in its search engine technology have become apparent.

“Google is very good at many types of search, but in many instances it produces nothing but spam and useless crap. Try searching for the term ‘Tampa hotels’, for example, and you will not get any useful results,” he said.

Spammers and commercial ventures are also learning how to manipulate Google’s computer-based search, he added.

Mr Wales believes that Google’s computer-based algorithmic search program is no match for the editorial judgment of humans.

To some extent, this makes sense to me. Although I'm a big fan and a heavy user of Google, over time I've noticed an increase in the number of spam/commercial links in the organic search results,especially for domains saturated with SEO-savvy web marketers. Using inbound links as a proxy for relevance and authority is no longer as effective as it was five years ago; marketers are getting increasingly better at fooling the algorithms.

Humans, on the other hand, can be far more effective at judging the meaning and "commercial-ness" of web pages, usually with a high degree of accuracy. And the model of leveraging the collective wisdom of a large group of people for solving very difficult problems has been proven to be effective in numerous cases - GoldCorp , Innocentive, the Netflix prize (currently ongoing), even Wikipedia itself.

But can a human-powered approach scale enough? Wikipedia has successfully proven the original model of large-scale participation by "citizen volunteers" to produce a comprehensive body of authoritative work, but it faces two major issues:
a. Ongoing attempts to manipulate content for marketing leverage
b. The opinion of the many is really expressed by the actions of a few (See Eric Goldman's posts here and here for reference.)

Spamming/gaming attempts are particularly pernicious, because they force editorial changes and restrictive content policies at Wikipedia - changes which strike at the heart of the egalitarian participation that made Wikipedia so wildly successful. (This is identical to issues at Digg - everyone gets one vote, but by the way, some votes count for more than others!).

It seems to me that these two issues will only worsen with an "open content/open source" search engine - the incentive for marketers to try to game the system will increase significantly, and it goes without saying that a great deal more effort will be needed from contributors. Given that contributors get no financial rewards and no significant marketing mileage for their efforts, I feel skeptical about the ability to scale up this effort to produce (and more importantly, maintain) a large-scale, general-purpose search engine.

Having said that, I hope I'm wrong, and my best wishes are with this project. I would love to see a great new alternative to the existing large-scale search engines, especially if it could produce spam-free, authoritative, relevant results. Regardless of how this turns out, the Search space is going to be an interesting one to watch in 2007!

---

Update:

1. Correction: Wikiasari is the name of the software, not that of the search engine.
2. As noted  by the Search Marketing Gurus, there is a lot of excitement and hysteria going on about this breaking story. The best place for definitive answers is on Wikia's search home page  .
3. Ashkan Karbasfrooshan has an interesting post   on his blog that compares search results from Google, MetaMojo, Ask, AOL, MSN and Yahoo! with the purported screenshot   of "WikiSearch" from TechCrunch (which itself may not be accurate).