Yahoo! SearchMonkey - Released to Developers

The good folks from Yahoo! unveiled their new open search platform Yahoo! SearchMonkey, at a developer launch party today at their Sunnyvale headquarters. In some ways, the SearchMonkey platform is revolutionary and a major step forward in search, allowing publishers to participate directly in improving the quality of their own information presented on the Yahoo! search results page (this is also implicitly a push for the bottom-up approach to the Semantic Web, which most industry observers have given up on in favor of a top-down approach). The platform also lets publishers and third-party developers build applications aimed at improving the search experience. Finally, and most important, if enough publishers and app developers participate in the program, it promises to improve the quality of search results for end users.

Features

At the simplest level, you can think of SearchMonkey as a community-powered set of rich information boxes (similar to the Google OneBox) that appear on the Yahoo! search results page. Publishers can provide this rich data to the Yahoo! search index in a variety of ways: through structured data feeds (RSS), through RDF or Microformat markup on web pages, or through simple page extraction. The "Information Bar" shows up underneath the main search results. The Yahoo! search team has also provided tools to enable developers to build search-based applications very simply and easily.

From Yahoo!'s documentation for site owners:

This is what an enhanced result would look like:

From the initial demo, it wasn't clear to me how the Yahoo! search engine decides when to display the rich results. Kevin Haas, Engineering Manager for Structured Web Search, explained that users have to opt-in to see enhanced search results from a specific publisher or developer. Publishers can add a button to their site that enables users to opt-in quickly and easily; from that point, these users will automatically see the enhanced results for any result URLs that match those specified by the publisher. In addition, for selected (major) publishers, Yahoo! could create a screening process to make their enhanced content visible to all users.

What if multiple publishers in the same vertical or domain have differing metadata or semantic structures? Haas clarified that the Yahoo! search team is planning to create schemas for various domains: they will either use existing microformats wherever they can, or else define something themselves.

What's in it for the Publishers, that will compel them to put time and resources into creating rich content for SearchMonkey? According to Amit Kumar, Director of Product Management for Yahoo! Web Search, it's all about finding new users and most importantly, about retaining and converting existing users for web sites. By providing richer, higher-quality results, publishers are more likely to get users to click on that particular content, regardless of where it is in the search results page, leading to higher click-throughs and conversion rates.

At the same time, this should improve the quality of search results in the long run; Kumar is convinced that it will help Yahoo! search gain market share.

Concerns

SearchMonkey is a major new feature for the Yahoo! search engine; if the publisher and developer communities rally behind it, it could significantly improve the search experience. However, there are some caveats:

• Users still need to opt-in to enhanced search results from specific publishers and developers. While this step is necessary to prevent spam and gaming of search results, any user action required automatically presents a barrier to entry, especially if the action seems to represent a perceived "installation" step.
• The SearchMonkey team is planning to create schemas for verticals and is going to try to reconcile varied semantic structures from different publishers within a given domain. I wish them luck with that!
• According to many sources, market share for Yahoo! search continues to fall, which makes it less attractive for publishers to spend time and resources in optimizing the quality of their results for Yahoo! search engine users; this in turn, makes users turn to other search engines, notably Google, for their casual search needs, which further affects the reduction in market share. With this vicious cycle going on, will publishers get behind this concept of providing enhanced search results to Yahoo!?
• Yahoo! has a well-deserved reputation for coming up with innovations in the web search space. For example, Yahoo! released their custom search API for publisher sites well before Google (although after core innovators Rollyo and Eurekster Swiki); yet Google's custom search engine seems to have become more popular with web publishers. Similarly, although this latest concept of opening up search results is leading-edge and innovative - how long before Google responds in kind?
   

Conclusion

With this new feature, Yahoo! search takes a big jump forward, both in supporting content publishers and in improving the quality of search results. But the biggest impact could be felt indirectly in the future.

Search improvements typically fall into one or more of these three areas: results display, query spec and relevance of results. With this new feature, Yahoo! will be gathering a great deal of metadata as part of the enhanced results specified by publishers. With all of this semantic content at its disposal: could Yahoo! enable parametric search queries in the future?

---