The Software Abstractions Blog

Yahoo! SearchMonkey - Released to Developers

The good folks from Yahoo! unveiled their new open search platform Yahoo! SearchMonkey, at a developer launch party today at their Sunnyvale headquarters. In some ways, the SearchMonkey platform is revolutionary and a major step forward in search, allowing publishers to participate directly in improving the quality of their own information presented on the Yahoo! search results page (this is also implicitly a push for the bottom-up approach to the Semantic Web, which most industry observers have given up on in favor of a top-down approach). The platform also lets publishers and third-party developers build applications aimed at improving the search experience. Finally, and most important, if enough publishers and app developers participate in the program, it promises to improve the quality of search results for end users.

Features

At the simplest level, you can think of SearchMonkey as a community-powered set of rich information boxes (similar to the Google OneBox) that appear on the Yahoo! search results page. Publishers can provide this rich data to the Yahoo! search index in a variety of ways: through structured data feeds (RSS), through RDF or Microformat markup on web pages, or through simple page extraction. The "Information Bar" shows up underneath the main search results. The Yahoo! search team has also provided tools to enable developers to build search-based applications very simply and easily.

Powerset Launches Wikipedia Search

Semantic search engine Powerset, which we've written about here before, has just launched its initial release. The current release is limited to indexing Wikipedia content, but it provides a great showcase for their technology and user experience.

For example, my search for "Alexander the Great" provided the following results page:

Cooperation of Alternate Search Engines: A Manifesto

( This post is inspired by my discussions with my friend, Charles Knight of AltSearchEngines )

Background

I'll be at the Alternative Search Engines Day tomorrow, a unique event in San Francisco put together by Charles and the AltSearchEngines team. The event is sponsored by SeeqPod, UpTake, Matchpoint, HealthPricer, GoPubMed and Blogdimension. (Unfortunately, it's not open to the general public.) If you're part of an Alternative Search Engine, I hope to see you there!

As I was getting ready for the event, it got me thinking about ASEs and how they can work together.

The Case for the Alts

I love the ASEs - Alts rock! Without them, there would be little innovation in Search, no new frontiers to be explored.

The Alts are the ones that keep pushing the envelope with new directions in search technology, whether it's algorithms, user interface, social search or something else.  Although Google has some fine technology and is synonymous with search, I firmly believe that we're still at Search 1.0, and have a long way to go. Because of all this competition from the Alts, and the resulting innovation, web search continues to improve.

WebGuild Web 2.0 Conference: Issues and Challenges for Crowdsourcing

I attended a really interesting session at the WebGuild Web 2.0 Conference and Expo last week: The Power of Crowdsourcing - moderated by Jeremiah Owyang of Forrester Research.

Participants on the panel were:

• Aaron Strout, Vice President, New Media, Mzinga
• Michael Sikorsky, CEO, Cambrian House
• Anil Rathi, CEO, Idea Crossing

This was one of the best panel sessions I attended at the conference, part of the reason being the bang-up job Owyang did as moderator. He took a very active role, bringing up provocative questions, directing those at specific members of the panel and not being shy about treading into the concerns and difficulties of using crowdsourcing and social media - this prevented the session from degenerating into a "Rah, Rah, Crowdsourcing is all good!" type of discussion.

The other reason was that the panel members were all knowledgeable, articulate and open in their remarks; the conversation never flagged, as it did with some of the other sessions I attended.

To kick off the session, Owyang put up a few slides entitled Social Technographics that were intriguing, but more on that in a future post.

One of the panelists, Michael Sikorsky of Cambrian House, listed the three legs of crowdsourcing as follows:

• Wisdom, which can be explicit (e.g. voting in American Idol) or implicit (e.g. links used for calculating PageRank)
• Participation, such as item submissions or code check-ins
• Funding, such as a prize or project funding

The Challenges

Based on the discussion at this session, I've compiled the following list of challenges in implementing crowdsourcing solutions and ways of addressing them.

• Wisdom of Crowds: How do you keep the input quality high?

For any crowdsourcing activity, the first step is to pick the right crowd! Equally important, you must ask the right question.

The next step is to use statistical methods to prioritize high-quality input. Finally, a self-policing community (possibly, with some moderation) can help weed out low-quality input and spam.

• Is there an inverse relationship between those who have the time to contribute, and the quality of the ideas presented?

One caveat to keep in mind is that the vocal minority may not be representative of the majority of users. But this type of forum may act as a funnel for identifying talented people who have not yet been discovered.

By providing rewards or incentives consistent with the value of the ideas being submitted, you can get greater participation from qualified users and a higher level of confidence in the quality of the ideas being submitted. Another alternative is to use some type of game mechanism; games have built-in rewards that encourage participation.

• What if some people don't want to be outsourced?

Tara Hunt, of Citizen Agency, recently wrote a blog post titled: Please Stop Crowdsourcing Me , questioning whether crowdsourcing is a good idea. She has a point - some users may not want to contribute or be involved in a crowdsourcing exercise, especially to benefit a large corporation.

The panelists agreed, and pointed out that you should carefully consider which tasks should be outsourced in this way - for example, product users love to help each other out with solving problems and difficulties, but if participants get the feeling that the company is simply using them to reduce customer service costs, then they will stop being helpful.

Any crowdsourcing program has to be thought through and managed carefully; you don't want to risk users having a bad experience.

• How do you manage and lead a crowd, to create a positive experience?

For the community to be truly engaged, it is extremely important for the company to be very transparent.

One key point to think about, especially for large companies, is that you have to be careful about what you share with the crowd. On the one hand, the more you share, the better the ideas you will get; on the other hand, you risk letting out corporate proprietary secrets.

Finally, some activities simply may not be amenable to crowdsourcing.

• How much control do you want to retain? Do you need a Product Manager as an expert?

A community of users can generate a lot of great ideas, but those don't all necessarily fit together; having an expert in place as a product manager can provide guard rails to keep things on track. The product manager can bring a single, unified vision and - this is critical - can communicate back to the community why a particular idea is not being used.

It's important to find a balance: the community generates the ideas, but the company or organization picks the ones to be used, refines them and implements them. Even the nuggets of ideas can be leveraged to create lots of value.

 

Successful Examples

The panelists also offered examples of actual crowdsourcing implementations:

• The Longitude Prize - one of the earliest examples, was a reward offered by the British government through an Act of Parliament in 1714 for a simple and practical method for the precise determination of a ship's longitude.
• Procter & Gamble has raised the level of outside design and significantly increased the success of product-related improvements.
• Intelpedia from Intel, is an example of crowdsourcing in the Enterprise space. The idea is to look internally for ideas, share best practices and preserve common knowledge. According to reports, Intelpedia has up to 20K pages already.
• Ace Hardware created a community for 300 dealers, whose ROI was measured at 500%; as a result, the community was rolled out to all 5000 of its dealers.
• The Hopelab Foundation created a global competition for kids of all ages and received submissions from 429 teams.
• American Idol has produced highly successful artists, some of whom have sold over 10M CDs; even the worst idol winner has sold 500K CDs.
• Innocentive is a well-known example where companies post complex problems and offer rewards for their solutions.

[Update :  An alert reader pointed out that the P&G name should be spelled with an "&" - this is now fixed. Thanks, John!]

---

Zvents makes Local Search pop!

There is a class of web search engines that can prove even more useful than Google within a certain context. I'm talking, of course, about Vertical Search engines - the writer and tech strategist Sramana Mitra considers them Google's Achilles heel and Profy.com's Cyndy Aleo-Carreira seems to agree. This blog also has long held the position that vertical search represents a powerful mechanism to find information on the web, and is a key category to watch in the search wars of the future. [see: The rise of Vertical Search Engines from Aug 2006].

Another way of achieving a similar focus, in order to improve the relevance of search results, is by segmenting by location rather than by industry vertical - i.e. create a hyperlocal search engine that limits its search results to a given geographical area.

One such alternate search engine is Zvents, which is relentlessly focused on local information, of any sort. This company, which has been around since early 2005, has just introduced an advanced feature called Federated Local search - basically, its own version of Universal Search (recall that Google introduced its Universal Search feature with much fanfare last May).

Federated Local Search: Multi-Dimensional Results for Local Information

What does Universal Search mean, for a local search engine? Initially this was not very clear to me; an email discussion with Paul O'Brien, Director of Marketing at Zvents, inspired me to draw the following diagram:

The basic idea is to enable the user to implement a general-purpose search within a local context. This allows the user to find local information about a given topic, across many different dimensions. For example, a sports fan living in San Jose, CA who tries a local search for the term "hockey", would get the following different types of results:

• Upcoming games for the San Jose Sharks, the local hockey team
• The location of Roosevelt Park Roller Hockey Rink
• The description and link for a local "Hockey Night" event
• Results about relevant personalities (what Zvents calls "performers")
• And other related links ...

Zvents has already partially implemented this vision, although some of the lower-ranked results could provide a better match. Hopefully these will improve in the future as the search index grows and the algorithm improves. A screen shot of this local Hockey search in Zvents is given below.

Similarly, here's a search for the term "Web 2.0" for Cupertino, CA:

Outcome: Relevance

The big advantage of this type of search, over a general-purpose Google or Yahoo! search, is that the user can obtain the benefits of a broad cross-section of results, while still constraining the search to a limited geographical area.

This is not a significant issue in highly developed, urban, technologically advanced areas like Silicon Valley, Boston or New York; but it could one day make a big difference for someone living in David Letterman's "home office" of Wahoo, NE , or even more important, someone trying to find the Boston Public School located in Boston, Ontario - as we've seen before, highly popular keywords tend to swamp nearby long-tail keywords in the search results for major search engines.

From a business model perspective, hyperlocal searches tend to provide highly qualified prospects for local merchants, so I would guess that this type of search is very easily monetizable in the long run.

From a user interface point-of-view, the NLP-like implementation of time period for the search engine ("when: tonight, this weekend, ...") is a nice touch; I tried different possibilities ("next month"), and it seemed to work just fine.

On a more technical note, Zvents has been making waves with the release of its open-source Bigtable clone called Hypertable, which adds a C++ option for this project.

Going forward, it will be interesting to see how Zvents scales to additional locations, and to additional dimensions within each locality. Will it make inroads into the market share for any of the major search engines, or into that of other locally-focused web sites like topix.com and craigslist?

---

Quintura Launches Site Search Widget

Alternative search engine Quintura, which I've mentioned before on this blog, has launched its site search widget. This widget allows site publishers to provide users with a specialized search limited to that specific site; it joins earlier offerings from Google, Yahoo!, Rollyo and Eurekster swiki in this space.

This blog was an early user of this widget. You can see a customized, Quintura-generated mini-tag cloud in the earlier post; a full-size tag cloud is also available. The widget is hosted by Quintura, so installation was a snap: once the site was indexed, all I had to do was to embed the widget code into my blog pages and provide some styling control.

The biggest benefit of using the Quintura solution, as I've said before, is the dynamic tag cloud that allows the user to navigate the search space; initial feedback from our readers here has been positive, but not enthusiastic.

The real benefits to both users and publishers will come when Quintura search results prove to be better than equivalent results from a mainstream search engine solution, such as Google; as long as the Google site search results are good enough, it will be hard for the Quintura widget to make significant inroads into the market share of the big-G juggernaut.

This widget release is currently in private beta; an invite for this beta is available over on ReadWriteWeb.

---

Disambiguation of Search Results? Yup, Google's got that

Just last week, in an email exchange with another search blogger, I wondered when Google would provide options for disambiguation of search results.

When you think about it, that's an obvious requirement for the Results page of any serious search engine. If I query for the search term "Java" - does it mean that I'm looking for results about the programming language, the coffee, or the island in Indonesia?

There's no way for the search engine to be able to tell, although personalization could provide clues. The easiest solution, as I wrote back in 2006, is for the search engine to just ask - which is why Wikipedia offers this page: Java (disambiguation) . Alternatively, the results can be grouped into various categories for the user to choose from, which is another way of doing the same thing.

Until now, Google has been mostly following a third option, which is to simply pick the most popular category regardless of the user's real preference; this can lead to some strange results, as highlighted in my earlier post on deconstructing real Google searches. But this approach doesn't really cut it, since it ignores all the unpopular search results - it's very possible that the long-tail searches can collectively make up a market share that rivals or exceeds the relatively few "popular" searches.

There has also been a limited amount of disambiguation offered by Google's "related searches" feature.

Well, no more. Google appears to be experimenting with offering disambiguation directly by grouping search results into categories. See the screen shot below, that shows Google search results for the query: "freebase" . Effectively, the results page seems to be asking: do you mean, the free semantic web database, or the other kind, associated with drugs? Or a third alternative: FreeBase - a free Windows software program to configure the Apple AirPort Base Station.

The use of horizontal ruled lines to separate the sections, is a nice touch!

Obviously this is some type of test; I certainly hope it's successful. I can't wait to see this feature become mainstream among the major search engines. It will be a big step forward in Search!

---

Deconstructing real Google searches - why Powerset matters

I was looking at the log files for my blog today, as I regularly do, and I was suddenly struck by the variety of search queries in Google for which users were getting referred to my posts. I write often about the different flavors of search - including vertical search, parametric search, semantic search, and so on - so users with queries about Search often land here. But do they always find what they're looking for?

Some Real-life Search Results

Let us examine some of the actual Google queries - in the form of referring URLs - that led users to my blog. In most cases, Google did a fine job of matching the content to the query; in some cases, it was a somewhat random match at best; finally, in a few cases, the Google search algorithms are clearly getting confused. It is this third case that is the most interesting.

The Good

In many cases, the match was quite straightforward and very relevant. Some examples are given below.
1.

Query: http://www.google.fr/search?q=Guru+Avinash+Kaushik& ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:fr:official& client=firefox-a
Result:  A conversation with Avinash Kaushik, Web Analytics Guru

Well, can't argue with that!

2.

Query: http://www.google.cn/search?sourceid=navclient&aq=t& hl=zh-CN&ie=UTF-8&rlz=1T4XNLA_zh-CNCN246CN247& q=vertical+search
Result:  The rise of Vertical Search Engines (VSEs)

Query:    http://www.google.com/search?
q=wikipedia+to+try+and+compete+with+google& ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official& client=firefox-a
Result:  Wikipedia Search to compete with Google

Again, can't argue with those.

3.

Query:  http://www.google.com/search?hl=en& q=search+technology+exits
Result:  So You've Built an Alternative Search Engine - Now What?

This is actually pretty awesome, the algorithm has figured out "search technology" and "exits"; in fact, this post does talk about exit strategies for search engines, so it's a great match.

The Bad

Some search queries are so vague that the matches you get are bound to be somewhat random. I don't blame Google for the following matches:

4.

Query:  http://www.google.com/search?hl=en& q=conceptual+architecture
Result:  A Conceptual Architecture for Search

Is the search string too vague? Although technically this post matches the search query, I'm guessing that this is not what the user intended to look for.

5.

Query: http://www.google.com/search?hl=en&safe=off& client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial& hs=pGP&q=disruptive+technologies+blog& btnG=Search
Result:  Disruptive technologies for 2007

While the words match, and possibly this may satisfy the user, I get the sense that the user was looking for a blog dedicated to discussing disruptive technologies, not a single post. But who knows? Again, too vague!

In the future, I wonder how soon Search technology will progress to the point where the UI will automatically ask the user for more information to qualify search terms that are too general or vague. A little while ago, I envisioned a similar scenario ( Vertical Search, with authority ) when taking a look at the search engine MetaMojo, which has taken some steps in this direction.

The Ugly

In a few cases, though, the proximity of certain keywords fools the search algorithms. Consider the following matches:

6.

Query: http://www.google.com/search? q=best+search+engine+for+directions&ie=utf-8& oe=utf-8&aq=t&rls=org.mozilla:en-US:official& client=firefox-a
Result:  Future Directions in Search

A post about "future directions in Search" is not a post about "search engines for directions", although the text itself is undoubtedly a close match.

7.

Query:  http://www.google.com/search?hl=en& q=people+search+software+compared&btnG=Search
Result:  Search and the Dumbness of Crowds

Hmm? This is a popular post, but I'm not sure if it helps the user, who is not trying to compare search strategies (as this post does); instead, the user appears to be trying to compare people search engines.

Are these good matches? While the content of the posts bears a superficial resemblance to the text in the respective queries, the results are not relevant to the requested user searches.

The Larger Problem

The samples given above are not that important; the matches from my blog do not always show up at the top of the search results and although these are real referrals, not many users will actually click on these links in the Results page. But these examples point to a deeper underlying issue, one that will be far from easy to fix in the general sense.

All the major search engines currently rely on the proximity of keywords and search terms to match results. But that approach can be misleading, causing the search engine to systematically produce incorrect results under certain conditions.

To demonstrate, let us take a look at three general use cases.

[Note: The examples given below are all drawn from Google. To be fair, all the major search engines use similar algorithms, and all suffer from similar problems. For its part, Google handles billions of queries every day, usually very competently. As the reigning market leader, though, Google is the obvious target - it goes with the territory!]

1. Difficulty of Finding Long Tail Results

Take Britney Spears. Given the current popularity of articles, pictures and videos of the superstar singer, the results for practically any query with the word "spears" in it will be loaded with matches about her - especially if the search involves television or entertainment in any way.

Let's say you're watching the movie Zulu and you start wondering what those large spears that all the extras are waving about, are made of. So, you go to Google and type in "movie spears material" - this is an obviously insufficient description, as the screen shot below shows.

What happens if you expand on the query further - say: "what are movie spears made out of?" - does it help? Here's a screen shot.

The general issue here is that articles about very popular subjects accumulate high levels of PageRank and then totally overwhelm long tail results. This makes it very difficult for a user to find information about unusual topics that happen to lie near these subjects.

2. Keyword Ordering

Since the major search engines focus only on the proximity of keywords without context, a user search that's similar to a popular concept gets swamped with those results, even if the order of keywords in the query has been reversed. For example, a tragic occurrence that's common in modern life is that of a bicycle getting hit by a car. Much less common is the possibility of a car getting hit by a bicycle, although it does happen. How would you search for the latter? Try typing "car hit by bicycle" into Google; here's a screen shot of what you get.  [Note the third result, which is actually relevant to this search!]

3. Keyword Relationships

Since the major search engines focus only on the keywords in the search phrase, all sense of the relationship between the search terms is lost. For example, users commonly change the meaning of search terms by using negations and prepositions; it is also fairly common to look for the less common members of a set.

This takes us into the realm of natural language processing (NLP). Without NLP, the nuances of these query modifications are totally invisible to the search algorithms.

For example, a query such as "Famous Science fiction writers other than Isaac Asimov" is doomed to failure. A screen shot of this search in Google is given below. Most of the returned results are about Isaac Asimov, even when the user is explicitly trying to exclude him from the list of authors found.

All of the searches shown above look like gimmicks - queries designed intentionally to mislead Google's search algorithms. And in a sense, they are; these specific queries can be easily fixed by tweaking the search engine. Nevertheless, these queries do point to a real need: the value of understanding the meaning behind both the query and the content indexed.

Semantic Search

That's where the concept of semantic search comes in. I attended a media event earlier this year at stealth search startup Powerset (see: Powerset is Not a Google-killer! ) which showcased a live demo of their search engine, currently in closed alpha, that highlighted solutions to exactly this type of issue.

For example, type "What was said about Jesus" into a major search engine, and you usually get a whole list of results that consist of the teachings of Jesus; this means that the search engine entirely missed the concepts of passive voice and "about". The Powerset results, on the other hand, were consistently on target (for the demo, anyway!).

In other words, when you look at just the keywords in the query, you don't really understand what the user is looking for; by looking at them within context, by taking into account the qualifiers, the prepositions, the negatives, and other such nuances, you can create a semantic graph of the query. The same case can be made for semantic parsing of the content indexed. Put the two together, as Powerset does, and you can get a much better feel for relevance of results.

What about Google? I'm sure the smart folks in Google's search-quality team are busily working on this problem as well. I look forward to the time when the major search engines handle long tail queries more accurately and make Search a better experience for all of us.

---

Enterprise 2.0: Battle of the IT Supply Chains

My dad pointed me to this fascinating article on IT Outsourcing from Information Week: The Second Decade Of Offshore Outsourcing: Where We're Headed by Mary Hayes Weier. It's a long article but well worth reading. The author captures a growing trend in the outsourcing of IT software: as offshore service providers move up the value chain, from tech support and quality assurance, to software design and development, business process re-engineering and eventually to providing direct business value, the relationship between these vendors and their Enterprise clients is changing. Instead of outsourcing specific, contained projects, companies are looking at IT development vendors as strategic partners - essentially, suppliers of IT business value. An example interaction, from the article:

A BT Wholesale executive team recently gave a presentation in Bangalore called "Joining The Dots," intended to help contract workers from Infosys, TCS, and Tech Mahindra understand BT's strategy and the role IT and process engineers play in it. The team explained BT Wholesale's market segments, key competitors, new products, and what it plans to launch over the next few years. "It's about spending time with people and explaining our objectives rather than just describing a piece of software they need to deliver by Friday," Selley says. "We want those employees to bring ideas back to us, too. It's been far too unidirectional between the U.S. and U.K. and suppliers in India."

The focus is no longer primarily on the cost savings obtained through offshoring, but rather on acquiring talent quickly, getting closer to customers - who are increasingly overseas as well - and creating global IT value chains that span people, processes & technology. Outsourcing companies (and in turn, their outsourcing vendors) are now valued partners in the software supply chain, and have a direct impact on the competitiveness of the enterprise.

This reminds me of the changes the automotive industry went through years ago.

Automotive Supply Chains

For example, authors Bill Jackson and Michael Pfitzmann compared different approaches to automotive supplier partnerships in their article Win-Win Sourcing in Strategy+Business magazine (via Evolving Excellence ):

This is one of many ways in which far-reaching manufacturers like Honda and Toyota rewrite the conventional rules of procurement.
...
With this approach, manufacturers and suppliers share a long-term commitment to improving each other’s capabilities, starting by working together to eliminate wasted effort and inefficiencies. The two sides, instead of being at odds, collaborate openly on lowering costs and raising overall performance, with the expectation that this mutuality will continue over many years, benefiting both companies.
...
Contrast this with the alternative ingrained in many companies’ purchasing departments: price-based sourcing. Essentially, this approach pits the interests of the supplier against those of the manufacturer. Each side reveals as little information as possible, for fear of giving the opposing side an edge.

Many others have made this point as well, such as the Economist's View blog: Why Toyota is Better than GM and Ford ; and MRO today: The supply chain is the difference maker .

This enlightened approach is already becoming an accepted view. Toyota sees its suppliers as partners; they, in turn, see their suppliers as partners; and so on. So the competition is not between the Toyota and General Motors; it is really a competition between Toyota's entire supply chain (Toyota, Tier 1 suppliers, Tier 2 suppliers, Tier 3 suppliers ...) and GM's supply chain (GM, GM Tier 1 suppliers, Tier 2 suppliers, Tier 3 suppliers, ...).

IT Supply Chains

Are there close parallels between these two approaches: BT Wholesale's interaction and shared strategy with its IT offshoring vendors - Infosys, TCS, and Tech Mahindra; and Honda and Toyota's closely cooperative relationship with their own vendors?

What are the implications of these changes for the IT outsourcing vendors, and for their Enterprises customers? If the parallel with the automotive industry holds, will we soon see the 1st tier IT software suppliers start to increasingly outsource pieces of their own projects, and so on, until there is a multi-tier supply chain in IT for software projects, business process optimization and quality enhancement? Projecting forward, how much further will offshore IT vendors go in providing value as a part of the business network of the larger enterprise?

---

Search Improvements 2008 - THAT'S IT?

A few days ago, Gord Hotchkiss, President and CEO of Enquiro , moderated a Webinar with the Search 2010 Panel; the panel is a who's who list of stellar participants in the Search space, including representatives from all the major search engines. You can find the actual Webinar and read Gord's post about it here: Search 2010 - A Review.

Gord writes:

I won’t steal the panelists thunder, but the first question I posed to them was what they see as the biggest change to search in the coming year. Most pointed to the continued emergence of blended search results on the page, as well as more advances in disambiguating intent. A few panelists looked at the promise of mobile, driven by advances in mobile technology such as multi touch displays, embodied in the iPhone.

He adds:

[One area]  ... is how search functionality will start showing up in more and more places. Already, we’re seeing search being a key component in many mash ups. The ability to put this functionality under the hood and have it power more and more functional interfaces, combined with other 2.0 and 3.0 capabilities, will drive the web forward.

Charles Knight of AltSearchEngines, in his reaction to the Webinar [ Thomas Jefferson Dines Alone ], writes:

So what did they see as the biggest change coming to Search in 2008?
...

Let’s break it down: 1) the continued emergence of blended search results 2) more advances in disambiguating intent, and 3) the promise of mobile…such as…the iPhone.

That’s it?  That’s what the key major search engine insiders and industry analysts predict for the roller coaster year ahead?  More of the same - and the iPhone?

Now, (disclosure) I'm an occasional contributor to ASE and Charles is a personal friend of mine, so I grant that I'm biased; but I'm with Charles on this one. That's it? Those are the key changes to search predicted by the major search engines for the next year? Is it just me, or do all of these changes seem - evolutionary, not revolutionary?

In a recent article on Future Directions in Search, I highlighted the major areas for potential advances in search: Query specification, Base Index, Relevance Algorithm, Results Visualization and Ongoing Interest (Notification). In that article, I was looking at a much longer time horizon, but I expect that some discontinuous changes will occur in one or more of those areas within the next year.

Search is a highly dynamic field that presently generates a tremendous amount of interest among scientists, engineers and entrepreneurs. (Google's stratospheric market cap has ensured that!). There are so many search startups coming up, many of which are introducing new concepts and technological innovation, such as - Vertical Search: indeed, Spock, and many, many others; semantic search: hakia, powerset; dynamic results visualization: quintura; ways to add value: trulia, zillow; ways to speed up search: vortexDNA; and so on. At least through acquisition, if nothing else, the mainstream search engines should be able to move ahead quickly.

As a specific example, let's look at the Video search space. I recently discovered Mark Robertson's web site, ReelSEO, which is dedicated to SEO/SEM of video content. On his site, Mark hosts the Comprehensive list of video search engines and video sharing sites, which lists over 100+ sites dedicated to video sharing and search. With so many players, surely there's someone who will introduce a new concept or significant change in video search?

Finally, let us acknowledge the elephant in the room. What about - discontinuous improvements to the heart of the Search Engine, the PageRank algorithm? After all, reduced to essentials, PageRank is only an approximation of the authority of a web page or site, based on the value and authority of incoming links. It was certainly an amazing insight on the part of Google's founders, and worthy of the success it attained; but just because all the major search engines use it today does not make it the right way or the only way to identify relevant results.

Perhaps there are other approximations which may work as well or better? Examples of alternative algorithms include: swarm intelligence (like Ant Colony Optimization), human algorithms (e.g. people-powered engines for popular searches and breaking news), brand authority (hey, we use it for everything else in life!), social graph, and many others.

Regardless of what Gord's A-list panel says, there's one thing I'm sure of: 2008 will be an exciting year for Search!

---