The Hidden Dangers of Special Casing Google Search Results

One of the key tests of the maturity of any software development organization, is how it deals with corner cases. Any software application, no matter how thoroughly thought-out and well-designed, tends to exhibit corner cases: relatively uncommon use cases or scenarios (often in a specific sequence) that lead to unexpected results.

A Question of Approach

One of the lessons that has deeply impressed me, over many years as a software architect and designer, is that the approach used to handle these corner cases determines the long-term stability and reliability of any software application.

The right approach to handling the unexpected results for these uncommon use cases (essentially the "long tail" of usage scenarios) is to look at the general case to see how each one fits in; then change the design appropriately to handle the outliers correctly. This algorithm can be expressed as follows:

1. Evaluate the unexpected result for a specific uncommon scenario
2. Generalize the problem - how does this use case fit into the big picture?
3. Change some part of the overall design to handle the issue
4. Rinse and repeat

Of course, changing the overall design is easier said than done. Sometimes, there are just too many specialized use cases to cover with a general design; the standard architectural approach to handle this problem is to create a general facility into which specific custom solutions can be integrated in a standard way.

The one thing you always try to stay away from, at all costs, is to tweak the code for that one special case, without evaluation of the general problem or consideration of the design. Yes, special-casing can be done and it will even work, for a short time; but do it often enough and pretty soon you have a mishmash of special cases interacting in unpredictable ways, the dreaded "spaghetti-code" syndrome.

I imagine that algorithmic solutions to the problem of Internet Search are similar; since there are many millions of possible search terms in the long tail, you want to approach the findability problem as generally as possible. Sure, there will be corner cases where the results are less than optimal; when that happens, the best approach is to solve for the more general case and modify the searching algorithms to try to improve the outcome even for the corner cases. The most risky option (albeit the one with the most instant gratification) is to special-case the results for that particular query - once you start down that path, it's a very slippery slope indeed. Apart from the obvious issues of lack of scalability of this approach and questionable ROI for the time invested, the real cost is long-term: with enough tweaking, the system rapidly becomes unstable until you can no longer be sure how the system will react in any general way.

Case Study

In this context, a recent article in the New York Times about Google's responses to unexpected query results (you can read it here via the 37 signals blog ) provide an interesting case study. It appears that Google uses both approaches: special-case tweaking, and algorithmic modifications - and frequently! As the article says:

... the search-quality team makes about a half-dozen major and minor changes a week to the vast nest of mathematical formulas that power the search engine.

After pointing out that the search engine has "many thousands of interlocking equations", the author gives us two specific examples, listed below.

Example 1:

Recently, a search for “French Revolution” returned too many sites about the recent French presidential election campaign — in which candidates opined on various policy revolutions — rather than the ouster of King Louis XVI. A search-engine tweak gave more weight to pages with phrases like “French Revolution” rather than pages that simply had both words.

Example 2:

In 2005, Bill Brougher, a Google product manager, complained that typing the phrase “teak patio Palo Alto” didn’t return a local store called the Teak Patio.

So Mr. Singhal fired up one of Google’s prized and closely guarded internal programs, called Debug, which shows how its computers evaluate each query and each Web page. He discovered that Theteakpatio.com did not show up because Google’s formulas were not giving enough importance to links from other sites about Palo Alto.

It was also a clue to a bigger problem. (Emphasis added.) Finding local businesses is important to users, but Google often has to rely on only a handful of sites for clues about which businesses are best. Within two months of Mr. Brougher’s complaint, Mr. Singhal’s group had written a new mathematical formula to handle queries for hometown shops.

These examples are ideal for illustrating our earlier discussion. The first one clearly indicates special-casing: yes, the "French Revolution" search works now; the change fixed that particular query, but how did it improve the search engine algorithms as a whole? The second example, on the other hand, is a textbook implementation of the general approach: tackling and solving the more general problem represented by the misbehaving query.

It was a suprise to me, looking at Example 1, that Google allows this type of special-casing for use cases so specific; it may be only a temporary solution. It's common knowledge that Google focuses on hiring PhDs and "bright people" in general (even Marc Andreessen says so ); perhaps they have figured out a way to avoid or overcome the long-term issues that normally accompany a large number of these one-off fixes - I sure hope so.

Greg Linden posted an incisive article on this topic a few days ago, which is a great read.

Conclusion

For my part, I believe that the best approach to this thorny problem of fixing corner use cases for search, is to utilize the power of social input (aka the "wisdom of crowds") to automatically improve results over time. The Unified Architecture for Search proposed in my earlier post, included the idea of combining additional inputs - both semantic and social - with the standard PageRank algorithm to continuously improve search results over time.

References

Other folks have also commented on the NYT piece:

• Matt Cutts gives his analysis in Five Things you didn't know about Google's search
• Philipp Lenssen has a concise summary for the time-constrained
• Matthew Hurst of the Data Mining blog questions whether this is really Innovation
• Donna Bogatin looks at the impact on Search Marketers