The Software Abstractions Blog

Zvents makes Local Search pop!

There is a class of web search engines that can prove even more useful than Google within a certain context. I'm talking, of course, about Vertical Search engines - the writer and tech strategist Sramana Mitra considers them Google's Achilles heel and Profy.com's Cyndy Aleo-Carreira seems to agree. This blog also has long held the position that vertical search represents a powerful mechanism to find information on the web, and is a key category to watch in the search wars of the future. [see: The rise of Vertical Search Engines from Aug 2006].

Another way of achieving a similar focus, in order to improve the relevance of search results, is by segmenting by location rather than by industry vertical - i.e. create a hyperlocal search engine that limits its search results to a given geographical area.

One such alternate search engine is Zvents, which is relentlessly focused on local information, of any sort. This company, which has been around since early 2005, has just introduced an advanced feature called Federated Local search - basically, its own version of Universal Search (recall that Google introduced its Universal Search feature with much fanfare last May).

Federated Local Search: Multi-Dimensional Results for Local Information

What does Universal Search mean, for a local search engine? Initially this was not very clear to me; an email discussion with Paul O'Brien, Director of Marketing at Zvents, inspired me to draw the following diagram:

The basic idea is to enable the user to implement a general-purpose search within a local context. This allows the user to find local information about a given topic, across many different dimensions. For example, a sports fan living in San Jose, CA who tries a local search for the term "hockey", would get the following different types of results:

• Upcoming games for the San Jose Sharks, the local hockey team
• The location of Roosevelt Park Roller Hockey Rink
• The description and link for a local "Hockey Night" event
• Results about relevant personalities (what Zvents calls "performers")
• And other related links ...

Zvents has already partially implemented this vision, although some of the lower-ranked results could provide a better match. Hopefully these will improve in the future as the search index grows and the algorithm improves. A screen shot of this local Hockey search in Zvents is given below.

Similarly, here's a search for the term "Web 2.0" for Cupertino, CA:

Outcome: Relevance

The big advantage of this type of search, over a general-purpose Google or Yahoo! search, is that the user can obtain the benefits of a broad cross-section of results, while still constraining the search to a limited geographical area.

This is not a significant issue in highly developed, urban, technologically advanced areas like Silicon Valley, Boston or New York; but it could one day make a big difference for someone living in David Letterman's "home office" of Wahoo, NE , or even more important, someone trying to find the Boston Public School located in Boston, Ontario - as we've seen before, highly popular keywords tend to swamp nearby long-tail keywords in the search results for major search engines.

From a business model perspective, hyperlocal searches tend to provide highly qualified prospects for local merchants, so I would guess that this type of search is very easily monetizable in the long run.

From a user interface point-of-view, the NLP-like implementation of time period for the search engine ("when: tonight, this weekend, ...") is a nice touch; I tried different possibilities ("next month"), and it seemed to work just fine.

On a more technical note, Zvents has been making waves with the release of its open-source Bigtable clone called Hypertable, which adds a C++ option for this project.

Going forward, it will be interesting to see how Zvents scales to additional locations, and to additional dimensions within each locality. Will it make inroads into the market share for any of the major search engines, or into that of other locally-focused web sites like topix.com and craigslist?

---

Quintura Launches Site Search Widget

Alternative search engine Quintura, which I've mentioned before on this blog, has launched its site search widget. This widget allows site publishers to provide users with a specialized search limited to that specific site; it joins earlier offerings from Google, Yahoo!, Rollyo and Eurekster swiki in this space.

This blog was an early user of this widget. You can see a customized, Quintura-generated mini-tag cloud in the earlier post; a full-size tag cloud is also available. The widget is hosted by Quintura, so installation was a snap: once the site was indexed, all I had to do was to embed the widget code into my blog pages and provide some styling control.

The biggest benefit of using the Quintura solution, as I've said before, is the dynamic tag cloud that allows the user to navigate the search space; initial feedback from our readers here has been positive, but not enthusiastic.

The real benefits to both users and publishers will come when Quintura search results prove to be better than equivalent results from a mainstream search engine solution, such as Google; as long as the Google site search results are good enough, it will be hard for the Quintura widget to make significant inroads into the market share of the big-G juggernaut.

This widget release is currently in private beta; an invite for this beta is available over on ReadWriteWeb.

---

Deconstructing real Google searches - why Powerset matters

I was looking at the log files for my blog today, as I regularly do, and I was suddenly struck by the variety of search queries in Google for which users were getting referred to my posts. I write often about the different flavors of search - including vertical search, parametric search, semantic search, and so on - so users with queries about Search often land here. But do they always find what they're looking for?

Some Real-life Search Results

Let us examine some of the actual Google queries - in the form of referring URLs - that led users to my blog. In most cases, Google did a fine job of matching the content to the query; in some cases, it was a somewhat random match at best; finally, in a few cases, the Google search algorithms are clearly getting confused. It is this third case that is the most interesting.

The Good

In many cases, the match was quite straightforward and very relevant. Some examples are given below.
1.

Query: http://www.google.fr/search?q=Guru+Avinash+Kaushik& ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:fr:official& client=firefox-a
Result:  A conversation with Avinash Kaushik, Web Analytics Guru

Well, can't argue with that!

2.

Query: http://www.google.cn/search?sourceid=navclient&aq=t& hl=zh-CN&ie=UTF-8&rlz=1T4XNLA_zh-CNCN246CN247& q=vertical+search
Result:  The rise of Vertical Search Engines (VSEs)

Query:    http://www.google.com/search?
q=wikipedia+to+try+and+compete+with+google& ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official& client=firefox-a
Result:  Wikipedia Search to compete with Google

Again, can't argue with those.

3.

Query:  http://www.google.com/search?hl=en& q=search+technology+exits
Result:  So You've Built an Alternative Search Engine - Now What?

This is actually pretty awesome, the algorithm has figured out "search technology" and "exits"; in fact, this post does talk about exit strategies for search engines, so it's a great match.

The Bad

Some search queries are so vague that the matches you get are bound to be somewhat random. I don't blame Google for the following matches:

4.

Query:  http://www.google.com/search?hl=en& q=conceptual+architecture
Result:  A Conceptual Architecture for Search

Is the search string too vague? Although technically this post matches the search query, I'm guessing that this is not what the user intended to look for.

5.

Query: http://www.google.com/search?hl=en&safe=off& client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial& hs=pGP&q=disruptive+technologies+blog& btnG=Search
Result:  Disruptive technologies for 2007

While the words match, and possibly this may satisfy the user, I get the sense that the user was looking for a blog dedicated to discussing disruptive technologies, not a single post. But who knows? Again, too vague!

In the future, I wonder how soon Search technology will progress to the point where the UI will automatically ask the user for more information to qualify search terms that are too general or vague. A little while ago, I envisioned a similar scenario ( Vertical Search, with authority ) when taking a look at the search engine MetaMojo, which has taken some steps in this direction.

The Ugly

In a few cases, though, the proximity of certain keywords fools the search algorithms. Consider the following matches:

6.

Query: http://www.google.com/search? q=best+search+engine+for+directions&ie=utf-8& oe=utf-8&aq=t&rls=org.mozilla:en-US:official& client=firefox-a
Result:  Future Directions in Search

A post about "future directions in Search" is not a post about "search engines for directions", although the text itself is undoubtedly a close match.

7.

Query:  http://www.google.com/search?hl=en& q=people+search+software+compared&btnG=Search
Result:  Search and the Dumbness of Crowds

Hmm? This is a popular post, but I'm not sure if it helps the user, who is not trying to compare search strategies (as this post does); instead, the user appears to be trying to compare people search engines.

Are these good matches? While the content of the posts bears a superficial resemblance to the text in the respective queries, the results are not relevant to the requested user searches.

The Larger Problem

The samples given above are not that important; the matches from my blog do not always show up at the top of the search results and although these are real referrals, not many users will actually click on these links in the Results page. But these examples point to a deeper underlying issue, one that will be far from easy to fix in the general sense.

All the major search engines currently rely on the proximity of keywords and search terms to match results. But that approach can be misleading, causing the search engine to systematically produce incorrect results under certain conditions.

To demonstrate, let us take a look at three general use cases.

[Note: The examples given below are all drawn from Google. To be fair, all the major search engines use similar algorithms, and all suffer from similar problems. For its part, Google handles billions of queries every day, usually very competently. As the reigning market leader, though, Google is the obvious target - it goes with the territory!]

1. Difficulty of Finding Long Tail Results

Take Britney Spears. Given the current popularity of articles, pictures and videos of the superstar singer, the results for practically any query with the word "spears" in it will be loaded with matches about her - especially if the search involves television or entertainment in any way.

Let's say you're watching the movie Zulu and you start wondering what those large spears that all the extras are waving about, are made of. So, you go to Google and type in "movie spears material" - this is an obviously insufficient description, as the screen shot below shows.

What happens if you expand on the query further - say: "what are movie spears made out of?" - does it help? Here's a screen shot.

The general issue here is that articles about very popular subjects accumulate high levels of PageRank and then totally overwhelm long tail results. This makes it very difficult for a user to find information about unusual topics that happen to lie near these subjects.

2. Keyword Ordering

Since the major search engines focus only on the proximity of keywords without context, a user search that's similar to a popular concept gets swamped with those results, even if the order of keywords in the query has been reversed. For example, a tragic occurrence that's common in modern life is that of a bicycle getting hit by a car. Much less common is the possibility of a car getting hit by a bicycle, although it does happen. How would you search for the latter? Try typing "car hit by bicycle" into Google; here's a screen shot of what you get.  [Note the third result, which is actually relevant to this search!]

3. Keyword Relationships

Since the major search engines focus only on the keywords in the search phrase, all sense of the relationship between the search terms is lost. For example, users commonly change the meaning of search terms by using negations and prepositions; it is also fairly common to look for the less common members of a set.

This takes us into the realm of natural language processing (NLP). Without NLP, the nuances of these query modifications are totally invisible to the search algorithms.

For example, a query such as "Famous Science fiction writers other than Isaac Asimov" is doomed to failure. A screen shot of this search in Google is given below. Most of the returned results are about Isaac Asimov, even when the user is explicitly trying to exclude him from the list of authors found.

All of the searches shown above look like gimmicks - queries designed intentionally to mislead Google's search algorithms. And in a sense, they are; these specific queries can be easily fixed by tweaking the search engine. Nevertheless, these queries do point to a real need: the value of understanding the meaning behind both the query and the content indexed.

Semantic Search

That's where the concept of semantic search comes in. I attended a media event earlier this year at stealth search startup Powerset (see: Powerset is Not a Google-killer! ) which showcased a live demo of their search engine, currently in closed alpha, that highlighted solutions to exactly this type of issue.

For example, type "What was said about Jesus" into a major search engine, and you usually get a whole list of results that consist of the teachings of Jesus; this means that the search engine entirely missed the concepts of passive voice and "about". The Powerset results, on the other hand, were consistently on target (for the demo, anyway!).

In other words, when you look at just the keywords in the query, you don't really understand what the user is looking for; by looking at them within context, by taking into account the qualifiers, the prepositions, the negatives, and other such nuances, you can create a semantic graph of the query. The same case can be made for semantic parsing of the content indexed. Put the two together, as Powerset does, and you can get a much better feel for relevance of results.

What about Google? I'm sure the smart folks in Google's search-quality team are busily working on this problem as well. I look forward to the time when the major search engines handle long tail queries more accurately and make Search a better experience for all of us.

---

Web Poll Results: What is the Most Important Component of a Search Engine?

Our last web poll had asked readers to vote on what they considered to be the most important component of a Search Engine - an indication of the areas a small search startup should focus on to help capture market share away from the major search engines.

27 readers voted (thank you!). The poll results are shown below.

These results are interesting because I had expected a higher percentage of votes for the Results Visualization choice, followed by the Algorithm choice, but the votes did not match my expectations. Part of the reason could be that readers of this blog are predisposed to have a stronger interest in the algorithms and strategies used by various search engines than in their UI paradigms.

As for the size of the Content Index, that's a metric that is slowly declining in importance. There was a time when the major search engines would fall all over themselves in trying to top each other in terms of the amount of data indexed; but as the content on the web explodes and grows progressively richer, it simply does not matter as much, and that is reflected by the votes.

As expected, the Query Spec choice got completely ignored. The search engine input spec could be so much richer than a minimal number of words or a single phrase. However, I fear that we're condemned to using keyword-ese for specifying our needs to the Search Engine for the long-term future, which is a pity; like the QWERTY keyboard, it may stick with us well after its usefulness has waned.

---

Search Improvements 2008 - THAT'S IT?

A few days ago, Gord Hotchkiss, President and CEO of Enquiro , moderated a Webinar with the Search 2010 Panel; the panel is a who's who list of stellar participants in the Search space, including representatives from all the major search engines. You can find the actual Webinar and read Gord's post about it here: Search 2010 - A Review.

Gord writes:

I won’t steal the panelists thunder, but the first question I posed to them was what they see as the biggest change to search in the coming year. Most pointed to the continued emergence of blended search results on the page, as well as more advances in disambiguating intent. A few panelists looked at the promise of mobile, driven by advances in mobile technology such as multi touch displays, embodied in the iPhone.

He adds:

[One area]  ... is how search functionality will start showing up in more and more places. Already, we’re seeing search being a key component in many mash ups. The ability to put this functionality under the hood and have it power more and more functional interfaces, combined with other 2.0 and 3.0 capabilities, will drive the web forward.

Charles Knight of AltSearchEngines, in his reaction to the Webinar [ Thomas Jefferson Dines Alone ], writes:

So what did they see as the biggest change coming to Search in 2008?
...

Let’s break it down: 1) the continued emergence of blended search results 2) more advances in disambiguating intent, and 3) the promise of mobile…such as…the iPhone.

That’s it?  That’s what the key major search engine insiders and industry analysts predict for the roller coaster year ahead?  More of the same - and the iPhone?

Now, (disclosure) I'm an occasional contributor to ASE and Charles is a personal friend of mine, so I grant that I'm biased; but I'm with Charles on this one. That's it? Those are the key changes to search predicted by the major search engines for the next year? Is it just me, or do all of these changes seem - evolutionary, not revolutionary?

In a recent article on Future Directions in Search, I highlighted the major areas for potential advances in search: Query specification, Base Index, Relevance Algorithm, Results Visualization and Ongoing Interest (Notification). In that article, I was looking at a much longer time horizon, but I expect that some discontinuous changes will occur in one or more of those areas within the next year.

Search is a highly dynamic field that presently generates a tremendous amount of interest among scientists, engineers and entrepreneurs. (Google's stratospheric market cap has ensured that!). There are so many search startups coming up, many of which are introducing new concepts and technological innovation, such as - Vertical Search: indeed, Spock, and many, many others; semantic search: hakia, powerset; dynamic results visualization: quintura; ways to add value: trulia, zillow; ways to speed up search: vortexDNA; and so on. At least through acquisition, if nothing else, the mainstream search engines should be able to move ahead quickly.

As a specific example, let's look at the Video search space. I recently discovered Mark Robertson's web site, ReelSEO, which is dedicated to SEO/SEM of video content. On his site, Mark hosts the Comprehensive list of video search engines and video sharing sites, which lists over 100+ sites dedicated to video sharing and search. With so many players, surely there's someone who will introduce a new concept or significant change in video search?

Finally, let us acknowledge the elephant in the room. What about - discontinuous improvements to the heart of the Search Engine, the PageRank algorithm? After all, reduced to essentials, PageRank is only an approximation of the authority of a web page or site, based on the value and authority of incoming links. It was certainly an amazing insight on the part of Google's founders, and worthy of the success it attained; but just because all the major search engines use it today does not make it the right way or the only way to identify relevant results.

Perhaps there are other approximations which may work as well or better? Examples of alternative algorithms include: swarm intelligence (like Ant Colony Optimization), human algorithms (e.g. people-powered engines for popular searches and breaking news), brand authority (hey, we use it for everything else in life!), social graph, and many others.

Regardless of what Gord's A-list panel says, there's one thing I'm sure of: 2008 will be an exciting year for Search!

---

Quintura: Alternative Search Engine of the Year

Over on the AltSearchEngines blog, Quintura has emerged as the winner of the The Alternative Search Engine of the Year - 2007 contest.

All the finalists are excellent search engines, so it must have been a tough call - as Charles readily admits. Regardless, I think it's a great choice, and a well-deserved honor for Quintura.

I've mentioned Quintura before on this blog ; the company offers some interesting and innovative technology. In addition to regular search results, one of their special features is a dynamic tag cloud, that allows the user to navigate visually through the search results space - as the user selects specific tags, the cloud automatically re-orients itself to highlight related concepts.

A mini tag cloud for this blog, powered by Quintura, is displayed below - you can play with it yourself to see how it works. Is it an intuitive way to surf through search results? Let us know in the comments.  A full-size Quintura cloud for this blog is also available on a separate page.

---

Future Directions in Search

[This article was originally posted as part of the Rising Star Dream Team Future of Search series on the VortexDNA blog. I'm deeply grateful to Kaila Colbin, VortexDNA's resident blogger, for the opportunity to participate!]

I've been writing about Search technologies for a while now, so when Kaila Colbin of VortexDNA offered me the chance to participate in a future-of-search marathon, I jumped at the chance!

What will the search for information look like in the future - in five years, ten, twenty? Is it just more of the same, or will it look radically different?

Looking Back

Before looking to the future, let us first look at how far we have come. Danny Sullivan has a great post looking at a decade of search history and the various tribulations of past and present search engines - AltaVista, Ask Jeeves, Microsoft, Yahoo! and of course, the early Google. We owe a huge debt of gratitude to the tremendous contributions of these and other early pioneers of Search; Google, in particular, deserves a great deal of the credit for making web search ubiquitous outside the tech community. Indeed, "to Google" as a verb has become virtually synonymous with the idea of Web Search, much as the Xerox brand became synonymous with the idea of the photocopier in a bygone era.

Google's venerable PageRank algorithm is certainly best-of-breed for the present, and Google keeps tweaking its results continuously. Given this progress in the Search area, can we still expect to see major improvements in search in the forseeable future? As an analogy, consider the DC-3 airplane - the first truly modern airliner, it was powerful, safe, reliable and economical (indeed, some of these are still flying today). It revolutionized air travel, and with its introduction, many considered the aviation age to have arrived for the general public. And yet, early jet aircraft had already appeared on the horizon, so to speak; within a decade, this reliable workhorse was obsolete, overtaken by jet aircraft in the competition for public air travel.

It could easily be the same with search. The key question that a search engine addresses is: what results do the maximum number of users find most useful for a given search query? PageRank is simply an approximation of the Wisdom of Crowds to answer this question. Is there a richer abstraction? Is Engagement the new black? Whatever the new approach is - in order to provide accurate results, it must work as implicitly as possible.

We have only to envision the possibilities ...

Looking Forward

So let us take a speculative look at search, circa 2015 . To look at it systematically, we can separate the search engine into the following components from a user perspective:   (To see this breakdown in visual format, check out my earlier post on an abstract architecture for search )

- Query specification
- Base Index
- Relevance Algorithm
- Results Visualization
- Ongoing Interest

Let us consider the possible future directions for each one in turn.

Future Directions

    Query Specification (Input)

Google pioneered the keyword-centric, minimalist approach for specifying the search query, and all the major search engines follow that lead. But the search criteria could be so much richer ; instead of experimenting with different types of keyword searches to find the information they need, users could simply provide additional criteria up front to qualify their request.

Admittedly, this approach does not work for everyone. The casual user would get reasonable defaults, which would automatically get updated with regular use to their favorite values; the topical researcher, on the other hand, would actively tinker with these widgets in a "power user" mode. (Google already supports this type of functionality in a limited fashion.)

Some possible advanced features for specifying the query, are given below:

1. Content Spec: Enabling the user to dynamically specify the data sources to be included, based on domain, reputation, social network, and so on
2. Scope:  Input for seamlessly limiting the scope of the search, to Enterprise or personal data
3. Qualifiers: Allow the user to add more information to disambiguate result matches, e.g. qualifying if "Java" means the programming language or the island
4. Parameter ranges: Domain-specific parameters can be extremely valuable even to a general-purpose engine (see #5 in the section on Relevance Algorithms below)
5. UI paradigms:  Text keywords are a limited form of input. The actual input mechanisms could be more visual, in the form of sliders, buttons, fields and other UI widgets. Imagine, for example, that as you move a slider, the search results change or an increasing number of results appear on the page!
6. Multiple Profiles: Personalization does not always have to be implicit. A user could explicitly set up profiles to represent different interests - professional, hobby, personal and so on, so that switching the profile would quickly change the areas of interest

    Base Index (Content)

This is a core area of concern for search engines: what is the scope of content to be considered when searching for information?

The standard approach currently is to build web crawlers that continuously scan as many web sites and web pages as possible; the scanned content is then used to build a master content index that is then updated regularly. This index is then used as the basis when searching for information.

For the base index, the big changes in the future are likely to involve both the scope and understanding of the content; here is a short list:

1. Rich media search, e.g. true indexing of audio and video content
2. Dynamic content search (searching the invisible web )
3. Integration of personal, web and corporate information
4. Perspective-based search, e.g. conservative vs liberal, hard news vs opinion, and so on
5. Subset creation, on-the-fly, e.g. to search for domain-specific data

    Relevance Algorithm (Mechanism)

This is, of course, the most-debated topic when discussing the future of search engines. Clearly, many different approaches and technologies show promise; some of these are noted below:

1. Personalization (but without storing personal info )
2. Social Input / Wisdom-of-Crowds (which has its pitfalls )
3. Social Graph: where your selected network of people help improve search results (Robert Scoble has recently gotten religion about this concept; Danny Sullivan rebuts )
4. Semantic Processing: of both, the query AND the content   (will this let the Search Engine find answers that we never knew we had?)
5. Parametric Search:  Vertical search engines already routinely offer domain-specific parametric search; for example, job search engines allow the user to specify the all-important location of the job as a primary criterion. Can this type of feature be generalized, so that as a user drills down deeper into search results, an increasing number of parameters can be offered?
6. Human-powered Search, for either the short head or the long tail of search
7. Swarm Intelligence: Mimicking biological search, such as Ant colony optimization, particle swarm optimization, and so on

    Results Visualization (Output)

Again, Google leads the way with its minimalist approach: simple headings, links and snippets of text. This is slowly changing, with the new "Universal Search" approach from Microsoft, Google and others; Ask.com is a leader in this area.

Search engines of the future will likely implement completely new paradigms for users to navigate and view search results. Often, meta-results - representing information about the results - are as important as the results themselves: users can figure out where a given result fits into the overall universe of results, and find the related results to an item of interest that has been found.

Some possibilities for results display are given below:

1. Tag clusters is not a new concept, but has yet to gain traction among the majors. Quintura, with its dynamic tag cloud display, has one of the best examples.
2. Organize results information by content type, is something every search engine will have to think about in the future. For example, should news stories be presented in an "overview capsule" fashion, or organized as a timeline-based view? Dale Dougherty at O'Reilly Radar has a brilliant article on this topic: Journalism is burning.
3. Follow-up actions - on viewing search engine results, a very common user action (as Greg Linden points out ) is to modify the current query, either to drill-down further or to try a different approach to find the required information. Google's "did you mean ..." feature is a step in this direction (although it leaves much to be desired ).
4. Domain-specific visualization can significantly enhance the understanding of results. This is similar to the data organization point above, but focused on the display itself; results from different vertical domains may require very different visualization techniques, such as colors, graphs, images, trend lines, heat maps, topographic charts, and so on. [For a list of the more exotic variations, check out this amazing list from Smashing Magazine.]
5. Dynamic scoping - enabling users to widen or narrow search results, based on different criteria - such as geography (local or global), site authority, timeliness, point of view, domain, and so on - is a powerful feature, that will continue to grow in importance.

    Ongoing Interest (Notification)

This can be best explained as a Reverse search, where it is the content that finds the user - thus turning the concept of search on its head.

Most of us have ongoing interests in certain areas; they could be professional, social or personal. It makes a great deal of sense for the search engine to keep track of these interests and pro-actively notify the user at some periodic interval of new items that fit those interests. Google Alerts is an early example in this direction. But enhancements to its functionality in the future could significantly boost its utility.

Some day, search engine notifications could support the following features:

1. Diverse Mediums: Many search engines already support email notifications. What's to stop them from adding support for many additional delivery mechanisms, such as IM, SMS, widgets, the twitter API, and so on?
2. Levels of Detail:  Allowing users to set the scope and organization of information presented.
3. Prioritization: This is a key feature! Once users are able to set priorities for different types of searches and for different areas, this can be used to drive the other features. For example, send me the headline about a breaking news event directly relevant to my blog, as an instant message, but email me a digest of the day's results for baseball scores.
4. Schedules: Some search results make sense only at certain times of the day; e.g. traffic search results are only relevant at commute times on work days.
5. Dynamic Control: Finally, empowering users to assert dynamic, granular control over their search alerts would make this functionality truly powerful. For example, once I've been notified about a breaking news story, I might want to artificially boost its priority and delivery method to continually get updates quickly and efficiently.

Power and Responsibility

As search engines start including a few or many of the features described above, search will grow increasingly more powerful. It will get easier to find any information we want, quickly and easily. Whether the information is high-level or detailed, global or local, general or specific, past or present, in any domain - no nugget of human knowledge shall escape this relentless spotlight.

Is shining a light on the darkest corners of the web always a good thing? As a webbed superhero once told me (and a few billion others) - "With great power comes great responsibility!". Privacy advocates are rightly concerned with the growing power of global web search engines; ongoing efforts from official and community channels are essential in minimizing abuse. A related issue is that web content can be archived and searched in perpetuity - the societal effects of this phenomenon have not yet begun to be understood.  A recent New York Times column highlighted this issue (paid content; here's a perspective on it from Slate magazine ).

Conclusion

Clearly, search engines will continue to evolve, and a future engine might well have many of the improvements described above, in the next ten years. But how about even further out - say, 2020 or 2030? Will disruptive changes in networking, computing and information technologies radically change the way search engines operate? A change in the nature of human thinking, interaction and social customs would be even more dramatic, and could cause a change in the nature of search itself.

This is, of course, a fertile area of speculation more in the realm of Science Fiction (for now): for example, will we one day need a galactic search engine? Can we create microscopic information-matching agents, either biological or atomic? Results that suddenly become available to the user as knowledge in the brain? An "implicit" search engine that finds information as we need it?  Why not?

---

What's More Important in Web Search: the Algorithm or the UI?

Over at Alt Search Engines, we've been having a debate about what it takes for a search engine startup to take on Google - a great Algorithm, a terrific new User Interface, or both?

My perspective is that it's the algorithm that is critical; Charles Knight (the editor of ASE), on the other hand, comes down heavily on the side of the UI, and makes a great argument for it in his post. Kaila Colbin, of the vortexDNA blog, has the last word - well, you have to read the posts for yourself before you make up your mind!

Links to the posts are given below:

• The Future of Web Search - Beauty or Brains?    by yours truly
• The Greatest Debate: Algorithms vs. Interfaces    by Charles Knight
• The Greatest Debate conclusion: Om Shanti Shanti    by Kaila Colbin

Which position do you agree with? Drop me an email or a comment and let me know!

---

Can the Semantic Web bring us Trusted Search Results?

Nova Spivack, during his recent talk about the Semantic Web (covered in my previous post ), made the point that addition of semantic processing to the underlying index for a search engine, make the issue of trust more serious. This was an intriguing statement, and I followed up with him via email to get further clarification; he was kind enough to respond at length. The questions and answers from our exchange are given below.

1. For Semantic web to really take off, the information on the web at large needs to become Semantic Web-compatible; i.e. web pages need to provide semantic information in the form of RDF, OWL etc. Do you see this happening in the forseeable future, given the huge mass of pages that already exist?
    Or is it more likely that technology will have to solve this problem for us, and we'll need to invent algorithms that can interpret currently existing web pages to extract and apply semantic knowledge on top (such as ClearForest Gnosis )?

The DBpedia.org is a good start. Also check out the emerging SPARQL and GRDDL standards at W3C -- they will bring existing data into the RDF world. There is also growing body of RDF already out there in the Dublin Core, FOAF and other ontologies on content on the Web. More will be coming from many big companies, Adobe, Oracle, Yahoo, etc. And of course other startups like Metaweb and Radar Networks will be adding a lot of content to the mix in different ways as well.

2. In your talk, you mentioned Trust - e.g. you referred to Powerset, wondering how they could add some sense of trust to the results they find through semantic processing (NLP) of web pages, because otherwise it would not be useful.
    I'm not sure I understand.

They are mining full text of the web and automatically building a knowledge base from that. So when they see a web page that says "Microsoft is a terrorist organization" or "Microsoft is a software manufacturer" how do they know which statement is true or false? Who do they trust? How do they determine who to trust? This determines what facts or assertions get what level of weight in their knowledge base. It's the crux of the issue really. You can mine in a lot of assertions, but if you have no good way to filter out the garbage, spam, erroneous statements, or deliberate deception, you can't use it for anything real. One solution is to only mine highly trusted sources -- such as encyclopedias and major newspapers for example. That's not a bad way to start. That would generate a decent knowledge base.

But the DBpedia.org might be a better way to start than mining free text. They've already done the heavy lifting of turning the wikipedia into RDF. I'm not sure you need natural language to get good, reasonably trustworthy knowledge, just use the DBpedia.

In the case of Powerset I believe their goals are different than the DBpedia/Wikipedia -- I think they don't just want almanac content, they want specialized vertical knowledge about travel, products, etc. That will require that they either are very selective of their data sources, or they have a sensitive way to measure trust and rank information accordingly.

        a] How does the addition of semantic processing to the underlying index make the issue of trust more serious? Google's PageRank is basically an approximation of the Wisdom of Crowds, using static links to represent votes; if Powerset uses some similar mechanism to rank the information sources in the underlying index, then their results should be no worse than Google's in terms of trust, and better than Google's in terms of relevance. [Incidentally, I've written about Powerset before when I attended their preview event.]

They could perhaps use a pagerank algorithm to attribute more trust to assertions they mine from various sites. That would be one solution. Unfortunately it gets more complex though -- because if their knowledge base has many grades of truth (statements ranked to varying degrees of trust) for a given assertion, then they will have to use modal logic or some other form of fuzzy reasoning to actually do any real reasoning or inferencing. That stuff is hard and uncharted territory to say the least. I don't think they will go there and if they did I think they would not be successful at it. So the question is what can they do without going there?

        b] How will the Semantic Web improve the trust situation for web search results?

First there is the issue of being able to assign a trust rank to each triple. Trust is relative so in fact there may not be a single global measure of trust that applies equally to everyone. I may trust someone that you don't trust and so I may take what they say to have more weight than you would for example. There isn't room within every triple to store that, but triples could be ratified by other triples that express "endorsement" of their content. So if I agree with something I can simply express that and now it is recorded that I (in an authenticated manner) have said I trust it. If lots of people do that with various assertions (triples), records (objects), and sites and people (sources), then we have a network of trust built in RDF. A network of trust can be reasoned on to determine weighted, socially relative trust rankings for triples. It can determine what triples I am likely to trust versus what you are likely to trust versus what everyone is likely to trust.

Second there is the issue of being able to trust reasoning performed by the system. For that the system needs to be able to explain to a human how it reached some logical conclusion and what data it used to do so. Work is being done both on computer-generated explanations and ways to record and show provenance to address these issues.

Many thanks to Nova for his detailed answers. You can find his blog here:  Minding the Planet .

---

Reactions: The Future of Alternative Search Engines

I recently wrote an article on this blog about the exit strategies for alternative search engines, that highlighted the recent and growing trend of publishers acquiring search engines; I also speculated about Charles Knight's quest to get these Alts to band together in order to grow overall traffic. I've gotten interesting reactions to the piece from some prominent bloggers.

 

I've been a fan of Ashkan Karbasfrooshan, of the HipMojo blog, for a long time. When I asked him about the Alts, his response was as follows:

No one has a crystal ball to predict with enough accuracy what will happen in the next few years with regards to search, but clearly, history suggests that there is always a period of consolidation after innovation and growth, so that should happen in search too.  And, we're seeing the big companies starting to have difficulty adding market share, it's not like adding one of these fringe search engines (including MetaMojo when I refer to alt engines as fringe) will add market share to a major one, but sometimes by buying a small player, a major company adds technology, know-how, but most importantly, brainpower.  That is the single most important variable for who will win or lose in search over the next decade: you can best deploy technology and market it, and not who can best develop technology.

One thing I was unsure of was whether traditional media would be doing the buying, I think ultimately traditional media will partner with search companies because it is not in the DNA to understand which search companies to acquire... which is a shame, because it's to old media that young search companies can most add value to...

 

David Berkowitz, of Inside The Marketers Studio stuck to his usual line where the Alts are concerned - he doesn't think most of them are good enough to get acquired or survive on their own:

Honestly, I don't think many of the alternative engines are good enough to be acquired, though a few will find that as an exit strategy, and a few others will survive as niche alternatives.

 

Microsoft's Don Dodge, who has a terrific blog The Next Big Thing, got so interested that he wrote a separate blog post to address this question. Here's a snippet from his post:

My thoughts? Some of these will be acquired by the big search engines or big content publishing networks. Most of them will fade away. I don't see any of them breaking out and creating a significant stand alone business with the possible exceptions of Powerset, Hakia, and Mahalo.

Read Don's post to find out where he thinks the big, untapped opportunities lie.

 

Bob Warfield, of the SmoothSpan blog, responded with an interesting blog post, proposing that:

Alt Search providers can get together and create an Open Sourced Collaborative Search-Oriented Social Network ...

In essence, he argues that the Alts could get together to share the costs and burdens of web crawling and the underlying infrastructure. This would help them to reduce the gap with the big players, which have a huge advantage in terms of resources. You can find his post here.

 

Conclusion

The overall consensus is that, unless something changes, most of the alternative search engines will not survive; some will get acquired and a few players may do well on their own, especially if they focus on a specific niche. A convergence in markets (e.g. local and mobile), or joint efforts on building the infrastructure, would give them a better chance.

Personally, I see the lack of web traffic as being the single biggest weakness of the alternative search engines; regardless of their cool technologies, innovative architectures or stunning visualizations, they cannot survive without getting the word out and capturing market share in search. There are simply too many web site destinations for the average user to remember. If the Alts could somehow cooperate to provide a single entry point that then branches off to different specializations, it would be a huge step forward!

---