A week-and-a-half ago I promised further details on a resurfacing project I wanted to do after I finish replumbing.  Last Friday I alluded to this when I wrote about upcoming changes in search.  Today I’d like to outline how the new search interface and experience will work.

Search in Genesis is a tricky problem.  In most search applications the entire corpus is available and indexed.  For example, when you do a search on Google, the search engine doesn’t go and find web pages that are related to your search.  The search engine already knows all the web pages that are out there (or at least all the web pages it has found), so it can quickly gather the most relevant results (supposedly) and present them, in order, almost immediately.  By contrast, Genesis may have some data cached and indexed from a previous search which it can show immediately, but there’s almost certainly a lot more out there, possibly better, which will take some time to locate.

Peer-to-peer applications such as Kazaa, Guntella, and the original Napster, face a similar problem.  Their approach, and the one used by Genesis until now, is to throw all the search results into one, big, scrollable list.  This doesn’t work very well for two main reasons:

1. The list can get really long, really fast, making it difficult to find what you’re looking for.  You could sort it, but…
2. When the list is sorted to help find something, new results continually displace old ones, forcing you to scroll down to find what you were looking at.  This is especially annoying if it happens just as you’re double-clicking on a result.

Most web search engines solve the first problem by sorting the results according to relevance and paging them.  Not only does this help the user find the best results quickly, but it is also much more efficient than showing all the results at once.  I want to do something similar with Genesis, but unfortunately I don’t know what the most relevant results are.  Also, the list is sorted by relevance, so won’t results be continually displaced if I try to do this dynamically?

Well, I may not know what the most relevant results are, but I do know what the most relevant results are at any given point in time, which suggests a novel approach.  Without going into too much depth, it operates a little something like this:

The moment the user clicks the “Search” button, Genesis gets to work.  It start by telling all the search agents (PGVAgent, for example) to start searching their respective domains.  The agents may take a while before they return any meaningful results, so Genesis performs a quick search over the local cache to see if it can give any results immediately.  If it finds any, it ranks them by relevance and displays them on the first page.  If there are more than 10 or so, it onlys shows the top 10 and puts links at the bottom of the page to access subsequent pages.  So far, nothing new.

While the agents continue to search in the background, the user looks over the first page of results and might go on to the second or even third page.  Genesis keeps track of which pages have been visited and doesn’t mess with them; if the user ever goes back to a page that has already been seen, the page will remain the same.  If, however, the user goes to a subsequent page and better results have become available through the work of the search agents in the background, then that page will show the current top results (excluding those that have already been seen).  Also, the user can at any time refresh the search, returning to the first page of results but with all the current results correctly ordered by relevance.

I don’t know of any other system that does this, but I’d be interested to learn of any that exist.  I’d also be interested in learning about any other, possibly-better solutions.  Any ideas?

Filed in Genesis
Comment feed for this post