Wednesday, August 16, 2006

New personalized web search at Findory

Findory just launched a new alpha test version of its personalized web search. You can get to it by using the search box in the upper right corner of the screen and selecting "Web" from the drop down menu.

In the web search results, any item with a personalized icon after the title is recommended by Findory. Findory recommends and reorders web search results based on each person's previous searches, the search results on which that person has clicked, and what other web searchers have found interesting.

The obvious question to ask about this is whether it is effective. That is somewhat hard to answer given the data, traffic, and resources of a tiny startup, but I do have some evaluation numbers.

My initial analysis used a larger data set and attempted to determine how often personalized web search could narrow in on the result on which the searcher would click. In this test, 52% of the time, the item that would be clicked was in the candidate recommendation set.

This first test merely shows that there is potential to recommend the item that would be clicked. It does not say that the recommender would successfully identify that item over other items or that, in reordering the search results, that it would not improperly reduce the rank of the to be clicked item by promoting other, less important items.

The second, more complicated analysis tried to get to these questions. It fired off queries to Google, reordered the results using the recommendations, and then looked at the new ranking of the item which would be clicked.

The results from this were promising. 32% of the time, the clicked result was in the top 5 of the reordered search results, slightly higher than the norm for the unchanged Google search results. About 3% of the time, the reordering moved the clicked result to the top slot or one of the top 5 slots when it was not there already. Only 1% of the time did it move the clicked item from the top slot (not a good thing to do), and it never moved the clicked item out of the top 5 results.

Because major search engines have limits on being hammered with queries, I was only able to run this second analysis over a small randomized sample of 500 queries. A longer run would be desirable, as would being able to A/B test variants on the algorithm under heavy, live traffic. As a tiny startup, both are impossible, at least for now.

If you want to try out Findory personalized web search, you can get to it using the search box of at the top right corner of every Findory page. Select "Web" from the drop down, enter your query, and look for results with a personalized icon after the title. Click on a few search results if you like, then search again or for the same or a related term.

Here are direct links to a few searches -- "video games", "amazon", "windows live" -- to get you started.

Please take it easy on this. It is pretty cool, but it is just an alpha stage product by a tiny little startup. I hope you enjoy trying it, and please let me know if you have any thoughts or comments.

Update: If you find yourself getting kicked out to a Google search result page instead of seeing the results on Findory, that is because Google failed to return search results to Findory. It is rare, only occurring when the Google API coughs up a smurf. When this happens, rather than display an error page, Findory asks Google to serve up the results directly.

Update: If you want to try making Findory the default web search in Firefox for a little while, there is a Firefox plug-in available.

4 comments:

Greg Linden said...

Oops! Silly typo! I fixed it. Thanks, Enkrates!

Anonymous said...

Greg, we get it ... Findory is a TINY LITTLE STARTUP :-)

mb said...

Pretty cool concept, Greg.

One of the things I love about Findory News is the visibility into *why* an article was recommended. Do you plan something similar for web search?

I tried about 25 web searches on a single subjet and it seems that Findory recommends sites that I've already visited. It also seems that Findory recommends pages that I haven't visited, if I've visited other pages for that site.

I'm guessing that unlike news, which is a fairly small and predictable domain, web searches will be far more sparse -- it's far more likely that I'm searching for something that few others are. As a "tiny little startup", how will you get the critical mass to make personalized web search work as well as news search?

Greg Linden said...

Thanks, MB! Glad you like it!

Good point on the explanations. Telling people "why" a recommendation was made can often increase the perception of the quality and relevance of the recommendations.

I started to implement the "why" feature for Findory personalized web search but I ran into two issues that made me stop.

First, the explanations looked like they would not be compelling or easily understood by searchers. They seemed far too technical, of interest only to geeks. Higher level, simpler explanations differed little from what is up there now, a general message about how the recommendations are generated.

Second, the detailed explanation gives away quite a bit about how the personalization is being done at a level that makes be a bit uncomfortable, especially when the detailed explanations seemed like they would be confusing to the average user.

So, in the end, I opted for a simple, general explanation of how the recommendations are generated.

I may revisit this if I get a clever idea on how to clarify and improve the messaging so non-geeks could appreciate the explanations.

As for what Findory recommends, it picks items you have visited before and items that other searchers with similar search and clickthrough behavior have visited. It is not merely picking items from the same domain.

By the way, on the simple technique of picking items you have visited before, there was an interesting poster (PDF) on the value of doing that at SIGIR 2006. The authors included Jaime Teevan, who has done some interesting work on personalized search.

On your last point, I agree the web is sparse and many techniques for personalized web search benefit from a lot of data. For Findory's efforts, yes, more data would help enormously.