This is very interesting. Yahoo Research MindSet is a search UI that includes a slider that lets you indicate your “intent” by moving a slider between shopping on one end and research on the other. It then re-ranks the search results accordingly. Works pretty well. Try a search for shoes or search for wind surfing.

The whole effect is surprisingly transparent, probably because it’s a single axis and a fairly natural one for web users at that. I’m not sure a slider (vs. a couple of radio buttons to represent each extreme of the spectrum) is the right UI presentation, but the slider’s a fun toy for the research guys at Yahoo!, I’m sure.

(Via Geeking with Greg, who includes a characteristic dismissal of anything not invented at Findory. I’m with him on the sliders, although I’m not against tuning knobs of all kinds, period.)

I think search has to get smarter, and I think users will know what to do with a few judiciously chosen and well-implemented control knobs, at least until some kind of consensus on a new “ideal search engine” emerges, a decade or so from now (if ever).

My car has a setting for the wipers that moves the wipers only when a sensor on the windshield detects a certain amount of moisture. I use that sometimes, and it works pretty well, but it’s not perfect and there’s also a manual control that activates the wipers at fixed intervals, a feature which I don’t think is going away anytime soon. The knobs currently available (collectively known as “advanced search”, e.g. date ranges and all words vs. any of the words) in search engines are not intuitive and they certainly fail the “grandma test”. But a few knobs that more intuitively allow the user to guide the search will be well received, I think. Yahoo! is going in the right direction here.

But while the shopping vs. research axis of intent is useful (it cuts down results spam — for now — and folds in a Froogle-like tab into the main search results), it’s only the Flatland of intent, a modest beginning. The next trick would be to accomodate additional axes of intent (maybe whole hyperspaces of intent?), without giving up the transparency and intuitiveness of the UI.

Grokker has resurfaced as a front-end for Yahoo! search results. Uses a Java plugin (why not Shockwave?) and devotes a sizeable chunk of the page to Overture ads. For some search use cases, the results are marginally more useful than Vivisimo’s clusters, because instead of clusters being completely data-driven (as they appear to be on Vivisimo), Grokker seems to use a shallow taxonomy for presenting high-level categories. That makes at least the labels more recognizable, but the overall effect is still to obscure rather than illuminate the search space.

For example, search for campaign finance on Grokker, and you get the following top-level categories:

Besides one or two promising (but sparsely populated) categories, these results consist mostly of vague (and overlapping) subsets that force the user into pruning the search tree without any good basis for his choices. The underlying cause of the weakness of this kind presentation is that the clustering system is superimposing an organization on a presumed topic with only a keyword query to go on. The organization is derived from keyword search results, which are notoriously short on precision.

Because I dragged Vivisimo into this, here, for completeness, are the top-level categories from a search for campaign finance on Vivisimo :

I said above that Grokker’s presumed shallow taxonomy makes its labels more recognizable (avoiding categories like “Board” in the Vivisimo results), but overall, I’d have to say that the absence of groups like “Information” and “General” in the Vivisimo results is actually refreshing. Unfortunately, each of the clusters only has between a dozen and twenty results, so again, given that all of this organization is based simply on a two-word query, pruning all 22 million Yahoo! Search Results down to fewer than twenty seems less than useful unless the results are selected in a partitioned in a very thoughtful way (and they usually aren’t in web-wide search).

There’s a huge spectrum of information needs out there (writing term papers, researching a product, finding a fix to a computer issue), and I’m sure some of them are well-served by clustering approaches like Vivisimo’s and Grokker’s. But I find that for most of my search use cases, a well-ranked list of all matching pages still yields the most transparency and flexibility. When clustering of web-wide search results approaches what you’d expect from a human reference librarian, that’s when this approach will get interesting.

This article about Maidenform (women’s underwear company) filing for an IPO showed up in the Topix.net ‘Search Engines’ channel today. It’s since been pulled, but it was already out on the RSS feed.

Two morals in this:

1. Nobody’s perfect. Or, put another way: In classification, you can’t win ‘em all.

2. It’s nice to have an intern who can pull stories from a channel when someone important complains (like a publishing partner?)

TheServerside.com has an interview with Doug Cutting, Lucene/Nutch founder.