Economically, this market is finally starting to take shape — the ideas and attempts have been out there for a few years, but consumer companies have been on the fence about whether the blogosphere is worth listening in on. Until recently, that is. Umbria claims they’ll have $2M revenue this year and will be profitable next year, but the overall market for this kind of service is still only $20M according to the article (Intelliseek has about 1/3rd of that market).

Technologically, Umbria also sounds pretty interesting. They claim to have a competitive edge in automating most of the process:

Umbria’s solution is entirely software-based. [Umbria's] competitors also meet with clients to interpret the data and suggest strategic responses. “Ultimately we rely on both technology and humans for analysis,” says Max Kalehoff, marketing director for BuzzMetrics [another Umbria competitor]. “Umbria takes an extremely automated approach.”

Umbria’s technology sounds like a pipeline of parsers that generates features that in turn drive product and sentiment classifiers (and those drive reporting):

Every few hours Umbria sends an application called a spider out over the web to scour the blogosphere for postings about the firm’s clients, most of which are big consumer companies, such as Electronic Arts, SAP, and Sprint. By analyzing keywords in blogs, Umbria can classify each citation thematically. In the case of Sprint, for example, Umbria’s software can tell whether a blogger is talking about customer service, the company’s advertisements, or a particular calling plan.

Another big challenge is to decipher what’s on a blogger’s mind. To figure out whether an opinion is strong or tepid, for example, it helps to know that “awesome” is a stronger endorsement than “pretty cool,” and that “shoddy” is less damning than “abominable.” Umbria has several employees with Ph.D.s in linguistics and artificial intelligence who are forever tweaking the software to make it better at categorizing opinions.

I can’t help thinking that more manual tweaking goes into each client’s setup than this description lets on, but still, I’m glad they’re seeing success, and I bet those linguists are having fun with the blogosphere, even if they have to do a bit of slumming to come up with their rules:

The software can also estimate the author’s age and gender. Elongated spellings (”soooooooo”), multiple exclamation marks (!!!), and acronyms such as POS (”parent over shoulder”) suggest a teenage female member of Generation Y (born after 1979). The blogger is probably a teenage boy if a posting is rife with hip-hop terminology such as “aight” (translation: “all right”) and “true dat” (”I agree!”).

There you have it, you don’t even have to know the language to have your voice heard by the people who want to sell you more stuff. Now that’s power. On one side of that function, at least.

Yahoo plans to announce Thursday that it is recruiting scientists who pioneered an advanced search-engine technology at I.B.M.’s Silicon Valley research laboratory.
…
Prabhakar Raghavan, a computer scientist who once led the Clever effort, joined Yahoo last week as head of research. He left I.B.M. in 2000 to become a vice president and chief scientist at Verity Inc., a maker of search and retrieval software for corporations; he was later named chief technical officer.
…
Yahoo offers one of the best opportunities to explore new ideas in search, Mr. Raghavan said
…
One area that will be pursued is new search technologies related to digital media.

It’s been fun to watch Google being forced from the position of category killer to more-or-less evenly matched contestant over the last year or two. There’s a mind-boggling amount of innovation happening in search, which is levelling the playing field for new entrants, but even the stuff we’re seeing now is only the beginning. Search, and other modes of information retrieval, will become even more ubiquitous and integrated than they are now, and we’ll wonder how an OS like Windows without integrated search ever came to dominate a market. The desktop market itself may go away (yes, I’ve been reading Paul Graham’s book Hackers and Painters, which contains this great essay on server-based software from 2001, which is still relevant and engaging, as are his many other essays).

Search is poised to become the great collective memory, and new research being brought to market in real services, along with the availability of public APIs, will speed progress toward that reality. But it won’t be just the extent of information covered by search that will grow, but also interconnectivity of seach services and, most importantly, new modes of retrieving information (the only mode now in widespread use is keyword search, which is as old computer science itself — or much older, if you count manual versions such as file cabinets and card catalogs and other manually compiled indexes). I don’t see any reason why search shouldn’t aim to duplicate in software all of the modes in which humans retrieve information in their own brains (by context, by association and so on) or from others, by interactive question answering or guided discovery.

Greg Linden wonders whether the small blog/feed search engines will survive the entry of the giants into the field:

… it is good for a startup to see the entry of a big company into its area since it attracts attention and legitimizes the field … but competing directly against these giants is scary if you have no differentiator.

While the small players have driven innovation and broad acceptance of concepts like link popularity and tagging, they continue to struggle with scalability. Also, the most compelling products to come out of the blog search startups, while they’ve been exciting and even revolutionary from a user’s point of view, have not been technologically deep in the sense of difficult to duplicate by the search giants. There have been exceptions, of course, but no really deep technology is in evidence among those services that have made the biggest splashes (technorati, bloglines, flickr, del.icio.us).

So, when a search giant comes in with equal-or-better features, scalability, and a huge engineering team that can relatively quickly merge ideas emerging from the programming part of the blogosphere into the vast search toolkit that the giants already have, that might just cast a bit of a cloud over the little guys.

Having said that, I believe there will continue to be a place for the little guys in the blog search ecosystem. They’re the real innovators and they have their ears to the ground. And even at the break-neck speed at which Yahoo and Google have been rolling out features lately, an army of little guys can still cover a lot more ground than the two giants in the search for the next cool thing that will make users’ lives (even) better.

(That’s Greg’s image. Don’t have Google Earth running here, waiting for the OS X version. Impatiently.) Blogdigger seems to have found its niche with Blogdigger Local, and it’s a good one.

Danny, from his exchange with Tristan:

Also, Google did say “of about” with the numbers it reports. That’s not an accident. They’re saying that this is an estimate. But no disagreement with me. If you put up a count, it would be nice if the count was as accurate as possible. Google’s have come under question.

Hmm. From what I’ve seen in Tristan’s data and my own testing, it’s Yahoo’s counts that ought to come under question, specifically for link: queries.

Danny to Tristan again:

The link: command is completely different than the site: command. The link command tells you nothing about the size of the index. As for a confirmation that all links aren’t reported, this past blog post from SEW gives you confirmation and this page on Google mentions links are only a sampling of what Google knows although this other Google page fails to make this clear.

link: and site: are very different, that’s true enough. And maybe the link command doesn’t tell you much about the size of an index, but if link collection methods are similar between Yahoo and Google (and why wouldn’t they be, it’s a relatively easy part of the whole game), then the counts ought to be similar. But they’re not, not by a long shot.

By the way, a big thanks to Tristan for posting his studies and kicking off this discussion. Most of us don’t take the time to do analysis of that depth to support our opinions, and to post the entire method and dataset so others can reproduce it, shoot holes in it, go off on tangents from it.

(I stumbled onto Danny’s post via John Battelle)

• For the query link:boingboing.net, Google reports “about 40,700 results”, i.e. pages in its index that link to BoingBoing.net
• Technorati has indexed 23,358 links to BoingBoing.
• Yahoo’s results claim that it has indexed about 1,320,000 pages linking to BoingBoing.

Interestingly, Technorati is the only one of the three that gives the same count whether the link query includes www before the domain name or not (I happen to think that’s the correct behavior).

So much for Tristan’s method, which is transparent and easily reproducible. The picture that emerges is that Technorati’s coverage of the blogosphere is worse than Google’s, which in turn is [much] worse than Yahoo’s. By the way, Tristan’s post has more depth than is relevant to this post, and it has some interesting statistics that pull apart this data. Read it.

Anyway, Dave smelled something fishy in Tristan’s data (he’s onto the right question, but he goes after a red herring and misses a different, interesting feature in the data):

… I believe that Tristan’s analysis begs a question that hasn’t been asked yet: How accurate are the numbers that search engines report about the size of their result sets? … For example, when you search for all the results for “Tristan Louis” on Google, it reports “about 575,000″.

Whoa, hold it. That’s a keyword query, which means Dave’s now running a different experiment from Tristan’s, which uses link queries. I recommend you read Dave’s entire post, but from this point forward, he’s on a different track, using keyword queries instead of link queries throughout.

Dave’s objection is to the limit on “viewable results” that Yahoo and Google implement (Technorati doesn’t). Both Yahoo and Google only serve up to about 1000 pages of a results set. Crunching through 1-N results not only gets more expensive for higher N, but the value for the user falls off pretty rapidly after a while. And as a bonus, this limit keeps pranksters with robots from chewing up bandwidth by paging through millions of results, wreaking havoc on caching. Not to mention that nobody wants to wade through more than a few pages of results anyway, instead of just rephrasing the query to get better results. [Someone should do something about the excessive recall of these keyword search engines...] Anyway, the 1000-page limit is an interesting discovery, but it’s an obvious optimization. The results count given at the top of the page (results 1-10 of 40,700) is of course an estimate, again as an optimization (getting exact counts from massively distributed indexes isn’t free, and who needs an exact count at this level of recall, anyway?)

So, the limit on viewable results is very straightforwardly explained as an optimization, benign to the user experience. I don’t think there’s anything to get worked up about in only being able to see the first 1000 results for a query. That’s what query refinement is for. The interesting thing is the estimated total number of results, specifically of link queries.

When I went through Tristan’s original experiment and ran some link queries, it became pretty obvious (as if it wasn’t obvious in Tristan’s post) that there’s something weird about Yahoo’s method of estimating the total number of results. The practice of estimating the total number of results (as opposed to computing it precisely) is a necessary optimization in a search engine that wants to scale to Google or Yahoo scale, and the estimated results counts seem plausible on Google and Yahoo for keyword queries. Counts from both engines were plausible (and within an order of magnitude of each other) for most keyword queries that I tried. But for link queries, that’s not the case. Let’s look again at the estimated total results counts for pages linking to BoingBoing:

• Google, about 40,700
• Technorati, 23,358
• Yahoo! , about 1,320,000

Now, the fact that Technorati found only half as many links to BoingBoing as Google isn’t a big deal and shouldn’t give Technorati an inferiority complex. A sizeable chunk of the links may be from sites that Technorati isn’t indexing because those sites aren’t blogs or don’t use ping services that Technorati is monitoring. Also, Technorati’s index isn’t as old as Google’s and other factors like multiple links per page (to the same blog) make the comparison even more difficult. In any case, the difference between Google and Technorati is relatively small (if the Technorati team spends some time on the back end now that the new UI is up, they’ll narrow that gap). What’s interesting, however, is Yahoo’s estimate for the number of results for this particular query. At 1.3 million, it’s about 30x larger than Google’s count and 60x larger than Technorati’s. That seems implausible to me, and it looks like some wacky calculations are happening in Yahoo’s estimation of results count for this query. For several blogs I tried, Google’s results count is plausible and roughly 2-4x Technorati’s, whereas Yahoo’s is, well, out there. Here are small, medium and medium-large examples (we covered extra-large above, with BoingBoing) :

annikrubens.de:

• Google, about 52
• Technorati, 29
• Yahoo!, about 318

blog.jackvinson.com:

• Google, about 228
• Technorati, 47
• Yahoo!, about 2,760

battellemedia.com:

• Google, about 10,300
• Technorati, 1,723
• Yahoo!, about 647,000

I told you, it’s wacky. Tristan’s conclusion is that Yahoo! is more focused on indexing the blogosphere and has more data. That may be true. But these counts are so far out there that I can’t help but think there’s a problem with the way they’re calculated. So there. Fix it. I may check

And, to end on a nitpicky note … as I mentioned somewhere above, if you add or subtract www to the link: query, both Google’s and Yahoo’s total counts jump around like crazy. I don’t know about you, but in this context of searching WWW content, I think www should be treated as a special hostname and equivalent to the domain, i.e. www.domain.com == domain.com.

Search engines are so powerful. And they are so pathetically weak.

He goes on to lament how ill-suited today’s KWS engines are at answering questions. His use case is trying to find figures for California’s school spending in their historical context and/or relative to other states’ school spending, and he finds no satisfaction from “normal search tools”.

… When I finally called an education expert on Monday, she gave me the answer off the top of her head … after I’d wasted what seemed like hours over the weekend with normal search tools …

Fallows casts the problem in terms of automated question answering and cites several projects working in that area, one of them a federal intelligence project named ACQUAINT, and the others web-wide efforts ranging from shallow question-answering technologies like Ask.com (now augmented with search refinement tools) and meta-search clustering like Clusty.com.

But there is another way to cast the problem, an alternative metaphor to the web as giant library or file cabinet. The web’s billions of pages of content are uploaded into the giant file cabinet by people, including an astounding number of experts in a vast array of subjects. Even if the web as seen through the scratched lenses of a search index can’t find the answer, or even if the answer isn’t even out there, there’s an expert out there who knows the answer, as Mr. Fallows realized.

Bloggers make up a rapidly growing population of experts who contribute content to the giant library. There are now somewhere between 9 million and 12 million blogs, dealing with a mind-boggling diversity of subjects, and with real depth in many of those subjects. We (our industry and our government) should push full-speed ahead in research on automatic question answering (our security may depend on it, as Mr. Fallows points out), but we should remember that the experts are already there. We just don’t know høw to find them yet, unless we have “education experts” in our address books.

Blog search is heading in the direction of web search, using the file cabinet metaphor and adding on top of that the stream of web pages metaphor, which is how RSS is most often treated and indexed. The unique nature of blogs, e.g. the 1-to-1 relationship of author to content and the historical record of expertise amassed by each blogger over the course of many posts, should be mined to provide ways to find answerers, not just answers.

The presentation reflects some real qualities of the blogosphere: In aggregate, the blogosphere is noisy, diverse, urgent, in-your-face, gah! Technorati gets across the busy-ness of the blogosphere of the last few hours, where bloggers continuously decant their paragraphs and photographs into the teeming “world live web”, as Technorati used to call it. Is this the best way to do blog search? Should blog search be a megaphone or an earphone? Should it be an amplifier, a repeater, a filter, or a tuner? Some of each? Something else entirely? A purple frog?

This flash movie shows some screenshots of what’s purported to be the Google eval UI. More or less what you’d expect, but not as nice as some others I’ve seen …