Sunday, January 21, 2007
I ran across a post by Greg Linden discussing a couple research papers on search. Especially interesting is the Bender paper on peer to peer search. Basically the paper discusses a strategy on how to create "specialized" peers with topic based crawls of the web. While I think this is problematic in a true peer to peer system, due to the unpredictable quality of peers (or predictably bad depending on your level of cynicism) and their limited bandwidth among other things, it did make me think of a related idea. What if some of the myriad "vertical search" providers proliferating on the web could follow a standard for communicating the contents of their verticals along the lines of thinking in the Bender paper. Its not peer to peer, but to some extent it addresses the monopolistic concerns of the big 4, and it does address the dependability of a search node problem. In other words, create a meta search infrastructure, with built in economic accommodations of some sort, that vertical search providers could plug into; unlike the meta crawlers of yore that focus on meta crawling general search engines (e.g. metacrawler, dogpile, ...), they metacrawl niche vertical engines constrained by topic. If done right, this might lower the barrier of entry for small players to make an impact in broader search.