Subscribing to the LazyWeb search engine (Idle Words)

01.27.2003

Subscribing to the LazyWeb search engine

Richard Soderberg came up with a great idea today for the LazyWeb search engine - why not offer an RSS feed for each search, so that subscribers can monitor the results over time?

Why not indeed? It was a bit of genius that turned out to be easy to implement. Now every search query on LazyWeb gives you a little orange XML button that you can click on to get an RSS feed. Just paste the link into your aggregator, and when the search results change, you'll find out about it.

That's one reason vector-model search engines are good: you can actually do things like offer an RSS feed without making your server catch fire. Vector engines keep all their search data in memory, so there's no need to read every entry from a disk every time you run a search. You only have to retrieve the documents that are in the result set, and those can even be served from a different machine.

Being able to subscribe to individual queries is cool. Now what we need is the ability to combine many different queries, to different places, into one integrated list. I should be able to subscribe to the same search query on LazyWeb, and Mark Pilgrim's blog, and Bruce Schneier's security newsletter, and get the results back in a meaningful ranked list that updates whenever the results change. That's actually not hard to do on the search side, but there's no room in the RSS protocol for the kind of handshake I need. And of course aggregators don't support distributed search (yet).

If anyone who knows about designing protocols can spare the brainpower, I could use some instruction.

Idle Words

brevity is for the weak

Your Host

Maciej Cegłowski
maciej @ ceglowski.com

Threat

Please ask permission before reprinting full-text posts or I will crush you.