Popdex Treachery (Idle Words)#

12.20.2002

Anil and others have already pointed out that Popdex demonstrates the sincerest form of flattery, both in design and in name. But there's yet another reason to be annoyed by the site.

For the past few weeks, I've seen the same IP address come up in my log stats with great frequency, sometimes six or ten times a day. The address resolved to a DSL connection in St. Louis, and I figured it must be a loyal—if somewhat obsessive—reader. After all, any self-respecting bot would ask for a robots.txt file, or at the very least identify itself in the User Agent field. That's just common courtesy, after all—the guidelines have been around since 1993.

Well, it turns out Popdex spiders in stealth mode, never asking for a robots.txt file, and pretending to be a regular browser (although once in a while, it does slip and reveal its Perl nature. Even then, it's only because the programmer was too lazy to change the default user agent setting on the Perl LWP module). 65.69.49.203, I have unmasked you!