I checked by 404 logs over at The New Leaf Journal. I found an interesting user agent among the myriad web crawlers:

PoorlyConfiguredWebCrawler

I appreciate that the crawler advertises that it is not well configured. You get the feeling that whoever released the poor thing on the internet is asking us to not be too hard on it. I did a bit of digging and learned it comes from a Swedish server, but information is otherwise scarce. According to my logs, it was only trying to crawl real articles (its only 404s were old links with no redirect) and did not seem to be misbehaving. Because it appears to be a well-behaved (albeit “poorly configured”) bot, I declined to stand athwart its crawling journey. I look forward to trying the Poorly Configured Search Engine it is clearly working toward.

Leave a Reply

Your email address will not be published. Required fields are marked *