Or at least, that’s what my access.log said.

Using this command via SSH:


tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n


to get the last 10,000 hits on this blog, I got this result (click for a better view):

The Yahoo crawler accessed this blog 1130 times for the last 45 minutes that this blog is up!

This is the reason why this blog went down again. Even if I have a dedicated server, Yahoo bot pounding this blog 1130 times every 45 minutes will bring this down. This blog is not ready for that kind of traffic.

So, what did I do? I blocked the Yahoo bot now on my .htaccess file. Hopefully, this will fix everything. I even put a crawl delay on my robots.txt for Yahoo bot and it seems that it is ignoring it. I’ll just unblock it next week when its insanity subsides.

This entry was posted on Wednesday, September 10th, 2008 at 11:08 pm and is filed under Internet. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.