• Dreamhost Banner Ad

Yahoo Bot Is Pwning This Blog

Or at least, that’s what my access.log said.

Using this command via SSH:

tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n

to get the last 10,000 hits on this blog, I got this result (click for a better view):

Yahoo Bot Log

The Yahoo crawler accessed this blog 1130 times for the last 45 minutes that this blog is up!

This is the reason why this blog went down again. Even if I have a dedicated server, Yahoo bot pounding this blog 1130 times every 45 minutes will bring this down. This blog is not ready for that kind of traffic.

So, what did I do? I blocked the Yahoo bot now on my .htaccess file. Hopefully, this will fix everything. I even put a crawl delay on my robots.txt for Yahoo bot and it seems that it is ignoring it. I’ll just unblock it next week when its insanity subsides.

Marhgil Macuha

Marhgil Macuha is a Computer Engineering graduate of Batangas State University. He is currently a Senior Solutions Developer at a Canadian IT company.

Facebook Comments

Leave a Reply