Or at least, that’s what my access.log said.
Using this command via SSH:
tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n
to get the last 10,000 hits on this blog, I got this result (click for a better view):
The Yahoo crawler accessed this blog 1130 times for the last 45 minutes that this blog is up!
This is the reason why this blog went down again. Even if I have a dedicated server, Yahoo bot pounding this blog 1130 times every 45 minutes will bring this down. This blog is not ready for that kind of traffic.
So, what did I do? I blocked the Yahoo bot now on my .htaccess file. Hopefully, this will fix everything. I even put a crawl delay on my robots.txt for Yahoo bot and it seems that it is ignoring it. I’ll just unblock it next week when its insanity subsides.
