The other day when I was looking at my site's stats, I noticed that I had a lot more bandwidth used than normal. Looking at the hosts section, I saw that 8 IPs in the same subnet had downloaded about 1.3G each, for a total of over 10G. It didn't look like they were going to stop downloading, so I banned those addresses. I looked up the addresses, and they belonged to Hurricane Electric.

I looked in my raw log file, and found out that these computers from Hurricane Electric were using gigabot 3.0 to download all this data from my site. I had never heard of gigabot before, so I googled it, and found Gigablast. Gigablast is a search engine written by Matt Wells. It was written from the ground up in C++ to be fast and scaleable. It has indexed billions of pages with only 8 computers. I guess those are the 8 computers I found in my logs. Mystery solved.