I've just updated SproutSearch's blog spider. It now tries to fetch the RSS feed for a blog before parsing it with my HTML parser. This should save some CPU time and give me better results. It also gets the date and time of the blog's latest post and some statistics about the number of words used. This new data will allow me to generate better pages in the future. I am also brainstorming some methods of data mining I can use to make SproutSearch a bit more interesting. With over 8 million blogs in the database there are lots of possibilities.

http://www.sproutsearch.com