What’s a Googlebot?

Think of Googlebot(sometimes referred as ‘Spider’) as a robotic visitor that likes to visit all sites that it can find. It visits a page and then recursively visits every other link that is not blocked by robots.txt or does not have a rel=”nofollow” meta tag.

Here is an always up to date article about the googlebot on Google webmasters support page