Google Bot: This is the bot used by Google to scour the web to identify new pages. It has two different versions. Deep bot and Fresh bot. As the name suggests, Deep bot tries to follow every link and downloads as many pages as possible. It tries to read minute details of a site including the internal structure.