A table of the most popular Robots (a.k.a. bots, user-agents) as it
should appear in the Robots.txt file. The table also includes its
description and a reference URL. Includes Google's web, image, mobile,
Adsense, Adwords crawlers, Yahoo and other popular search engines.
Useful if you want to disallowone type of bot e.g. the Adsense crawler
but allow the rest etc.
