Names of the most popular web Robots (bots, user-agents) to use in the robots.txt file

A table of the most popular Robots (a.k.a. bots, user-agents) as it should appear in the Robots.txt file. The table also includes its description and a reference URL. Includes Google's web, image, mobile, Adsense, Adwords crawlers, Yahoo and other popular search engines. Useful if you want to disallow one type of bot e.g. the Adsense crawler but allow the rest etc.

Names of the most popular web Robots (bots, user-agents) to use in the robots.txt file

Posted Fri, 03/20/2009 - 23:24 by deepak
Table of Contents
  1. Table of robot names

Table of robot names

 

Bot Name

(As it should appear in the robots.txt file for attribute 'User-agent:' )

 Owner Description Reference URL
Googlebot Google Web and news index bot (for google web search) http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
Googlebot-Mobile Google Mobile index (for google mobile search) http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
Googlebot-Image Google Image index (for google image search) http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
Mediapartners-Google Google Content for Adsense ads (only if the website is displaying adsense ads) http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
Adsbot-Google Google To measure Adwords landing page quality (only if advertising through Google Adwords) http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
Slurp Yahoo Web crawler bot (for Yahoo's search) http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html;_ylt=AtXtz6zwHzB6qCky4l3Kh8KygiN4
YahooSeeker/M1A1-R2D2 Yahoo Mobile crawler bot http://help.yahoo.com/l/us/yahoo/search/mobilecrawler/mobilecrawler-01.html
MSNBot Microsoft Web crawler (for Live search) - http://www.live.com http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
MSNBot-Media Microsoft Web crawler for images and all other media - http://images.live.com http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
MSNBot-NewsBlogs Microsoft Web crawler for news and blogs - http://search.live.com/news http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
MSNBot-Products Microsoft Web crawler for products and shopping- http://products.live.com http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
MSNBot-Academic Microsoft Web crawler for Academic Search - http://academic.live.com http://blogs.msdn.com/livesearch/archive/2006/11/29/search-robots-in-disguise.aspx
Teoma Ask Ask.com/Teoma's web crawler (for Ask.com search) - http://ask.com http://about.ask.com/en/docs/about/webmasters.shtml#5

Note: This is not an exhaustive search. This table will be updated with more popular user-agents frequently.

Comments

dsi r4

by Guest - 07/09/2009 - 08:25

Hi,

A robots.txt file can make a huge impact on your WordPress blogs traffic and search engine rank. This is an SEO optimized WordPress robots.txt file. Keep in mind that if you mess up the robots.txt file by blocking too much, you could lose all of your rank.

http://www.r4-ds-karte.de/

Social network marketing

by Guest - 06/18/2009 - 06:33

Hi,
Usual SEO advice is to contentrate on link backs from high pagerank sources.  If you want to improve your overall rankings I think this is good general advice because pageank is logarithmic, such that a link from a higher pagerank site will have more of an effect than one from a lower pagerank site than you might otherwise think.  
Social network marketing

SEO

by Guest - 06/09/2009 - 15:30

Good post.Great stuff in this article.Thanks!!!
SEO

Feedback or Questions?

We welcome feedback and questions and will try our best to attend to it as quickly as possible!

Please post it in our forums.

Please note that you would have to register before you can post in our forums and this is purely to guard us from the spam-bots. Be assured that we do not send spam mails and our website registration only takes minutes.