Every time a search engine sends a spider to your website to crawl your pages, your website can detect the spider’s fingerprint, called a user agent. Here are the most common user agents for the major search spiders to enable them to crawl the site.
| Search Engine | User Agent String |
| Baidu China | Baiduspider+(+http://www.baidu.com/search/spider.htm)BaiDuSpider |
| Baidu Japan | Baiduspider+(+http://www.baidu.com/search/spider_jp.html) |
| Bing | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)msnbot/2.0b (+http://search.msn.com/msnbot.htm)Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm) |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Googlebot/2.1 (+http://www.googlebot.com/bot.html)Googlebot/2.1 (+http://www.google.com/bot.html) | |
| Google Mobile | Feature phone Googlebot-MobileSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Smartphone GooglebotMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Naver/Yeti | Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)Yeti/1.0 (+http://help.naver.com/robots/) |
| Yahoo | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
| Yahoo China | Yahoo! Slurp China Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
| Yandex | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
This site has the most comprehensive list of user agents.