搜索引擎爬虫蜘蛛的UserAgent收集

百度爬虫
    * Baiduspider+(+www.baidu/search/spider.htm”)

google爬虫
    * Mozilla/5.0 (compatible; Googlebot/2.1; +le/bot.html)
    * Googlebot/2.1 (+lebot/bot.html)
    * Googlebot/2.1 (+le/bot.html)

雅虎爬虫（分别是雅虎中国和美国总部的爬虫）
    *Mozilla/5.0 (compatible; Yahoo! Slurp China; misc.yahoo/help.html”)
    *Mozilla/5.0 (compatible; Yahoo! Slurp; help.yahoo/help/us/ysearch/slurp”)

新浪爱问爬虫
    *iaskspider/2.0(+iask/help/help_index.html”)
    *Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)

搜狗爬虫
    *Sogou web spider/3.0(+www.sogou/docs/help/webmasters.htm#07″)
    *Sogou Push Spider/3.0(+www.sogou/docs/help/webmasters.htm#07″)

网易爬虫
    *Mozilla/5.0 (compatible; YodaoBot/1.0; dao/help/webmaster/spider/”; )

MSN爬虫

    *msnbot/1.0 (+search.msn/msnbot.htm”)

GOOGLE
---------------------------------------------------------------------
66.249.70.212 - - [11/Jan/2009:00:03:35 -0700] "GET www.vidun/user-f2fc990265c712c49d51a18a32b39f0c.html?umid=f2fc990265c712c49d51a18a32b39f0c HTTP/1.1" 200 8148 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +le/bot.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Googlebot/2.1; +le/bot.html)"

66.249.70.212 - - [11/Jan/2009:03:27:23 -0700] "uxigao/images/pink/demo.gif HTTP/1.1" 200 2367 "-" "Googlebot-Image/1.0"
Referer: ""
UserAgent: "Googlebot-Image/1.0"

209.85.238.7 - - [11/Jan/2009:00:02:58 -0700] "uxigao/rss/c/1009 HTTP/1.1" 404 37 "-" "Feedfetcher-Google; (+le/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"
Referer: ""
UserAgent: "Feedfetcher-Google; (+le/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"


百度
---------------------------------------------------------------------
60.28.22.38 - - [11/Jan/2009:01:28:09 -0700] "GET www.vidun/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "www.vidun/" "Baiduspider+(+www.baidu/search/spider.htm)"
Referer: ""
UserAgent: "Baiduspider+(+www.baidu/search/spider.htm)"


YAHOO
---------------------------------------------------------------------
202.160.180.81 - - [11/Jan/2009:00:02:44 -0700] "GET www.vidun/ HTTP/1.0" 200 14250 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp China; misc.yahoo/help.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp China; misc.yahoo/help.html)"

67.195.37.167 - - [11/Jan/2009:00:23:00 -0700] "GET www.vidun/postmsg-tech-2-120.html?type=tech&id=2&tid=120 HTTP/1.0" 200 12609 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; help.yahoo/help/us/ysearch/slurp)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp; help.yahoo/help/us/ysearch/slurp)"


有道
---------------------------------------------------------------------
2008-03-04 09:54:12 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 61.135.219.7 Mozilla/5.0+(compatible;+YodaoBot/1.0;+dao/help/webmaster/spider/;+) - 200 0 0
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; dao/help/webmaster/spider/; )"


61.135.249.120 - - [11/Jan/2009:09:44:46 -0700] "GET HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; YodaoBot/1.0; dao/help/webmaster/spider/; )"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; dao/help/webmaster/spider/; )"


SOSO
---------------------------------------------------------------------
58.61.164.207 - - [11/Jan/2009:03:13:53 -0700] "GET www. HTTP/1.1" 404 - "www." "Sosospider+(+help.soso/webspider.htm)"
Referer: ""
UserAgent: "Sosospider+(+help.soso/webspider.htm)"

2008-03-04 10:48:28 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 124.115.4.218 Sosoimagespider+(+help.soso/soso-image-spider.htm) www.tatabao/ 200 0 0
Referer: ""
UserAgent: "Sosoimagespider+(+help.soso/soso-image-spider.htm)"


Sogou
---------------------------------------------------------------------
219.234.81.41 - - [11/Jan/2009:03:26:49 -0700] "GET www.vidun/ HTTP/1.0" 200 14250 "-" "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"
Referer: ""
UserAgent: "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"


220.181.61.217 - - [11/Jan/2009:13:10:57 -0700] "uxigao/play/3615?id=3615 HTTP/1.1" 302 5 "-" "Sogou web spider/4.0(+www.sogou/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou web spider/4.0(+www.sogou/docs/help/webmasters.htm#07)"

220.181.19.74 - - [11/Jan/2009:06:20:37 -0700] "GET www.vidun/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "-" "Sogou Orion spider/4.0(+www.sogou/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou Orion spider/4.0(+www.sogou/docs/help/webmasters.htm#07)"


220.181.19.78 - - [11/Jan/2009:10:55:18 -0700] "GET www. HTTP/1.1" 404 - "pic.sogou/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
Referer: "pic.sogou/"
UserAgent: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

219.234.81.27 - - [11/Jan/2009:23:53:41 -0700] "GET www.vidun/ HTTP/1.1" 200 14271 "-" "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"
Referer: ""
UserAgent: "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"

2008-03-04 17:23:59 W3SVC226223753 222.33.192.54 HEAD /index.php - 80 - 220.181.19.107 Sogou+head+spider/3.0(+www.sogou/docs/help/webmasters.htm#07) - 200 0 0