In actually no time (just 3 days after the site is on-air), search engine spiders arrived. A couple of observations:
The client string is set to Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp). The IPs used where in the following subnets: 66.196.90., 66.196.91. and 66.196.101.* Yahoo! robot(s) performed 68 requests today, and always looked for robots.txt.
Googlebot (identified by Googlebot/2.1 (+http://www.google.com/bot.html)) performed only 4 requests today and 2 yestarday, from the following IPs: 184.108.40.206, 220.127.116.11, 18.104.22.168. It will be interesting to see how much time will pass until I can see results when searching google...
vrypan — panayotis vryonis
My blog is a digital scrapbook of ideas, thoughts and personal events. My home is vrypan.net.