windows live - search problem

[mikel2k3]

Wasnt sure what to put as the title, but anyay.

When viewing my website statistics with Awstats, i seem to be getting 5000+ unique hits from http://www.live.com . I dont know why this is happening, and will it make any difference if i create a robots.txt file that will dissallow everything?

Something like this:
User-agent: *
Disallow: /
Crawl-delay: 10

and has anybody else had this problem?

 

[Chris Z]

I believe that would just suggest windows live not to crawl your website, I don't think that it would actually force it to back off.

 

[Zenax]

The 5000+ hits from live.com are from bots that crawl websites, and obviously yours seems to be quite a popular one.

I believe the robots.txt would tell it to back off and not scan your site but I have no idea if that will stop it all together.

 

[Cubeform]

A similar thing has occurred in the past with Yahoo's very aggressive Slurp crawler. I've heard the crawler would often create miniature denial of service attacks on some small websites with shared hosting plans.

Windows Live should be robots.txt compliant. Try this:

User-agent: msnbot
Disallow: /
Crawl-delay: 10

This should stop Windows Live from indexing your page.

Edit: Here is a useful FAQ made by Microsoft:http://search.msn.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_FAQ_MSNBotIndexing.htm#D Try the robots.txt examples they give you there.

 

[mikel2k3]

The part that really confuses me is:

Robots/Spiders visitors:
Yahoo Slurp - 4+3 hits
Googlebot - 5+1 hits
MSNbot - 4+1 hits

Coonect to site from - Links from internet search engine:
Windows Live - 5618
Google - 2

Ill presume that they are not related in anyway, but ill also put the robots.txt file blocking MSNbot anyway to see if it helps.

 

[t2t2t]

Coonect to site from - Links from internet search engine:
Windows Live - 5618
Google - 2

Isn't that number the amount of people that come through Windows Live Search to your site?

 

[mikel2k3]

exactly what I was thinking. But surely im not that popular, lol.

And if i was so popular, people wouldnt JUST be using windows live search engine