robots.txt help

[phazzedout]

I need some help with robots.txt.

I know you could use it to stop the search engine robots from going through parts or all of your website with this file but I need something explained.

Ok let's say I have this:

website.x10hosting.com
/public_html/
robots.txt - do not go into /public_html/othersite/

othersite.x10hosting.com
/public_html/othersite/

Would the search engines look at othersite.x10hosting.com as a different website or will it see it as a root of website.x10hosting.com and not scan othersite.x10hosting.com.

If this did not make sense please tell me. I am interested.

 

[descalzo]

You are saying that you have your maindomain website.x10hosting.com
You added an add-on domain otherdomain.x10hosting.com which has as its document root public_html/otherdomain

Then, place in public_html

User-agent: *
Disallow: /otherdomain/

This should keep bots coming in via maindomain.x10 from descending into the directory

and in public_html/otherdomain place

User-agent: *
Disallow: /

This should keep bots entering via otherdomain.x10 out.

"Should" .... robots do not have to honor your robots.txt file.