Help with robots.txt!

divebuddha

New Member
Messages
56
Reaction score
0
Points
0
hi there everyone

after a few months of having the web site up and running i wanted to ask a few questions about

about whether the use of robots.txt within my web page enviroment will help with the broswers like google

to index my pages easier or will it be more hassle than its worth allowing all the user agents to visit my

pages?

ive tried to get all the meta txts in the corect places but again if anyone want to take a look and tell me

if im doing anything wrong and how to put it write i will only be too glad to take that info onboard?

also do you allow all bots

or do you only allow some?

and do yu dissallow the bots to visit certain parts of your site?

any help would be apreciated?


http://www.divebuddha.x10hosting.com
 
Last edited:

Zubair

Community Leader
Community Support
Messages
8,766
Reaction score
305
Points
83
Add your site to Google web master tools. Then confirm your domain by uploading file given by google.

Then go in crawler access option and generate your robot.txt there
 

divebuddha

New Member
Messages
56
Reaction score
0
Points
0
Add your site to Google web master tools. Then confirm your domain by uploading file given by google.

Then go in crawler access option and generate your robot.txt there

yes went through 6that process last night and was very supprised how easy it was to get the info i needed ive just got to create the robot.txt file now then i will see what it brings

the site has been indexed as its a strange name i.e dive buddha
 
Top