Google has started to ignore my robots.txt

galaxyAbstractor

Community Advocate
Community Support
Messages
5,508
Reaction score
35
Points
48
why?

Code:
User-agent: *
Disallow: /bigjoe/

That's one part of my robots.txt that got indexed. I belive it is strongly against the rules that search engines index sites in the robots.txt.
Edit:
see:
namnls-46.png


Tillåten=allowed
 
Last edited:

DeadBattery

Community Support Team
Community Support
Messages
4,018
Reaction score
120
Points
0
Sometimes, it takes time to have Google process things.
That was probably from the last time GoogleBot went to your forum.
I suggest waiting a few days and then see what happens.
:)
 

galaxyAbstractor

Community Advocate
Community Support
Messages
5,508
Reaction score
35
Points
48
Sometimes, it takes time to have Google process things.
That was probably from the last time GoogleBot went to your forum.
I suggest waiting a few days and then see what happens.
:)


I made that robots.txt at january and google bot can access all. The Bigjoe thing was added 2 weeks ago and google dled it 3 hours ago.
 

tittat

Active Member
Messages
2,478
Reaction score
1
Points
38
How you save your file is it robots.txt or robot.txt ???
please double check that 's' .Many people often make this mistake.
 

galaxyAbstractor

Community Advocate
Community Support
Messages
5,508
Reaction score
35
Points
48
How you save your file is it robots.txt or robot.txt ???
please double check that 's' .Many people often make this mistake.

it is robots.txt but I noticed there is difference between /bigjoe/ and /bigjoe
 

tittat

Active Member
Messages
2,478
Reaction score
1
Points
38
But i think Disallow: /bigjoe/ is correct itself. Am i right?

where is your robots.txt located. Is it right inside your public_html folder itself?
Please give your site URL. Is it an Addon-Domain or Parked one?
 

ASPX.King

New Member
Messages
155
Reaction score
0
Points
0
you need to disallow /bigjoe/* and /bigjoe/sub_dir/* and all the other sub-folders
don't forget the *
to be on the safe side, just put in every single filename!
 
D

dWhite

Guest
http://www.robotstxt.org/robotstxt.html

About /robots.txt
In a nutshell

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.


So don't try to use /robots.txt to hide information.
 
Top