Tyler
Retired Staff
- Messages
- 8,564
- Reaction score
- 0
- Points
- 0
For more and more websites you need to register or pay to have full access. The odd thing is that Google has the complete and full index of the website. So what’s going on here? Why must regular users pay or register to have access when the google search engine bot has full access?. The reason is simple; every site wants to use the benefits of the wonderful world of Google, for webmasters free advertising is always welcome. But there is a simple way to be the Google (search)Bot. In this little article i will try to explain it.
For the full article go Here
EXAMPLE:
This actually does work (well on some sites)
go to http://www.windowsitpro.com/WindowsScripting/Article/ArticleID/46794/46794.html and you will not see the FULL article.
Then change the agent info to Googlebot/2.1, go back to that same article and you can see the whole thing!
If anyone finds a good site that this works with let us know.
Last edited: