Simple robots.txt question
If I have "Disallow: /gallery/" in my robots.txt file, will that stop search engines (who listen to the rules) from indexing all the files and subfolders within that directory? Like /gallery/album5/ or do I need to make seperate disallows for every folder within /gallery/?
Thanks in advance.
Busy posted this at 21:10 — 23rd January 2005.
He has: 6,151 posts
Joined: May 2001
If the robots reads the robots txt file it wont go into or follow the path
I take it you have the complete thing:
User-agent: *
Disallow: /gallery/
depending on your servers stats you may also want to throw up an .htacces file to stop people viewing your folders (if you dont have a default file like index.html in there)
james.thornton posted this at 21:45 — 3rd March 2005.
They have: 42 posts
Joined: Jan 2005
For creating robots.txt files I use the free tool at:
http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm
Creates exactly what you want in your robots.txt file and has the useful capability of blocking 'nasty' robots (ones which don't index your site, just dowload your entire site to somebodies machine, wasting a large amount of your bandwidth)
James
bja888 (not verified) posted this at 22:20 — 3rd March 2005.
They have: 5,633 posts
Joined: Jan 1970
wow! thankx for the information!!
anything else I can do to block personal content?
Busy posted this at 09:08 — 4th March 2005.
He has: 6,151 posts
Joined: May 2001
Just remember robots.txt shouldn't be used as the only means of protection, bad bots are bad bots for a reason, because they ignore the robots file.
The robots file is only a guideline for bots, the .htaccess is a command which can't be ignored
Depends what you want/need bja888
bja888 (not verified) posted this at 10:34 — 4th March 2005.
They have: 5,633 posts
Joined: Jan 1970
Well... I password protected my blog site. oldbja888.com?Web-forums The mian page is a frame set. But you can get around the password by typeing in individual pages in the url. I can add ASP to all the pages and make sure they have thr framset open before they load but thats too much work. I just whan them to stop showing up when I search the net on google.
Busy posted this at 22:27 — 4th March 2005.
He has: 6,151 posts
Joined: May 2001
if you are using cookies from the form, include a file on each page to see if the cookie is set, same for sessions, javascript is no good because you can just disable it
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.