block all robots
Is there a way to block all robots from visiting a site?
I am working on a new project, and dont want any search engine listings taken yet, as I am changing and updating live (i cannot work on localhost for this as I have to allow viewing access for a few other users)
Or is there another trick to work on a site live (on a live server) and avoid robots?
I have the full site (domain) password protected from the server control panel.
cheers
kazimmerman posted this at 19:34 — 22nd July 2007.
He has: 698 posts
Joined: Jul 2005
This is very simple. Create a file in your root directory named robots.txt. Then put the following lines in the file:
If you simply want to restrict one page, you can either use the above code and specify an exact page instead of a directory, or you can place the following in the actual file:
<META NAME="*" CONTENT="noindex" />Kurtis
greg posted this at 17:28 — 23rd July 2007.
He has: 1,581 posts
Joined: Nov 2005
I thought some robots ignored the robots.txt file?
I was hoping to use .htaccess.
I have the server's public.html folder password protected (which is of course where all the website files are)
will this keep out all robots?
Busy posted this at 21:36 — 23rd July 2007.
He has: 6,151 posts
Joined: May 2001
All bots ignore the robots file at least once, sadly there is no quick and easy .htaccess method (you have to name all the bots).
About the only real thing you can do is use the robots file and password protect the entire site (all pages you'd have to be logged in for)
greg posted this at 22:56 — 23rd July 2007.
He has: 1,581 posts
Joined: Nov 2005
ok thanks
I have the full site password protected for now while its in testing stage
after that I will just block out the bad bots
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.