Link Blocking
I am looking for a way to block a link completely from being spidered, I have the folder in my robots.txt, and I know the noindex, nofollow doesnt work, and doesnt block links from being spidered.
Is there any other way besides not linking the link on the site anywhere, that I can get this link to not be spidered?
EDIT: Never Mind, I figured something out.
JeevesBond posted this at 01:13 — 19th June 2007.
He has: 3,956 posts
Joined: Jun 2002
Any chance you could tell us what you did in the end, or is it a secret?
If the search engine spiders are not honouring your robots.txt you could block access using .htaccess, using a username/password. Or you could block using the user agent string. My guess is you did one of those.
a Padded Cell our articles site!
jennbell posted this at 13:53 — 19th June 2007.
They have: 23 posts
Joined: May 2007
Well I ended up just for now, using the robots.txt I am trying to figure anything more out, I want that extra protection, but I dont want to have to use a username and password type thing. I am going to see for about a month what the robots/txt does, and then in the meantime look for anything else that might accompany that to work as well.
I wish it was something more elaborate than that though.
Professional Link Management System
koitech posted this at 10:42 — 1st July 2007.
They have: 24 posts
Joined: Jun 2007
this is the only way to stop spidering your links.
but there is another way. you can delete those files from your website.
and there will be no link and no search engine will add those links to their database
this is the best solution i guess
I recommend this reliable web host: MarbleHost.com
benj posted this at 16:32 — 1st July 2007.
They have: 36 posts
Joined: Apr 2007
For the links, maybe try a similar approach with the Email Riddler: http://www.dynamicdrive.com/emailriddler/
pr0gr4mm3r posted this at 19:43 — 1st July 2007.
He has: 1,502 posts
Joined: Sep 2006
I have my own way for dealing with naughty spiders. I have a directory that's only mentioned in my disallowed section of my .htaccess file. If the spider navigates to it, the IP is blocked.
marblehost posted this at 10:39 — 6th July 2007.
They have: 23 posts
Joined: Jul 2007
I don't want block the spider all.I want it go to some specific web page.
Can I use you method let the spider go where I want.
I highly recommend reliable web hosting
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.