robots.txt?
Hey everybody!
I've been checking my visitor logs and error logs, and it seems google and other search engines have been crawling my site (YES!)... but my error log displays a "file does not exist" error for "robots.txt" which I never created, or have linked to in any way. What's this file do? Do I need one?
Suzanne posted this at 01:23 — 3rd October 2003.
She has: 5,507 posts
Joined: Feb 2000
http://www.robotstxt.org/wc/norobots.html
m3rajk posted this at 03:20 — 3rd October 2003.
They have: 461 posts
Joined: Jul 2003
kewl
kb posted this at 01:46 — 4th October 2003.
He has: 1,380 posts
Joined: Feb 2002
thanks
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.