robots.txt?

He has: 1,380 posts

Joined: Feb 2002

Hey everybody!

I've been checking my visitor logs and error logs, and it seems google and other search engines have been crawling my site (YES!)... but my error log displays a "file does not exist" error for "robots.txt" which I never created, or have linked to in any way. What's this file do? Do I need one?

They have: 461 posts

Joined: Jul 2003

kewl

He has: 1,380 posts

Joined: Feb 2002

thanks

Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.