Problem with bot-indexing of my .asp pages...
Hello,
I need advice of professional, who knows about indexing of ASP pages by MSNbot, Slurp etc.
My worry:
Today I replaced on FTP my html-site (static) to asp-site (dynamic). Soon I saw in logs strange behavoir of both bots.
MSNBot - 207.46.98.59 - - [09/Oct/2004:16:42:29 +0000] "GET /company_sell.asp?|3|800a01a8|Object_required:_'[undefined]' HTTP/1.0" 500 0 "-" "msnbot/0.3 ( http://search.msn.com/msnbot.htm)"
There is something about error 500 - but links and functionality of my new site has no problem, so what this error meens?...
Slurp bot:
1. 66.196.91.40 - - [09/Oct/2004:18:32:05 +0000] "GET /download/regform.txt HTTP/1.0" 200 26442 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
File 'regform.txt' was removed about month ago, but Slurp reads it as 200.
2. 66.196.90.84 - - [09/Oct/2004:18:32:29 +0000] "GET /download/legend.asp HTTP/1.0" 200 26387 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
Strangely to see data above, because in folder Download I never had such file (legend.asp). But when I enter such URL into address bar of browser, I see broken page 'legend.asp', when it doesn't exist at this folder at all....
Please, explain me, what's going on with my site?
Regards,
Spaniard
Busy posted this at 23:18 — 9th October 2004.
He has: 6,151 posts
Joined: May 2001
MSNbot is a rude bot, very unbehaved and will take more than just the kitchen sink if you don't watch them. I personally blocked msnbot several months ago for several reasons, 1/ it has nothing to do with msn search site, 2/ doesn't only take .html,.php files etc it takes .zip, .txt, .avb, .bgb, .tar as well as images etc 3/ is a bad bandwidth hog
Slurp is usually pretty well behaved but gets confused sometimes I think. It's not uncommon for it to look for outdated files. keep an eye on it, it could be looking for something.
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.