What is causing our ecommerce server downtime?

We are running IIS 5.0 with Windows 2000 server on dual 2.4 GHz Xeon Processors with 1 GB of RAM and a 10 Mbit half duplex network connection on a dedicated box hosted with Verio Hosting.

Over the past year we upgraded to this machine because about once a month we would encounter down time and severe lag for a period of minutes and or hours on and off. Verio confirmed that traffic / bandwith was not the problem.

We were able to see in the task manager that our ecommerce application Softcart (from Mercantec) was eating up all the available RAM. The program runs on demand when a visitor uses the navigation on our site, starts a shopping cart or checks out. However, the application usually processes each request and then closes. It appeared during times of severe lag that the processes were not closing. Since we rewrote our URLS about two years ago which removed the application out of the URL, we now use Perl scripts to talk to the Softcart application. So, for each instance of the softcart application that we saw in the task manager, we also saw an instance of Perl. One of the theories that has some evidence to support it is that we have noticed search engine bots present during several of the lag periods. Since our site is dynamic and relies on Softcart to access the flat file database of more than 6,000 products, we thought it was possible that the bots were overwhelming our server. Whether it was Softcart, Perl or the processor we didnt know. So we upgraded to the server as described above and the problem has not gone away (though it is still possible that we need a faster server). We did have Verio test opening our connection to 100 Mbit full duplex, but the problem did not go away. We have since instituted a robots.txt file to keep the robots from visiting our site and for the past two days that has appeared to work. However, we obviously want to be a bot friendly site and Google has indexed most of our pages at this point so at least they have found a way to not take our site down.

We are currently trying to find some process management software that will allow us to track memory allocation and running time to all the processes on our server. If anyone has any software suggestions or suggestions of any other kind please let us know.