Expected server outage(s)
Megan's '21 ways to streamline your CSS' article is currently on Digg (no, we didn't submit it, so it's not our fault! ).
This has so far caused two server outages: the first because I had Apache's MaxClients set waaaay too low.
The second outage was due to a huge influx of visitors from Digg, causing the server to get load averages of ~100 and MySQL to get too many connections. Since the second outage, I've tweaked down Apache's MaxClients and increased MySQL's max_connections. The VPS is handling the current load nicely:
load average: 0.87, 3.16, 15.61
However, another large influx of visitors, will probably kill the VPS again. I'm monitoring closely, and will step-in and rescue the server if possible.
The forums may go down for a short period of time today. Blame Digg.
a Padded Cell our articles site!
JeevesBond posted this at 21:16 — 13th November 2008.
He has: 3,956 posts
Joined: Jun 2002
top - 21:13:12 up 1:43, 2 users, load average: 2.50, 2.24, 5.35Forgot to mention, I've also stopped courier, postfix, cyrus-sasl and any other services not essential to the OS or directly related to serving Web pages.
a Padded Cell our articles site!
decibel.places posted this at 21:53 — 13th November 2008.
He has: 1,494 posts
Joined: Jun 2008
Your success is killing you!
(Time to up your advertising rates?)
JeevesBond posted this at 22:18 — 13th November 2008.
He has: 3,956 posts
Joined: Jun 2002
I wish! Digg users all have adblock switched on. Still, we may get some decent backlinks to that article, and maybe a few more visitors.
Edit:
top - 22:19:05 up 2:49, 2 users, load average: 0.81, 0.61, 0.60Looks like we're through the worst of it.
a Padded Cell our articles site!
pr0gr4mm3r posted this at 22:56 — 13th November 2008.
He has: 1,502 posts
Joined: Sep 2006
Easy for you to say.
Why would MySQL be crashing by a load spike such as this. Don't you have some sort of caching enabled?
decibel.places posted this at 23:48 — 13th November 2008.
He has: 1,494 posts
Joined: Jun 2008
I met recently with developers from Northpoint Solutions in NYC who build Drupal sites capable of 1,000,000+ page views/minute.
They separate the dynamic and static content, and they use not only Drupal caching but also memcache and distributed servers.
The point is that Drupal is configurable for heavy traffic.
I thought Jeeves mentioned some kind of cache persistence problem a while back, maybe you left caching off?
JeevesBond posted this at 23:45 — 13th November 2008.
He has: 3,956 posts
Joined: Jun 2002
Well, it wasn't crashing, but it's max_connections were set quite low (80). The problem is that Drupal uses a few complicated queries, and lots of simple ones too. So we've got MySQL optimised to cache tables/queries, meaning it doesn't take many connections to gobble all the RAM on our server.
There was approximately a five-minute interval, as the story hit the front page of Digg, when everything went haywire. Otherwise the server seemed happy to handle 5-10 pages per second.
Yes, an article page only uses one SQL query. This probably saved the VPS from getting utterly destroyed.
a Padded Cell our articles site!
pr0gr4mm3r posted this at 23:58 — 13th November 2008.
He has: 1,502 posts
Joined: Sep 2006
Does Drupal have anything similar to wp-super-cache? This plugin for Wordpress uses generatic htm pages and .htaccess rules, so the PHP engine isn't even loaded when serving a cached page. It helps several Wordpress sites survive the Digg effect.
JeevesBond posted this at 02:25 — 14th November 2008.
He has: 3,956 posts
Joined: Jun 2002
Yes, it has Boost. We were using it for a while, but there were a few problems (not that I can remember what they are now of course).
We ended up using the APC Opcode cache, some MySQL caching (this tuning primer script is handy). However, I'm glad you mentioned that, I should check again whether Boost could at least work for aPaddedCell. The Drupal caching is good, but still couldn't handle the worst of the Digg effect.
a Padded Cell our articles site!
Abhishek Reddy posted this at 18:19 — 14th November 2008.
He has: 3,348 posts
Joined: Jul 2001
What's the status now? Are the caching and connection settings still up, and will you keep it that way or revert?
pr0gr4mm3r posted this at 21:17 — 14th November 2008.
He has: 1,502 posts
Joined: Sep 2006
I think any caching that still loads the PHP engine is not good enough to handle a huge traffic spike on a VPS. Something that can serve static htm files in place of even calling a PHP script is best.
JeevesBond posted this at 03:49 — 17th November 2008.
He has: 3,956 posts
Joined: Jun 2002
Am going to keep things as they are for now. Maybe tweak things back a bit and do some testing with siege.
True. Will have to look into Boost (as mentioned above) again, see if we can use it on just aPaddedCell.
a Padded Cell our articles site!
pr0gr4mm3r posted this at 05:12 — 17th November 2008.
He has: 1,502 posts
Joined: Sep 2006
Boost doesn't quite do that I'm talking about.
From one of their site pages:
The fact that they are calling queries at all shows that they are still using the PHP engine & a MySQL connection.
There is a better way to cache your site, and it's at the Apache level. Let's say that, using the current permalink structure on this site, you create a filesystem version as it is being browsed, so you end up with a crawled version of the site on your local file system, stored in /cache/. So by viewing this page, some PHP script would save a copy of the output to /cache/twf-feedback-and-announcements/expected-server-outages/index.html.
With a couple .htaccess entries, we can direct visitors to this cached page before PHP is even loaded for this next request.
RewriteCond %{DOCUMENT_ROOT}/cache/$1/index.html -fRewriteRule ^(.*) /cache/$1/index.html [L]
Now the static html page will be served. The PHP extension doesn't need to load, and no MySQL connection is needed to serve the cached page. So even if your database server crashed from visitors browsing the rest of the site, the initial Digg article landing page will remain intact.
To make sure we don't cache a page with someone's signin information, place this line above the two htaccess lines above:
RewriteCond %{HTTP_COOKIE} !^.*(signin).*$Replace "signin" with the name of the cookie you give authenticated users.
Apache is more than able to saturate several T1 lines serving html pages, and using a cache method like this should solve any server crashes due to traffic spikes.
On a footnote, there would still have to be a method of clearing out expired pages.
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.