Server Meltage

Nonpirate Notes

Ok, my server did not stand a snowball’s chance in hell in the onslaught last night. That’s kind of sad since I had prepared for so long — but no use crying over spilled milk.

I topped Slashdot, was under “Hot News” on Digg, the top news item on TorrentFreak and was quite retweeted. The server just grinded to a halt. You might as well have unplugged it. (Which, coincidentally, I had to do to let it cool down. That took about an hour.)

Currently, the server is hanging in there — but just barely. It is working at some 95% of memory capacity, and when it hits 100, a downward spiral of death begins immediately. As it can’t service requests, the present requests take longer to process, and so capacity goes down while the flood stays constant, and so the backlog increases, et cetera.

It hits the memory ceiling a couple times a day and when that happens I need to cool it down for a while. I have ordered more memory, but until it arrives (it’s the holidays…) I need to babysit the server.

Oh, and for those who wonder what “average load 150″ means: it means that the server would have needed, on average, 150 processors to handle the load over the past 15 minutes. The machine has four (!) processors. That conveys an image of just how hopelessly melted it was.

Ping @opassande.

You've read the whole article. Why not subscribe to the RSS flow using your favorite reader, or even have articles delivered by mail?

About The Author: Rick Falkvinge

Rick is the founder of the first Pirate Party and is a political evangelist, traveling around Europe and the world to talk and write about ideas of a sensible information policy. He has a tech entrepreneur background and loves whisky.

Book Falkvinge as speaker?

Liked This?

By participating in the discussion and posting here, you are placing your contribution in the public domain (CC0). If you are quoting somebody else, credit them.

Contributors take own responsibility for their comments.

Name (required)

January 3, 2011 - 03:09

Why not just let some webhotel host it?

RESPOND TO THIS

1.1

Henry Rouhivuori

January 3, 2011 - 17:09

Monthly quota….

RESPOND TO THIS

Hans J

January 3, 2011 - 18:32

I finally succeeded in adding the RSS-feed to my reader.

If one could be permitted to wish for something: The RSS-”backlog” is now set to 10 messages/postings as well as 10 comments. Please increase that to at least 25?

2.1

Rick Falkvinge

January 3, 2011 - 18:34

I’ll see if there’s a magic button somewhere for that.

RESPOND TO THIS
2.2

Rick Falkvinge

January 3, 2011 - 18:56

Fixed!

RESPOND TO THIS

Johan Lindh

January 4, 2011 - 14:12

If you’re not already using lighttpd, switch to it. A lot faster than apache, and uses a lot less resources. Works pretty well with WordPress too. I run it on http://www.linkdata.se/.

Magnus

January 4, 2011 - 17:42

Well of course. Just take a look at your page code and think for a while. Every user that visit a page on your site triggers quite a lot of processes and quite a lot of files to be served out. Much of this could be simplified without sacrificing any info or even one bit of the layout.

Just think about all that dynamic content, many of them causing database activity, that dynamic content stands probably alone for the bulk of the load. You have 11 linked scripts and 20 inline, where many probably is written on the fly by the server. Is it good and useful info? Yes. Does it have to be on the fly second fresh info? No. Can it be done in other ways reducing load? Yes. You could have the server update all that dynamic content to some template every second minute or so, even variable depending on current load, not once for every page served. You would get quite a huge gain only on that little stunt and visitors wont even notice it.

Etu

January 5, 2011 - 00:13

I see that the slashdot effect hitted you :)

Patric

January 5, 2011 - 00:37

This post might be of interest:

http://mocko.org.uk/b/2011/01/02/how-to-survive-a-slashdotting-on-a-small-apache-server/

Benny L

January 7, 2011 - 16:03

Well, as of course you already know, the “load” figure in a unix/linux system really doesn’t mean that you need more processors.

It means that at any given time (or, as in your case, as an average over the last fifteen minutes) there are X processes in the *run* *queue*. That means, there are X processes with something to do, that for one reason or another waits to get its opportunity to run some userland code.

And in your case, I’m willing to bet that your disk subsystem is saturated, either by database activity or, more likely, by swapping activity.

Anyway, having hundreds of processors instead of your four cores wouldn’t have helped you one bit. :-) You would have had a load of 1 but hundreds of processes in an i/o-wait state instead.

Hur som helst, lycka till med vad än framtiden har att bjuda. :-)