Ok, my server did not stand a snowball’s chance in hell in the onslaught last night. That’s kind of sad since I had prepared for so long — but no use crying over spilled milk.
I topped Slashdot, was under “Hot News” on Digg, the top news item on TorrentFreak and was quite retweeted. The server just grinded to a halt. You might as well have unplugged it. (Which, coincidentally, I had to do to let it cool down. That took about an hour.)
Currently, the server is hanging in there — but just barely. It is working at some 95% of memory capacity, and when it hits 100, a downward spiral of death begins immediately. As it can’t service requests, the present requests take longer to process, and so capacity goes down while the flood stays constant, and so the backlog increases, et cetera.
It hits the memory ceiling a couple times a day and when that happens I need to cool it down for a while. I have ordered more memory, but until it arrives (it’s the holidays…) I need to babysit the server.
Oh, and for those who wonder what “average load 150” means: it means that the server would have needed, on average, 150 processors to handle the load over the past 15 minutes. The machine has four (!) processors. That conveys an image of just how hopelessly melted it was.
Ping @opassande.
Why not just let some webhotel host it?
Monthly quota….
I finally succeeded in adding the RSS-feed to my reader.
If one could be permitted to wish for something: The RSS-“backlog” is now set to 10 messages/postings as well as 10 comments. Please increase that to at least 25?
I’ll see if there’s a magic button somewhere for that.
Fixed!
If you’re not already using lighttpd, switch to it. A lot faster than apache, and uses a lot less resources. Works pretty well with WordPress too. I run it on http://www.linkdata.se/.
Well of course. Just take a look at your page code and think for a while. Every user that visit a page on your site triggers quite a lot of processes and quite a lot of files to be served out. Much of this could be simplified without sacrificing any info or even one bit of the layout.
Just think about all that dynamic content, many of them causing database activity, that dynamic content stands probably alone for the bulk of the load. You have 11 linked scripts and 20 inline, where many probably is written on the fly by the server. Is it good and useful info? Yes. Does it have to be on the fly second fresh info? No. Can it be done in other ways reducing load? Yes. You could have the server update all that dynamic content to some template every second minute or so, even variable depending on current load, not once for every page served. You would get quite a huge gain only on that little stunt and visitors wont even notice it.
I see that the slashdot effect hitted you 🙂
This post might be of interest:
http://mocko.org.uk/b/2011/01/02/how-to-survive-a-slashdotting-on-a-small-apache-server/
Well, as of course you already know, the “load” figure in a unix/linux system really doesn’t mean that you need more processors.
It means that at any given time (or, as in your case, as an average over the last fifteen minutes) there are X processes in the *run* *queue*. That means, there are X processes with something to do, that for one reason or another waits to get its opportunity to run some userland code.
And in your case, I’m willing to bet that your disk subsystem is saturated, either by database activity or, more likely, by swapping activity.
Anyway, having hundreds of processors instead of your four cores wouldn’t have helped you one bit. 🙂 You would have had a load of 1 but hundreds of processes in an i/o-wait state instead.
Hur som helst, lycka till med vad än framtiden har att bjuda. 🙂
/B
Big on Slashdot, Digg… still not in Swedish parliament. Reality check.