memory usage rambling

general = { about, articles, links, projects } meta = { date-posted: 2006-08-06 }

I run typo on a shared host, dreamhost in fact. Dreamhost lets me install software of my choice, giving me a compiler and shell access. So, I got rails running on it. It took me a while to get around a few intricacies with the dreamhost setup, but I managed to get typo running very smoothly eventually.

Well, rails seems to use quite a bit of memory. You have to 'pay' for all that magic somewhere. I find rails to have problems in a shared hosting environment. High memory usage makes it tough for some smaller webhosts to make hosting rails worth while. Many scalability problems can be solved by throwing more hardware at the problem. I have heard rails proponents tell me that you cannot scale development time, and that since rails cuts down development time, and that hardware scaling is mostly a 'solved problem', then it is a moot point. Just buy more hardware.

Hardware does scale, and programmers do not. The wisdom of 'The Mythical Man Month' showed us that many years ago. Yet a shared host needs to be able to fit more than a few apps on a box. It becomes an issue of financial scalability for them. Pocketbooks do not always scale well.

Now you say, it can't be that bad. I present the following output from running 'top'.

 VIRT  RES  SHR DATA S %CPU %MEM    TIME+  COMMAND
44696  43m 3448  43m S  0.0  1.1   0:08.71 ruby1.8
43160  42m 3448  42m S  0.0  1.1   0:12.98 ruby1.8
37592  36m 3452  36m S  0.0  0.9   0:03.95 ruby1.8
 7884 7876 2804 6980 S  0.0  0.2   0:00.95 trac.fcgi
 7884 7876 2804 6980 S  0.0  0.2   0:00.09 trac.fcgi

You can see above, that each fastcgi instance of rails is taking up an average of 40MB of system memory. That is quite alot. By comparison, the trac bugtracking fastcgi instances are currently taking up around 7MB each. This is only a loose comparison, however, because my typo site has considerably more traffic than does the trac instance, uses a different language, does different things, ad infinitum.

Each typo instance is using about 1% of the memory on a box with 4 gigs of ram. With fastcgi spawning about 3 per blog instance, that stacks up. My single typo blog above, comprising of 3 fastcgi listeners, is using over 100MB of system memory. Ouch.

Just to make sure nothing goofy was going on, I killed all the fcgi instances, and after surfing around my site for about 40 seconds, I had the same results. An average of 40MB per typo instance.

Now, I realize that typo imports quite a few gems, such as bluecloth, redcloth, rubypants, and a few other filters and macros. I would just chalk it up to typo being a bit bloated for a blog, but I know people that do rails hosting and development, and they report similar memory usage.

A few links to some random sites showing similar memory usage numbers:

What does this all mean? Likely not much for most of the people out there. For shared hosting providers, it means you should be using fewer fcgi spawns. Two is plenty for most people. Dreamhost has their fastcgi set to spawn up to 5 fcgi listeners. This is far too many.

Mostly just food for thought.