Benchmarking mod_rails against mongrel
I’d like to use mod_rails instead of the apache->haproxy->mongrel configuration, but before I do I wanted to make sure I don’t lose to much speed, so I decided to benchmark mod_rails against mongrel. mod_rails was already benchmarked, but I don’t believe it if I don’t see it :)
To benchmark it I created a new rails application with a single controller and a hello.html.erb view with only ‘Hello World’ in it.
For the test I took a very low end machine, because the virtual server I rent for the production site is not much better anyway:
s2@fresh:~$ cat /proc/cpuinfo processor : 0 vendor_id : CentaurHauls cpu family : 6 model : 9 model name : VIA Nehemiah stepping : 8 cpu MHz : 532.000 cache size : 64 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr cx8 sep mtrr pge cmov pat mmx fxsr sse rng rng_en ace ace_en bogomips : 1067.72 clflush size : 32
with 483608 bytes of RAM on Ubuntu 7.10 and ruby 1.8.6.
The mongrel setup consists of 3 mongrels running in production mode behind haproxy balancing the requests coming from apache.
This is the apache config:
<VirtualHost *:80> ServerName benchmark.fresh ProxyRequests Off ProxyPass / http://127.0.0.1:8010/ ProxyPassReverse / http://127.0.0.1:8010/ </VirtualHost>
I keep things simple because I want to test the speed of the app running with mongrel against mod_rails. I don’t care about static stuff, url rewriting and other things in this test.
haproxy is configured like this:
... listen rails :8010 server rails-1 localhost:8011 maxconn 1 server rails-2 localhost:8012 maxconn 1 server rails-3 localhost:8013 maxconn 1
I left the sessions active, and the configured database is a postrges connection. As this will be the same in the two setups, I hope this will not make any difference.
Now I get the mongrels running and start the benchmark (after a dry run of 1000 requests):
$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking benchmark.fresh (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software: Mongrel
Server Hostname: benchmark.fresh
Server Port: 80
Document Path: /hello/hello
Document Length: 12 bytes
Concurrency Level: 100
Time taken for tests: 25.184929 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 480000 bytes
HTML transferred: 12000 bytes
Requests per second: 39.71 [#/sec] (mean)
Time per request: 2518.493 [ms] (mean)
Time per request: 25.185 [ms] (mean, across all concurrent requests)
Transfer rate: 18.58 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 107 375.1 0 1874
Processing: 47 2272 771.9 2414 11236
Waiting: 5 2264 740.8 2413 11236
Total: 483 2380 711.4 2415 13110
Percentage of the requests served within a certain time (ms)
50% 2415
66% 2456
75% 2484
80% 2562
90% 2793
95% 2863
98% 3291
99% 4396
100% 13110 (longest request)
Now it’s time for mod_rails. I stopped the mongrels and haproxy to free up some precious RAM.
Apache config:
<VirtualHost *:80> ServerName benchmark.fresh DocumentRoot /home/s2/tmp/benchmark/public/ </VirtualHost> LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/ext/apache2/mod_passenger.so RailsSpawnServer /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/bin/passenger-spawn-server RailsRuby /usr/bin/ruby1.8 RailsEnv production
and after a dry run of 1000 requests:
$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking benchmark.fresh (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software: Apache/2.2.4
Server Hostname: benchmark.fresh
Server Port: 80
Document Path: /hello/hello
Document Length: 12 bytes
Concurrency Level: 100
Time taken for tests: 22.329544 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 514001 bytes
HTML transferred: 12000 bytes
Requests per second: 44.78 [#/sec] (mean)
Time per request: 2232.954 [ms] (mean)
Time per request: 22.330 [ms] (mean, across all concurrent requests)
Transfer rate: 22.44 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 4.9 0 41
Processing: 48 2069 2922.9 1445 22101
Waiting: 45 2064 2922.5 1443 22101
Total: 61 2071 2922.8 1445 22114
Percentage of the requests served within a certain time (ms)
50% 1445
66% 1589
75% 1735
80% 1808
90% 2339
95% 3249
98% 17180
99% 19692
100% 22114 (longest request)
A pretty graph (because every serious benchmark has at least one):
mod_rails can handle 44,78 req/sec, and mongrel 39,71. I think I can ditch the complicated mongrel setup after I make sure mod_rails is as stable as the mongrels are.
Update: Dan and Sean pointed out in the comments that a
RailsMaxPoolSize 3
for mod_rails would be more appropriate. Here you go:
$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
...
Concurrency Level: 100
Time taken for tests: 20.94626 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 514000 bytes
HTML transferred: 12000 bytes
Requests per second: 49.76 [#/sec] (mean)
Time per request: 2009.463 [ms] (mean)
Time per request: 20.095 [ms] (mean, across all concurrent requests)
Transfer rate: 24.93 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 8 28.1 0 113
Processing: 131 1902 1518.9 1240 6861
Waiting: 127 1900 1518.9 1238 6860
Total: 149 1910 1515.4 1253 6861
Percentage of the requests served within a certain time (ms)
50% 1253
66% 2038
75% 3334
80% 3656
90% 4322
95% 4644
98% 5431
99% 5696
100% 6861 (longest request)
49.76. It’s actually quicker?! I could not believe this, so I repeated the test twice, with and without RailsMaxPoolSize 3.
With: 46.92, 46.51
Without: 10.22, 22.32
Ok. Without RailsMaxPoolSize set to 3 the box started to swap, that’s why it was slower.
Might be more fair to add
RailsMaxPoolSize 3to your apache conf for mod_rails, just to keep the number of actual rails instances the same as with mongrel. I believe the default max is 20.
I agree with Dan. It would also be interesting to see the memory footprints.
@Dan: there you go :)
@Sean: how? running top while benchmarking?
Interesting stuff. Of course benchmarks can only be trusted so much but it’s cool to see that mod_rails is at very least in the same league as mongrel speed-wise, and even maybe a little better. After reading this I’m more interested in mod_rails. :-)
haproxy will take a small bite out of performance. It would be nice to see the same benchmark without haproxy.
In my personal benchmarks, Mongrel was a shade faster than mod_rails. Thin or Ebb are even faster.
Geoffrey, without haproxy you have the problem of mongrels working on a slow requests getting hit by an second request.
The apache balancer does a simple round-robin on the workers, so a request could hit a busy mongrel serving another request. Haproxy on the other side forwards the request only to a free mongrel (not currently serving a request (and that is why the nginx fair balancer exists)), so taking out haproxy of the equation is not really an option for me.
Thinking on the results of this benchmark again, I think it is almost obvious that mongrel is a tad slower, because it has to parse the request again, while mod_rails does not.
With limited amount of RAM it is also very important to limit the mod_rails pool size, as pointed out by Dan, or you risk starting to swap out and kill your server.
Do you have a link to your benchmarks?
To see the memory footprint of your httpd and Rails processes, use pmap:
Have a look at the httpd processes in particular – in my testing with ab, httpd’s start up using around 40M, but after 200+ requests to Rails, they vary wildly from 143M to 174M. And as I run more ab, they keep growing to 349M and 533M. My Rails processes spawned by mod_rails stay at a consistent size.
This doesn’t happen at all when I ab test with a non-rails url like
Seems like a memory leak in the mod_rails module C++ code. Can anyone confirm?
The thing that makes me wonder then is, if I’m using mod_rails and I need more performance, where do I turn? Using the mongrel setup, I can add more to the cluster. It is a similar case with mod_rails?
I ask because I hate the pattern prescribed by using mongrel-clusters. Given, I’m no expert on these things so I’m sure there are some areas I could improve in. Nonetheless, keeping track of the ports and everything else involved (monit) is really suboptimal (IMHO), which makes mod_rails very appealing.
@Jonathan: With pmap, you’re only measuring the VM size of the heap, not the actual memory usage. So consider the following memory layout:
UUUffffffU
‘f’ is a free memory block, and ‘U’ is a used (allocated) memory block. pmap would report a memory usage of 10, even though actual memory usage 4. And this explanation doesn’t even take things into account, such as copy-on-write mappings. A more detailed explanation can be found here: http://tinyurl.com/5fd6mj
Passenger also makes use of threads. Because each thread has a stack, the VM size (not actual memory usage) will increase significantly. Many people mistake this for being a memory leak.
Passenger 1.0.2 will have a memory analysis tool. You can also get it from the git repository. It’s in ‘bin/passenger-memory-stats’. Please run that tool for more accurate memory statistics. In particular, the “private dirty RSS” field is important.
@Eric:
“if I’m using mod_rails and I need more performance, where do I turn?”
You don’t. :) Passenger will do it automatically for you. You may want to configure RailsMaxPoolSize to set an upper bound, but other than that, everything’s automatically managed for you.