[Home] You are not logged in. Login here

Just my stuff

Benchmarking mod_rails against mongrel

I’d like to use mod_rails instead of the apache->haproxy->mongrel configuration, but before I do I wanted to make sure I don’t lose to much speed, so I decided to benchmark mod_rails against mongrel. mod_rails was already benchmarked, but I don’t believe it if I don’t see it :)

To benchmark it I created a new rails application with a single controller and a hello.html.erb view with only ‘Hello World’ in it.

For the test I took a very low end machine, because the virtual server I rent for the production site is not much better anyway:

s2@fresh:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : CentaurHauls
cpu family      : 6
model           : 9
model name      : VIA Nehemiah
stepping        : 8
cpu MHz         : 532.000
cache size      : 64 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr cx8 sep mtrr pge cmov pat mmx fxsr sse rng rng_en ace ace_en
bogomips        : 1067.72
clflush size    : 32

with 483608 bytes of RAM on Ubuntu 7.10 and ruby 1.8.6.

The mongrel setup consists of 3 mongrels running in production mode behind haproxy balancing the requests coming from apache.
This is the apache config:

<VirtualHost *:80>
  ServerName benchmark.fresh

  ProxyRequests Off

  ProxyPass / http://127.0.0.1:8010/
  ProxyPassReverse / http://127.0.0.1:8010/
</VirtualHost>

I keep things simple because I want to test the speed of the app running with mongrel against mod_rails. I don’t care about static stuff, url rewriting and other things in this test.
haproxy is configured like this:
...
listen rails :8010
  server rails-1 localhost:8011 maxconn 1
  server rails-2 localhost:8012 maxconn 1
  server rails-3 localhost:8013 maxconn 1

I left the sessions active, and the configured database is a postrges connection. As this will be the same in the two setups, I hope this will not make any difference.
Now I get the mongrels running and start the benchmark (after a dry run of 1000 requests):
$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark.fresh (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software:        Mongrel
Server Hostname:        benchmark.fresh
Server Port:            80

Document Path:          /hello/hello
Document Length:        12 bytes

Concurrency Level:      100
Time taken for tests:   25.184929 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      480000 bytes
HTML transferred:       12000 bytes
Requests per second:    39.71 [#/sec] (mean)
Time per request:       2518.493 [ms] (mean)
Time per request:       25.185 [ms] (mean, across all concurrent requests)
Transfer rate:          18.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  107 375.1      0    1874
Processing:    47 2272 771.9   2414   11236
Waiting:        5 2264 740.8   2413   11236
Total:        483 2380 711.4   2415   13110

Percentage of the requests served within a certain time (ms)
  50%   2415
  66%   2456
  75%   2484
  80%   2562
  90%   2793
  95%   2863
  98%   3291
  99%   4396
 100%  13110 (longest request)

Now it’s time for mod_rails. I stopped the mongrels and haproxy to free up some precious RAM.
Apache config:

<VirtualHost *:80>
  ServerName benchmark.fresh
  DocumentRoot /home/s2/tmp/benchmark/public/
</VirtualHost>

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/ext/apache2/mod_passenger.so
RailsSpawnServer /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/bin/passenger-spawn-server
RailsRuby /usr/bin/ruby1.8
RailsEnv production

and after a dry run of 1000 requests:

$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark.fresh (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software:        Apache/2.2.4
Server Hostname:        benchmark.fresh
Server Port:            80

Document Path:          /hello/hello
Document Length:        12 bytes

Concurrency Level:      100
Time taken for tests:   22.329544 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      514001 bytes
HTML transferred:       12000 bytes
Requests per second:    44.78 [#/sec] (mean)
Time per request:       2232.954 [ms] (mean)
Time per request:       22.330 [ms] (mean, across all concurrent requests)
Transfer rate:          22.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   4.9      0      41
Processing:    48 2069 2922.9   1445   22101
Waiting:       45 2064 2922.5   1443   22101
Total:         61 2071 2922.8   1445   22114

Percentage of the requests served within a certain time (ms)
  50%   1445
  66%   1589
  75%   1735
  80%   1808
  90%   2339
  95%   3249
  98%  17180
  99%  19692
 100%  22114 (longest request)

A pretty graph (because every serious benchmark has at least one):

mod_rails can handle 44,78 req/sec, and mongrel 39,71. I think I can ditch the complicated mongrel setup after I make sure mod_rails is as stable as the mongrels are.


Update: Dan and Sean pointed out in the comments that a
RailsMaxPoolSize 3
for mod_rails would be more appropriate. Here you go:

$ ab -n 1000 -c 100 http://benchmark.fresh/hello/hello
...
Concurrency Level:      100
Time taken for tests:   20.94626 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      514000 bytes
HTML transferred:       12000 bytes
Requests per second:    49.76 [#/sec] (mean)
Time per request:       2009.463 [ms] (mean)
Time per request:       20.095 [ms] (mean, across all concurrent requests)
Transfer rate:          24.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    8  28.1      0     113
Processing:   131 1902 1518.9   1240    6861
Waiting:      127 1900 1518.9   1238    6860
Total:        149 1910 1515.4   1253    6861

Percentage of the requests served within a certain time (ms)
  50%   1253
  66%   2038
  75%   3334
  80%   3656
  90%   4322
  95%   4644
  98%   5431
  99%   5696
 100%   6861 (longest request)

49.76. It’s actually quicker?! I could not believe this, so I repeated the test twice, with and without RailsMaxPoolSize 3.

With: 46.92, 46.51
Without: 10.22, 22.32

Ok. Without RailsMaxPoolSize set to 3 the box started to swap, that’s why it was slower.

Anonymous Dan Walters said...

Might be more fair to add

RailsMaxPoolSize 3

to your apache conf for mod_rails, just to keep the number of actual rails instances the same as with mongrel. I believe the default max is 20.

Tue Apr 22 18:57:38 +0200 2008
Anonymous Sean McCleary said...

I agree with Dan. It would also be interesting to see the memory footprints.

Tue Apr 22 19:04:02 +0200 2008
Avatar S2 said...

@Dan: there you go :)
@Sean: how? running top while benchmarking?

Tue Apr 22 19:43:01 +0200 2008
Anonymous Norman Clarke said...

Interesting stuff. Of course benchmarks can only be trusted so much but it’s cool to see that mod_rails is at very least in the same league as mongrel speed-wise, and even maybe a little better. After reading this I’m more interested in mod_rails. :-)

Tue Apr 22 19:54:14 +0200 2008
Anonymous Geoffrey Grosenbach said...

haproxy will take a small bite out of performance. It would be nice to see the same benchmark without haproxy.

In my personal benchmarks, Mongrel was a shade faster than mod_rails. Thin or Ebb are even faster.

Tue Apr 22 20:48:49 +0200 2008
Avatar S2 said...

Geoffrey, without haproxy you have the problem of mongrels working on a slow requests getting hit by an second request.
The apache balancer does a simple round-robin on the workers, so a request could hit a busy mongrel serving another request. Haproxy on the other side forwards the request only to a free mongrel (not currently serving a request (and that is why the nginx fair balancer exists)), so taking out haproxy of the equation is not really an option for me.
Thinking on the results of this benchmark again, I think it is almost obvious that mongrel is a tad slower, because it has to parse the request again, while mod_rails does not.
With limited amount of RAM it is also very important to limit the mod_rails pool size, as pointed out by Dan, or you risk starting to swap out and kill your server.
Do you have a link to your benchmarks?

Tue Apr 22 21:04:25 +0200 2008
Anonymous Jonathan said...

To see the memory footprint of your httpd and Rails processes, use pmap:

sudo pmap <pid> |tail -1

Have a look at the httpd processes in particular – in my testing with ab, httpd’s start up using around 40M, but after 200+ requests to Rails, they vary wildly from 143M to 174M. And as I run more ab, they keep growing to 349M and 533M. My Rails processes spawned by mod_rails stay at a consistent size.

This doesn’t happen at all when I ab test with a non-rails url like

ab -n 5000 -c 100 http://example.com/images/rails.png

Seems like a memory leak in the mod_rails module C++ code. Can anyone confirm?

Tue Apr 22 22:04:08 +0200 2008
Anonymous Eric Larson said...

The thing that makes me wonder then is, if I’m using mod_rails and I need more performance, where do I turn? Using the mongrel setup, I can add more to the cluster. It is a similar case with mod_rails?

I ask because I hate the pattern prescribed by using mongrel-clusters. Given, I’m no expert on these things so I’m sure there are some areas I could improve in. Nonetheless, keeping track of the ports and everything else involved (monit) is really suboptimal (IMHO), which makes mod_rails very appealing.

Tue Apr 22 23:41:08 +0200 2008
Anonymous Hongli Lai said...

@Jonathan: With pmap, you’re only measuring the VM size of the heap, not the actual memory usage. So consider the following memory layout:

UUUffffffU

‘f’ is a free memory block, and ‘U’ is a used (allocated) memory block. pmap would report a memory usage of 10, even though actual memory usage 4. And this explanation doesn’t even take things into account, such as copy-on-write mappings. A more detailed explanation can be found here: http://tinyurl.com/5fd6mj
Passenger also makes use of threads. Because each thread has a stack, the VM size (not actual memory usage) will increase significantly. Many people mistake this for being a memory leak.

Passenger 1.0.2 will have a memory analysis tool. You can also get it from the git repository. It’s in ‘bin/passenger-memory-stats’. Please run that tool for more accurate memory statistics. In particular, the “private dirty RSS” field is important.

Wed Apr 23 01:30:54 +0200 2008
Anonymous Hongli Lai said...

@Eric:

“if I’m using mod_rails and I need more performance, where do I turn?”

You don’t. :) Passenger will do it automatically for you. You may want to configure RailsMaxPoolSize to set an upper bound, but other than that, everything’s automatically managed for you.

Wed Apr 23 01:32:50 +0200 2008
 
Comments for this post have been disabled