Using The Grinder to load test Blog Software packages
In my quest to find blog software for www.javasanity.org, I decided to use The Grinder for comparing blog package performance. This article covers configuring The Grinder for my tests as well as the transaction per second figures from each blog package analyzed.
You might find this article interesting if you are either looking to use The Grinder for test purposes, or if you are planning to select Blog Software for your web site.
Table of Contents:
1 Blog software candidate selection process
2 Test system configuration and software setup
3 Grinder installation and configuration
4 Running Grinder
5 Test results
6 Conclusions
1. Blog software candidate selection process
When I started looking, I had only one requirement: The software I chose had to be something that could be ran by the various inexpensive web hosting sites out there. This restricted the options down to perl and php based blog software. (Yes, it's a shame considering this site Java related!)
To find individual candidates, I went to the cms comparison site OpenSourceCMS and picked out choices that appealed to me.
I wanted something clean, less cluttered and professional. I was also curious to see how the "lite" blog software candidates which used a flat file database would stack up against the ones that used a SQL database.
2. Test system configuration and setup
A dual-core 1.6ghz Socket-939 Athlon 64with 2GB RAM. Apache 2.2 and MySQL 5.0. Operating system was Fedora Core 5. All software was configured with out of the box settings for the reported FPS figures. Some of the blog software titles do support caching to increase performance to near-flat-file levels, but those features were not included in these tests.
The Grinder was installed on a separate dual-core Athlon 64 machine.
3. Grinder installation and configuration
Installing grinder was really simple:
$ unzip grinder
$ cd grinder-2.8.6
$ cp ./examples/http/grinder.properties .
$ vi grinder.properties
I changed the following parameters:
| Parameter | Default Value | Chnaged Value |
|---|---|---|
| grinder.threads | 5 | 25 |
| grinder.cycles | 5 | 50 |
| grinder.grinderAddress | 228.1.1.1 | Machine IP Address |
| grinder.test0.parameter.url | http://localhost:7001/security | http://www.javasanity.org/textpattern |
Of course, the test0 url was changed for each blog package directory prior to running the test.
Because I only planned on testing the rendering of the main page with 2 articles on it, I removed all of the entries for test1 through test99.
4. Running the Grinder
Running grinder was simple as well:
$ java -cp lib/grinder.jar net.grinder.Console &
$ java -cp lib/grinder.jar net.grinder.Grinder
Then, in the Grinder console, I selected:
Action-> Collect Statistics
Action-> Start processes
Being impatient, I only allowed Grinder to run through 50 samples. At the end of 50 samples, I recorded the mean transactions per second, stopped the processes and stopped collection.
For the index.html control test, it was so fast that it completed after collecting only 7 samples.
5. Test Results (TPS)
It's probably best to just let the table do the talking here.
| Test | TPS | DB Type | CPU Utilization |
|---|---|---|---|
| index.html | 900 | N/A | Low |
| pivot_cms | 32 | Flat File | 2x100% |
| jaf_cms | 856 | Flat File | Low** |
| AngelineCMS | 460 | Flat File | |
| WordPress | 19 | MySQL | 2x100% |
| textpattern | 46 | MySQL | 2x100% |
| etomite | 87 | MySQL | 2x100% |
| b2evolution | 10 | MySQL | 2x100% |
| nucleus | 87 | MySQL | 2x100% |
| bmachine | 140 | MySQL | 2x100% |
| simplog | 54 | MySQL | 2x100% |
6. Conclusions
It's already obvious that I selected textpattern. Non-performance factors that ruled out some of the other options were:
a. I decided after running these tests that widespread adoption was important.
b. Some of the blog software items didn't meet the minimum feature set that I needed for this site. (E.g. write an article without publishing it immediately was missing from some of the lite versions)
c. Some were excluded because it took me longer to get the hang of the administrative interface than others. This one is very objective though.
**JAF was incredibly fast and I'm not entirely sure why. I ran it several times and could not get the CPU utilization to go much beyond 15%. It was nearly as fast as load testing just the flat-file index.html. Tuning Apache allowed me to get slightly higher TPS but the CPU still stayed low.
On further analysis, I found the 100mbit network to be the bottleneck. Moving grinder to the system that was hosting the web site resulted in higher TPS (2600TPS) and I finally got the CPU's to max out.
[Update - 12/26/2007 - I've decided to switch to Drupal because I like the way it organizes content. I may perform benchmark tests on Drupal at some point in the future as well]
7. Other Benchmarks
Sencer.de had results surprisingly similar to mine prior to enabling the cache plug-ins.
Posted by: BrianHayward 25 Feb 2007 09:12:30