Async frameworks “Hello World” showdown

This is not intended to be a proper comparison between these frameworks. However, since the “Hello World” test is the lowest common denominator, it is a pretty clear indicator that an application can’t exceed in performance these numbers. Also, what Guillermo did not understand from my comment is the fact that 1000 requests at the concurrency of 10 is way to few for get a proper picture of a “Hello World” showdown.

Tested frameworks:

  • node.js – v0.6.17
  • vert.x – v1.0 final + OpenJDK 7 installed from the Ubuntu repository – using the JavaScript bindings
  • luanode – built from the master branch using the Ubuntu provided lua dependencies
  • luvit – built from the master branch
  • react – cloned the master branch

I also wanted to test node.native, but it kept crashing on me. You can see that it is a pretty old issue. I didn’t have the patience to make the v0.1.0 branch to work with the previously used code. But I’d like to give it a run for its money.

The system used for the testing is a modest Athlon II X2 240e (2.8GHz) with 4GB or DDR2 800MHz running the latest Kubuntu 12.04 LTS amd64. Since ab pretty much takes a CPU core for itself, the frameworks ran a single process that occupied a single CPU core. I tried running a node.js HTTP server wrapped with the cluster module. Or passing -instances 2 to the vertx framework. The results were pretty much the same, therefore using just a single CPU core is a fair comparison.

The ab command that I used to hammer the Hello World! output:

ab -r -k -n 1000000 -c 1000{port_name}/

The command ran at least a couple of times before saving the results. Just to make sure that everything is properly warmed up.

The averages graph:

The test sources and full ab output is available on this gist. There’s interesting output in the results.txt file for the stats nerds.

PS: I have the impression (but did not test) that vert.x may be a little bit faster, but ab is the actual bottleneck.

Update: added React (node.php) to comparision. Too lazy to plot another graph. But at 1573.40 req/s, it is harly a match even for luanode. Used the PHP 5.3.10 from the Ubuntu repositories.

Update: added another React (node.php) to comparision, but with a custom build of PHP 5.4.3. This time, it managed to get 3727.49 req/s.

2 thoughts on “Async frameworks “Hello World” showdown

  1. Daniele Alessandri

    Out of curiosity, which PHP interpreter did you use to test React? The PHP 5.3 binaries from Canonical’s repositories have always been slow compared to a self-built PHP interpreter. Also, method dispatch in PHP 5.4 is ~30% faster than PHP 5.3 and this could make a difference, albeit slight, given the nature of React which is currently 100% userland PHP code.

    Anyway as a Lua junkie I’m quite pleased to see luvit doing better than NodeJS, let alone the fact that memory consumption should be a considerable fraction 🙂

  2. SaltwaterC Post author

    I was too lazy to build a 5.4 at the time. I did that, and the results somehow speak for themselves: 3727.49 req/s with a custom 5.4.3 build. However, I am curious if it can use the pecl/libevent bindings.

Leave a Reply

Your email address will not be published. Required fields are marked *