Real-World WordPress Benchmarks with PHP5.5 PHP5.6 PHP-NG and HHVM

TL;DR In a local, Vagrant-based environment HHVM lost, probably due to a bug; it’s still investigated with the help of the HHVM guys! However on a DigitalOcean 4GB box it beat even the latest build of PHP-NG!

phphhvm

Update: Please take a look at the results at the end of the article! They reflect the power of HHVM better (after the JIT warmup), for some reason we cannot get these results with all setups though.
The tests below were done in a Vagrant/VVV environment, the results are still interesting, it might be a bug in HHVM or the Vagrant setup that’s preventing it from kicking into high speed, we’re investigating the issue with the HHVM guys.

If you remember we wrote an article a good couple of months ago when WordPress 3.9 came out that HHVM was fully supported beginning with that release, and we were all happy about it. The initial benchmark results showed HHVM to be far more superior than the Zend engine that’s currently powering all PHP builds. Then the problems came:

  • HHVM can only be run as one user, which means less security (in shared environments)
  • HHVM does not restart itself after it crashes, and unfortunately it still does that quite often
  • HHVM uses a lot of memory right from the start, and yes, it per-request memory usage will be lower once you scale compared to PHP-FPM

Obviously you have to compromise based on your (or rather your sites’) needs but is it worth it? How much of a performance gain can you expect by switching to HHVM?

At Kinsta we really like to test everything new and generally optimize everything to provide the best environment to our clients. Today I finally took the time to set up a test environment and do some tests to compare a couple of different builds with a fresh out of the box WordPress install and one that has a bunch of content added plus runs WooCommerce! To measure the script running time I simply added the

<?php timer_stop(1); ?>

line before the /body tag of the footer.php’s.

Note:
Previously this section contained benchmarks made with Vagrant/Virtualbox/Ubuntu14.04 however for some reason HHVM was really underperforming, probably due to a bug or a limitation of the virtualized environment. We feel that these test results do not reflect the reality so we re-run the tests on a cloud server and consider these valid.

Here are the exact setup details of the environment:

  • DigitalOcean 4GB droplet (2 CPU cores, 4GB RAM)
  • Ubuntu 14.04, MariaDB10
  • Test site: Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1
  • PHP 5.5.9, PHP 5.5.15, PHP 5.6.0 RC2, PHP-NG (20140718-git-6cc487d) and HHVM 3.2.0 (version says PHP 5.6.99-hhvm)

Without further ado, these were my test results, the lower the better, values in seconds:

DigitalOcean 4GB droplet

Seconds, 10 runs, lower the better.

It looks like that PHP-NG achieves its peak performance after the first run! HHVM needs a couple more reloads, but their performance seems to be almost equal! I can’t wait until PHP-NG is merged into the master! :)

Hits in a minute, higher the better.

PHP 5.5.15 OpCache Disabled

Transactions: 236 hits
Availability: 100.00 %
Elapsed time: 59.03 secs
Data transferred: 2.40 MB
Response time: 2.47 secs
Transaction rate: 4.00 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 9.87
Successful transactions: 236
Failed transactions: 0
Longest transaction: 4.44
Shortest transaction: 0.48

PHP 5.5.15 OpCache Enabled
Transactions: 441 hits
Availability: 100.00 %
Elapsed time: 59.55 secs
Data transferred: 4.48 MB
Response time: 1.34 secs
Transaction rate: 7.41 trans/sec
Throughput: 0.08 MB/sec
Concurrency: 9.91
Successful transactions: 441
Failed transactions: 0
Longest transaction: 2.19
Shortest transaction: 0.64

PHP 5.6 RC2 OpCache Disabled
Transactions: 207 hits
Availability: 100.00 %
Elapsed time: 59.87 secs
Data transferred: 2.10 MB
Response time: 2.80 secs
Transaction rate: 3.46 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 9.68
Successful transactions: 207
Failed transactions: 0
Longest transaction: 3.65
Shortest transaction: 0.54

PHP 5.6 RC2 OpCache Enabled
Transactions: 412 hits
Availability: 100.00 %
Elapsed time: 59.03 secs
Data transferred: 4.18 MB
Response time: 1.42 secs
Transaction rate: 6.98 trans/sec
Throughput: 0.07 MB/sec
Concurrency: 9.88
Successful transactions: 412
Failed transactions: 0
Longest transaction: 1.93
Shortest transaction: 0.34

HHVM 3.2.0 (version says PHP 5.6.99-hhvm)
Transactions: 955 hits
Availability: 100.00 %
Elapsed time: 59.69 secs
Data transferred: 9.18 MB
Response time: 0.62 secs
Transaction rate: 16.00 trans/sec
Throughput: 0.15 MB/sec
Concurrency: 9.94
Successful transactions: 955
Failed transactions: 0
Longest transaction: 0.85
Shortest transaction: 0.23

PHP-NG OpCache Enabled (built: Jul 29 2014 )
Transactions: 849 hits
Availability: 100.00 %
Elapsed time: 59.88 secs
Data transferred: 8.63 MB
Response time: 0.70 secs
Transaction rate: 14.18 trans/sec
Throughput: 0.14 MB/sec
Concurrency: 9.94
Successful transactions: 849
Failed transactions: 0
Longest transaction: 1.06
Shortest transaction: 0.13


 

Note:
These are the previous test results, they’re faulty. I left them here for future reference but please do NOT consider these values a truthful representation!

Here are the exact setup details of the environment:

  • Apple MacBook Pro mid-2011 (Intel Core i7 2 GHz 4 cores, 4GB RAM, 256GB Ocz Vertex 3 MI)
  • Current Varying Vagrant Vagrants build with Ubuntu 14.04, nginx 1.6.x, mysql 5.5.x, etc.
  • Test site 1: WordPress 3.9.1 bare minimum
  • Test site 2: Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1
  • PHP 5.5.9, PHP 5.5.15, PHP 5.6.0 RC2, PHP-NG (20140718-git-6cc487d) and HHVM 3.2.0 (version says PHP 5.6.99-hhvm)

Default Theme, Default WordPress 3.9.1, PHP 5.5.9-1ubuntu4.3 (with OpCache 7.0.3)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.

Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1 (OpCache Disabled)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.

Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1 (OpCache Enabled)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.

Siege
parameters: 10 concurrent users for 1 minute: siege -c 10 -b -t 1M

Faulty results. Please read the note above! Hits in a minute, higher the better.

PHP5.5 OpCache Disabled (PHP 5.5.15-1+deb.sury.org~trusty+1)Faulty results. Please read the note above!
Transactions: 35 hits
Availability: 100.00 %
Elapsed time: 59.04 secs
Data transferred: 2.03 MB
Response time: 14.56 secs
Transaction rate: 0.59 trans/sec
Throughput: 0.03 MB/sec
Concurrency: 8.63
Successful transactions: 35
Failed transactions: 0
Longest transaction: 18.73
Shortest transaction: 5.80

HHVM 3.2.0 (version says PHP 5.6.99-hhvm)Faulty results. Please read the note above!
Transactions: 44 hits
Availability: 100.00 %
Elapsed time: 59.53 secs
Data transferred: 0.42 MB
Response time: 12.00 secs
Transaction rate: 0.74 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 8.87
Successful transactions: 44
Failed transactions: 0
Longest transaction: 13.40
Shortest transaction: 2.65

PHP5.5 OpCache Enabled (PHP 5.5.15-1+deb.sury.org~trusty+1 with OpCache 7.0.4-dev)Faulty results. Please read the note above!
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 59.30 secs
Data transferred: 5.81 MB
Response time: 5.69 secs
Transaction rate: 1.69 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 9.60
Successful transactions: 100
Failed transactions: 0
Longest transaction: 7.25
Shortest transaction: 2.82

PHP5.6 OpCache Enabled (PHP 5.6.0RC2 with OpCache 7.0.4-dev)Faulty results. Please read the note above!
Transactions: 103 hits
Availability: 100.00 %
Elapsed time: 59.99 secs
Data transferred: 5.98 MB
Response time: 5.51 secs
Transaction rate: 1.72 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 9.45
Successful transactions: 103
Failed transactions: 0
Longest transaction: 6.87
Shortest transaction: 2.52

PHP-NG OpCache Enabled (20140718-git-6cc487d)Faulty results. Please read the note above!
Transactions: 124 hits
Availability: 100.00 %
Elapsed time: 59.32 secs
Data transferred: 7.19 MB
Response time: 4.58 secs
Transaction rate: 2.09 trans/sec
Throughput: 0.12 MB/sec
Concurrency: 9.57
Successful transactions: 124
Failed transactions: 0
Longest transaction: 6.86
Shortest transaction: 2.24

What do you think about this test? Did I miss something? What would you like to see in the next benchmarking article? Please leave your comment below!

Written by
Mark is a DevOps guy and head honcho at Kinsta, a performance WordPress hosting company.
  • http://paulisageek.com Paul Tarjan

    I’m pretty surprised by your results as they fly in the face in all of our previous benchmarking. Can you post the exact commands you used for both HHVM and PHP please, starting from a raw ubuntu 14.04 VM? Or post your exact VM? I’d like to reproduce it and debug it.

    Some things that come to mind:
    1) Did you warm up HHVM? We only JIT after the 12th request and only get really good after 100 or so requests.
    2) HHVM does indeed use all your cores (otherwise our machines in our data center would be very very idle). Were you not seeing that?
    3) I’m surprised by your memory. Everyone else deploying us sees a large memory decrease per request. Yes our base memory is higher (as we have x86 translated code in RAM) but per-request should be much lower.
    4) Can you please open issues for any crashes? I’m surprised to hear it crashes very often.

    • http://www.kinsta.com/ Mark Gavalda

      Hi Paul, thanks for stopping by! 1) Yep, but to be honest for some reason it doesn’t seem to warm up even after the 50th reload. On any other server (from a cloud provider) that happens after the 10th(ish) reload. It’s weird and I think therein lies the reason of these poor results!

      2) Good to know, I think I read that in a GH issue and was explained that it’s running on one thread only, I must have misinterpreted it. I will update the article right away!

      3) It’s a bit harder to compare that, I will try. Generally speaking HHVM takes up ~500-700MB after launch compared to the ~25MB of a PHP-FPM worker. I’m not sure what’s the concurrency number where the balance would tip to HHVM’s favor?

      4) For example: https://github.com/facebook/hhvm/issues/2851#issuecomment-49955592 but you already know about that ;) Also there’s a WP plugin with a captcha feature that’s giving us segfaults every day. Didn’t have the time to report that yet, I will though.

      Please keep an eye on the results that I’m updating the article’s end with! On the DO server it seems that HHVM is winning as it’s supposed to?

      • http://paulisageek.com Paul Tarjan

        1) Very weird. I’d love to have your exact VM so I can reproduce it and debug it.

        3) That sounds very much like your translation cache is taking up all that space. It looks like from your `ini` settings on the bug you posted sums to 800 Megs so that will be our fixed memory. Try turning those down until HHVM crashes. At FB I think we run with 2 GBs fixed memory but our site has a ton of code.

        4) Yup, I’m very aware of that issue sadly. We really don’t deal well with running out of TC space. A person on our team has an idea, but we haven’t had a chance to see if it works yet.

        What is a DO server?

        • http://www.kinsta.com/ Mark Gavalda

          It’s a DigitalOcean virtual server :)
          1) I can send it to you, no problem. Would love to get to the bottom of this and add more updates to the article! (Already fixed the problems that you clarified, thanks!) My email is markgavalda@kinsta.com

          • http://paulisageek.com Paul Tarjan

            Replied. Thanks.

  • http://asif.im/ Asif2BD

    Its good to see such benchmark and good conversation. Looking forward how to get over HHVM crash and maybe a bit more stable version in 3.2.x branch.

    • http://www.kinsta.com/ Mark Gavalda

      Thanks for stopping by and commenting, Asif!

  • http://www.synet.sk/ lubosdz

    Benchmarks where opcache is disabled are useless actually for the purpose of this article since HHVM always uses natively bytecode after warm up. Hence objective comparison against PHP is also with opcache always enabled. Also note, that PHP-NG is still without JIT, that might come in the future. I am wondering whether PHP-NG with build-in JIT could beat HHVM… Thanx a lot for interesting results.

    • http://www.kinsta.com/ Mark Gavalda

      Yep I get what you’re saying re the results without opcache, they were included “just for fun”. Obviously you’d keep opcache switched on in 99.9% of the time! And yes it’s “unreal” that they could achieve these performance gains without JIT! Hopefully they’ll explore that possibility in the near future! :) Thanks for commenting!

  • roman

    did you use vagrant with virtualbox or vmware provider?

    • http://www.kinsta.com/ Mark Gavalda

      It was the default OSX provider so I believe it’s virtualbox! Do you know about a shortcoming? Cheers!

      • roman

        our team found that hhvm works really slow with virtualbox provider. It works slower than fpm+nginx in our tests. But with vmware provider it showed the best results.

  • LouWii D.

    Thanks for these tests !
    It would have been interesting to see how performs PHP 5.4 with APC compared to these. I think a lot of webservers still use this “old” configuration (mine does, yes you can blame me) so it should be interesting to see the gain we can have by upgrading those old 5.4.

  • Laruence

    Actually, PHP NG is underworking, the performance now might be more better then July, you can follow this if you are interesting in it : https://wiki.php.net/phpng#performance_evaluation :), thanks for the benchmarkng

    • http://www.kinsta.com/ Mark Gavalda

      Yep, it’s on my todo list! I’ve seen the news that PHP7 has been optimized a bit more since July, as soon as I get some free time I will do more tests! :) Cheers

  • Laruence

    Or, I should say PHP7 instead of PHP NG :)

  • Yosh Drov

    Have you done any tests with HHVM 3.3?

    • http://www.kinsta.com/ Mark Gavalda

      Yes, I don’t really see a performance difference between HHVM releases. However I’ll be posting a new benchmark soon with lots of real world data.

      • Yosh Drov

        I see, just moved to a new server and decided to go for:

        Ubuntu -> Nginx 1.6.2 + HHVM 3.3 + Varnish 4 + Cloudflare w/railgun

        And all was going better than before that i used php-fpm and today saw a huge jump in the cpu after posting a new article, waiting impatiently for your new benchmark

        any eta?

      • Yosh Drov

        Would be cool to see quite light themes like:

        Genesis or Thesis clean themes

      • pjv

        Thanks for posting all this real-world data. Do you guys see HHVM memory leaking and the ram consumed by the HHVM process growing over time? I’m seeing that and thinking about a pragmatic solution to it. Maybe a cron that looks at the memory usage every few minutes and restarts hhvm server over a threshold. Wondering if you already have experience with this issue and a real-world solution for it.

        • http://www.kinsta.com/ Mark Gavalda

          Yes, unfortunately that’s quite common. There are multiple memory leaks that are well known yet not fixed. Hopefully they’ll be soon.
          Also a regular issue that comes up is a crash because “the TC is full”. See https://github.com/facebook/hhvm/issues/2851
          It’s a delicate issue because with the right settings you can find the golden balance, however it’s definitely not for shared hosting environments, only for when you have full control over a VPS/dedicated box.

          • pjv

            Yes, I have seen that TC issue – in fact swiped your .ini memory settings from that github issue when I was first deploying HHVM on my server (thanks).

            here’s my current solution to the memory leaks (script called from cron every 10 minutes):


            #!/usr/bin/env bash

            PID=`pidof hhvm`
            MEM=`ps_mem.py -p $PID -t`
            regex="^([0-9]+.?[0-9]*)(.+)$"

            [[ $MEM =~ $regex ]]

            num=${BASH_REMATCH[1]}
            units=${BASH_REMATCH[2]}

            # restart hhvm daemon if using more than 3.0 GB
            if [[ $units == "GiB" && $num > 3.0 ]] ; then
            service hhvm restart
            echo "`date -u` restart hhvm over 3 GB" >> hhvm_restart.log
            fi

  • http://CamClare.com/ Cameron Clare

    Hi Mark,

    Fantastic to see a benchmark such as this. I was considering running my own but I think it’s time for me to forget about my site’s load speed and get back to writing content!!

    Cheers
    Cam

Written by LondonKinsta Wordpress Hosting
Mark is a DevOps guy and head honcho at Kinsta, a performance WordPress hosting company.

Subscribe to our FREE monthly newsletter and receive our favorite WordPress tips, tricks and tutorials!

We'll send you posts that will help you profit from your online business with WordPress.