Real-World WordPress Benchmarks with PHP5.5 PHP5.6 PHP-NG and HHVM

Updated on June 28, 2017

Update: Please take a look at the results at the end of the article! They reflect the power of HHVM better (after the JIT warmup), for some reason we cannot get these results with all setups though.
The tests below were done in a Vagrant/VVV environment, the results are still interesting, it might be a bug in HHVM or the Vagrant setup that’s preventing it from kicking into high speed, we’re investigating the issue with the HHVM guys.

If you remember we wrote an article a good couple of months ago when WordPress 3.9 came out that HHVM was fully supported beginning with that release, and we were all happy about it. The initial benchmark results showed HHVM to be far more superior than the Zend engine that’s currently powering all PHP builds. Then the problems came:

  • HHVM can only be run as one user, which means less security (in shared environments)
  • HHVM does not restart itself after it crashes, and unfortunately it still does that quite often
  • HHVM uses a lot of memory right from the start, and yes, it per-request memory usage will be lower once you scale compared to PHP-FPM

Obviously you have to compromise based on your (or rather your sites’) needs but is it worth it? How much of a performance gain can you expect by switching to HHVM?

At Kinsta we really like to test everything new and generally optimize everything to provide the best environment to our clients. Today I finally took the time to set up a test environment and do some tests to compare a couple of different builds with a fresh out of the box WordPress install and one that has a bunch of content added plus runs WooCommerce! To measure the script running time I simply added the

[code lang=”php”]<?php timer_stop(1); ?>[/code]

line before the /body tag of the footer.php’s.

Note:
Previously this section contained benchmarks made with Vagrant/Virtualbox/Ubuntu14.04 however for some reason HHVM was really underperforming, probably due to a bug or a limitation of the virtualized environment. We feel that these test results do not reflect the reality so we re-run the tests on a cloud server and consider these valid.

Here are the exact setup details of the environment:

  • DigitalOcean 4GB droplet (2 CPU cores, 4GB RAM)
  • Ubuntu 14.04, MariaDB10
  • Test site: Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1
  • PHP 5.5.9, PHP 5.5.15, PHP 5.6.0 RC2, PHP-NG (20140718-git-6cc487d) and HHVM 3.2.0 (version says PHP 5.6.99-hhvm)

Without further ado, these were my test results, the lower the better, values in seconds:

DigitalOcean 4GB droplet

php hhvm

It looks like that PHP-NG achieves its peak performance after the first run! HHVM needs a couple more reloads, but their performance seems to be almost equal! I can’t wait until PHP-NG is merged into the master! :)

Hits in a minute, higher the better.

hits per minute php hhvm

PHP 5.5.15 OpCache Disabled

Transactions: 236 hits
Availability: 100.00 %
Elapsed time: 59.03 secs
Data transferred: 2.40 MB
Response time: 2.47 secs
Transaction rate: 4.00 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 9.87
Successful transactions: 236
Failed transactions: 0
Longest transaction: 4.44
Shortest transaction: 0.48

PHP 5.5.15 OpCache Enabled
Transactions: 441 hits
Availability: 100.00 %
Elapsed time: 59.55 secs
Data transferred: 4.48 MB
Response time: 1.34 secs
Transaction rate: 7.41 trans/sec
Throughput: 0.08 MB/sec
Concurrency: 9.91
Successful transactions: 441
Failed transactions: 0
Longest transaction: 2.19
Shortest transaction: 0.64

PHP 5.6 RC2 OpCache Disabled
Transactions: 207 hits
Availability: 100.00 %
Elapsed time: 59.87 secs
Data transferred: 2.10 MB
Response time: 2.80 secs
Transaction rate: 3.46 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 9.68
Successful transactions: 207
Failed transactions: 0
Longest transaction: 3.65
Shortest transaction: 0.54

PHP 5.6 RC2 OpCache Enabled
Transactions: 412 hits
Availability: 100.00 %
Elapsed time: 59.03 secs
Data transferred: 4.18 MB
Response time: 1.42 secs
Transaction rate: 6.98 trans/sec
Throughput: 0.07 MB/sec
Concurrency: 9.88
Successful transactions: 412
Failed transactions: 0
Longest transaction: 1.93
Shortest transaction: 0.34

HHVM 3.2.0 (version says PHP 5.6.99-hhvm)
Transactions: 955 hits
Availability: 100.00 %
Elapsed time: 59.69 secs
Data transferred: 9.18 MB
Response time: 0.62 secs
Transaction rate: 16.00 trans/sec
Throughput: 0.15 MB/sec
Concurrency: 9.94
Successful transactions: 955
Failed transactions: 0
Longest transaction: 0.85
Shortest transaction: 0.23

PHP-NG OpCache Enabled (built: Jul 29 2014 )
Transactions: 849 hits
Availability: 100.00 %
Elapsed time: 59.88 secs
Data transferred: 8.63 MB
Response time: 0.70 secs
Transaction rate: 14.18 trans/sec
Throughput: 0.14 MB/sec
Concurrency: 9.94
Successful transactions: 849
Failed transactions: 0
Longest transaction: 1.06
Shortest transaction: 0.13


Note:
These are the previous test results, they’re faulty. I left them here for future reference but please do NOT consider these values a truthful representation!

Here are the exact setup details of the environment:

  • Apple MacBook Pro mid-2011 (Intel Core i7 2 GHz 4 cores, 4GB RAM, 256GB Ocz Vertex 3 MI)
  • Current Varying Vagrant Vagrants build with Ubuntu 14.04, nginx 1.6.x, mysql 5.5.x, etc.
  • Test site 1: WordPress 3.9.1 bare minimum
  • Test site 2: Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1
  • PHP 5.5.9, PHP 5.5.15, PHP 5.6.0 RC2, PHP-NG (20140718-git-6cc487d) and HHVM 3.2.0 (version says PHP 5.6.99-hhvm)

Default Theme, Default WordPress 3.9.1, PHP 5.5.9-1ubuntu4.3 (with OpCache 7.0.3)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.

php hhvm

Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1 (OpCache Disabled)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.
php hhvm

Munditia Theme with Demo Content Imported, WooCommerce 2.1.12 & WordPress 3.9.1 (OpCache Enabled)

Faulty results. Please read the note above! Seconds, 10 runs, lower the better.

php hhvm

Siege
parameters: 10 concurrent users for 1 minute: siege -c 10 -b -t 1M

Faulty results. Please read the note above! Hits in a minute, higher the better.

php hhvm

PHP5.5 OpCache Disabled (PHP 5.5.15-1+deb.sury.org~trusty+1)Faulty results. Please read the note above!
Transactions: 35 hits
Availability: 100.00 %
Elapsed time: 59.04 secs
Data transferred: 2.03 MB
Response time: 14.56 secs
Transaction rate: 0.59 trans/sec
Throughput: 0.03 MB/sec
Concurrency: 8.63
Successful transactions: 35
Failed transactions: 0
Longest transaction: 18.73
Shortest transaction: 5.80

HHVM 3.2.0 (version says PHP 5.6.99-hhvm)Faulty results. Please read the note above!
Transactions: 44 hits
Availability: 100.00 %
Elapsed time: 59.53 secs
Data transferred: 0.42 MB
Response time: 12.00 secs
Transaction rate: 0.74 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 8.87
Successful transactions: 44
Failed transactions: 0
Longest transaction: 13.40
Shortest transaction: 2.65

PHP5.5 OpCache Enabled (PHP 5.5.15-1+deb.sury.org~trusty+1 with OpCache 7.0.4-dev)Faulty results. Please read the note above!
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 59.30 secs
Data transferred: 5.81 MB
Response time: 5.69 secs
Transaction rate: 1.69 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 9.60
Successful transactions: 100
Failed transactions: 0
Longest transaction: 7.25
Shortest transaction: 2.82

PHP5.6 OpCache Enabled (PHP 5.6.0RC2 with OpCache 7.0.4-dev)Faulty results. Please read the note above!
Transactions: 103 hits
Availability: 100.00 %
Elapsed time: 59.99 secs
Data transferred: 5.98 MB
Response time: 5.51 secs
Transaction rate: 1.72 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 9.45
Successful transactions: 103
Failed transactions: 0
Longest transaction: 6.87
Shortest transaction: 2.52

PHP-NG OpCache Enabled (20140718-git-6cc487d)Faulty results. Please read the note above!
Transactions: 124 hits
Availability: 100.00 %
Elapsed time: 59.32 secs
Data transferred: 7.19 MB
Response time: 4.58 secs
Transaction rate: 2.09 trans/sec
Throughput: 0.12 MB/sec
Concurrency: 9.57
Successful transactions: 124
Failed transactions: 0
Longest transaction: 6.86
Shortest transaction: 2.24

What do you think about this test? Did I miss something? What would you like to see in the next benchmarking article? Please leave your comment below!

This article was written by Mark Gavalda

Mark has many years of experience leading teams in the fields of marketing, web design and development. As a dev guy he used his WP expertise to collect the know-hows of creating a reliable and customer friendly hosting company to satisfy the increasing demand of clients. He is an urban cyclist and autodidact who never stops learning new skills.

  1. Gravatar for this comment's author
    Paul Tarjan July 29, 2014 at 11:15 pm

    I’m pretty surprised by your results as they fly in the face in all of our previous benchmarking. Can you post the exact commands you used for both HHVM and PHP please, starting from a raw ubuntu 14.04 VM? Or post your exact VM? I’d like to reproduce it and debug it.

    Some things that come to mind:
    1) Did you warm up HHVM? We only JIT after the 12th request and only get really good after 100 or so requests.
    2) HHVM does indeed use all your cores (otherwise our machines in our data center would be very very idle). Were you not seeing that?
    3) I’m surprised by your memory. Everyone else deploying us sees a large memory decrease per request. Yes our base memory is higher (as we have x86 translated code in RAM) but per-request should be much lower.
    4) Can you please open issues for any crashes? I’m surprised to hear it crashes very often.

    1. Gravatar for this comment's author
      Mark Gavalda July 29, 2014 at 11:23 pm

      Hi Paul, thanks for stopping by! 1) Yep, but to be honest for some reason it doesn’t seem to warm up even after the 50th reload. On any other server (from a cloud provider) that happens after the 10th(ish) reload. It’s weird and I think therein lies the reason of these poor results!

      2) Good to know, I think I read that in a GH issue and was explained that it’s running on one thread only, I must have misinterpreted it. I will update the article right away!

      3) It’s a bit harder to compare that, I will try. Generally speaking HHVM takes up ~500-700MB after launch compared to the ~25MB of a PHP-FPM worker. I’m not sure what’s the concurrency number where the balance would tip to HHVM’s favor?

      4) For example: https://github.com/facebook/hhvm/issues/2851#issuecomment-49955592 but you already know about that ;) Also there’s a WP plugin with a captcha feature that’s giving us segfaults every day. Didn’t have the time to report that yet, I will though.

      Please keep an eye on the results that I’m updating the article’s end with! On the DO server it seems that HHVM is winning as it’s supposed to?

      1. Gravatar for this comment's author
        Paul Tarjan July 29, 2014 at 11:29 pm

        1) Very weird. I’d love to have your exact VM so I can reproduce it and debug it.

        3) That sounds very much like your translation cache is taking up all that space. It looks like from your `ini` settings on the bug you posted sums to 800 Megs so that will be our fixed memory. Try turning those down until HHVM crashes. At FB I think we run with 2 GBs fixed memory but our site has a ton of code.

        4) Yup, I’m very aware of that issue sadly. We really don’t deal well with running out of TC space. A person on our team has an idea, but we haven’t had a chance to see if it works yet.

        What is a DO server?

        1. Gravatar for this comment's author
          Mark Gavalda July 30, 2014 at 12:07 am

          It’s a DigitalOcean virtual server :)
          1) I can send it to you, no problem. Would love to get to the bottom of this and add more updates to the article! (Already fixed the problems that you clarified, thanks!) My email is markgavalda@kinsta.com

          1. Gravatar for this comment's author
            Paul Tarjan July 30, 2014 at 12:09 am

            Replied. Thanks.

  2. Gravatar for this comment's author
    Asif2BD July 30, 2014 at 1:55 pm

    Its good to see such benchmark and good conversation. Looking forward how to get over HHVM crash and maybe a bit more stable version in 3.2.x branch.

    1. Gravatar for this comment's author
      Mark Gavalda July 30, 2014 at 3:06 pm

      Thanks for stopping by and commenting, Asif!

  3. Gravatar for this comment's author
    lubosdz July 30, 2014 at 6:33 pm

    Benchmarks where opcache is disabled are useless actually for the purpose of this article since HHVM always uses natively bytecode after warm up. Hence objective comparison against PHP is also with opcache always enabled. Also note, that PHP-NG is still without JIT, that might come in the future. I am wondering whether PHP-NG with build-in JIT could beat HHVM… Thanx a lot for interesting results.

    1. Gravatar for this comment's author
      Mark Gavalda July 30, 2014 at 6:52 pm

      Yep I get what you’re saying re the results without opcache, they were included “just for fun”. Obviously you’d keep opcache switched on in 99.9% of the time! And yes it’s “unreal” that they could achieve these performance gains without JIT! Hopefully they’ll explore that possibility in the near future! :) Thanks for commenting!

  4. Gravatar for this comment's author
    roman July 31, 2014 at 3:18 am

    did you use vagrant with virtualbox or vmware provider?

    1. Gravatar for this comment's author
      Mark Gavalda July 31, 2014 at 8:47 am

      It was the default OSX provider so I believe it’s virtualbox! Do you know about a shortcoming? Cheers!

      1. Gravatar for this comment's author
        roman July 31, 2014 at 2:40 pm

        our team found that hhvm works really slow with virtualbox provider. It works slower than fpm+nginx in our tests. But with vmware provider it showed the best results.

  5. Gravatar for this comment's author
    LouWii D. September 15, 2014 at 6:29 pm

    Thanks for these tests !
    It would have been interesting to see how performs PHP 5.4 with APC compared to these. I think a lot of webservers still use this “old” configuration (mine does, yes you can blame me) so it should be interesting to see the gain we can have by upgrading those old 5.4.

  6. Gravatar for this comment's author
    Laruence October 22, 2014 at 9:40 am

    Actually, PHP NG is underworking, the performance now might be more better then July, you can follow this if you are interesting in it : https://wiki.php.net/phpng#performance_evaluation :), thanks for the benchmarkng

    1. Gravatar for this comment's author
      Mark Gavalda October 22, 2014 at 2:57 pm

      Yep, it’s on my todo list! I’ve seen the news that PHP7 has been optimized a bit more since July, as soon as I get some free time I will do more tests! :) Cheers

  7. Gravatar for this comment's author
    Laruence October 22, 2014 at 9:41 am

    Or, I should say PHP7 instead of PHP NG :)

  8. Gravatar for this comment's author
    Yosh Drov November 16, 2014 at 7:44 pm

    Have you done any tests with HHVM 3.3?

    1. Gravatar for this comment's author
      Mark Gavalda November 16, 2014 at 8:03 pm

      Yes, I don’t really see a performance difference between HHVM releases. However I’ll be posting a new benchmark soon with lots of real world data.

      1. Gravatar for this comment's author
        Yosh Drov November 17, 2014 at 12:33 am

        I see, just moved to a new server and decided to go for:

        Ubuntu -> Nginx 1.6.2 + HHVM 3.3 + Varnish 4 + Cloudflare w/railgun

        And all was going better than before that i used php-fpm and today saw a huge jump in the cpu after posting a new article, waiting impatiently for your new benchmark

        any eta?

      2. Gravatar for this comment's author
        Yosh Drov November 17, 2014 at 12:40 am

        Would be cool to see quite light themes like:

        Genesis or Thesis clean themes

      3. Gravatar for this comment's author
        pjv November 23, 2014 at 1:52 pm

        Thanks for posting all this real-world data. Do you guys see HHVM memory leaking and the ram consumed by the HHVM process growing over time? I’m seeing that and thinking about a pragmatic solution to it. Maybe a cron that looks at the memory usage every few minutes and restarts hhvm server over a threshold. Wondering if you already have experience with this issue and a real-world solution for it.

        1. Gravatar for this comment's author
          Mark Gavalda November 23, 2014 at 8:47 pm

          Yes, unfortunately that’s quite common. There are multiple memory leaks that are well known yet not fixed. Hopefully they’ll be soon.
          Also a regular issue that comes up is a crash because “the TC is full”. See https://github.com/facebook/hhvm/issues/2851
          It’s a delicate issue because with the right settings you can find the golden balance, however it’s definitely not for shared hosting environments, only for when you have full control over a VPS/dedicated box.

          1. Gravatar for this comment's author
            pjv November 23, 2014 at 9:06 pm

            Yes, I have seen that TC issue – in fact swiped your .ini memory settings from that github issue when I was first deploying HHVM on my server (thanks).

            here’s my current solution to the memory leaks (script called from cron every 10 minutes):


            #!/usr/bin/env bash

            PID=`pidof hhvm`
            MEM=`ps_mem.py -p $PID -t`
            regex="^([0-9]+.?[0-9]*)(.+)$"

            [[ $MEM =~ $regex ]]

            num=${BASH_REMATCH[1]}
            units=${BASH_REMATCH[2]}

            # restart hhvm daemon if using more than 3.0 GB
            if [[ $units == "GiB" && $num > 3.0 ]] ; then
            service hhvm restart
            echo "`date -u` restart hhvm over 3 GB" >> hhvm_restart.log
            fi

  9. Gravatar for this comment's author
    Cameron Clare December 23, 2014 at 8:53 am

    Hi Mark,

    Fantastic to see a benchmark such as this. I was considering running my own but I think it’s time for me to forget about my site’s load speed and get back to writing content!!

    Cheers
    Cam

Leave a Reply to Yosh Drov Cancel reply

Use WordPress?

Join 20,000+ others who get our FREE weekly newsletter with WordPress tips on how to drive more traffic and revenue to your business!

You have Successfully Subscribed!

Send this to a friend