Amasty Research | Full Page Cache vs. Varnish Cache

Amasty Team has recently issued a new extension Full Page Cache, aimed to speed up Magento store pages, known for their huge weight. Full Page Cache is a great success, it considerably reduces page load time and offers a number of other cool features. But actions speak louder than words. And statistics speak even louder. Check Amasty's new research on how fast can Magento be with Full Page Cache?

→ Have already switched to Magento 2? Try the Speed Optimization extension we’ve recently released. Improve your site performance by numerous showings.

How fast can Magento be with Full Page Cache?

TEST CONFIGURATION

HARDWARE

• CPU: Intel Xeon L5520 2.27GHz (16 core)
• RAM: 16 GB
• Storage: SSD INTEL SSDSC2BB480G4

SOFTWARE

• Magento Community 1.8.1.0
• Magento Sample Data 1.2.0
• Apache 2.2.22
• PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
• MySQL 5.5.35

For testing Apache jMeter with the following scenario was used:

1. A visitor goes to the website home page;
2. Searches for “shirt” in the search field;
3. The result is filtered by price ($0.00 — $99.99);
4. After that the result is filtered by color (white color);
5. As a result, only 1 Product is left, the user opens its page;
6. The Product is added to the cart at the Product Page;
7. After that, the user goes to the Category Page «Electronics / Cameras / Digital Cameras»;
8. The user reloads the Category Page «Electronics / Cameras / Digital Cameras»;
9. At the Category Page camera Canon PowerShot A630 is added for comparison;
10. At the Category Page camera Kodak EasyShare C530 is added for comparison;
11. The Products Comparison Page is opened;
12. From the Products Comparison Page, the camera Canon PowerShot A630 is added to the cart.

The test was performed for 10, 20, 30, 40, and 50 simultaneous visitors in the variants “Only Magento cache on” and “Magento cache and FPC on”. Before every launch, the Magento cache was cleared and a test for 10 simultaneous visitors was launched, but its results were not taken into account (this is done to prevent a difference in the results connected with the increasing amount of cache during the successive test launches).

In order to compare all possible variants, we have additionally tested a variant when Magento has caching turned off. The most impatient readers can see its result in the diagram below (the result for the Product Page with 50 simultaneous visitors).

To prevent the influence of the data transfer speed on the Internet the test was launched in the mode “Remote start”. In this case in the server where the tested Magento is installed Apache jMeter is launched in the server mode, which receives the test from the managing client, performs it, and sends the results back to the client. The client part of Apache jMeter is launched on the desktop, it manages only the server part and shows the results. This way we have prevented the influence of the Internet channel load on the test results.

Every test was performed 10 times to provide the correct calculation of the average value of the test results. This was for 50 simultaneous users jMeter will perform the test scenario 500 times. The test scenario contains 12 requests to the server (download of CSS page resources/JS/images are not taken into account in the test results), this way during the test with 50 simultaneous visitors 6000 requests are sent to the server.

The test results in ms come in the table as follows:

You have probably noticed that Category Page opens twice. This can be put to the fact that when a product is added to the cart the status of the dynamic blocks, the content of which should be reloaded during the next request to them, is changed. In this case, the first Page impression from the cache is slower than the successive ones. To see the difference in the load speed (when the dynamic blocks are already reloaded) the test goes to the Category page one more time. And the page is downloaded completely from the cache.

According to the table data, the following diagrams have been created for a better visual assessment of the difference in the server reply speed.

The last diagram shows the average results in all the mentioned categories. Usage of the Full Page Cache Magento extension lets us on average increase the speed by 5 times. In some tests, the difference is even bigger.

But what makes our extension better than Varnish Cache and whether it is better at all?

Magento Varnish Cache vs. Full Page Cache: comparison test

If you need to speed up your Magento store, just ‘faster’ is not enough to describe the performance of your website. A fair contest of two Magento cache extensions with detailed speed results is at your service.

→ Check how to configure varnish cache in Magento 2

TEST CONFIGURATION

We have changed the test configuration a little bit in comparison to our previous Full Page Cache test.

Hardware

• CPU: Intel Xeon L5520 2.27GHz (16 core)
• RAM: 16 GB
• Storage: SSD INTEL SSDSC2BB480G4

Software

• Magento Community 1.9.0.0
• Magento Sample Data 1.9.0.0
• Apache 2.2.22
• Varnish 3.0.2
• PHP 5.4.4 (mod_php + Zend OpCache 7.0.3)
• MySQL 5.5.35

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. It is installed in front of any server that speaks HTTP.

Varnish Cache itself is not designed for working with Magento directly, so we took the Nexcess Turpentine module for Magento and Varnish integration for the speed test. Based on the recommendations of the Turpentine module producer, we added «-p esi_syntax=0×2 -p cli_buffer=16384» to Varnish startup options. As a result, Varnish startup options looked as follows:

DAEMON_OPTS=”-a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m -p esi_syntax=0x2 -p cli_buffer=16384″

To prevent Apache and Varnish conflict Apache was switched to port 8080.

For testing we used the list of 14 URLs of Category and Product pages: http://our-test-site/

• http://our-test-site/women.html
• http://our-test-site/women/new-arrivals.html
• http://our-test-site/women/new-arrivals/tori-tank-460.html
• http://our-test-site/women/new-arrivals/elizabeth-knit-top-484.html
• http://our-test-site/women/new-arrivals/lafayette-convertible-dress.html
• http://our-test-site/women/tops-blouses.html
• http://our-test-site/women/tops-blouses/nolita-cami-484.html
• http://our-test-site/women/tops-blouses/black-nolita-cami.html
• http://our-test-site/women/pants-denim.html
• http://our-test-site/women/pants-denim/dumbo-boyfriend-jean.html
• http://our-test-site/women/dresses-skirts.html
• http://our-test-site/women/dresses-skirts/essex-pencil-skirt-527.html
• http://our-test-site/women/dresses-skirts/ludlow-sheath-dress.html

Before every test launch, the Magento cache was cleared and a single visit to all URLs was performed to “warm the cache up” before taking the test results. After the warm-up test for 10, 20, 30, 40, and 50 simultaneous visitors was launched.

Comparison test results (in ms) are in the table as follows:

To make the results easier to comprehend and let you vividly see the page load speed each Magento extension provides we have built some diagrams.

As you can see, the speed of Full Page Cache exceeds the combination of Varnish Cache and Turpentine in 2-4 times. The biggest difference can be seen during the first visit of the website – at this moment both caches create a session for the user and have to initialize Magento. Since Full Page Cache initializes at the early stage, it can give the cached result quickly and Turpentine uses a lot of time in vain on extra initialization. This is an important point, as its impact deals with first visits and users’ impression of your website.

According to Kissmetrics, a 1-second delay in website load speed can decrease conversions by 7%. You might even want to count the money you lose using a Magento cache extension providing worse page load time.

This way Amasty Team recommends you using the Full Page Cache Magento extension for providing the minimal page load time possible.