One of the most important buzz features of Magento 2 is the built-in full page cache. What is more, Magento 2 FPC supports Varnish as backend. While Magento 2 is declared to be a much faster platform in comparison with the first version, we decided to check Magento 2 vs Magento 1 performance with various cache settings to see which combination gives the best results.
We set up two servers, with 6 CPU cores and 8GB RAM each. Both servers are using SSD for data storage and correspond to Amasty recommendations regarding server configuration for Magento, as well as minimal recommended Magento system requirements.
#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30, MySQL 5.5.46 + Magento 184.108.40.206 + Sample data 220.127.116.11 + Amasty Full Page Cache 1.5.6
#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14, MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2
In Magento 2, the full page cache settings are changed to use Varnish as backend when needed.
Speed test tools
In the previous articles we performed various tests and used our own Apache Jmeter scenarios.
People asked us why we weren’t using Magento Performance Kit for testing. The thing is, Magento Performance Toolkit does not support Magento CE 18.104.22.168 and is designed for Magento EE, so it isn’t suitable for our case.
Starting from Magento 2.x, the toolkit supports CE, but it is different from Magento 1.x version, and it would be incorrect to compare them. What is more, the toolkit measures only the server response time without taking AJAX into account, which is actively used in Magento 2.x.
To correctly test a dynamic website on Magento 2.x, we chose Selenium Webdriver plugin for Apache JMeter. This plugin allows using a fully functional browser, such as Google Chrome, for performance tests. In this case, we are able to measure the full page loading time, including AJAX.
Performance test scenario
The tests were performed using the following scenarios:
- The guest visits the home page
- The guest goes to the category
- The category is filtered by color
- The category is filtered by material/purpose
- Search by ‘shirt’ keyword is performed
- The guest goes to the product page
- The product is added to cart
Each test has been performed 100 times, and then we counted the average value for each stage. As the web driver uses the real browser, the number of simultaneous tests for a single computer is limited by the number of CPU cores. I used Intel Core i5 with 4 cores and ran only 5 simultaneous tests so the CPU performance wouldn’t influence the test results.
Please note that this experiment is not a load test. It shows the time difference between how fast the page is loaded in the browser with full page cache variants and without it.
In our previous tests we measured the server response time, and today we’re focusing on page load time in the browser. This time consists of server response generation, network data transfer, page rendering by the browser and AJAX queries execution time.
This kind of test is very close to what a user actually sees on your website. In reality, the user sees the page before it fully loads, and the data we have received in this performance test can be treated as the worst case scenario. The results will also be similar to what webpagetest.org or pingdom.com will show.
Performance test results
Magento 22.214.171.124, no FPC:
Magento 126.96.36.199 , Amasty FPC on:
Magento 2.0.0, no FPC:
Magento 2.0.0, FPC on:
Magento 2.0.0 , FPC (Varnish) on:
We use the average value here ( see ‘Average’ column). The difference between sample data for Magento 188.8.131.52 and Magento 2.0.0 can be neglected, because the browser uses its own local cache after the first query.
Let’s see the difference between page load time for Magento 184.108.40.206 and Magento 2.0.0 with FPC off. Magento 220.127.116.11 doesn’t have a built-in FPC, so we count it as ‘off’ (all results in ms).
Then, we compared different variants of FPC usage in Magento 2.0.0:
- FPC off
- FPC on, Varnish off
- FPC on, Varnish on
FPC with Varnish shows up to 30% increase of page load speed if compared to the variant with FPC off, and around 3% increase if compared to the variant with FPC on, but no Varnish.
Thus, FPC in Magento 2.0.0 gives at least 27% increase for page load in the browser. It’s a great result for a free built-in feature!
Now, let’s compare the page load speed in Magento 18.104.22.168 + Amasty FPC and in Magento 2.0.0 + FPC + Varnish (ms).
FPC for Magento 2 significantly increases page load in the browser. The difference between Varnish on/off combinations is insignificant with low load and comes as 3%, as our test has been using only 5 simultaneous users.
With higher load, Varnish will give you a bigger advantage by lowering the server load, but with smaller traffic the margin is quite thin.
Do you have any questions about the performance test results? We’re happy to answer!
A couple of weeks ago we published this article to compare Magento 1 and Magento 2 performance. We were contacted by Magento representatives regarding the results and the settings that can influence the speed of the installation. It’s great that we have real communication here, and it’s a unique experience for our team to work with the new product and to receive info straight from the source.
At the same time doing maximal optimization only for Magento 2 seemed uneven, so we did our best to apply the improved settings to Magento 1 test installation as well.
Here’s the configuration of the test servers we used:
#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30 (Zend Opcache), MySQL 5.5.46 + Magento 22.214.171.124 + Sample data 126.96.36.199 + Amasty Full Page Cache 1.5.6
#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14 (Zend Opcache), MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2
Magento 188.8.131.52 no FPC/Amasty FPC performance comparison (ms)
Magento 2.0.0 new performance tests
For Magento 2, we applied the recommended list of optimizations:
- php bin/magento setup:static-content:deploy
- php bin/magento setup:di:compile
- Install the production mode for .htaccess (SetEnv MAGE_MODE production).
Magento 2 no FPC/Varnish FPC performance tests (ms)
Magento 184.108.40.206 vs Magento 2.0.0 performance comparison
- Using Magento 2 with the turned off FPC is a bad idea.
- Production mode is much faster than Developer mode.
- Magento 2 doesn’t cache search results. We tried to turn on query_cache in MySQL, but that didn’t influence the result at all. We suppose that using Solr instead of MySQL would give a better result, but Solr for Catalog Search is available only for the enterprise version of Magento 2.