Magento 2 vs Magento 2 performance comparison: speed test results

 

One of the most important buzz features of Magento 2 is the built-in full page cache. What is more, Magento 2 FPC supports Varnish as backend. While Magento 2 is declared to be a much faster platform in comparison with the first version, we decided to check Magento 1 and 2 performance with various cache settings to see which combination gives the best results.

In spite of all those vanilla Magento 2 features like FPC and Varnish – the performance won’t be perfect without magic sorcery. We won’t offer you to speed up your website with a magic wand but with our new Magento 2 Speed Optimization extension. Asynchronous indexing; HTML, CSS, JS minification; lazy load and images optimization can help you gain higher scores in Google Page Speed Optimizer in just a few days. We guarantee that moving your JS to footer will never be a problem anymore. Render your pages faster, attract visitors with speedy performance.

We set up two servers, with 6 CPU cores and 8GB RAM each. Both servers are using SSD for data storage and correspond to Amasty recommendations regarding server configuration for Magento, as well as minimal recommended Magento system requirements.

Server setup

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30, MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14, MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

In Magento 2, the full page cache settings are changed to use Varnish as backend when needed.

Speed test tools

In the previous articles we performed various tests and used our own Apache Jmeter scenarios.

People asked us why we weren’t using Magento Performance Kit for testing. The thing is, Magento Performance Toolkit does not support Magento CE 1.9.2.2 and is designed for Magento EE, so it isn’t suitable for our case.

Starting from Magento 2.x, the toolkit supports CE, but it is different from Magento 1.x version, and it would be incorrect to compare them. What is more, the toolkit measures only the server response time without taking AJAX into account, which is actively used in Magento 2.x.

To correctly test a dynamic website on Magento 2.x, we chose Selenium Webdriver plugin for Apache JMeter. This plugin allows using a fully functional browser, such as Google Chrome, for performance tests. In this case, we are able to measure the full page loading time, including AJAX.

Performance test scenario

The tests were performed using the following scenarios:

  • The guest visits the home page
  • The guest goes to the category
  • The category is filtered by color
  • The category is filtered by material/purpose
  • Search by ‘shirt’ keyword is performed
  • The guest goes to the product page
  • The product is added to cart

Each test has been performed 100 times, and then we counted the average value for each stage. As the web driver uses the real browser, the number of simultaneous tests for a single computer is limited by the number of CPU cores. I used Intel Core i5 with 4 cores and ran only 5 simultaneous tests so the CPU performance wouldn’t influence the test results.

Please note that this experiment is not a load test. It shows the time difference between how fast the page is loaded in the browser with full page cache variants and without it.

In our previous tests we measured the server response time, and today we’re focusing on page load time in the browser. This time consists of server response generation, network data transfer, page rendering by the browser and AJAX queries execution time.

This kind of test is very close to what a user actually sees on your website. In reality, the user sees the page before it fully loads, and the data we have received in this performance test can be treated as the worst case scenario. The results will also be similar to what webpagetest.org or pingdom.com will show.

Performance test results

Magento 1.9.2.2, no FPC:

Performance test results for Magento 1.9 without full page cache

Magento 1.9.2.2 , Amasty FPC on:

Magento 1.9 + Amasty FPC page load time

 

Magento 2.0.0, no FPC:

Magento 2 with disabled FPC speed test results

Magento 2.0.0, FPC on:

Magento 2 + FPC performance results (page load speed)

Magento 2.0.0 , FPC (Varnish) on:

Magento 2 + FPC (Varnish) performance test results

We use the average value here ( see ‘Average’ column). The difference between sample data for Magento 1.9.2.2 and Magento 2.0.0 can be neglected, because the browser uses its own local cache after the first query.

Let’s see the difference between page load time for Magento 1.9.2.2 and Magento 2.0.0 with FPC off. Magento 1.9.2.2 doesn’t have a built-in FPC, so we count it as ‘off’ (all results in ms).

Magento 1.9 and Magento 2 performance test result without FPC

 

Then, we compared different variants of FPC usage in Magento 2.0.0:

  • FPC off
  • FPC on, Varnish off
  • FPC on, Varnish on

Magento 2 page load performance results

FPC with Varnish shows up to 30% increase of page load speed if compared to the variant with FPC off, and around 3% increase if compared to the variant with FPC on, but no Varnish.

Thus, FPC in Magento 2.0.0 gives at least 27% increase for page load in the browser. It’s a great result for a free built-in feature!

Now, let’s compare the page load speed in Magento 1.9.2.2 + Amasty FPC and in Magento 2.0.0 + FPC + Varnish (ms).

Magento 1.9 and Magento 2 performance test result with FPC

Takeaway

FPC for Magento 2 significantly increases page load in the browser. The difference between Varnish on/off combinations is insignificant with low load and comes as 3%, as our test has been using only 5 simultaneous users.

With higher load, Varnish will give you a bigger advantage by lowering the server load, but with smaller traffic the margin is quite thin.

Do you have any questions about the performance test results? We’re happy to answer!

21.02.2016 update

A couple of weeks ago we published this article to compare Magento 1 and Magento 2 performance. We were contacted by Magento representatives regarding the results and the settings that can influence the speed of the installation. It’s great that we have real communication here, and it’s a unique experience for our team to work with the new product and to receive info straight from the source.

At the same time doing maximal optimization only for Magento 2 seemed uneven, so we did our best to apply the improved settings to Magento 1 test installation as well.

Here’s the configuration of the test servers we used:

#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30 (Zend Opcache), MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6

#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14 (Zend Opcache), MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2

Also, on both servers JavaScript, CSS, XML, etc compression was turned on, and Expires were added. These settings guarantee that the browser receives the content fast and caches it. We ran 5 simultaneous tests at each server and performed 100 iterations total.

Magento 1.9.2.2

For Magento 1.9.2.2 we turned compilation on at System / Tools / Compilation. Minification and merging of JavaScript/CSS were off, because the server minifies the data while delivering it, and Expires is set up as well. In case of Expires, the browser downloads CSS/JavaScript files only once and uses them from the cache afterwards.

Magento 1.9.2.2 no FPC performance

Magento 1.9.2.2 FPC performance

Magento 1.9.2.2 no FPC/Amasty FPC performance comparison (ms)

Magento 1.9.2.2 performance chart

Here FPC gives 50% more speed everywhere but the Add to Cart step. I wanted to point out once again that now we are measuring page load time, not server response time. It is the time that the browser needs to receive the data, render the page and execute javascript scenarios.

Magento 2.0.0 new performance tests

For Magento 2, we applied the recommended list of optimizations:

  • php bin/magento setup:static-content:deploy
  • php bin/magento setup:di:compile
  • Install the production mode for .htaccess (SetEnv MAGE_MODE production).
  • Turn Javascript Bundling on (Stores / Configuration / Advanced / Developer / JavaScript Settings / Enable Javascript Bundling).

Still, we didn’t use JavaScript/CSS minification and merging, because it doesn’t help with using compression while transferring the data and Expires, which was set up correctly.

Magento 2 no FPC performance test

Magento 2 in production mode with Varnish FPC performance test

Magento 2 developer mode with Varnish FPC performance test results

Magento 2 no FPC/Varnish FPC performance tests (ms)

Magento 2 no FPC vs Varnish FPC production mode performance tests

Magento 1.9.2.2 vs Magento 2.0.0 performance comparison

Magento 1.9 vs Magento 2 performance comparison

Takeaway:

  • Using Magento 2 with the turned off FPC is a bad idea.
  • Production mode is much faster than Developer mode.
  • Magento 2 doesn’t cache search results. We tried to turn on query_cache in MySQL, but that didn’t influence the result at all. We suppose that using Solr instead of MySQL would give a better result, but Solr for Catalog Search is available only for the enterprise version of Magento 2.