One of the most important buzz features of Magento 2 is the built-in full page cache. What is more, Magento 2 FPC supports Varnish as backend. While Magento 2 is declared to be a much faster platform in comparison with the first version, we decided to check Magento 1 and 2 performance with various cache settings to see which combination gives the best results.
We set up two servers, with 6 CPU cores and 8GB RAM each. Both servers are using SSD for data storage and correspond to Amasty recommendations regarding server configuration for Magento, as well as minimal recommended Magento system requirements.
Server setup
#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30, MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6
#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14, MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2
In Magento 2, the full page cache settings are changed to use Varnish as backend when needed.
Speed test tools
In the previous articles we performed various tests and used our own Apache Jmeter scenarios.
People asked us why we weren’t using Magento Performance Kit for testing. The thing is, Magento Performance Toolkit does not support Magento CE 1.9.2.2 and is designed for Magento EE, so it isn’t suitable for our case.
Starting from Magento 2.x, the toolkit supports CE, but it is different from Magento 1.x version, and it would be incorrect to compare them. What is more, the toolkit measures only the server response time without taking AJAX into account, which is actively used in Magento 2.x.
To correctly test a dynamic website on Magento 2.x, we chose Selenium Webdriver plugin for Apache JMeter. This plugin allows using a fully functional browser, such as Google Chrome, for performance tests. In this case, we are able to measure the full page loading time, including AJAX.
Performance test scenario
The tests were performed using the following scenarios:
- The guest visits the home page
- The guest goes to the category
- The category is filtered by color
- The category is filtered by material/purpose
- Search by ‘shirt’ keyword is performed
- The guest goes to the product page
- The product is added to cart
Each test has been performed 100 times, and then we counted the average value for each stage. As the web driver uses the real browser, the number of simultaneous tests for a single computer is limited by the number of CPU cores. I used Intel Core i5 with 4 cores and ran only 5 simultaneous tests so the CPU performance wouldn’t influence the test results.
Please note that this experiment is not a load test. It shows the time difference between how fast the page is loaded in the browser with full page cache variants and without it.
In our previous tests we measured the server response time, and today we’re focusing on page load time in the browser. This time consists of server response generation, network data transfer, page rendering by the browser and AJAX queries execution time.
This kind of test is very close to what a user actually sees on your website. In reality, the user sees the page before it fully loads, and the data we have received in this performance test can be treated as the worst case scenario. The results will also be similar to what webpagetest.org or pingdom.com will show.
Performance test results
Magento 1.9.2.2, no FPC:
Magento 1.9.2.2 , Amasty FPC on:
Magento 2.0.0, no FPC:
Magento 2.0.0, FPC on:
Magento 2.0.0 , FPC (Varnish) on:
We use the average value here ( see ‘Average’ column). The difference between sample data for Magento 1.9.2.2 and Magento 2.0.0 can be neglected, because the browser uses its own local cache after the first query.
Let’s see the difference between page load time for Magento 1.9.2.2 and Magento 2.0.0 with FPC off. Magento 1.9.2.2 doesn’t have a built-in FPC, so we count it as ‘off’ (all results in ms).
Then, we compared different variants of FPC usage in Magento 2.0.0:
- FPC off
- FPC on, Varnish off
- FPC on, Varnish on
FPC with Varnish shows up to 30% increase of page load speed if compared to the variant with FPC off, and around 3% increase if compared to the variant with FPC on, but no Varnish.
Thus, FPC in Magento 2.0.0 gives at least 27% increase for page load in the browser. It’s a great result for a free built-in feature!
Now, let’s compare the page load speed in Magento 1.9.2.2 + Amasty FPC and in Magento 2.0.0 + FPC + Varnish (ms).
Takeaway
FPC for Magento 2 significantly increases page load in the browser. The difference between Varnish on/off combinations is insignificant with low load and comes as 3%, as our test has been using only 5 simultaneous users.
With higher load, Varnish will give you a bigger advantage by lowering the server load, but with smaller traffic the margin is quite thin.
Do you have any questions about the performance test results? We’re happy to answer!
21.02.2016 update
A couple of weeks ago we published this article to compare Magento 1 and Magento 2 performance. We were contacted by Magento representatives regarding the results and the settings that can influence the speed of the installation. It’s great that we have real communication here, and it’s a unique experience for our team to work with the new product and to receive info straight from the source.
At the same time doing maximal optimization only for Magento 2 seemed uneven, so we did our best to apply the improved settings to Magento 1 test installation as well.
Here’s the configuration of the test servers we used:
#1: CPU 6 cores, RAM 8G, SSD storage, Apache 2.2.22, PHP 5.5.30 (Zend Opcache), MySQL 5.5.46 + Magento 1.9.2.2 + Sample data 1.9.1.0 + Amasty Full Page Cache 1.5.6
#2: CPU 6 cores, RAM 8G, SSD storage, Apache 2.4.10, PHP 5.6.14 (Zend Opcache), MySQL 5.6.25 + Magento 2.0.0 + Sample data + Varnish 4.0.2
Also, on both servers JavaScript, CSS, XML, etc compression was turned on, and Expires were added. These settings guarantee that the browser receives the content fast and caches it. We ran 5 simultaneous tests at each server and performed 100 iterations total.
Magento 1.9.2.2
For Magento 1.9.2.2 we turned compilation on at System / Tools / Compilation. Minification and merging of JavaScript/CSS were off, because the server minifies the data while delivering it, and Expires is set up as well. In case of Expires, the browser downloads CSS/JavaScript files only once and uses them from the cache afterwards.
Magento 1.9.2.2 no FPC/Amasty FPC performance comparison (ms)
Here FPC gives 50% more speed everywhere but the Add to Cart step. I wanted to point out once again that now we are measuring page load time, not server response time. It is the time that the browser needs to receive the data, render the page and execute javascript scenarios.
Magento 2.0.0 new performance tests
For Magento 2, we applied the recommended list of optimizations:
- php bin/magento setup:static-content:deploy
- php bin/magento setup:di:compile
- Install the production mode for .htaccess (SetEnv MAGE_MODE production).
- Turn Javascript Bundling on (Stores / Configuration / Advanced / Developer / JavaScript Settings / Enable Javascript Bundling).
Still, we didn’t use JavaScript/CSS minification and merging, because it doesn’t help with using compression while transferring the data and Expires, which was set up correctly.
Magento 2 no FPC/Varnish FPC performance tests (ms)
Magento 1.9.2.2 vs Magento 2.0.0 performance comparison
Takeaway:
- Using Magento 2 with the turned off FPC is a bad idea.
- Production mode is much faster than Developer mode.
- Magento 2 doesn’t cache search results. We tried to turn on query_cache in MySQL, but that didn’t influence the result at all. We suppose that using Solr instead of MySQL would give a better result, but Solr for Catalog Search is available only for the enterprise version of Magento 2.
Hello!
Have you tried the similar test with installed bunch of 3rd extensions on the stores?
Thanks.
Hi there, thanks for the question and your interest. No, we didn’t do such a test, because we believe these results are applicable for the particular store and can be used for optimization of this very installation. Each store has its own set of extensions of different quality, functionality and by different vendors. Even a single poor quality extension can slow down the whole installation, and a bunch of well-developed modules will hardly give any changes if compared to the clean installation. So, the test you’re talking about is more about extensions, than Magento itself, if we’re speaking about such… Read more »
Hi, As you are using WebDriver for benchmarks, you need to perform several additional actions before measuring performance for Magento2. Here is a list of actions, that should be performed in order to switch M2 instance into production (most performance) mode. I highly recommend to pass this flow prior to testing: 1. Activating of client side performance features (is not the reason of slow server response, but recommended) a. JS bundling b. JS minification c. CSS minification d. HTML minification 2. Running static content deployment tool (to avoid real time less compilation, themes fallback, etc) php bin/magento setup:static-content:deploy 3. Running… Read more »
Hey Dmytro,
thanks so much for your attention and for your suggestions. We will definitely test it once again and will post an update on the results!
It’s funny how much stuff you need to do in order to make a humpy M2 running smoother … 10 steps … well …
I’d like to mention that you should turn on these features in M1 as well:
Compiler Mode
JS/CSS merger
Running all pages at least twice to measure the cached performance
Reindex everything.
Rico
Thanks for your interest, Rico. We will also take these things into account when performing tests once again.
Happy holidays!
Great results! The mode change really makes a huge difference.
Yes, and we’re sure that in the future with the new updates the results will get even better.
This enticed me to buy Amasty FPC and try it in my Magento 1.9.2.3 site. 😀
Ha-ha, very happy to hear it, Bryan =)
Results between magento 1.9.2 and magento 2 discourage me to upgrade some of the shops i maintain…
If we read between the lines, don’t expect to improve performance of your magento site if you don’t use Varnish…. compare to 1.9.x versions…
it’s the point which can make switch off to other ecommerce solutions…
Hey Aurélien,
thanks for your comment!
I think that it’s too early to judge – because the tests reflect the current state of the platform. Meanwhile, Magento 2 is and will be updated, and I’m sure that the updates will be affecting performance, too.
While ver 1 has been on the market for years, and the developers had plenty of time to polish everything.
We’ll be keeping our eye on the M2 updates, maybe even doing more tests for some future versions.
How about enabling the External Full Page Cache in Magento 1? Does that help speed up Magento 1 more? The default one at System > Configuration > System > External Full Page Cache > Zend Full Page Cache. 😀
Hi there and thanks for your comment!
Zend Pagecache is a component of Zend server, and it’s a paid solution. At this moment we’re not likely to test it because honestly we have never met a single client using this tool =)
I’m using Vagrant to make coding locally, it is very slow. Any tips that can help me streamline my tests?
Setting the current VM:
* 2GB memory;
* Two processor cores;
* PHP 5.6
* MySQL 5.6
I used the script “MySQL Tunner” to see if improved income, and still remains too reading.
Hey Israel,
thanks for your question.
Which Vagrant backend are you using (virtualbox, vmware, etc)? Also, did you turn on the hardware visualization in BIOS?
I’m asking because the second thing is extremely important for the virtual machine performance.
Thank you for Ksenia answer, I am using Parallels on these options BIOS, I will give a searched for using the MAC.
Unfortunately, we can’t help you with MAC settings because we haven’t enough experience with it. But, depending on the backend type, you can play with your disk caching for better results.
Thanks ! =)
Why does you not use php 7 for Magento 2?
Hey Andrii, thanks for your question.
At the time the article was in process, PHP 7 was still pretty much ‘raw’. Now 7.0.4 is out, where ZendOpcache issues were fixed, and we don’t know if there will be any critical fixes in the nearest future.
Yep, we also know that Magento 2 is faster on PHP 7 (as for Magento 1, too). We’re looking at the situation and the PHP 7 updates and are considering extending this topic in the future blog posts.
Thanks for answer, will wait for future posts.
I have more questions.
Pages response time for page in varnish is ~ 10ms. And this time should not depend on Magento 2 mode because request is not processed by php. Can you explain why your measurements show a different behavior?
Sure!
10ms are actually results of synthetic tests, when both the test and Varnish are run on the same machine. Our test is not synthetic, we performed it under real life conditions. And the reality is that 30-50ms are added to each query because of the Internet connection.
Also, in our test the result is the page load time in the browser, not only the Varnish response.
If you have more questions about this topic, we will be happy to answer them as well!
Magento answered by a good april Joke: “no no, we assure you add to cart is faster on Magento2”.
https://magento.com/resources/white-paper-magento-2-site-performance-and-scalability-optimizations
It’s just manipulation of numbers, and way of testing things… It’s just a shame.
Long live magento 1.x
Hey there, thanks for sharing the link. We haven’t scanned the doc yet, but we will do this in the nearest time.
Hi,
Your article has not been updated for a long time.
Maybe you can test it with php7 Magento 1.9.3.1 and Magento 2.1.2
Or Magento EE 1.14.3.1, Magento-EE-2.0.10, Magento-EE-2.1.2.
If you need Magento EE contact me to e-mail.
Hey there and many thanks for your interest.
Indeed, we have a plan to test those versions in the nearest future. Not sure with the date, but most probably it’ll be in 2016. Thanks for your suggestion about the EE – I’ll check with our admin who will write the post, and if he has plans on including EE on the list, I’ll definitely contact you. Cheers!
Do you have a comparison between Magento CE 1.9.3.1 vs Magento CE 2.1.3. Maybe the results are now different. ?
Hey Jehzeel,
thanks for your interest. We will be posting some new tests right after the New Year. Stay tuned!
Are those new tests still coming?
Matt, feel free to see the post we’ve recently published: https://amasty.com/blog/magento-1-2-performance-speed-php-5-php-7/
I thought one of the benefits of Magento 2 was speed? how come it is slow! also, I do not like to rely fully on FPC for fast page loads. The application itself should be fast too.
Hey Joshua, this test was performed more than a year ago. I’m pretty sure right now Magento 2 is much faster than before. We’re planning to do the similar tests, and also you’re welcome to check this post as well:
https://amasty.com/blog/magento-1-2-performance-speed-php-5-php-7/
Thanks for sharing this article, Your efforts are shown here about your research and findings to get the best result to your readers, I was confused in selecting version of Magento because many said to go with Magento 1 and many said to go with Magento 2, but after reading this post and another article (https://www.cloudways.com/blog/magento-1-vs-magento-2-survey/), I have now selected to go with Magento 2 .