This September, Amasty took part in Meet Magento NYC. Valeria Shevtsova, Head of QA department, explained how MFTF works and how we use it at Amasty. And today, we share her speech and presentation with you.
MFTF stands for Magento Functional Testing Framework.
MFTF is an open-source cross-platform solution. MFTF enables you to quickly develop functional tests for the Magento application.
And now, we will discuss the reasons why the Magento Functional Testing Framework is suited for tests automation, important phases that should not be omitted in the course of tests development, specific features of automation test-cases, some of the best practices and lifehacks.
Note: Read our step-by-step guide to learn how to make a smooth start with MFTF.
Reasons to apply MFTF to your project
Here are the reasons to automate testing via MFTF:
- to reduce testing time;
- this is a tool that works from scratch and suits to different platform;
- long-term projects development;
- multiple product versions support;
- to test products on different Magento versions and detect version-specific bugs etc.
We have 250+ extensions, hence regression testing, as part of updating them, is time-consuming. We decided to accelerate this process and develop MFTF tests to avoid delaying releases.
Here is an example: manual regression testing of the Free Gift for Magento 2 took 7 hours, the time has been reduced to 2 hours (excluding test execution time) as soon as we covered it with autotests. Even when half of the autotests has been ready, we already have been running them and seeing the results. For the larger Improved Layered Navigation extension, the regression testing time has been decreased from 50 to 20 hours, though autotest development is not completed yet.
Moreover, you can use MFTF tests out of the box and skip some of the manual testing without developing own tests. For instance, you can test such important features as user registration; products, categories, attributes creation; product search; check-out, and many more using default tests. Of course, if changes have been made and the functionality differs from the default one, some of the tests may crash. But, selecting default tests that function correctly and using them as part of the regression testing may come in useful. You can do it even if you don’t have an automation QA engineer in your team since MFTF is .xml-based and easy to use.
Therefore, the main advantages of the MFTF are:
- The availability of ready-to-use tests;
- The simplicity of mastering;
- Scalability.
Additional tests for specific functionality may be written, as MFTF enables this. We at Amasty are developing tests to cover the functionality of our extensions.
Test development strategy
If you’ve never done this before, it is not always clear what to start from. By trial and error, we have found out that certain steps are absolute must-haves.
- Project profitability calculation. That means ROI calculation. Updates frequency and time required to develop and support tests are worth considering. However, bear in mind that autotests not only save manual testing budgets but are also winning in terms of reliability, the time required to pass the quality assurance phase (which results in faster updates rollout), and “human factor” elimination.
- Definition of what exactly you need to automate. There is a certain consensus that regression is the best candidate for automation, thus we decided to start with it. In some cases, these may be the important functionality blocks or detected places that accumulate defects. It is difficult to overestimate the importance of the planning step, because the better the input data, the better the final result.
- Test-Case composition. Go to the next part to read the full description of this step.
Choose the order in which to develop the tests. For example, we split extension tests into “Smoke”, “Regression”, and “Nice to have” groups and began with the first group. - Tests development.
- Review.
- Include tests into extensions.
- Analysis. It is vital to track how often tests are used and whether any bugs are found. The result of the analysis may be compared to the calculations from step 1.
- We move on to refinement and maintenance based upon the information obtained in the course of the activities that are part of the previous step.
Specific features of the test-cases designed for auto-tests
As we have previously mentioned, test-cases designed for automated testing services are an entirely different thing that is definitely worth paying attention to.
We choose test-cases because they are quite detailed and therefore are suitable for automation.
Other types of test documentation are way too unspecified. With the hands-on Test Survey, an automation engineer, no matter how experienced they, risks missing important checks. Hence the test won’t attain its main target and detect the defects.
Let’s consider the most important point of test-cases.
Recurrent checks and steps
Test-cases must contain as few recurrent checks and steps as possible. Indeed, if there was a bug in a product, several autotests would crash, not just one. For sure, making all steps unique is barely possible.
Preconditions
“Preconditions” are a list of actions that result in a status that is suitable for running the main test. While testing manually, the quality assurance engineer barely thinks of test execution preconditions. However, they are crucial for automation.
Let's analyze an example. Say, we have a test-case on product editing. We would select any existing product and edit it when testing manually. An auto-test should not act that way. We are risking to break the co-dependence of tests! If one test edits a product, while the other will try to use it the same way, it can crash. The solution is as follows: it should be created via an autotest! It is important to understand that adding such data is not the test itself, these are just secondary actions. Preconditions are important. They make tests self-sufficient, stable, independent and reduce test runtime.
Test-case optimization
It is important to use the shortest path of the test-case. Transition through pages, tabs or buttons clicks, dialog opening or closing takes time. Minimize these steps to accelerate the test.
One step should not be referenced to another step. For instance, you shouldn't write: “Repeat steps 1-10 for a configurable product” in step 11. This check is either not needed at all, or we include it into a separate test for this type of product.
Connection between autotest and a test-case
The next important point is the correspondence of an autotest with a test-case. Each step of the autotest should be the same as in the test-case. This is the only way to examine the results of autotests, find bugs, and maintain tests.
Size optimization
The pretty complicated task is the optimization of a test-case size.
What size should a test-case for automation be? There is no definite answer.
We stick to the following approach:
- One test means one significant check. For instance, check whether the cart price rule can be edited.
- You can add a few less significant checks after a significant check, providing they use the same preconditions and do not increase the test runtime significantly.
- In case there is an indivisible sequence of steps, they are taken out to the action group.
MFTF best practices
As devdocs state:
Use an action group to wrap a set of actions to reuse them multiple times. Use an extension when a test or action group needs to be repeated with the exception of a few steps.
This is a fundamental principle when developing tests!
Some other tips:
- Independence. This means that the result of one test does not affect the result of the others. Even if the tests are performed simultaneously (for instance, they are running in different browsers), they should not interfere with each other.
- A Magento test should delete critical data after it has been executed, for it may interfere with the subsequent executions of other tests or this same test (this does not always mean deletion as such, sometimes the field that was mandatory as part of the test may be set optional).
- All the data that a Magento test creates must be unique when it comes to IDs, URLs, keys, names, etc. for the same purpose.
- Standard Magento Action Groups should not be neglected.
- Self-sufficiency. Autotests should not depend on any data that may disappear or change over time or at any given moment.
- Stability. If there are no bugs in the application, the test should always be completed. This seems to be obvious, but not always easily achieved.
Teamwork
When several people collaborate on an automation project, the correct organization of teamwork becomes even more crucial. We have undertaken the following steps to achieve it:
- We’ve introduced Git Workflow. You can find in our presentation how it looks like. This helped fostered collaboration a lot, especially when it comes to several people developing tests for the same extension.
- We use the same Action Groups not just within the same extension, but for different extensions as well. When Action Groups, which may be common for several or all of our extensions, increase in number, we placed them into Base.
- Larger extensions (this may be a large project or a store) should be divided into logical parts and automated consequently part by part. Several people may write tests for different parts at the same time. That’s what we did when working on our ILN extension.
Lifehacks
1. Committing to the same value options by everyone working on tests for the “group” annotation seems to be an unsophisticated yet a very useful approach for maintaining order in tests and running them later. Therefore, we have compiled a list of possible values and use them in each test (there may be several groups in one test):
- A common group for all tests developed within the company; e.g., the Amasty group for all our tests in order to conveniently run everything together;
- A group that matches the name of the extension/project, that combines all the tests of this extension (for instance, Orderattr for the order of attributes);
- A group corresponding to the functionality of Magento (optional). For instance, tests in any module that affect checkouts are combined into a Checkout group. These groups may correspond to native Magento modules;
- Logical groups of tests may be selected within a larger extension.
2. If several tests have the same precondition, don’t neglect the use of test suites. This dramatically reduces the time it takes to run tests.
Note: You can learn more about this tool in the MFTF documentation.
3. Running tests automatically after the master branch update is a great practice.
4. Keeping the statistics of how the tests performed so that you could see how well they detect bugs in the future - also rocks! In case tests are running a significant number of times without detecting bugs - well, good for you, you did a regression. However, but this may be a signal for you to analyze test scenarios and improve them.
5. Do not forget that all tests, including MFTF tests, require maintenance, fixes, and refactoring due to product changes. Therefore, allocate some time for these activities when planning.
6. Don’t forget to use the reports!
In conclusion
Developing tests is a creative process that will improve the quality of the site or extension. It is worth contributing to the platform to make it better and build your skills to the next level.