How to make a smooth start with MFTF [checklist]

Posted in: Magento 2 Guides
Hello to our blog readers!

Today we turn to testing with  Magento Functional Testing Framework (MFTF), a powerful native tool that can help you facilitate testing.  

So what does this framework provide to testers?


What is MFTF?

MFTF is a set of ready-made tests that check if a Magento 2 system functionality runs as expected. In case the default functionality was expanded, you need to cover the new features with autotests, and notably, even manual testers can do it. The thing is, MFTF only uses XML for writing test cases, no PHP involved. PHP phase comes later when XML test cases are auto-converted into PHP scripts and tested in Codeception, Selenium or Allure. These are the tools MFTF employs.

This autoconversion spares QA specialists the trouble of scripting and enables them to write test cases on the fly. Besides, the framework provides a set of functions, and the existing tests can be used as a viable template for writing new tests. However, to use the whole range of functions MFTF offers, you should configure it correctly. 

Planning to work with MFTF or just started testing with it? Here at Amasty, we have decided to share some useful tips from our  MFTF experience. They will help you avoid stumbling blocks and make a fast and efficient start.

So let’s begin.

Prep steps

Though the framework is relatively easy to use, you should tread carefully from the very start. Remember that small mistakes and even typos you make at the beginning may cost you dear later.

So here are the 3 preparatory steps for seamless testing:

Step #1

Testing a Magento 2.3 system, you should store the test cases in the module folder, directory Test/Mftf.

Step #2

Apart from the directory that stores your test cases, you should set up such subdirectories as ActionGroup, Data, Page, Section and other in the MFTF directory. These will store the data structures necessary to run the tests.

Prep steps 1 and 2

Step #3

You should set noNamespaceSchemaLocation correctly. For Magento 2.3 test cases, it should start with urn:magento:mftf. Otherwise your test won’t start. Please, see the examples of incorrect and correct implementations below.

Prep step 3. Incorrect implementation


Prep step 3. Correct implementation

Simple prep steps complete, you are ready to proceed with writing test cases. And there comes another surprise: annotation section. In MFTF test annotations are of special importance.

Mind annotation section

Annotations explain each test case (or a group of test cases) in MFTF. Annotations should be easily understandable, as you will have to submit test reports to other team members,  and they should see what a test case does exactly. So here are best practices for writing annotations. Stories and Title should be unique to help you and other team members understand what the test case checks. For the same reason, TestCaseId should also be unique.

  • Make sure the Value field in a Group annotation is meaningful. Value permits to run a separate Group of tests independently. Relying on our testing practice, we recommend that each test case covers the following Groups. (Note that one test can employ several Groups.):
    • VendorName Group for all your tests.
    • Module Group that corresponds to the name of the module folder and unites all tests belonging to this module. For example, the group of Order Attributes module is called Orderattr.
    • Magento functionality Group (optional). This group includes test cases that cover native Magento features, such as Checkout.
    • You can also single out separate logical groups of tests within some module or new custom features. This was the case with testing Improved Layered Navigation module when we introduced several groups of test cases: for filters (FiltersILN), for brands (ShopByBrandILN), for SEO (SeoILN), and more.

Here are the bad and the good example of the Value field.

A reasonably descriptive annotation will help the whole project team to locate the detected issues swiftly. But good annotations are not the only aspect ensuring a smooth start.

Testing with MFTF: cleansing

It takes ages to find the necessary object in an old cluttered house, and the same holds true for testing, both manual and automated. A tidy testing environment is one of the keys to fast and efficient testing. So how to ensure it in MFTF? 

First of all, upon completion, a test case should clean up the data that may hamper repeating it or running other tests.

Example:  A test sets up a field in ?heckout, and in case this field is empty, you can’t place an order. Logically, this will affect the tests related to order placement functionality.

Remember that “cleaning up” is not always about deleting the data. You can simply render a field necessary in the first test optional in the second test. To do so, go to Magento or module settings.

 The data a test creates should be unique. This applies to IDs, URLs, keys, names, etc.

Summing up

In this post, we’ve decided to share some ideas on making the start with MFTF painless and swift. So here’s a short sum-up based on our practice:

  • Prep steps: Do invest some time in the correct naming and storing of your test cases.
  • Informative annotation section: Make annotations to your test cases reasonably detailed. This will ensure that the whole team understand what a test case is about.
  • Tidy test environment: Clotting your test environment with unnecessary data will only make your test cases stumble. So clean it up timely.

This neat checklist helps us set off to testing quickly.

Maybe you have something to add? Please, leave your comments.

For more specific questions arising in the process of testing, you can consult a comprehensive MFTF guide.

December 22, 2018
January 29, 2019
December 13, 2018
Jaya Venkat
May 2, 2019
Hi Thank you for the check list to start Mftf test. I am new to Mftf and it was really useful to verify my test. I tried my first custom test. As per requirement above , I created the required folders under Mftf folder and kept my test xml under Test folder. Here is my test xml. <!-- /** * Copyright © Magento, Inc. All rights reserved. * See COPYING.txt for license details. */ --> I verified all the requirements you mentioned in the article. But when I ran my test using command 'vendor/bin/mftf run:test StorefrontLoginRequiredFieldTest --remove' , I am getting error In TestObjectHandler.php line 81: Test StorefrontLoginRequiredFieldTest not defined in xml. I am using the tests from module-customer under {magento_root}/vendor/magento folder. What is the adjustment I need to do in .env file to access tests from {magento_root}/vendor/magento folder? Your solution is really appreciated.
Jaya Venkat
May 2, 2019
My test XML was some how not pasted. so trying again <!-- /** * Copyright © Magento, Inc. All rights reserved. * See COPYING.txt for license details. */ -->
Alina Bragina
May 17, 2019
Hey Jaya, thanks for reading our blog! You see, to help you out we need to see more details on the problem you're trying to solve and understand what the problem is. First off, there're some precautions: make sure you ran generate command before the ‘vendor/bin/mftf run:test StorefrontLoginRequiredFieldTest –remove’ one. The problem could be caused by missing the step. If not, then we need to see your .xml file and screenshots to get the location of your test .xml in the file system, as your description isn't enough. Unfortunately, you can't attach them to the comments below, therefore, you can subscribe to our blog and let me know about this here, I'll try to further put you in touch with the author of the article via email. All the best!
Leave your comment

Your email address will not be published

This blog was created with Amasty Blog Pro

This blog was created with Amasty Blog Pro

© 2009-2022 Amasty. All Rights Reserved.