Hello, reader!
So you need to test a feature/fix/customization, but there’s no free QA specialist around? It happens. :(
In the article below we describe basic testing techniques that’ll help you resolve these situations independently.
Here is one of the 7 principles of testing:
Exhaustive testing is impossible
Why so? The thing is, for any real-life application, the number of possible test cases is just too large. To reduce this number to a reasonable optimum, we use a range of tried-and-true test design techniques.
We’ll explain these techniques with practice-based examples featuring testing of Magento 2 Out Of Stock Notification. How it works? See the box below.
What is equivalence partitioning?
This technique is used to test the processing of inputs, outputs, internal values and time-related values efficiently. How does it work? You have to break a multitude of values into equivalence classes or equivalence partitions relying on the specifics of the application under test. All the values pertaining to the same equivalence partition are processed in the same way.
Example. The OSN module allows a user to subscribe to out-of-stock products. So we need to test if the subscription field (Subscribe) is visible to a customer, i.e. if it is displayed on the frontend.
In a test Magento 2 with a pre-set sample date, there are 2046 products with different settings and attribute values. This potentially provides us with 2046 test cases:
As a rule, equivalence partitioning is used for smoke tests that you run to check if the application works at all.
This allows us to simplify the task: we’ll consider only simple products with visibility = Catalog, Search.
There are 32 of them. Still a lot.
Let’s try to single out the equivalence classes:
- Products with Quantity > 0
- Products with Quantity <= 0
Now we need only 2 test cases:
- test case 1 checks that the field is displayed when needed;
- test case 2 checks that the field is not displayed when it's not needed.
We’ve assumed that the field display only depends on the quantity of the product (quantity field value). This means that all the products within the given ranges should behave identically.
The problem is, this assumption is incorrect.
One of the key problems you come across when using this technique is the additional conditions. Test analytics don’t know about them, as product specifications rarely mention them.
In our case, these additional conditions boil down to Out-of-Stock Threshold and Backorders in the Advanced Inventory tab.
Now as we know about them, we can improve our model. We’ll design more test cases using another technique - Decision tables.
This technique helps to structure the information you get from the product specification/your experience/other sources.
Mind that:
- "-" means that the value doesn’t impact the result for the given conditions.
- We don't use any negative numbers for Out-of-Stock Threshold.
What do we get? The model became more complex, and the number of test cases doubled, reaching 4. On the positive side is that the number of products in the store doesn’t matter, as long as we can assign them to one of the four subsets having the combos of properties described above.
Types of defects detected: functional defects that cause incorrect data processing.
Test coverage: the ratio of the number of No. of equivalence classes divided by the No. of test cases.
Now let's turn to another technique suitable for independent Magento testing - boundary value analysis.
What is boundary value analysis?
The values located at the equivalence partition boundaries make a special point of interest. This is where the defects are settled.
Boundary value analysis makes a good choice for ordered sets, such as:
- Stored data structures
- Time-dependent activities
- Numeric attributes of non-numeric variables (e.g., length);
- Loops, including those in use cases and more
This technique perfectly fits testing Qty field in the example above. Let’s see how it works.
Out-of-Stock Threshold = 10
This means that as soon as the number of products reaches 10, we’ll start displaying the Subscription field (Subscribe). So we can single out 2 equivalence partitions:
- Range 1: x<=10;
- Range 2: x>10.
To make the example more appealing, we’ll allow to purchase the product in parts:
- Qty Uses Decimals = Yes
Magento allows users to save Qty values up to 0.01. In our example, it’s minimal delta (increment).
For each boundary value, we can single out 3 remarkable points:
- ON - a boundary point. For the 1st range, it’s 10;
- OFF - point just above the max (10) - 10.01;
- IN - point just below the max - 9.99 (the least interesting one, we can let it go. It’s included in the equivalence class).
Now let's consider Range 2.
For the second range, we get the following points:
- ON = 10.01
- OFF = 10
- IN = 10.02
The values repeat. So we can design the following 2 test cases for each of these ranges.
After checking with the database, we’ll learn that there Qty can be stored to the nearest 0,0001.
So what happens in Magento 2 if the Qty is greater than 9.99 and lesser than 10?
Will the field be displayed if the Qty is 9.9999? No, it won’t.
Nevertheless, you should remember that the minimal increment in Magento 2 and in the database may be different.
Boundary value analysis: an add-on
Testing with this technique, we usually add checking the Qty = 0 and the related boundary conditions, even if they don’t exist for a particular condition. Why so?
First of all, it’s a tradition. The zero has always been a trap when calculating, especially at the dawn of programming.
Secondly, you can’t be too cautious when testing.
So here’s how we can check Subscribe field display for a product with the attributes below:
- Qty Uses Decimals = Yes
- Out-of-Stock Threshold = 10
We can use the following values:
Types of defects: errors with less-than and greater-than logic, missed boundaries and errors connected with exceeding permissible values (like value limit in the database).
Test coverage: the ratio of the number of No. of equivalence classes divided by the No. of test cases. In our example, we reached a 100% coverage with the help of 4 test cases (we don’t usually count a 0 test case in the metrics.)
Pairwise testing
This technique suits for testing diverse combinations of parameter values. Instead of checking all possible combinations, you test only those of each parameter pair.
It was empirically established that the majority of defects occur due to faulty interaction of 2 parameters. The interactions of 3+ parameters result id defects far more rarely.
Why so? Read A Practitioner's Guide to Software Test Design by Lee Copeland for more information.
And we turn back to our OSN module. We’ll consider settings.
In this block, you set the 3 WHs (What, where and to whom) for the Subscription field, as well as the related notification settings.
We don’t consider the latter setting, as other techniques (e.g.boundary values and equivalence partitioning) suit testing it better.
For the 1st setting (Allow subscription for Selected Groups), we single out 3 meaningful values.
Let’s assume that General, Wholesale и Retailer values are equivalent for our purposes.
For other settings, the available values are Yes/No.
We use the Cartesian product to get the number of all possible combinations:
3*2*2*2 = 24 possible combos (potential test cases)
Pairwise testing involves complex mathematics. The combos below were set with the help of a specific tool:
A - means that in this case, the value is not important, i.e. you can choose any.
Mind that some test cases should be ignored, as the combos don’t always make sense.
What do we have? Using pairwise testing, we reduced the number of test cases by 3.
Types of defects: defects related to combined values of several parameters.
Afterword
In this post, we considered some basic testing techniques you can use to run testing in case a QA specialist is not available. Have you ever tried them?
Please leave comments below.
There are many other test design techniques, but they prove useful in other situations.
You’re likely to learn about them in our upcoming posts.
Stay tuned.