Any Magento 2 website well-populated with high-quality content and a whole bunch of products variety has to perform fast on the frontend. Why can’t we just say ‘should’ rather than ‘has to’?

It’s better to see once than to hear one hundred times:

Graphic websites speed

Source: (based on HubSpot research telling how conversions depend on page load time).

Thus, can we say your frontend speed performance influences conversions? Sure, we can.

What are the top reasons your website can brake?

  • Unminified JS and CSS files 68% (see the solution here)
  • Slow HTML load speed 43%
  • Uncompressed JS and CSS files 43%
  • Uncached JS and CSS files 26%

To see a complete list of problems, use the source.

What’s this got to do with indexing?

Indexing directly affects your storefront performance. Without indexers, we would force our store work on the fly. So, read on to learn what is indexing in Magento 2 and how to manage the process correctly.

 

What is indexing in Magento 2?

Put simply, indexing is the way Magento 2 converts refreshable data (products, categories, prices, etc.) to enhance the storefront performance. When you refresh the converted data it needs an update or reindex.

More specifically, read the definition from Magento DevDocs:

Magento has a very sophisticated architecture that stores lots of merchant data (including catalog data, prices, users, stores, and so on) in many database tables. To optimize storefront performance, Magento accumulates data into special tables using indexers.

Source

How does it look like in practice?

Let’s take an example from our Magento 2 Product Labels extension.

Say, you’ve created a product label and specified 4 conditions to determine cases when the label should be applied. Consequently, without indexers, while loading a product page, Magento 2 would have to check each label to verify it matches the conditions you preset. Definitely, this would take too much time, which could increase your abandoned cart rate and would decrease conversions in the foreseeable future.

In the case with indexing, information about what labels should be displayed on this or that product page is already saved in tables taking into account all the conditions.

By and large, if Magento 2 worked without these special tables with indexers, the performance would be slowed down and, as a result, you would lose a significant part of conversions, which is bad.

Now let’s dig deeper into the technical part.

Magento 2: Indexing terminology

When laying it all out, indexing terminology comprises three main terms. They are:

  1. Dictionary;
  2. Index;
  3. and Indexer.

What is dictionary?

The term includes all the original data entered into your system. In order to simplify the maintenance, these dictionaries are normalized, which means they are restructured to cut data redundancy and better data integrity.

What is index?

The term means the representation of the original data for improved reading/searching. They can include aggregations results, as well as different calculations. By using a special algorithm, an index content can be restored from a dictionary.

What is indexer?

Quite simply, the term implies an object creating an index.

When you understand why you need indexers and know the indexing terminology, it’s time for little coding. So, the next part we have dedicated to the creation of a custom indexer.

Why do you need custom indexers?

Getting back to Magento 2 Product Labels extension, when developing the module we came to the conclusion that we need to create a custom indexer for it.

What made us think so?

Just imagine multiple labels created on numerous conditions for a wide range of products.

Can Magento 2 make the analytical work (upon what label on what product pages to display) on the fly?

Yes, it can.

Will your storefront perform well at that.

We tested, it won’t.

That’s why we decided to create a special indexer and added a setting to the backend named ‘Use Indexes for Better Performance’, which allows you to enable the process of indexation:

Magento 2 Product Labels index enable

Note that after the latest update the setting has been removed and now indexers are used by default.

How to create a Magento 2 custom indexer?

For you to devise a custom indexer, complete the next steps.

Step 1: Devise a configuration file

Create a config file app/code/Amasty/Example/etc/indexer.xml first:

The id attribute is used to identify this indexer. You can call it when you want to check the status, mode or reindex this indexer by the command line.

The view_id is the id of view element, which will be defined in the mview configuration file.

The class attribute is the name to the class, which processes the indexer method.

The shared_index attribute is used for combining several indexers into groups. While reindexing one of the group indexers, the others will be labeled as valid.

Magento 2 indexing has some child elements like the title, which is applied to show the name of the indexer in the grid of indexers; description, which is applied to define the description of the indexer in the grid of indexers.

The dependencies element is used to mark the dependencies between various indexers. While calling reindexRow, reindexList, invalidate methods of the cataloginventory_stock indexer after the reindex of the cataloginventory_stock indexer, the indexers that are marked as dependent on the indexer will be initiated. In our case, it will be the amasty_example indexer. The option is available in Magento 2.2.0. Starting from Magento 2.2.6, a required condition of the dependent indexer for reindexRow, reindexList methods will be the work mode – Update On Save.

Step 2: Create an MView configuration file

Create an app/code/Amasty/Example/etc/mview.xml config file, which will be responsible for the indexation by Cron.

Also, you need to identify the class specified in the config files of Amasty\Example\Model\Indexer\ExampleIndexer, it will contain the logic of our instance indexation. The class should implement the interfaces:
\Magento\Framework\Indexer\ActionInterface, \Magento\Framework\Mview\ActionInterface. Where the \Magento\Framework\Indexer\ActionInterface interface contains the main indexation methods and the \Magento\Framework\Mview\ActionInterface interface contains the methods that are applied for the Update By Schedule mode of an indexation.

See the code below, as the example of indexing methods and why you need them:

Step 3: Clear the cache

After all the files mentioned above are created, clear the Magento 2 cache. Then go to System > Index Management to see a new indexer in the Index Management grid:

Magento 2 clear cashe index management

Update On Save vs. Update By Schedule

Update On Save mode

In the indexing mode, Update On Save has the sense to initiate the reindex for a definite instance after it’s been changed. In our case (with product labels) we depend on products. This means that while a concrete product is changed we need to reindex our instance for this product. For this, you can do the next:

  • Create an app/code/Amasty/Example/etc/adminhtml/events.xml file:

  • Create an app/code/Amasty/Example/Model/Observer/Backend/CatalogProductSaveAfterObserver.php file:

By doing so, we have added an observer for a product save action.
After a product is changed, the indexer will be called and the ID of the updated product will be transferred.

Update By Schedule mode

If you change the Update On Save mode for the Update By Schedule one, the app/code/Amasty/Example/etc/mview.xml config file will be read and MySQL triggers will be created. This means that while deleting, adding, updating rows in the defined subscriptions tables, the entity_id column will be added and amasty_example_cl will be written into the table (it’s created automatically for the work in the Update By Schedule mode):

select from Magento 2 indexers

Also, there is a version_id (AUTO_INCREMENT) field in the amasty_example_cl table. The field is compared to the version_id field from mview_state for a current indexer.
As a result, in our execute($ids) processor will get only those instances that have been changed after the previous сrone launch (amasty_example_cl.version_id > mview_state.version_id).

How to set up Cron job in Magento 2 for working with indexers?

The main Cron jobs for working with indexers are defined in the vendor/magento/module-indexer/etc/crontab.xml file.

When launching Cron job by indexer_reindex_all_invalid (every minute), executeFull will be started for invalid indexers (they are marked in the Indexers grid with REINDEX REQUIRED status). It is intended for a full reindex, that is, in this case for all the invalid indexers, the table with the saved information will be rewritten.

When starting Cron job by indexer_update_all_views (every minute) all the indexers that work in the Update On Schedule mode will be launched. While launching it by indexer_clean_all_changelogs (every hour) the _cl tables (in our example it’s amasty_example_cl) indexers will be purged of entries with version_id below the latest version in the mview_state table.

All the presented Cron jobs belong to the index group. The work of the Cron job group can be set up here Stores -> Advanced -> System:

Magento 2 set up cron

How to manage Magento 2 indexers through the command line?

To see a complete list of indexer commands, click the link.

Note: While executing the bin/magento indexer:reindex amasty_example command, you may face the error:
‘Amasty Example index is locked by another reindex process. Skipping.’

You can solve it in two ways:
#1. Either you have to wait for the end of the reindex process, which is now being executed;
#2. or the reindex process has been interrupted for any reason and the status meaning hasn’t been changed in the table, as the process wasn’t finished:

Magento 2 indexers table

If this occurs, you can execute the bin/magento indexer:reset amasty_example command. It will change the working status to invalid and while launching the next reindex it will work.

Wrap-Up

That’s all for today.

If you have questions or suggestions, feel free to pose them in the comments below.

And stay tuned for new technical posts!