Comprehensive Guide to Magento 2 Index Management

Magento 2 indexers by Amasty
Posted in: Magento 2 Guides

Any Magento 2 website well-populated with high-quality content and a whole bunch of products variety has to perform fast on the frontend. Why can’t we just say ‘should’ rather than ‘has to’?

It’s better to see once than to hear one hundred times:

Source

Thus, can we say your frontend speed performance influences conversions? Sure, we can.

What are the top reasons your website can brake?

  • Unminified JS and CSS files 68% (see the solution here)
  • Slow HTML load speed 43%
  • Uncompressed JS and CSS files 43%
  • Uncached JS and CSS files 26%

To see a complete list of problems, use thesource.

What’s this got to do with indexing?

Indexing directly affects your storefront performance. Without indexers, we would force our store work on the fly. So, read on to learn what is indexing in Magento 2, why do you need it, how to create a new indexer and manage the process correctly. 

What is indexing in Magento 2?

Put simply, indexing is the way Magento 2 converts refreshable data (products, categories, prices, etc.) to enhance the storefront performance. When you refresh the converted data it needs an update or Magento 2 to be reindex programmatically or manually.

More specifically, read the definition from Magento DevDocs:

 

Magento has a very sophisticated architecture that stores lots of merchant data (including catalog data, prices, users, stores, and so on) in many database tables. To optimize storefront performance, Magento accumulates data into special tables using indexers.

 

Source

How does it look like in practice?

Let’s take an example from our Magento 2 Product Labels extension.

Say, you’ve created a product label and specified 4 conditions to determine cases when the label should be applied. Consequently, without indexers, while loading a product page, Magento 2 would have to check each label to verify it matches the conditions you preset. Definitely, this would take too much time, which could increase your abandoned cart rate and would decrease conversions in the foreseeable future.

In the case with indexing, information about what labels should be displayed on this or that product page is already saved in tables taking into account all the conditions.

By and large, if Magento 2 worked without these special tables with indexers, the performance would be slowed down and, as a result, you would lose a significant part of conversions, which is bad.

Now let’s dig deeper into the technical part.

Magento 2: terminology of indexing

When laying it all out, indexing terminology comprises three main terms. They are:

  1. Dictionary;
  2. Index;
  3. and Indexer.

What is dictionary?

This is all the original data entered into your system. In order to simplify the maintenance, these dictionaries are normalized, which means they are restructured to cut data redundancy and better data integrity.

What is index?

This is the representation of the original data for improved reading/searching. They can include aggregations results, as well as different calculations. By using a special algorithm, an index content can be restored from a dictionary.

What is indexer?

The indexer implies an object creating an index.

When you understand why you need indexers and know the indexing terminology, it’s time for little coding. So, the next part we have dedicated to the creation of a Magento 2 custom indexer.

Why do you need Magento 2 custom indexers?

Getting back to Magento 2 Product Labels extension, when developing the module we came to the conclusion that we need to create a custom indexer for it.

What made us think so?

Just imagine multiple labels created on numerous conditions for a wide range of products.

Can Magento 2 make the analytical work (upon what label on what product pages to display) on the fly?

Yes, it can.

Will your storefront perform well at that.

We tested, it won’t.

That’s why we decided to create a special indexer and added a setting to the backend named ‘Use Indexes for Better Performance’, which allows you to enable the process of indexation:

How to create Magento 2 custom indexer?

For you to devise a custom indexer, complete the next steps.

Step 1: Devise a configuration file

Create a config file app/code/Amasty/Example/etc/indexer.xml first:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation=
"urn:magento:framework:Indexer/etc/indexer.xsd">
<indexer id="amasty_example" view_id="amasty_example" class=
"Amasty\Example\Model\Indexer\ExampleIndexer" shared_index="amasty_example">
<title translate="true">Amasty Example</title>
<description translate="true">Amasty Example</description>
<dependencies>
<indexer id="cataloginventory_stock"/>
</dependencies>
</indexer>
</config>

The id attribute is used to identify this indexer. You can call it when you want to check the status, mode or reindex this indexer by the command line.

The view_id is the id of the view element, which will be defined in the mview configuration file.

The class attribute is the name to the class, which processes the indexer method.

The shared_index attribute is used for combining several indexers into groups. While reindexing one of the group indexers, the others will be labeled as valid.

Magento 2 indexing has some child elements like the title, which is applied to show the name of the indexer in the grid of indexers; description, which is applied to define the description of the indexer in the grid of indexers.

The dependencies element is used to mark the dependencies between various indexers. While calling reindexRow, reindexList, invalidate methods of the cataloginventory_stock indexer after the reindex of the cataloginventory_stock indexer, the indexers that are marked as dependent on the indexer will be initiated. In our case, it will be the amasty_example indexer. The option is available in Magento 2.2.0. Starting from Magento 2.2.6, a required condition of dependent indexer for reindexRow, reindexList methods of the indexer will be the work mode - Update On Save.

Step 2: Create an MView configuration file

Create an app/code/Amasty/Example/etc/mview.xml config file, which will be responsible for the indexation by Crone.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation=
"urn:magento:framework:Mview/etc/mview.xsd">
<view id="amasty_example" class="Amasty\Example\Model\Indexer\ExampleIndexer" group="indexer">
<subscriptions>
<table name="catalog_product_entity" entity_column="entity_id"/>
</subscriptions>
</view>
</config>

Also, you need to identify the class specified in the config files of Amasty\Example\Model\Indexer\ExampleIndexer, it will contain the logic of our instance indexation.

The class should implement the interfaces: \Magento\Framework\Indexer\ActionInterface, \Magento\Framework\Mview\ActionInterface.

Where the \Magento\Framework\Indexer\ActionInterface interface contains the main indexation methods and the \Magento\Framework\Mview\ActionInterface interface contains the methods that are applied for the Update By Schedule mode of indexation.

See the code below, as the example of indexing methods and why you need them:

php custom indexer example

Step 3: Clear the cache

After all the files mentioned above are created, clear the Magento 2 cache. Then go to System > Index Management to see a new indexer in the Index Management grid.

Update On Save vs. Update By Schedule

Update On Save mode

In the indexing mode, Update On Save has the sense to initiate the reindex for a definite instance after it’s been changed. In our case (with product labels) we depend on products. This means that while one concrete Magento 2 product is changed we need to reindex our instance for this product programmatically. To programmatically reindex the instance in Magento 2, you need to do the following:

  • Create an app/code/Amasty/Example/etc/adminhtml/events.xml file:
<?xml version="1.0"?>

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation=
"urn:magento:framework:Event/etc/events.xsd">
<event name="catalog_product_save_after">
<observer name="reindex_event" instance=
"Amasty\Example\Model\Observer\Backend\CatalogProductSaveAfterObserver" />
</event>
</config>
  • Create an app/code/Amasty/Example/Model/Observer/Backend/CatalogProductSaveAfterObserver.php file:
<?php

namespace Amasty\Example\Model\Observer\Backend;

use Magento\Framework\Event\Observer;
use Magento\Framework\Indexer\IndexerRegistry;
use Magento\Framework\Event\ObserverInterface;

class CatalogProductSaveAfterObserver implements ObserverInterface
{
/**
* @var IndexerRegistry
*/
private $indexerRegistry;

public function __construct(IndexerRegistry $indexerRegistry)
{
$this->indexerRegistry = $indexerRegistry;
}

public function execute(Observer $observer)
{
$product = $observer->getEvent()->getProduct();

if ($product) {
$this->indexerRegistry->get('amasty_example')->reindexRow($product->getId());
}
}
}

By doing so, we have added an observer for a product save action.
After a product is changed, the indexer will be called and the ID of the updated product will be transferred.

Update By Schedule mode

If you change the Update On Save mode for the Update By Schedule one, the app/code/Amasty/Example/etc/mview.xml config file will be read and MySQL triggers will be created. This means that while deleting, adding, updating rows in the defined subscriptions tables, the entity_id column will be added and amasty_example_cl will be written into the table (it’s created automatically for the work in the Update By Schedule mode):

Also, there is a version_id (AUTO_INCREMENT) field in the amasty_example_cl table. The field is compared to the version_id field from mview_state for a current indexer.
As a result, in our execute($ids) processor will get only those instances that have been changed after the previous ?rone launch (amasty_example_cl.version_id > mview_state.version_id).

How to set up cron job in Magento 2 for working with indexers?

The main Cron jobs for working with indexers are defined in the vendor/magento/module-indexer/etc/crontab.xml file.

When launching Cron job by indexer_reindex_all_invalid (every minute), executeFull will be started for invalid indexers (they are marked in the Indexers grid with REINDEX REQUIRED status). It is intended for a full reindex, that is, in this case for all the invalid indexers, the table with the saved information will be rewritten.

When starting Cron job by indexer_update_all_views (every minute) all the indexers that work in the Update On Schedule mode will be launched.

While launching it by indexer_clean_all_changelogs (every hour) the _cl tables (in our example it’s amasty_example_cl) indexers will be purged of entries with version_id below the latest version in the mview_state table.

Note: Check out how to reindex Magento 2 indexers manually without cron.

All the presented Cron jobs belong to the index group. The work of the Cron job group can be set up here Stores -> Advanced -> System:

How to manage Magento 2 custom indexers through the command line?

To see a complete list indexer commands in Magento 2, click the link.

Note: While executing the bin/magento indexer:reindex amasty_example command, you may face the error:
‘Amasty Example index is locked by another reindex process. Skipping.’

You can solve it in two ways:

  1. Either you have to wait for the end of the reindex process, which is now being executed,
  2. or the reindex process has been interrupted for any reason and the status meaning hasn’t been changed in the table, as the process wasn’t finished:

If this occurs, you can reset indexer in your Magento 2 via the bin/magento indexer:reset amasty_example command. It will change the working status to invalid and while launching the next reindex it will work.

Wrap-up

That’s all for today.

If you have questions or suggestions, feel free to pose them in the comments below.

And stay tuned for new technical posts!

November 9, 2018
November 14, 2018
October 18, 2018
Comments
André
July 2, 2019
Wow, that was a very complete guide and explanation. I really loved it! Thank you so much!
Reply
Alina Bragina
July 9, 2019
Hi André, thanks for reading us and sharing your kind words! <strong><a href="http://eepurl.com/Mv5tD" rel="nofollow">Click here</a></strong> to stay updated on all our new posts! ;)
Hakeem
August 30, 2019
I DON’T UNDERSTAND HOW I SHOULD CHECK IF CRON IS DONE OR NOT???
Reply
Polina Litreyeva
August 30, 2019
Hello, Hakeem! Please, go to your Magento 2 database table named cron_schedule. There you will see all pending and completed cron tasks. Or you can use our <a href="https://amasty.com/cron-scheduler-for-magento-2.html">Cron Scheduler</a> to manage your cron tasks.
Angela Flores
September 6, 2019
Hi! What settings should I choose in Cron configurations?
Reply
Polina Litreyeva
September 6, 2019
Hello, Angela! Thanks for reading! We recommend using default Magento settings.
bHagyaraju
July 26, 2021
I have created a new custom indexer process-related product price. so I need the product entityId along with its Store ID (i mean in which store scope the product price was updated). Currently, I am getting the only product entity IDs only. how do I get a store ID also?
Reply
Alina Bragina
August 2, 2021
Hi there, unfortunately, it's too little information to give you a quick reply here. Feel free to write us at support@amasty.com, and we'll try to help you out. Cheers.
Caesar
September 24, 2022
Hello, what is backlog in indexer?
Reply
Vladimir
September 27, 2022
Hi there, unfortunately, it's too little information to give you a quick reply here. Feel free to write us at support@amasty.com, and we'll try to help you out.
Leave your comment

Your email address will not be published

This blog was created with Amasty Blog Pro

This blog was created with Amasty Blog Pro

© 2009-2022 Amasty. All Rights Reserved.