Nothing is created equal, they say. They are wrong. And if you own an online shop, you are the one who knows that perfectly well as many pages on your site are literary equal, aren’t they? Our extensive guide will help you to solve the problem of Magento duplicate content and never allow it again.
So, you may ask: ‘Why do I need this if Google doesn’t ban websites with duplicate content?’. The arguments for are more than just auto-suggestions and overreacting:
- Google can’t understand that all the Magento duplicate product URL addresses target the same location. Therefore, different variants of your website links won’t play out and get the page authority as it was expected;
- Users will see the most relevant (according to Google) version of your website URL but not the one you need;
- You are risking to lose crawler cycles as once Google bots discover the duplicate content, they won’t crawl your fresh content.
If you’re still not sure whether this guide is for you, just search for a keyword you rank for in Google and look for the results. If you see a not user-friendly URL or multiple URLs of pages with the same content, you have duplicate content on the website. Also, log in to your Google Webmasters search console account to see alerts about duplicate content. Examine the crawler metrics (GW account>Crawl>Crawl Stats) to see how many pages are already crawled, indexed. Then compare the stats with the real page quantity. If the number of those crawled, indexed pages is many times more than a real one, read on as there are most likely duplicate content problems.
Article summary
What are the types and most common examples of duplicate content in Magento?
Partial duplicates are the case when only minor part of the content or its layout is unique.
Here are the most common examples of partial duplicates in Magento stores (you can click on any example and and jump to its detailed overview).
Full duplicates are the case when the content on two or more pages is identical.
The most common example of full duplicates in Magento is:
Are you ready? Then let’s get down to the details!
Partial Duplicates in Magento
1. Product Sorting
That’s great when users can sort the products in your store by bestsellers, by newest, by Magento 2 price filter, number of reviews, etc. It’s even better if people can decide how many products should be displayed on the page: 20? 50? 100? But all these sorting options create pages with different characters (?, =, |) in the URLs:
http://site.co.uk/category/products.htm?sortby=total_reviews|desc http://site.co.uk/category/products.htm?sortby=total_reviews|asc http://site.co.uk/category/products.htm?sortby=relevance|desc
The problem comes out when sorting pages get indexed and even cached by Google. Imagine how many such pages can exist! Thousands! And Google crawlers spend time indexing them while they could concentrate their resources on indexing more important pages of your site: categories, products, etc.
1.2. How to find product sorting pages
First go to your product pages and sort them by any option. Now you can see the parameters added to the URL after sorting (e.g., dir, sortby). Go to Google and search for site:yourdomain.com inurl:dir
Most likely you’ll see this:
Just click to include the omitted results and you’ll see the pages in your store containing “dir” in the URLs. It’s bad when these pages with parameters are indexed.
1.3. How to remove product sorting duplicates
1.3.1. Google Webmaster Tools
Go to Google Webmaster Tools => Crawl => URL Parameters. Here you will see the parameters Google has found in the URLs of your store and how it crawls them. “Let Googlebot decide” is the default option there.
But when it comes to crawling your Magento store, it’s you but not Google who should decide which pages should be indexed, right? So if you haven’t decided this before, it’s high time you did it! Click “edit”, choose “Yes” in the dropdown menu and then – “No URLs”.
You can also add parameters that are not listed in GWT and set crawling options for Google. But be careful and check twice (or even three times) before blocking the URLs with these parameters.
1.3.2. Rel=canonical
You can also use canonicalization for the sorting pages in your Magento store. This way they will be accessible for users but redirect the crawlers to pages without parameters.
Canonicals? Great! But how do I implement them?
You should add this code to the the sorting pages:
<link href="CategoryURL " rel="canonical" />
where CategoryURL is the address of the same category page without parameters. For example, the following pages:
http://site.co.uk/category/products.htm?sortby=total_reviews|desc http://site.co.uk/category/products.htm?sortby=total_reviews|asc http://site.co.uk/category/products.htm?sortby=relevance|desc
should canonicalize this page
http://site.co.uk/category/products.htm
After adding the code, give Google some time to re-index the pages and follow the new instructions, usually it takes a few days. Canonicalization set up correctly, you’ll see the cache of the canonicalized page (http://site.co.uk/category/products.htm) even if you check the cache of the sorting pages (cache:http://site.co.uk/category/products.htm?sortby=relevance|desc)
2. Pagination
Your Magento store is big as you have lots of great products there, right? But even you have only a few products (that are also great!), they are still placed on the pages with pagination options.
For example:
http://www.site.com/category1.htm?page=2 http://www.site.com/category1.htm?page=3
2.1. How to find paginated duplicates
To find paginated pages in your Magento store, go to Google and search for site:yoursite.com inurl:page. This search returns all the pages containing “page” in the address within your site.
2.2. How to remove duplicates
2.2.1. Canonicalization
You already know a lot about canonicalizing pages in your store, right? Just make rel=canonical tags on those paginated pages:
http://www.site.com/category1.htm?page=2 http://www.site.com/category1.htm?page=3 http://www.site.com/category1.htm?page=4
point to the category
http://www.site.com/category1.htm
2.2.2. Pagination with rel=”next” and rel=”prev”
This option was specifically created by Google to fight duplicate paginated results. The idea is that all paginated pages are connected like links in a chain:
If we take these pages as an example:
http://www.site.com/category1.htm http://www.site.com/category1.htm?page=2
Then we should put the prev/next instructions in the following way:
<link href=" http://www.site.com/category1.htm?page=2" rel="next" />
in the <head> of http://www.site.com/category1.htm.
<link rel="prev" href=" http://www.site.com/category1.htm " /> <link rel="next" href=" http://www.site.com/category1.htm?page=3" />
in the <head> of http://www.site.com/category1.htm?page=2.
<link rel="prev" href=" http://www.site.com/category1.htm?page=2" />
in the <head> of http://www.site.com/category1.htm?page=3 (let’s imagine this is the last page).
2.2.3 Pagination with Robots meta tag
If you don’t want Google to crawl all the paginated pages, you can use this meta tag: <meta name=”value” content=”value”> within the head tag <head>…</head> of a duplicate content page. E.g.: <meta name=”robots” content=”noindex,nofollow”>. In doing so, search engines won’t ignore the links on the duplicate content. The value of robots with the name of a search engine can be changed depending on your needs.
Besides, make sure not to block URLs with the robot.txt file. Some Magento users indicate URLs that contain duplicate content in the robot.txt files to block the pages from crawling. However, Panda Update ignores practices that block crawlers and bots consider the pages as unique ones. But what is even worse, other websites will still be able to link to the blocked pages and as a result, you won’t get any benefit for your SEO from the backlinking. Therefore, it’s better to mark the pages with duplicate content with the ‘canonical’ tag.
There are a few things to remember:
- The first paginated page should contain only rel=”next”
- The last paginated page should contain only rel=”prev”
- Google allows using both canonicals and rel=”next” and rel=”prev” on one page
3. Variations of the same product
Imagine you sell mugs (or you really sell them?) and have landing pages for each color:
The characteristics are the same, description is the same, layout is the same… So what’s new? Just picture! Unfortunately, it’s too little for Google to treat such pages as unique. This means that all product variations found on different pages are partial duplicates that act like a magnet for Google Panda.
3.1. How to find product variations
As a Magento shop owner, you probably know all your products and can make a list of their variations. Alternatively, you can search in Google for:
site:yoursite.com “here comes a short excerpt from the product description”
This way you will find all the pages of your site containing this very excerpt.
3.2. How to remove duplicates
3.2.1. One page for all variations
You can create a single page for a particular product and list all its variations there.
This way you have one unique page instead of several duplicate ones.
3.2.2. Rel=canonical
You can also use rel=canonical. Just choose one variation page and put canonical to it from other variations. This way the content will be seen by users but Google will have a copy of only one page that you’ve chosen.
3.2.3. Make each variation page unique
The hardest way to solve duplication issues with product variations is to make each variation page unique. You will have to add different product descriptions and meta info. Yes, this is extremely time consuming. So be sure to grab a cup of coffee before starting this long journey!
3.2.4. Rel=hreflang
When running several Magento localized store views in different languages, you can apply the rel=hreflang tag to show the search engines how to choose a needed version of the content. E.g.: If you have two store views in English and Italian, you should add the tag: “<link rel=”alternate” href=”https://example.com” hreflang=”en-it” />” to the Italian store view. The same method should be applied to all the rest localized store views.
What does it give? It allows you to avoid the risks when search engines consider the content as duplicated.
Full Duplicates in Magento
Now when you’ve found all the partial duplicates, it’s time to look for full ones. Ready, steady, go!
The same product in different categories
You may have one product that can be found in two or more categories. For example:
http://www.site.com/jewellery/necklace.html http://www.site.com/for-her/necklace.html http://www.site.com/gifts/necklace.html
There’s only one necklace but 3 different URLs! In case with Magento duplicate product URL addresses, no matter how great your product is, Google will consider it thin content. It’s so unfair! So let your great products be great to Google by making them unique.
1.1. Rel=canonical
Just like with product variations I’ve talked above, you can choose one URL to show your product (necklace, as seen in the example) and the other pages will canonicalize it.
1.2. Remove category from URL
Alternatively, you can remove the category path from the URL, so that each product will have only one address no matter in how many categories it can be found:
http://www.site.com/necklace.html
Don’t know how to do that in Magento? There was a good tip in one of our post on SEO in Magento. Just go to System => Configuration => Catalog => Search Engine Optimization and switch “Use Categories Path for Product URLs” field to “No” and both “Canonical Link Meta Tag” fields to “Yes”.
1.3. Leave only one category path in a product URL
If you have a red T-Shirt in 2 categories at once: T-Shirts and New, you can choose which category to use in the URL: either the longest one (T-Shirt) or the shortest one (new). This is possible with Unique Product URL extension.
Summary
Now you know how Google sees duplicate content on your store. Don’t let it decide how to treat your pages, suggest the best way possible instead. Take control over your site crawling!
![How to make a smooth start with MFTF [checklist]](https://amasty.com/blog/wp-content/uploads/2018/12/Blog-3-440x264.png)
![Shipping Table Rates for different postal code systems [Expert tips]](https://amasty.com/blog/wp-content/uploads/2018/05/shipping-table-rates-m2-and-postcodes-440x264.jpg)
Great helpful article, thanks for posting!
Thanks, Darryl! Let’s fight duplicate content once and forever!
Great article! What about duplicated content over multiple stores (one magento installation) ?
Thanks for your comment, Nghia!
The best strategy is to keep content unique as much as possible. If you have the same products in different stores, you’d better write unique descriptions for them. This is quite time consuming but rewarding.
hey this is really interesting & helpful post. Thanks for sharing this.
Our pleasure, Andrew
Thanks for sharing fabulous information.It’ s my pleasure to read it.I have also bookmarked you for checking out new posts.
Happy you liked it.
How do you make your paginated pages canonical point back to the url without the ?page=2 for example?
Hello Scott!
You just need to insert a canonical tag pointing to the main category page (i.e. without ?page=n) to all the paginated results of this category. You can see an example on our site at https://amasty.com/magento-extensions.html?p=2 – you will find there a canonical tag pointing to http://amasty.com/magento-extensions.html.
Dear Kristina,
I was wondering, currently you don’t have this exact canonical on your website. Now it points to itself, and that is something we also want to do. Can you tell me how i can achieve that? (we use your improved layered navigation module).
Hope to hear from you soon.
Greetings Martijn
Hello Kristina! What I really wanted to say is “Bundle of Thanks”.
You helped me a lot by sharing such a nice and detailed post. I’m a webmaster and didn’t know the Goolge Webmasters Crawling “URL parameters” technique to remove duplicated product pages. I’ve implemented these duplication removing tips on my magento site.
Thanks again!
You’re welcome! I’m happy I’ve helped you!
I blog often and I seriously thank you for your information. This great article has truly peaked
my interest. I’m going to book mark your blog and keep checking for new information about once a week.
I opted in for your Feed as well.
Thanks, Gonzalo 🙂
We’ll do your best to post the most interesting information, so stay connected!
Great article!
Another tip that we often use to help GWT warnings of high number of URLs indexed is to hide the layered navigation from search bots. This would also help with section 1 of this article We normally create a theme exception for bot user agents with a small layout XML file which removes the layered nav.
Thanks for your tip, Zak!
Thanks Kristina. I saved this article to study later – lots of stuff needs to be done…
You’re right, good results always require hard work.
Good luck!
Hi, great post ! Thank you very much, simple and efficiente 🙂
Thanks, Julien! 🙂
Thanks so much for this blog post, Kristina, you’re going to make me look like a hero tomorrow at work. If I read your above response to Nghia correctly and apply it to my own situation, it sounds like the best method for dealing with a large number of products whose only difference is their size would be boil them down to one product with some method of selecting the correct size before adding to a cart.
Does that hold water?
Hello Matt,
Placing product variations on a single page is a painless solution in most cases. However, it still depends on your niche. If you customers tend to look for particular sizes of your products, it might be better to have multiple pages with more or less unique content and meta info.
Good luck to you tomorrow!
Hi! Again! Thank you for the last reply relating to bodygoal.co.uk ! Ok here is the most confusing thing. Absolute best practices have been applied, any I now have 48000 indexed! Oh my lord! Which software are you using to identify how many are listed in SERP’s? And how would I increase the quantity sat in SERP’s? This site will mean the world to me? Eg allow me to eventually spend more time with the children. I will make it work, absolute determination as the driving force! It is still in development, however I am trying hard to stagger so that Google can see a steady progression! Also, where is the best place to put articles in Magento? Is there an extension that works well with SEO? I have started to write an article within Cms pages…. bodygoal.co.uk/vitamin-d (take a look) I have added to this every 1 to two weeks for progression. I will continue with these articles, however if this is the wrong place to start growing a knowledge base, I will stop now (leave the vitamin D page for index reasons) but start growing the rest elsewhere! Help! :0)
Kind Regards
Paul
Hi Paul,
you can find how many pages are found in main Google index by adding /& (please see more info about it at the beginning of that article).
I would also recommend adding your site to Google Webmaster Tools to track duplicate titles/descriptions there. It seems you have variations of 1 product on different pages, this may be an issue.
As for a blog, you can use Magento CMS pages or use WordPress for blog and connect it with Magento.
Hope this helps.
Thanks so much for this great post. Was great timing as I was cleaning up my GWT! For the URL Parameters I have fixed the ones you have listed with the examples of what to put as the Effect. I have some others but don’t want to get it wrong. Can you please tell me what I would put as the effect for the following: manuf, em_x, artist, size_materials, options. Thanks so much for your help.
Hello Amy!
There’s one thing you should remember: when you add a new parameter to GWT, you must be sure it’s not used in canonical URLs. For example, if you add “options” as a parameter, any page containing it in the URL (like example.com/all-product-options) will be de-indexed. So at first make a list of “pure parameters”. In your case these ones em_x, size_materials seem to match.
Moreover, pages usually contain more than 1 parameter, so it’s enough to block the most obvious of them. For example, for that URL example.com/gifts-for-her?sortby=manuf it’s enough to block just sortby.
Hope that helps!
Great information! Regarding removing the category from the URL on an existing site, I assume one will need to build a table of the inbound links and redirect them to the desired landing page? Is it better to handle it this way in Magento rather than doing a NOINDEX in the category landing pages?
Hello Mar,
you actually don’t need to build a list of inbound links. What you need is to make sure all visitors will reach the necessary product page. If you decide to remove category path on an existing site, you can use a ready solution or make a list of multiple product URLs with categories in them and 301 redirect all of them to the URL without any category path (it will become a canonical one).
This applies to product pages only. When it comes to category pages and pagination, there are some other ways to handle duplication (they are covered in the article).
If you still have questions, don’t hesitate to ask them 🙂
the article is helpful for me.
can u share me how to fix step by step process ppt
we are getting same issue
site:eposdirect.co.uk
site:eposdirect.co.uk/&
please help waiting for reply
my mail is rakhi2b8@gmail.com
Hi Rakesh,
thanks for your comment. Sorry, but we do not consult via email.
It’s not clear what issue you are getting. The whole article is a step by step instruction, so maybe if you describe your problem we could give you a few hints.
Cheers!
ok,Thanks 🙂
Hi Kristina Azarenko,
This is my website (www.eposdirect.co.uk/uk_en/) have the duplicate content,
site:www.eposdirect.co.uk/uk_en – 164 indexed and site:www.eposdirect.co.uk/uk_en/& – 97indexed in google, Please send me file resolve this issue in ppt.
Hi there and thanks for the comment!
Although I don’t understand what ppt file you’re talking about. We don’t have any presentation files, the whole post is the instruction on how to get rid of the duplicate content (which you can certainly apply to your store).
Cheers!
Thanks for your great post. I looked through the mentioned items and found a lot of things to be done on my shop.
Thanks for reading, Karen!
Hi, I have been patiently building a site in a live environment, bodygoal.co.uk ….. Fantastic post first of all! Ok, so I have followed all the instructions which should knock about 2300 products off the duplicate indexes. I have 17629 products, Google has indexed according to the site: command , 31400 products……pheweeeee! However the plot thickens, within the site map area of webmaster tools Google claims to have only indexed 377 pages! No errors?!!! Help!
Hello Paul,
Thanks for commenting.
As I can see, Google really shows that 31000+ pages of your site are indexed. However, only 26 pages of Google SERPs are available, i.e. about 260 indexed pages. The best thing you can do in this case is to relax, apply the best practices of fighting duplicate content and wait for the results. Google may be slow at showing such information, especially if your site is new.
Good luck!
Hey,
Great article by the way, something i have been meaning to do for ages but didnt quite know how to go about it so thank you for that. Quick question though in step 1.3.2 where you say add
to the sorting pages. in magento im sure you know most pages work off of .phtml files. What file would i add this code to to make sure it works, the layout file, the view.phtml which sets up whats on the page or the list.phtml which gets the products?
Thank you
same question for the 2.2.1 section as well.
Thanks
Hello Adam, thanks for your comment.
You can turn on canonicals for category pages by going to System >> Configuration >> Catalog:
Thank you
Hi
thanks for this great article ! Please have a look in this article from GOOGLE http://googlewebmastercentral.blogspot.de/2013/04/5-common-mistakes-with-relcanonical.html
if you use category canonical in Magento you’ll have the same canonical url of page 1 for all other pages, so GOOGLE just uses your page 1 and not the other content & products of the other category pages. Thats what Google says. Have you made other experiences than Google?
Hello Steffen, thanks for your comment!
You are right about rel=canonical, it shows only the canonicalized page to Google and states that other duplicate pages should be ignored. That’s why I wrote in the article “If canonicalization set up correctly, you’ll see the cache of the canonicalized page even if you check the cache of the sorting pages” – this can be applied to sorting, pagination, etc.
This is how canonicalization works for Google, so hardly anybody had other experiences with that 🙂
Hello,
I have a doubt about csv file template to import coupons.
Anyone can help me?
Hey Ricardo, are you having a trouble with an extension? Could you please submit a ticket here so our support managers could help you? https://amasty.com/contacts/
Thanks in advance!
Hi there,
First of all, thanks A LOT for your article. I have been searching about this issue for a while and this article is exactly what I needed!
I just have 3 quick questions about your solutions. In case it helps, my website is tictoque[dot]com. My attributes/filters create a huge amount of duplicate content despite canonical links being long enabled in Magento Admin… So here are my questions:
1) Do you recommend using canonicalization for the sorting pages or using URL Parameters in GWT or both? (My goal is to prevent duplicate content from bein indexed in the future AND to de-index all these sorting pages that are already indexed and cached by Google…)
2) If I use canonicals, how do I add the canonical tags to the canonical pages and to the sorting pages? Do I have to edit phtml files via FTP (and if so, in which directories) or can I do it in Admin?
3) If I edit URL Parameters in GWT and set them to “No URLs” like you suggest, will that make Google de-index the corresponding pages or do I have to additionnaly go to GWT “Google Index / Delete URL” and ask for URL deletion?
Thanks a lot again!!
Hi Sebastien,
I’ve looked through your site and will try to answer your questions.
1. I would recommend using both solutions just to be on the safe side.
2. You’d better ask your developer about that. I’m not sure that it is possible in Magento without any customization or extension.
3. Yes, if you let Google know that the pages with particular parameters are duplicates and should not be indexed, it will de-index them. It will take time but at the end they will be removed from the search. Of course, you can use URLs removal requests on the URL-by-URL basis but it’s quite time-consuming.
As for your site, I see that the URLs like “__store=default&couleur_vin=30&dir=asc&order=position&___from_store=default” were last visited more than a month ago, so if your canonicals were implemented not long ago, Google hasn’t found them yet. Your site is quite new, so be patient 🙂
Hi Kristina,
Thanks a lot for your answer. I will follow your advice then 😉
And actually, I can already see the canonical links pointing to my main category in the head section of my sorting pages, so you are right, I just have to wait for Google to find them…
What worries me is that canonicals were actually implemented quite a long time ago, before the last crawl from Googlebot. Do you advise me to wait for a few weeks or to already use GWT and set the relevant parameters to “No URLs”?
Thanks again 🙂
Hi Sebastien,
It’s up to you. But I would rather wait for the canonicals to get noticed by Google and add parameters exclusions in GWT after that.
Kristina,
Thank you for the helpful article! We have category paths for URLs currently set to “yes”. So the full category path is being indexed. We realize that it should have been be set to no. Our site has been indexed and is getting SEO traction. However, we would like to improve and since everyone states this should be set to no we feel we should change the setting. Our concern is what the affect will be on page rank after we make the switch. Any advice or suggestions?
Thanks again!
Hi Kristian,
If you remove duplicates, the effect will be positive. In your case you have 2 options: exclude all the category paths from the products URLs or leave only 1 category path per product, so it’s up to you which way to go. You can get back to the above article for more details on these options.
Hi Kristina,
I have been dealing with many canonical issues for awhile, but had a question. Take a look at mothersfamilyrings.com/mothers-rings/two-stone-mothers-rings.html?limit=15. See the canonical url, it appears wrong to be because of having the parameter included. Would it be better have my developer set these to the default url (mothersfamilyrings.com/mothers-rings/two-stone-mothers-rings.html) and/or adjust it in Google Webmaster Tools?
Hi Jeff,
It seems I’ve already answered that question above. You can go for the canonicals first. Once Google notices them, adjust the parameters in GWT.
Hi Kristina,
I need to get products which are having same values in their attributes.
For example,
If any product attribute (color, size, manufacturer) values matches with another product attribute values, then need those products to be added in product collection.
Thanks
Hi Umesh,
thanks for the question. Looks like you need to ask a Magento developer to do this task for you.
Thank you for this post. I though I was going to experience a duplicate content hell with magento when I noticed google crawling all those url versions. You saved me. I used robots.txt to remove all unwanted url paths that were already indexed. You did not mention anything about index.php duplicates where all you website could get duplicated trough the index.php path. I removed the index.php with robots.txt. Is it best to redirect it?
Hi Ricardo,
Yes, you are right, sometimes homepages can be accessed with index.php as well, so it’s better to set a 301 redirect pointing to the version without it.
I have 1000 pages on my sites. I changed the magento suffix from “- myshop” to “myshop .com” now google has indexed 2000 pages. A few week later I changed again the suffix of my store to “| mishop” and google indexed again 1000 pages. Now I have 3000 pages indexed instead of 1000. Google ignores completely that it is the same page and that only the title changed. This search engine is completely STUPID.
Hey Chris,
thanks for sharing your experience.
However, I’m not sure that changing the suffix can cause the duplicate issue. These things are not connected at all. Are you completely sure that your shop does not create duplicate pages with various titles?
You’re probably right (I was tired) however magento is generating duplicate content even with all SEO rules like someone posted earlier. canonical links, link rel and NOINDEX on any filters/searchs doesn’t fix the problem. I ‘m working on it for months, saw lots of comments in magento forums since 2014 and nobody cannot precisely explain why. I tried another CMS like prestashop (without mods) and I do not have any duplicate issue with google. For startups and very small businesses who rely on natural indexing, magento is a nightmare if you don’t have enough backlinks.
Hey Chris,
if your installation still generates duplicate content and settings/extensions aren’t helping, maybe it can be a good idea to ask a developer to have a look at what exactly causes duplicates and to fix it manually?
I’m a developper myself (Since 1992) and PHP since 2000 😉
Can you provide just one Magento site used in production without duplicates confirmed by google with site:www.websiten-ame.com – I checked a dozen of magento sites and all of them are full of duplicates in Google. Thanks for the help.
(doesn’t mean I’m good)
Chris,
you can check our site amasty.com – it’s on Magento.
I mean a classic ecommerce magento www shop with at least 500 products, filters, colors, ajax etc… for amasty.com google says 2400 indexed pages (1900 without your blog) for just 124 products, does it sound right?
Chris,
I understand what you mean, but the number of products doesn’t mean anything here. If the settings are right, there won’t be duplicate issues even for a large store, and if the settings are wrong, there will be duplicate issues for a small store as well.
As for our site, if you look through the indexed pages, excluding blog and support articles, here’s what you’ll see: the products themselves, CMS pages, categories, knowledge base articles, user guides and demos for each extension. According to Google Webmaster tools, we have around 500 pages indexed by Google (this is a more exact data if compared to what the SERP tells you). If you search for any product and limit the search by our site, you’ll see we have absolutely no duplicates.
I know that you’re frustrated by what’s happening by your site, but it’s difficult to say what’s happening without seeing the site itself. Are you using any SEO extensions? Have you looked through Google Webmaster Tools?
Yes, titles are unique.
The author needs to brush up his knowledge about use of canonical tags on paginated pages. Paginated pages are not the case of content duplication. More info. on Official Google Webmaster Blog here.
https://webmasters.googleblog.com/2013/04/5-common-mistakes-with-relcanonical.html
Hello,
Thanks for your comment. You are absolutely right about paginated results and using rel=canonical (though paginated URLs ARE partial duplicates as they generally have the same navigation, sidebars, footer, meta data and similar content).
Using rel=next/prev is also mentioned in the article as a way to indicate paginated results. Although this is a preferable way of handling such URLs, it’s not always possible to implement it, and using canonicals in this case is possible if to choose between them and no changes at all.
Howdy! I simply want to give a huge thumbs up for the nice info you will have here on this post. I will be coming back to your weblog for extra soon.
You are a life saviour.. This article is gonna my all time favorite.
I am working on a large ecommerce website with millions of pages. And duplicate content is the major issue, I am trying to overcome.
I am sure using some of your advice, may surely help to deal with duplicate content to a large extent.
Thanks Heaps..I really appreciate..!!
Joining all the comments above, this is a highly helpful article indeed. You just touched the spot with this topic, thank you so much!
Thank you, Lola, for your comment! We are glad to hear that.
thanks god i’ve finally found a truly useful article about the duplicate content. for some magento store owners it is a really worth to read information!
Thanks, Kevin! You can subscribe to our blog, there is a lot of other useful info for merchants.
Thanks for sharing such a nice information. Actually, i have lots of Duplicate meta for booksmela.com and I was looking for something you shared and it resolved my issue. Thanks a lot again
Hi Robert,
I’m happy that the tips I’ve shared are useful for you. Good luck!
We are finding duplicate content in Magento for our UV Gel store a real problem, I hope that with these suggestions we can reduce Magento duplicate content.
Hi,
Great post, very helpful!
I have a question regarding section 1.3 (LEAVE ONLY ONE CATEGORY PATH IN A PRODUCT URL).
We are on Magento 2.1.4 (M2) Community, and we have exactly this situation where multiple categories would be associated to same groups of products:
FIBER OPTIC >> FIBER TOOLS >> FIBER CLEAVERS
specialized.net/fiber-optic/fiber-tools/fiber-cleavers.html
TOOLS >> FIBER OPTIC TOOLS >> FIBER CLEAVERS
specialized.net/tools/fiber-optic-tools/fiber-cleavers.html
How would we solve this problem in M2? If I’m correct “Unique Product URL” extension is not available for M2.
In addition, according to what I’ve been able to find online, apparently you cannot set NOINDEX, NOFOLLOW meta tags on only certain landing pages in M2.
Thanks in advance!
Hi Immanuel,
I’ve covered this point in another article on Magento 2 SEO https://amasty.com/blog/magento-2-seo-full-overview-default-settings/
The point ‘Category path in product URLs’ answers your question 🙂
For clarification: if I do the test described: site:youronlineshop.com and site:youronlineshop.com/& and I find almost the same number of pages indexed, will that mean that there is no full duplicate content issue and there may be a partial duplicate content issue or can I discard the duplicate content worry altogether? Thank you
Hi Sebastian,
Thanks for a good question. In your situation I would still search for site:youronlineshop.com and look at the last pages of Google index, most URLs with parameters are listed there. If even last pages list only clean URLs with unique content, then everything is ok.
That’s all plain, thnx. And what about utm links I use for blog channel will google see them as a duplicate content?
Hi Michael, thanks for reading us and for the question! In your Google Search Console, there is a section where you can tell Google how to handle your URL parameters (consider as unique content, crawl, index or not). Cheers!
I added canonicals and waited for two days as you say but I don’t see Google crawled them. What should I do?
Hi there, thanks for leaving your question! Pity you’ve come up with the issue. Yes, despite the settings seem easy, they may cause you troubles. First off, stop panicking as two days is not a limit for Google bots. Check if you haven’t misprinted any sign in the tag. Make sure you put it into the
tag of the desired page’s code or made it correctly via the backend. Good luck with that!Hola Amasty, thanks for useful tips! pagination is real disaster.
Hi Correia! Thanks for reading and sharing your opinion. We are glad to be helpful to our readers.
You offer a lot against duplicate content saying no words of robots.txt. Doesn’t it work?
Hi Mathew, thanks for your question! As soon as Panda Update can ignore all the URLs included into the robot.txt file and consider them as unique. Therefore, a set of other variants is offered in the post. Hope this helps!
Could you help me with this? I set canonical for a page with mobile an desktop versions but Google chooses a different url as canonical.
Hey there, thanks for the question! Sometimes Google can choose another URL to consider as a canonical. It happens due to different reasons. It could occur in case the bots decided another page is more relevant for the topic, complete and useful. The search engine decides basing on multiple parameters: http or https, quality, presence of the URLs in the sitemap, “rel=canonical” and others. You can apply different techniques telling the bots what to do, but, unfortunately, the result can be surprising.
What can you say about mobile version of a main site?Will google see it as duplicate content and do I need to rewrite the content for it?
Hey Harry, thanks for your commenting! You probably know that Google has separate bots that crawl and index mobile site versions. They won’t be considered as duplicate content. Cheers!
Hi Amasty, thanks for the tips they are handy! Will you update it for M2?
Hi there, thanks for reading and commenting! We’ll see what we can do with it in our future posts. Though, the principals used for duplicate content avoidance are mostly the same for Magento 2. Cheers!
don’t know why someone is sweating it I ain’t have any problems
Hi Evans, thanks for commenting! We are glad to be helpful.
You say about rel=canonicals, but don’t say where I shoudl put it?
Hey there, thanks for commenting! The tags like rel=”canonical” are usually placed in within the head tag in a duplicate page HTML code. However, usually, the labeling is placed using different tools but not manually. Hope that helps!
How can I find the duplicate pages?
Hi Kinzel, thanks for reading us and for the question! You may use one of the methods described in the post, use screaming frog or any other special tool. Cheers!
The best solution for a brand to grow nowadays is Digital Marketing so I think it’s very much essential for brands to go for digital marketing which will increase their awareness and elevate its sales.
Hi Logo, thanks for commenting! You’re absolutely right, and nobody here will deny the importance of digital marketing tools. However, as time goes we come across with new and new marketing techniques like growth hacking, for instance. In this post, we just wanted to stress the importance of simple SEO showings, such as duplicate content on your Magento website. I hope you’ve found something interesting for you in this article.
Thanks for sharing this valuable information
Hello!
Thanks for your feedback!