SEO Case Study – Improper Indexing of Product Pages
CategoriesSEO
I originally wrote this as a case study for Boomtown Internet Group’s website here when I worked there as their Director of Digital Marketing.
Background
A local home décor business with an ecommerce website built with Magento signed with Boomtown Internet Group hoping to improve their presence in Google’s search results. I, Brian Donohue, was assigned to be their SEO Account Manager. They had previously been with another SEO agency, but after several consecutive months of declining traffic decided to move on.
Within my first month (April 2015) of working with the client, I discovered a major technical issue with the website and how search engines were indexing it.
Problem
The previous SEO agency had submitted a Robots.txt file requesting search engines not to index any URL containing /index.php/
Many sites require there to be /index.php/ URLs in addition to the clean URLs and it is common to block search engines with Robots.txt
However, in addition to the Robots.txt file, the previous SEO agency had also added canonical tags on every clean URL specifying that the URL with /index.php/ was the preferred version.
Canonical tags are used when a website has 2 identical, or very similar, webpages, which is common with ecommerce sites. The tag tells the search engine which of the 2 pages is the preferred page to show up in search results.
Essentially, the website was telling search engines that they preferred the /index.php/ URLs instead of the clean URLs, but at the same time were telling search engines NOT to index those /index.php/ URLs.
Consequences of the Problem
Most of the product and product category pages were not being indexed by the search engines. In fact, Google had only 14 pages indexed for the entire site even though the site actually had over 1,200 pages. This resulted in the vast majority of their webpages being left out of search results, meaning that they received zero visits from Google searches. So, if you were to conduct a Google search for a very specific product, while you were located in the business’s hometown, their product page would not be listed in the results. The only page that was ever included in search results was the homepage.
Solution
I had Boomtown’s web developers change the canonical tags so that the clean URL was the preferred version, rather than the /index.php/ version. For example, we set http://www.example.com/products/ as the preferred version instead of http://www.example.com/index.php/products which was not being indexed due to the Robots.txt file.
Results
The next time search engines crawled the site they indexed all of the product and product category pages. Soon, these pages began to show up in search results and receive traffic.
If you’re finding that your website is not showing up in Google’s search results, please contact me and I’ll see if we can fix the problem together and get your business back on track.
Thanks for reading! Be sure to follow me on Twitter at @BrianEDonohue or find me on LinkedIn here.
By Brian Donohue