John Mueller Says It Is Normal For 20% Of A Site’s Pages To Not Get Indexed

Online businesses and site owners optimise their websites for smooth crawling and indexing in SEO. A publisher recently noticed how Google had not indexed 20 per cent of their website. In response, Google’s John Mueller shared his insights on indexing and how site quality affects a search engine’s indexing patterns. Moreover, he told the SEO community that it is completely normal if Google does not index 20 per cent of a website’s pages.

The SEO told Mueller that 20 per cent of their web pages are not getting indexed. He said that these pages are only “discovered” but not “crawled”. They also shared their website’s background information, saying that there is a possibility the server was overloaded, which could affect the number of pages that Google indexes. Overloaded servers usually experience 500 error responses because they cannot serve a web page the standard response.

However, the person did not confirm nor mention anything about receiving Google Search Console reports regarding Googlebot receiving 500 error response codes. If this is the case, it is highly unlikely that the server overload issue prevented Googlebot from indexing 20 per cent of the website’s pages.

The person also asked if the issue has something to do with the site’s overall quality.

Mueller Explains Factors That Affect Indexing

Mueller replied to the SEO about the several factors that could affect Google’s indexing.

Firstly, Mueller made it clear that the crawl budget is not why Google refuses to index pages on a small website. However, if a bigger site with millions of pages is involved, one should consider the crawl budget.

Secondly, he explained how the overall site quality affects the number of web pages that Google crawls and indexes. He also touched lightly on how Google evaluates and indexes a website in terms of its overall impression and quality.

Mueller said that Google strongly considers a site’s overall quality when crawling and indexing its web pages. However, it is not necessarily relevant to the individual URL.

For instance, if Google did not index five pages on a website, it doesn’t necessarily mean that those pages are of low quality. Instead, Google views the website as a whole as low quality. As a result, the search engine will not bother crawling and indexing everything on the website.

Because of this, site owners need to make sure that their website’s overall quality is good. According to Mueller, for smaller sites containing a huge number of non-indexed pages, webmasters need to reconsider the site’s overall quality instead of focusing on the technical issues.

Technical Factors And Indexing

Mueller also discussed things on the technical side. He said modern websites are generally technically reasonable and can easily get that part right since content management systems are so common. With this in mind, we can assume that technical issues are not the main cause of unindexed site pages; it is more a matter of a site’s overall quality.

Mueller reassured the SEO community that it is normal for Google to leave several web pages as non-indexed. Site owners usually think that it is a bad sign, but Mueller is more knowledgeable about the number of site pages that are typically indexed. Not to mention, he is speaking from Google’s perspective.

Whether it is a large or small website, there will always be fluctuations in indexing. There is never a case where Google indexes 100 per cent of the pages discovered on the website.

So, if a website has a hundred pages, and Google only indexes 80 pages, then that is not a problem that one should be worried about. As time goes by, the website could grow until it has 200 pages. Google would then index 180 of them, and the percentage gets a bit smaller.

Mueller shared a lot of information about indexing. Here are the key points:

• It is completely normal if Googlebot does not index 20 per cent of a website’s pages.
• Technical issues do not hinder Google from indexing web pages.
• The site’s overall quality can determine how much of a website gets indexed.
• There will always be fluctuations in indexing, no matter the size of the website.
• Small websites do not have to be concerned about the crawl budget.

Troubleshooting Non-Indexed Pages

In some situations, issues might hinder Google from indexing a site’s web pages. Here are some troubleshooting steps from Google Search Central.

Step 1: Verify that the web page is missing from Google’s Index.

Sometimes, people assume that their page is missing from Google’s index only to find out that their page has low rankings in search results. One can do a site: search to check if individual pages are in Google’s index. If the page does not appear, it confirms that Google has not indexed the web page.

Step 2: Fix the problem.

If the web page is new, it is best to give Google some time to crawl and index it; the time frame could be anywhere from a day to a few weeks, depending on several factors.

On the other hand, problems could have occurred during the process if the webmaster recently restructured the site or moved to a new domain. One can use 301 redirects to redirect users and search engine crawlers.

Google also recommends setting up a Google Search Console account to make it easier to diagnose and solve indexing problems. The tool comes with documentation that tells online businesses and site owners the root cause of each issue and how to fix them.

Looking For The Best SEO Agency? Work With Position1SEO Today!

Position1SEO can help you optimise your web page so that your website is ready for crawling and indexing in SEO. We can improve your site’s overall quality using proven white hat SEO tactics and Google-recommended tools, such as Search Console.

By choosing us, you will receive a free in-depth SEO audit, which looks at the following areas on your website:

• Google Search Console overview report
• Mobilegeddon report (mobile-friendliness)
• Keyword research
• Domain health check report
• IP address review report
• Backlink reports
• SEO review report
• Etc.

If you are interested in acquiring our affordable SEO packages, you can contact us via office@position1seo.co.uk or 0141 404 7515.