John Mueller Discusses Possible Lag In Google Search Console

The best SEO consultants use Google Search Console to analyse their site's indexing report, allowing them to know if the search engine has indexed their pages. However, SEO consultants recently noticed an issue where some URLs are excluded from the list of indexed pages. But upon examining the said URLs, the tool listed them as indexed.

Google's John Mueller said that he has also seen reports on this problem, and he has an idea of what could be the cause.

Pages Were Crawled But Not Indexed

An SEO asked Mueller about an issue where Google Search Console reports that they have many Crawled Not Indexed Pages under the Excluded category. However, when they clicked through them in another report, the tool indicated that the links were converted into indexed pages.

The problem has made it difficult for them to accurately monitor and analyse their website's crawling and indexing statistics. Moreover, they do not know if it is affecting their crawl budget.

Crawl Budget Impact

The SEO who asked the question was concerned that the "crawled but not indexed" situation will cause a problem with their crawl budget.

A crawl budget refers to the number of links that Google allots to crawl on a website. A site's crawl budget is partly determined by its server's ability to serve pages, which is called Crawl Capacity Limit.

Therefore, if a server has difficulty serving pages, Google might limit how much it crawls the site. Thanks to this, the server's capability to serve pages will not be hampered. On the other hand, servers with a faster response time can allow more requests from GoogleBot. A faster response time convinces Google's system to increase the crawl budget so that the search engine can crawl more pages.

Another factor that influences the crawl budget is how frequent an SEO updates their website. If they rarely update their site, Google will crawl it less often.

The person then later revealed that their website has thousands of web pages, but Google only indexes 2,000 pages every day, which means that there are many pages GoogleBot could not crawl.

The underlying concern that the SEO didn't raise before was more about why Google didn't index their other pages and whether or not the "Crawled Not Indexed" problem was related to the crawling issue. However, these have not been brought up yet in the hangout.

So, Mueller only focused on answering what the person asked him at that time, which was about the "crawled but not indexed" issue and whether it affected their crawl budget. He said that the situation most likely would not affect their crawling budget. Next, Mueller discussed why Google would flag a crawled page as "not indexed" when it has, in fact, been indexed.

Mueller noted that he had read several Twitter threads where SEOs saw that their URLs were flagged as not being indexed in Search Console. But when they check the pages individually, they find out that they have been indexed. He admits that he does not know the main cause of this problem, but he suspects that it is more a matter of timing.

It could be that Google shows the URLs in the Search Console as crawled but not indexed, and they get indexed as time goes by. At some point, the pages would get excluded from the report again, and this could take a little bit longer than it should.

Verify Index Coverage Issue

Mueller suggested a method that SEO consultants can use to verify if their Google Search Console reports are caused by a real index coverage problem or just a lag in reporting.

He advised online businesses and site owners to determine if their pages show up for regular searches. To do this, SEOs can take some words from the web page in question and type them into the search engine box. If the piece of content shows up, it confirms that there is just a lag in the Google Search Console reporting. In such cases, there is nothing else they need to do.

SEOs are hoping that Google would fix the lag issue as soon as possible because it gives a poor experience and false information to its users.

Reason For Excluded Pages In Google Search Console

Online businesses and site owners should monitor their web pages under the excluded category. This section shows the corresponding reason for the exclusion of a web page in Search Console.

The feature also allows SEOs to see why Google excluded the page from their index. Here are some of the most common reasons why a page is under the excluded category:

1. Crawled - Currently Not Indexed

It's possible for Google to crawl a page, but they choose not to index it for whatever reason. SEOs can use the site:search in Google to see if the page is indexed. If not, they should confirm that the page is in the sitemap, not a duplicate page, not canonicalised to another page, or disallowed in the robots.txt.

2. Crawl Anomaly

Sometimes, Google may be unable to access a page. One can use the URL Inspection tool to find any errors or crawl the page using an SEO tool like ScreamingFrog.

3. Duplicate Without User-Selected Canonical

GoogleBot may find more than one version of a page, but if none of them have a canonical tag, the search engine will think it is not a primary page and won't index it. With this in mind, SEOs should always use the proper canonical tag on each page.

Work With The Best SEO Agency!

Position1SEO is comprised of a team of the best SEO consultants around. With our help, you can be sure that all your web pages will be primed and ready for crawling by Google.

Thanks to our years of experience in the SEO industry, we have developed tried-and-tested white hat tactics that will get you a spot on Google Page 1 without any risk of penalties!

We take a no-stone-unturned approach to all kinds of projects - big or small - and we promise to identify and fix every possible issue that could be holding back your site's rankings.

Some of the common tasks we can do for you include but are not limited to:

• Google Search Console
• Google Local set up
• Website load speed, performance, and accessibility
• Sitemap installation
• Accelerated mobile pages
• Directory citations
• Positive link building

If you are interested to know more about our services, email us at office@position1seo.co.uk or dial 0141 846 0114.