Googlebot’s Crawling and Indexing Activity Reportedly Drops

what are the changes in crawling and indexing in SEO

There have recently been numerous discussions within the SEO community about crawling and indexing in SEO, particularly about a decline in Googlebot’s crawling activity over the previous weeks. This led some SEO experts to speculate on the real cause of this phenomenon.

One web crawl analysis service creator tweeted an infographic explaining how Google’s crawl rate has been declining since 11 November this year. Although this slowdown does not impact all websites, many people on Reddit and Twitter believe something changed in Google’s way of indexing and proved it with images of activity from Googlebot.

Proof of a Decline in Google’s Crawling

A lot of anecdotal findings regarding Google crawls have been accumulating on social media. However, social media may not be the most accurate source of information as any remark about Google may be expected to receive agreements. Anecdotal evidence, while fascinating, is nothing compared to observations backed up by data.

The founder of a crawling and log analysis service tweeted a graph showing a stunning drop in Google’s crawling activity on 11 November. They said that many big websites were involved in this phenomenon.

The Relationship Between Googlebot Crawling and 304 Server Response Codes

Some users have pointed out that Googlebot no longer crawls pages with a 304 server response code. This type of response code means 304 (Not Modified), anda server generates it when Googlebot or a browser submits a conditional request for a web page.

It means that Googlebot or the browser tells the server that it saved a page in the cache, so there is no need to serve it unless it has been modified or updated.

The HTTP Working Group states that the status code indicates that a conditional HEAD or GET request has been submitted, resulting in a 200 (OK) response if the condition was evaluated to false. This means that the request indicates that the client made the request conditional and has a valid representation already. Therefore, the server doesn’t need to transfer a representation of the target resource and redirects the client to use that stored representation as if it were a part of the 200 response.

Claims That Less Googlebot Crawling is Caused by 304 Response

An SEO stated that they monitored a number of websites with AMP, and a drop occurred on pages with a 304 response. The person who originally tweeted about it posted a graph demonstrating how Google almost stopped crawling web pages with a 304 server response code.

Many SEOs also shared similar experiences where web pages with a 304 response code had a decline in Googlebot crawling. Someone else noticed that their travel pages had a decline in crawls, while their e-commerce web pages had a crawl increase. Others in the SEO community were also sharing screenshots of data from their Google Search Console and Analytics.

Crawling Not Affected  by 304 Response Codes

The Google developer help page on crawling says that a 304 response should not hinder crawling. According to Google’s official documentation, they signal to the indexing pipeline that the page content is similar to how it was when it was previously crawled. The indexing pipeline may change the URLs’ signals, but the response code does not affect it. This statement reassures that the response code should have no impact, but it could also be that Google temporarily changed things or that the developer page isn’t up-to-date.

Cookie Consent Theory

The 304 Server Response theory is one of several theories and approaches to explaining why a web page is not indexed. One person wrote that, after they removed a cookie consent bar, Google began to index more pages. Many SEOs wonder why a cookie response bar would prevent Google from indexing a web page. It could be that the cookie consent bar may trigger a 304 response and cause Google not to index the page.

Reddit Discussions About Reduced Googlebot Crawls

The decline in Googlebot crawls also reached Reddit. A Reddit user recounted how their successful site’s articles were indexed within 10 minutes of uploading them using Google Search Console. However, things changed in November as less than half of their newly-published articles were indexed, even if they had already manually submitted them after publishing.

Many other Reddit users shared a similar issue, and they concluded that there could be something going on with Google.

Mueller Says Nothing Is Broken

Google’s John Mueller also shared insights. He said that at the moment, he doesn’t believe there is anything wrong with Google’s indexing methods. However, he thinks Google is just more critical about what they choose to include in their indexes, just as any search engine should.

The Possibility of Google Testing New Crawling Patterns

In October, Bing revealed the new IndexNow, an open-source indexing protocol that aims to reduce the number of times search engines crawl web pages and the amount of energy consumed at data centres by crawling and powering servers. The new method benefits publishers since it shortens the time to notify search engines when sites are created or modified, resulting in faster indexing of web pages.

In November, Google said it would test the new IndexNow method to see if they could gain any advantages. One of the most pressing problems today is energy conservation and carbon reduction. It may be possible that Google is developing ways to reduce crawling without necessitating a substantial change in protocol.

There are reports that Google has stopped indexing entirely, but this is incorrect. However, there is evidence that Googlebot’s indexing behaviour has been altered in response to the current debate on social media, with screenshots of data to support this matter.

Position1SEO Can Help You With Crawling And Indexing

Position1SEO is a UK-based SEO company that specialises in optimising web pages, making your website ready for crawling and indexing in SEO. Our services help your website attract more clicks and traffic, bringing more potential clients in for your business.

We are experts in converting site traffic to paying customers. Our affordable SEO packages ensure that you can increase your profits without any upfront costs or contracts.

Our services are tailored to suit your specific business needs, so you can be sure that you’ll get the best results possible. Whether it’s an ecommerce store, blog, content marketing project, or anything else, we can get started on it right away!

changes in crawling and indexing in SEO min1
Author: Jason Ferry
Jason Ferry is an SEO specialist based in the United Kingdom. Taking pride in his many years of experience in the search engine optimisation industry, he has honed his skills in various digital marketing processes. From keyword marketing, website auditing, and link building campaigns, to social media monitoring, he is well-versed in all of them. Jason Ferry’s excellent skills in SEO, combined with his vast experience, makes him one of the best professionals to work with in the industry today. Not only does he guarantee outstanding output for everyone he works with, but also values a deep relationship with them as well.

Related  Posts

the benefits of an search engine optimisation audit
Search Engine Optimisation (SEO) is an essential ingredient for website success in the digital arena. The process of indexing and ranking websites on Search Engine Results Pages (SERP) requires constant evaluation. Therefore, it is vital to conduct an in-depth SEO audit to identify areas that need improvement. SEO audit is the process of evaluating a […]
search engine optimisation company
Search Engine Optimisation (SEO) is a crucial aspect of building a strong online presence. While many website owners focus on using keywords to rank higher on search engines, an SEO company can optimise your website in more ways than one. In this blog, we will explore the three areas an SEO company can improve upon […]
The importance of hiring a search engine optimisation consultant
Are you struggling to get your business noticed online? Is your website buried under a sea of search results? If your answer is yes, then it might be time to consider hiring an SEO consultant. With the ever-growing importance of online presence for businesses, it has become crucial to employ the right strategies to make […]