John Mueller Explains How 500 Error Codes Affect Google Indexing

Professional SEO experts always have to deal with error response codes, particularly the 500 error codes. A person asked Google’s John Mueller about the starting point where 500 error response codes will affect Googlebot and cause it to crawl the pages less often. Mueller gave SEO experts some scenarios where 500 response codes will not affect crawling and situations where these error codes would cause a drop in search visibility.

The person who asked the question told Mueller that they intentionally set up their content delivery system to block bad bots while allowing legitimate bots and human users to access their website. They said everything was under control until they made changes to their “server monitoring suite”. That’s when they noticed that they could not apply the bot blocking filters to their new system. Moreover, they observed that Google Search Console began listing 500 error response codes after the Googlebot crawled their site.

The person explained that the question came from their technology department, whether it represented real user impact and why they would focus on Googlebot instead of real user metrics to prove there is an issue.

With that context, the SEO laid out a few questions. Firstly, they wanted to get the technology concern solved. They asked Mueller about how Googlebot views 500 series errors and specific thresholds where the bot will crawl source content less based on those errors.

What Are 500 Error Codes?

Servers use various status response codes to relay information to browsers and bots. For instance, the response codes tell the search engine whether a web page is successful, blocked or unsuccessful, or redirected to another page. The 200 response code tells the search engine that the request for a page was successful, while a 500 response code means it was unsuccessful. The formal name of the latter is “500 Internal Server Error”.

The 500 response code is quite generic; it simply means that the page request was unsuccessful. Many other versions of the 500 response code tell search engines why the request for a page was unsuccessful. Most of the time, a site uses this response code because its server was down, but there are also other reasons a website would trigger a 500 response code.

Google Search Console And 500 Error Responses

Professional SEO experts and site owners should constantly monitor Google Search Console and watch out for 500 error responses because these strongly indicate that there’s an issue. Sometimes, it warns SEOs that their server is under heavy load because of too many bots, causing it to be incapable of serving web pages.

In this situation, the issue is due to a misconfiguration in the Content Delivery Network (CDN), which blocks Google from gaining access to web pages.

How 500 Errors Affect Googlebot Indexing

Mueller gave a clear explanation of the steps Google takes in response to 500 error codes and how these responses could result in web pages dropping from Google’s search index.

Mueller said they do not have any strong thresholds regarding the matter, but Google retries 500 errors. He added that it is not a big deal if their system sees a 500 error on individual pages now and then. If they continue encountering the 500 response error, they will drop those links from their index.

Next, Mueller said that Google slows down the crawling because their bots could cause the server to overload. If a large part of the website consistently has 500 errors, Google’s system might think that it is causing the problem. Therefore, it may slow down when crawling a website. At some point, it will begin to think that the pages are gone and eventually drop them from the index.

When it comes to bigger websites, Mueller admits that he does not know the percentage of 500 errors that are considered acceptable. He said that if an SEO sees more than one per cent, then there is a possibility that there is an issue, and the system would start slowing down with their crawling. However, Mueller thinks they do not have any solid thresholds where a specific number of requests or errors could cause the bots to slow down by a certain amount.

It is up to each publisher to decide what they want to work on for the day. Some check their Search Console once a week, while others get peace of mind if they check the tool every day so they can respond to unforeseen issues and solve them quickly.

Whether an SEO often or seldom checks their Search Console, monitoring the tool to look for 500 error codes is an important task. Large-scale 500 errors could be a sign that something is indeed wrong and needs immediate attention.

It is crucial to fix 500 internal server errors as soon as possible. This is because if site visitors keep seeing these error response codes, it could chase them away, leading them to turn to competitor websites instead. Aside from this, Google could also lower the website’s rankings and visibility in their search results.

Having Issues With Your SEO? Position1SEO Can Help!

Position1SEO is comprised of a team of professionals who can help you solve all kinds of SEO issues. We can fully optimise your site, boosting your rankings and online presence and giving you a higher return on your investment.

Our SEO experts only use proven white hat tactics, allowing you to achieve the best results without getting penalties from Google. We also focus on these four main factors in SEO to boost your rankings:

• The number of your site visitors
• The time those visitors spent on your website
• The number of pages they visited within a session
• The bounce rate of site visitors

We will provide you with accurate and detailed reports after conducting an in-depth SEO audit on your website. You can choose which tasks you want us to handle for you to create the perfect SEO package that suits your business’s specific needs.

With our help, you can choose any key phrase you want to target at any competition level. Moreover, you can target as many or as few keywords as you like, and your site will still be guaranteed a spot on Google Page 1!

For more information about our services, you can contact us on 0141 404 7515 or email us at office@position1seo.co.uk.