Google’s John Mueller answered a question about at what point will 500 error response codes cause Googlebot to crawl content less frequently. Mueller explained scenarios where 500 response codes do not affect crawling and when they can cause pages to drop from Google’s index.
Accidentally blocked Googlebot and caused 500 errors
The person who asked the question explained that their content delivery system was configured to block bad bots while allowing the actual human users and legitimate bots to browse the site.
Everything worked fine until they updated their “server monitoring suite”, and apparently the previous bot blocking filters and settings were not transferred to the new system.
Afterwards, they noticed that the Google Search Console started listing 500 error response codes in response to Googlebot crawling.
The person finished explaining what happened and then asked the question:
“The question came from our technology team as to whether this represented real user impact and why we would specifically look at Googlebot and not real user metrics to prove that there is a problem here.
So given that context, I have a few questions.
The first is just to get the technology concern out of the way.
From your perspective, how does Googlebot 500 view serial errors, and can you provide some clarity on established… thresholds, at what point will Googlebot gennem review source content… less, based on these errors? ”
Continue reading below
What is a 500 Error Code?
Servers communicate a variety of status response codes to browsers and bots. These response codes communicate, for example, whether the request for one page was successful, redirected to another page, blocked, or failed.
The 200 reply code indicates that the request for a web page was successful. The 500 response code means that the request for a web page failed.
The formal name of the 500 response code is:
“500 internal server error”
There are other versions of the 500 response code that convey a specific reason why the request for a web page failed.
The 500 response code is somewhat generic because it simply means that the page request failed. But that often means the server was down, but not always.
Google Search Console and 500 error answers
It is a good idea to monitor the Search Console for 500 error responses because they are a strong indicator that something is wrong.
Sometimes it can mean that the server is under heavy load, for example from too many bots, and because of this it cannot display web pages.
Continue reading below
In this case, it appears that the problem was due to a configuration error in the Content Delivery Network (CDN) that inadvertently blocked Google from accessing Web pages.
How Googlebot indexing is affected by 500 errors
Mueller provided a helpful explanation of the steps Google is taking in response to 500 error codes, and how, in the end, continued errors could lead to these web pages being dropped from Google’s search index.
Google’s John Mueller replied:
“We have no strong thresholds for that.
But what happens with 500 errors is that we try to try them again.
And if we continue to see the 500 errors, we will slow down the review.
And if we continue to see that there are 500 errors, then we’ll remove those URLs from the index.
So it’s something, if every now and then there are a few pages with a 500 error, it’s no problem.
We try them again, they stay indexed, and the next time we try them again, it’s fine. ”
Next, Mueller explains that the reason Googlebot slows down the crawl is in the event that Googlebot itself causes the server to overload.
“But if a large part of a site consistently has 500 errors and we can assume that we might be causing the problem and we will slow down the crawl of the entire site and at some point we say well it looks like pages are really away, we’re going to drop them.
So those are essentially the effects that you would see there, and if you’re talking about a big site and wondering what percentage of 500 errors are okay … I do not know.
My feeling is that if you see anything more than one percent, it sounds like something is broken and would probably be something where we would start to slow down.
But I do not think we have any hard thresholds where we will say so many requests and many mistakes mean it slows down a lot. ”
Continue reading below
Monitor search console for 500 errors
It is up to each publisher to decide what their daily routine is. For some people, it is enough to check the search engine once a week. Some may find it reassuring to check the Search Console once a day to respond to unforeseen circumstances.
In any case, 500 error codes are something to monitor the Search Console for. As John Mueller mentioned, 500 big mistakes can be a flag that something is wrong and needs immediate attention.
World Wide Web Consortium (W3C) List of server response codes
How does Google respond to 500 response codes?
Watch the discussion at the 13:48 minute mark