Google’s John Mueller answered a question about the Google Search Console (GSC), which shows crawling errors on pages that load well in the browser. John Mueller explained that the problem typically exists on the server side and not a problem with Googlebot.
The person asking the question tried to validate the web pages in the Google Search Console (GSC), but the pages stubbornly refused to validate. It’s as if something is blocking Googlebot and showing an error message.
Here is the question:
“I had a server error on some of my pages. When I checked the few pages shown in the example, they worked fine.
I also used the validation option a few times, but Google persistently marks the pages with an error.
It’s been a month since then.
I’ve been waiting for Googlebot to index these pages without success.
This has affected my organic impressions and clicks.
Is there anything I can do here? ”
John Mueller confirmed that if GSC shows server errors when Googlebot crawls the web pages, they really do exist and are not an error on Google’s page.
Continue reading below
Here’s what John said:
“We do not invent errors on pages. When Googlebot checks the page and there is a server error, we really see a server error there. ”
Temporary crawl errors
Next Mueller noted that some of these problems are temporary.
Here is what he said:
“And it may be that this is something that is temporary on your site, and if it is temporary, with one of the future reviews, we will … try again.
And if the error is gone, we can index the page normally. ”
Sometimes these errors are actually temporary.
For example, a server may go down for maintenance, there may be problems in DNS that remove part of the Internet, or it may also be that the server is overloaded and prevents crawling.
But this is not the problem that the person asking the question is experiencing.
If the web pages load normally, but every time you validate with GSC (or try to test it with one of Google’s tools like the test result for rich results or mobile friendly tests) and it does not validate, it’s a problem on the server.
Continue reading below
Server problems can cause GSC indexing errors
John Mueller then suggested that the problem with the person asking the question could be server-related.
“So it’s something if you see these kinds of issues come up regularly, and especially you use the validation feature in the search console, and the validation comes back and says there are still server errors, then that’s something I would address with your host.
And try to see if there is anything they can do to diagnose this problem to double check what might be happening here to give you a sense of how many URLs it is actually affecting.
For sometimes it can be difficult where if Googlebot crawls millions of pages from your site and hundreds of them have a server error, then it’s probably irrelevant, because as if there’s always some error somewhere.
But if Googlebot crawls 200 pages from your site, and hundreds of these are server errors, then it’s a little more about what you’re likely to fix, and make sure that does not happen.
So that’s the direction I would go.
This is not something that Google can fix.
It really is something you need to correct on your side with regard to hosting. ”
Diagnosing Googlebot Crawl Errors
There is a diagnostic trick that you can do to determine if this is a server-wide configuration issue if your IP is shared by other sites on the server.
What can be done is to identify the IP address that the site is at, and then run that ISP through a reverse IP check that can show which other sites are hosted at that IP address.
Then you take the list of domain names and run them through one of Google’s tools like AMP Checker or Google’s rich results checker tool.
If the tool reports an error response for one or more of the domains, it may indicate that there is a server-wide error that
Each server has a server log and it is a good place to start diagnosing what is causing the problem. These server logs show the date and time when an error occurred, as well as the IP address of the visitor who triggered the error.
Continue reading below
A typical problem could be a bug in how to create a firewall that may be set too strict and blocking Google.
If you do not have access to server logs, a web hosting call is OK, as recommended by John Mueller.
See Google Office Hours Hangout
Why does GSC show server errors on pages that are fine?
Look at the 47:05 minute mark