Google Cannot Crawl Content Hidden Behind Captchas

SEO experts and website owners sometimes make use of captchas when they plan an SEO content strategy. But according to Google, websites can encounter issues if their SEO content is hidden behind captchas. This is because captchas essentially prevent Googlebot from crawling the content.

Why Use Captcha

“Captcha” is actually an acronym that stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. Just like what the name suggests, a captcha test is a tool that distinguishes human users from bots. Websites often use these tests to avoid receiving spam through contact forms and the like.

The original captcha forms in the late 1990s looked like a panel of obscure numbers and letters that either looked stretched, blurred, or warped. The page would require the user to identify these numbers and letters and type them into a box. If they interpret the letters correctly, they pass the test.

As time goes by, captcha tests evolve. Some captchas now show an item image – such as a street sign or a car – with its bigger version in grid form below. These captchas require users to identify all areas of the grid that show the given item. Some captchas may also present simple sums, like 4+1, while others are audio captchas for users with vision impairments.

Bots are unable to interpret the obscure numbers and letters on a captcha. In contrast, human users – who are used to seeing and interpreting letters in different handwritings and fonts – can identify them. Often, many bots will only input random letters, which will result in them failing the captcha test. Because of this, bots will usually be locked out of accessing the web page or application, while human users can proceed with ease.

Advanced bots, however, can identify distorted numbers and letters, so other websites use more complex captcha tests for such bots, like the reCAPTCHA.

reCAPTCHA refers to the most recent type of captcha, which is more inquisitive than the previous versions. This type of captcha analyses user behaviour. If the user’s behaviour is quite life-like, for instance, it will not require the user to complete the captcha test. Instead, it will ask the user to tick a box to confirm “I am not a robot”. On the other hand, if the behaviour is seen as robotic, the page will require the user to solve a more complicated captcha test.

Google’s crawlers do not interact with other elements when it crawls pages. Let’s say that Googlebot visits a web page with the main content hidden behind a captcha. In this case, search engine crawlers will detect no content – just captchas.

Captchas can be problematic depending on the situation. However, there is no reason to avoid using them. Moreover, Google’s John Mueller said there is a way to solve this issue.

Mueller discussed the matter in the Search Central SEO office hours on 24 September 2021. A directory site owner asked him if the captchas they used to avoid scraping could affect their SEO. The short answer is yes; it can affect SEO. Nevertheless, one can opt to use captchas that block content but do not interfere with crawling or indexing.

Mueller explained that Google’s crawlers do not fill out any captchas – even Google-based captchas. So, if the website requires users to fill out the captcha before accessing the content, it would hinder Googlebot from accessing the page. However, if the captcha is displayed on top and the SEO content is available without requiring the visitor to do anything, then it is fine.

Although online businesses and site owners can use captchas on their websites, Mueller recommends using Google Search Console’s Inspect URL tool. This tool allows SEOs to verify that a captcha is not blocking Google’s view.

With the tool, SEOs can analyse specific pages and see their data. They should also check the visible page to ensure that it matches the visible content. Afterwards, they should look at the HTML rendered on the page to ensure it includes the content they want Google to index. Mueller said this is the best approach to take.

What Mueller suggested was just one solution. There is another way to solve this problem.

Some publishers strive to completely block their main content with a captcha while maintaining a Google-friendly page at the same time. This technique requires one to serve Googlebot a version of the page that’s different compared to what regular visitors would see. Mueller confirmed that this method does not violate any of Google’s guidelines and policies as long as it is done correctly.

Websites can serve Googlebot a captcha-free version of the web page while requiring users to complete a captcha before giving them access to the main content. This way, one can reap the benefits of using captchas while making sure that Google is able to crawl and index the content.

Benefits Of Using Captcha In Content Pages

Using captchas in content pages can offer many advantages. Here are some of them:

• - They deter hackers from abusing online services by blocking robot software from sending nefarious or fake online requests.
• - They protect the integrity of online polls by preventing hackers from sending in multiple votes.
• - They stop hackers from using hundreds of passwords to log into accounts.
• - They help make online shopping safer.
• - They prevent robots from buying show and concert tickets in bulk.
• - They stop cybercriminals from spamming news or blogs with spammy comments and links.
• - They prevent hackers from signing up for multiple accounts that they will use for nefarious purposes

Provide A Great User Experience With Position1SEO

Here at Position1SEO, we understand that providing a great user experience is one of the keys to successful SEO. Page elements like interstitial pop-ups and captcha tests that block the content may lead to frustration. Most Internet users have a short attention span, so they might immediately leave the page at the sight of captcha tests and other intrusive site elements.

If you want to improve your website’s user experience, work with Position1SEO today! We are a team of experts who can help you plan the best SEO content strategy, including providing a great user experience so readers can easily access your content.

For more information about our services, please do not hesitate to call us on 0141 404 7515 or email at office@position1seo.co.uk.