Robots.txt

Home / Robots.txt

This Is How Google Ranks Pages Blocked By Robots.txt

11 July, 2019
Jason Ferry
Robots.txt

In a recent Google Webmaster Central hangout, a question about “how does Google figure out query relevancy for ranking pages blocked by robots.txt” was asked. This is a good thing to ask, especially for site owners and SEO experts who are also curious about what types of search queries do these pages rank for. In […]

This Is Why Google Has Decided To Stop Supporting Unsupported Directives In Robots.txt Files

5 July, 2019
Jason Ferry
Robots.txt

SEO experts and webmasters should take note that starting September 1, Google will not be supporting unpublished and unsupported rules in the robots exclusive protocol anymore. With this change, the search engine giant will not consider robots.txt files that have noindex directive listed within them. Google has decided to proceed with this change for possible […]

This Is Why Google Aims To Set An Official Standard For Using Robots.txt

2 July, 2019
Jason Ferry
Robots.txt

SEO experts and webmasters have been following the unofficial rules outlined in the Robots Exclusion Protocol (REP) for the past 25 years when using robots.txt. With this, the publishers are allowed to pick what they want to be crawled on their website, and interested users can view them. The said rules are also observed by […]

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.