robots.txt Validator and Testing Tool

Test and validate your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!









URL Path:

Result: Allowed


Sitemap URL Status Code

Resource URL Status Code File Type Result Host robots.txt robots.txt URL


Learn more about controlling crawling and indexation indexing

Robots.txt

Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy. Learn more about Robots.txt files and their effect on SEO.

Learn more about the robots.txt file

Robots Tags

Robots tags are directives that allow you to control how search engines crawl and index the content of your website. They are page-level signals and can be implemented in a meta tag or in an HTTP response header.

Learn more about robots tags