Test and validate your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS, JavaScript, images) are disallowed.

Invalid URL
Live Editor
Check Resources

robots.txt: {{data.robotstxt.url == "Editor" && data.robotstxt.url || data.robotstxt.url + ' (' + data.robotstxt.status_code + ' ' + data.robotstxt.status_text + ')'}}

URL Path: {{data.url.path}}

Result: {{data.url.result}}

Sitemap URL Status
{{sitemap.url}} {{sitemap.status_code}} {{sitemap.status_text}}
Resource URL Status Type Result Host robots.txt URL
{{resource.url}} {{resource.status_code}} {{resource.status_text}} {{resource.type}} {{resource.crawl.status}} by {{resource.crawl.applied_rule.rule}} {{resource.crawl.host}} {{resource.crawl.robots_txt}}