Invalid URL
{{user_agent.name}}{{user_agent.token}}
Obey robots.txt
Render
PhantomJS2.1.1 Chromev
None {{language.name}}{{language.code}}
None {{region.name}}{{region.code}}
{{test_button}}
URL Crawlable robots.txt Directive Status Redirect Type
{{location.url}} {{location.crawl.applied_rule.rule || 'Not blocked via robots.txt'}} {{location.crawl.applied_rule.rule}} {{location.status_code}} {{location.status_text}} {{location.redirect_type && location.redirect_type || !$last && 'HTTP' || 'N/A'}}



The following URLs {{!robots_txt && 'would ' || ''}}have been blocked from crawling

Resource URL File Type Disallow Rule robots.txt Status
{{resource.url}} {{resource.file_type}} {{resource.crawl.applied_rule.rule}} See robots.txt {{!robots_txt && resource.status_code + ' ' + resource.status_text || 'N/A'}}

The following URLs could not be accessed

Resource URL File Type Status
{{resource.url}} {{resource.file_type}} {{resource.status_code + ' ' + resource.status_text}}

Comments