Invalid URL
Obey robots.txt
PhantomJS2.1.1 Chromev
None {{}}{{language.code}}
None {{}}{{region.code}}
URL Crawlable robots.txt Directive Status Redirect Type
{{location.url}} {{location.crawl.applied_rule.rule || 'Not blocked via robots.txt'}} {{location.crawl.applied_rule.rule}} {{location.status_code}} {{location.status_text}} {{location.redirect_type && location.redirect_type || !$last && 'HTTP' || 'N/A'}}

The following URLs {{!robots_txt && 'would ' || ''}}have been blocked from crawling

Resource URL File Type Disallow Rule robots.txt Status
{{resource.url}} {{resource.file_type}} {{resource.crawl.applied_rule.rule}} See robots.txt {{!robots_txt && resource.status_code + ' ' + resource.status_text || 'N/A'}}

The following URLs could not be accessed

Resource URL File Type Status
{{resource.url}} {{resource.file_type}} {{resource.status_code + ' ' + resource.status_text}}