{"id":821,"date":"2019-09-09T06:00:10","date_gmt":"2019-09-09T13:00:10","guid":{"rendered":"https:\/\/technicalseo.com\/insights\/?p=821"},"modified":"2019-09-05T08:20:25","modified_gmt":"2019-09-05T15:20:25","slug":"robots-txt-ambiguities","status":"publish","type":"post","link":"https:\/\/technicalseo.com\/insights\/blog\/robots-txt-ambiguities\/","title":{"rendered":"Blocked or not? Wildcards, typos and other robots.txt ambiguities"},"content":{"rendered":"\n
\"robots.txt<\/figure>\n\n\n\n

Older than most of the search engines we use today, the robots.txt file is a dinosaur of the web. Although there are many articles discussing the protocol and how to use the file, in this post we will cover some ambiguities, corner cases and other undocumented scenarios.<\/p>\n\n\n\n

Why should you care?<\/h2>\n\n\n\n

The robots.txt is a convenient, yet complex tool for SEO that can easily cause unexpected results if not handled carefully. While it makes sense to avoid any typos and ambiguous or conflicting rules when using Allow and Disallow directives, we should also know how search engines behave in these situations. <\/p>\n\n\n\n

robots.txt: the puffer fish of the web<\/h2>\n\n\n\n
\n
\n

robots.txt is like a puffer fish: delicious, yet toxic. It is safe to eat as long as the SEO chef knows how to cook it. They know the file format, the syntax and the rules of grouping. They understand the order of precedence for user-agents and how URL matching based on path values works.<\/p>\n\n\n\n

The Robots Exclusion Protocol (REP) is celebrating 25 years and its specification is finally being formalized with some new rules<\/a>. Yet, we can still observe misuse and confusion in the SEO and webmaster communities, which, unfortunately, result in sub-optimal performance in search for some websites. <\/p>\n\n\n\n

So, let\u2019s clarify and expand on some specific puzzling rules.<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n