Robots Tags

Robots tags are directives that allow webmasters to control how search engines crawl and index the content of a website. They are page-level signals and can be implemented in a meta tag (robots meta tag) or in an HTTP response header (X-Robots-Tag).

Robots Meta Tags

Robots meta tags have two attribute-value pairs, name and content. The value of the name attribute must specify a robot's name (e.g. googlebot) or simply "robots" to include all of them. The value of the content attribute must be one or more valid directives.

Among the few existing directives, here are the ones primarily used for SEO.

Directive Meaning
all There are no restrictions for indexing or serving. Equivalent to index, follow. Note: this directive is the default value and has no effect if explicitly listed.
noindex Do not show this page in search results and do not show a "Cached" link in search results.
nofollow Do not follow the links on this page.
none Equivalent to noindex, nofollow
Source: Google Developers

Note that the absence of robots tag or an empty value of the "content" attribute is also equivalent to the default index, follow value.

Implementation of robots meta tags

Robots meta tags must be inserted in the <head> section of the HTML page.

Code examples

<head>
  <meta name="robots" content="noindex">
</head>
<head>
  <meta name="googlebot" content="noindex, nofollow">
</head>

X-Robots-Tags

X-Robots-Tags serve the same purpose as meta robots tags, they simply are a different way to implement robots tags directives. They are particularly useful in absence of an HTML document (e.g. PDFs or images) or if a directive has to be implemented site-wide.

When in place, the HTTP response header will look as follow:

HTTP/1.1 200 OK
...
X-Robots-Tag: noindex, nofollow
...


Related Topics

rel="nofollow"

rel="nofollow" tags are used to prevent the flow of equity across certain links and help prioritize how bots crawl a site.

Learn more about the rel="nofollow" tag