Robots tags are directives that allow webmasters to control how search engines crawl and index the content of a website. They are page-level signals and can be implemented in a meta tag (robots meta tag) or in an HTTP response header (X-Robots-Tag).
Robots meta tags have two attribute-value pairs, name and content. The value of the name attribute must specify a robot's name (e.g. googlebot) or simply "robots" to include all of them. The value of the content attribute must be one or more valid directives.
Among the few existing directives, here are the ones primarily used for SEO.
||There are no restrictions for indexing or serving. Equivalent to
||Do not show this page in search results and do not show a "Cached" link in search results.|
||Do not follow the links on this page.|
Note that the absence of robots tag or an empty value of the "content" attribute is also equivalent to the default
index, follow value.
Robots meta tags must be inserted in the
<head> section of the HTML page.
<head> <meta name="robots" content="noindex"> </head>
<head> <meta name="googlebot" content="noindex, nofollow"> </head>
X-Robots-Tags serve the same purpose as meta robots tags, they simply are a different way to implement robots tags directives. They are particularly useful in absence of an HTML document (e.g. PDFs or images) or if a directive has to be implemented site-wide.
When in place, the HTTP response header will look as follow:
HTTP/1.1 200 OK ... X-Robots-Tag: noindex, nofollow ...
rel="nofollow" tags are used to prevent the flow of equity across certain links and help prioritize how bots crawl a site.