Docs
Robots Meta & X-Robots-Tag

Robots Meta & X-Robots-Tag

How noindex and nofollow work, when to use a meta tag versus an HTTP header, directive combinations, and the errors that hide pages by accident.