# Allow crawlers to read the noindex directives in HTML and HTTP headers. # Disallowing crawling would prevent those directives from being discovered. User-agent: * Disallow: