Applying noindex or nofollow to an entire path or directory helps control how search engines handle your site’s content. These directives can be set with meta tags, HTTP headers, or server rules depending on your needs.
What Noindex Does
Noindex prevents a page from appearing in search results while still allowing crawlers to access and follow its links. It’s useful for login pages, thank-you pages, or duplicate content.
Example for all pages in a directory:
<meta name="robots" content="noindex, follow">
Or via HTTP header for non-HTML files:
X-Robots-Tag: noindex, follow
What Nofollow Does
Nofollow tells crawlers not to follow links or pass link equity. It’s best for sponsored or untrusted outbound links, but not recommended for internal navigation.
Page-level example:
<meta name="robots" content="nofollow">
Link-level example:
<a href="example.com" rel="nofollow">Example</a>
Applying to a Directory
Apache
<FilesMatch "\.(html|htm)$">
Header set X-Robots-Tag "noindex, follow"
</FilesMatch>
NGINX
location /private/ {
add_header X-Robots-Tag "noindex, follow";
}
Common Mistakes
- Blocking with
robots.txt while also using noindex prevents crawlers from seeing the directive.
- Using
nofollow internally can weaken site authority.
- Accidentally applying
noindex to important content can remove it from search results quickly.
Join The Discussion
Have you used noindex or nofollow on directories in your site? Share whether it helped manage duplicate content or caused unexpected indexing issues.
