There are two distinct tools for keeping web spiders away from your content, and using the wrong one — or combining them incorrectly — can actually backfire and leave your pages exposed. Here’s exactly how each method works and when to use which.
Understand the Two Different Tools First
The critical distinction to get right from the start: robots.txt is not for hiding pages from Google — use noindex or password protection for that instead. The robots.txt file is mainly used to avoid overloading your site with requests; it’s not a mechanism for keeping a page out of search results entirely.
The functional difference matters here. Disallow means that spiders aren’t allowed to visit the page, but they can still index the page if they discover it without visiting it directly — while noindex means search engines aren’t allowed to index the page at all. That’s why relying only on robots.txt can produce a strange outcome: a Google search can turn up an archived page anyway, showing a message reading “No information is available for this page,” because the page was still indexed even though crawling was blocked.
Method 1: Block Crawling With robots.txt
If your goal is to manage crawl traffic rather than fully hide content from search results, robots.txt is the right tool. The structure is simple — you create a plain text file listing a “User-agent” field identifying the spider in question, followed by “Disallow” lines instructing that spider not to access certain parts of your site.
To block all spiders from your entire site:
User-agent: *
Disallow: /
To block all spiders from a specific folder, such as an archive of posts:
User-agent: *
Disallow: /archives/
Sitemap: /sitemap.xml
You can also target specific bots this way — for example, if you want to block ChatGPT’s crawler from accessing your whole site or just a particular section of it, you’d write a dedicated User-agent line for GPTBot.
Method 2: Block Indexing With the Noindex Tag
If you actually want your posts kept out of search results entirely — not just uncrawled — this is the correct method. Add this meta tag to the head of each page you want hidden:
<meta name="robots" content="noindex, nofollow">
The “nofollow” portion adds an extra layer of control. “Noindex” tells the search engine not to include the page in its list, while “nofollow” tells it not to follow any links to other pages found on that particular page.
The Critical Rule: Don’t Combine Them Incorrectly
This is the mistake most people make. If you’re using noindex, you must also remove any robots.txt rule that blocks the same page — because Google needs to actually be able to read the page in order to see your noindex instruction in the first place.
HubSpot’s documentation states this even more directly: this noindex method should not be combined with the robots.txt method, since doing so will prevent search engines from seeing the noindex tag at all.
For WordPress Sites Specifically
If your entire site runs on WordPress, there’s a built-in shortcut that avoids manually editing files.
To hide your whole site from indexing: go to Settings, then Reading, and check “Discourage search engines from indexing this site” — this adds a noindex tag to every page on your site automatically.
To hide specific posts, pages, or categories only: you can use a free SEO plugin like Yoast or The SEO Framework to noindex individual content selectively rather than site-wide.
What to Do If You Want the Entire Site Hidden From People Too
If your goal goes beyond search engines and you want to block human visitors as well, neither robots.txt nor noindex will help — both only affect well-behaved bots. In that case, putting a password on your website is the best approach, which on WordPress can be done with a free plugin called Password Protected.
Verifying It’s Actually Working
Once you’ve made changes, don’t just assume they took effect. You can use Google Search Console to test whether your robots.txt file is behaving as expected. It’s also worth remembering that anyone can view your robots.txt file directly by typing it into their browser, which means it isn’t a private or secure way to hide sensitive content — it only signals intent to well-behaved crawlers.
Common Mistakes to Avoid
- Using robots.txt Disallow when you actually wanted noindex, then being confused when the page still shows up in search results without a description.
- Combining Disallow and noindex on the same page, which prevents search engines from ever seeing the noindex instruction.
- Assuming robots.txt keeps content private or secure — it’s publicly viewable and only a request, not an enforcement mechanism.
- Applying noindex to a page involved in an A/B test combined with a permanent redirect, which sends search engines conflicting signals about whether the page still exists.
Join The Discussion
Have you run into the classic robots.txt-versus-noindex confusion on your own site, or found a page still showing up in search results after you thought you’d blocked it? Share what ended up fixing it for you, whether it was switching to noindex, removing a conflicting Disallow rule, or using a plugin instead of editing files directly. If you’ve had to deal with newer bots like GPTBot specifically, it’d be useful to hear how you approached blocking those.
