Business owner and marketing advisor reviewing website pages at a desk

Robots.txt vs Noindex: Choosing Which Pages Appear in Google

If a page should stay available to visitors but stay out of Google Search, use noindex and let Google crawl it. If the content is private, protect access with a login or password. Robots.txt controls what a crawler can visit; blocking a web page there can still leave its address in search results.

That difference matters when a customer finds an old campaign page, a download page, or a document you never meant to promote. Before someone changes your website settings, agree on the outcome: should the page remain public, disappear from search, or be available only to authorized people?

Business owner and marketing advisor reviewing website pages at a desk
Start with the purpose of each page, then choose how people and search engines should reach it.

Robots.txt vs noindex: choose the outcome first

Think of crawling as Google visiting a page, and indexing as Google adding it to the collection it can show in search results. They are separate steps. Stopping the visit does not reliably remove the page’s address from that collection.

A public workshop handout is a useful example. You might want customers to open it from an email without finding it through a Google search. Noindex fits that goal. A client proposal containing private pricing needs access protection instead, because anyone with a public link could still open a noindexed file.

Use the comparison below to explain the business goal to your website provider. The right choice depends on what people should be able to see, as well as what Google should do.

Which control fits your goal?
What you want What to ask your provider for
A public page stays usable but leaves Google Search
Example: a consultation confirmation page
Noindex, with crawling allowed.
Google must visit the page to read the rule. People can still open its link.
Google should avoid particular media files
Example: selected public library images
A targeted robots.txt rule.
This can keep those media files out of Google’s results. It does not make them private.
Only authorized people should open the content
Example: a confidential client proposal
Login or password protection, or removal from public hosting.
Test access to the file itself. Search settings do not secure it.

For web-page addresses, a robots.txt block alone is not a reliable search-removal method. Check each live URL before applying changes widely.

Why a robots.txt block can leave a page in Google

Robots.txt is a file that gives crawlers instructions about which parts of your site they may access. It can help manage crawling, but it does not reliably hide a web-page address from Google. Google may learn about a blocked address through links elsewhere and show that address without a description.

Imagine an old promotion page that another website still links to. Your developer blocks the page in robots.txt, expecting it to disappear. That block stops Google from reading the page, but the outside link can still tell Google the address exists.

Adding noindex to the blocked page creates another problem: Google cannot visit the page to read the new instruction. For a public page you want removed from search, your provider should check that Google can reach it and see noindex. Private content should have its access secured before any crawl restriction is reconsidered.

When robots.txt does make sense

Robots.txt can be appropriate for controlling crawler traffic or keeping specific image and video files from appearing in Google’s results. For example, a public media library might contain images you do not want surfaced separately in image search. That decision applies to those media files, rather than automatically hiding every web page that contains them.

Ask for a precise explanation of the paths being blocked. A broad rule can also stop Google from reading important pages or resources it needs to understand the site. Our technical SEO services cover these crawl and website-access checks.

When noindex is the better fit

Noindex tells Google to leave content out of its search results once Google has crawled and processed the instruction. Visitors can still open the page through a direct link or a link on your site. That makes it useful for selected public pages that serve existing visitors without needing to attract search traffic.

Consider a simple confirmation page shown after someone requests a consultation. You may want the person who completed the form to see it, while keeping it out of Google. A noindex rule can fit that purpose, provided the page contains nothing confidential and Google can read the instruction.

For a regular web page, a developer can add the rule in the page’s settings or code. For a public PDF, it can be delivered through a server response called an X-Robots-Tag header. You do not need to manage the code yourself; ask the provider to show that the final page or file actually sends the intended instruction.

A removal is not immediate

Google has to revisit the page to discover a new noindex rule. Saving a setting in WordPress does not prove that Google has already processed it. Ask for the last crawl information and a follow-up check, rather than a promise that the result will disappear on a particular day.

If your problem is getting a useful service page into Google, that calls for a different review. Our guide to how long Google indexing takes explains when to wait and when to investigate.

Private content needs access protection

Neither robots.txt nor noindex is a lock on your website. They do not stop a person who already has a public URL from opening it. A confidential proposal, customer record, or internal staff document belongs behind properly configured access controls, or should be removed from public hosting.

For example, hiding a client proposal from Google still leaves a problem if someone can forward the link and anyone can open it. Ask your website or IT provider to confirm that an unauthorized visitor cannot retrieve the content. Check the actual file address as well as the page that links to it.

If private information is already exposed, secure or remove it first and review Google’s search-removal options with your provider. Search visibility and access protection need separate checks; a disappearing search result alone does not prove the file is protected.

What changed in Google’s guidance this October?

On October 21, 2026, Google refreshed its overview of the ways site owners can control content in Search. The update brought existing options together, including search-preview settings and the Search generative AI control. It did not introduce robots.txt or noindex as new features.

There is also a separate decision if you want ordinary search visibility while limiting participation in Google’s generative AI search features. As checked on October 30, Google’s Search Console guidance says its Search generative AI control covers AI Overviews, AI Mode, and generative AI features in Discover. Google says this choice does not act as a ranking or inclusion signal for other parts of Search.

For example, a business may want its service pages in ordinary Google results while considering whether to exclude them from those AI features. Noindex would remove those pages from Google Search entirely once processed. The separate AI control has a different purpose, and it does not control model training; Google’s guidance points to Google-Extended for that choice.

Google states that the Search generative AI control had rolled out worldwide by August 31, 2026. Treat the October documentation refresh as a reason to review your options. If your goal is to earn visibility in those answers, see our guide to appearing in AI Overviews.

What to ask before your provider changes anything

Start with a short list of affected addresses and the intended outcome for each. Include one valuable service page that must remain findable, so your provider can check that a broader setting will not hide it accidentally.

Ask for evidence that answers four practical questions:

  • What should happen to each page? Stay in search, leave search while remaining public, or require authorized access?
  • What does the live address return? Ask for the actual crawl rule, page instruction, or access check, rather than only a screenshot of an admin setting.
  • Can Google read the instruction? A noindex rule needs to be reachable by Google’s crawler; a robots.txt block can prevent that.
  • How will we confirm the result? Agree on the pages to inspect in Search Console, who will follow up, and which important pages must remain visible.

For a public confirmation page, a useful handoff would identify its exact URL, show that Google can reach it and read noindex, and record when the result will be checked again. A private proposal needs a different handoff: evidence that an unauthorized visitor cannot open the file.

Avoid applying noindex to large groups of pages just because the site feels cluttered. If two service pages compete for the same searches, review their purpose and content first. Our keyword cannibalization guide covers that separate problem.

Get the right pages working for your business

If you are unsure which pages should appear in Google, start by mapping the pages that bring customers in and the pages that support people after they arrive. An SEO audit can identify conflicting instructions and help you decide what deserves a change.

Motivated Digital Marketing can review the affected pages and explain the options in plain English. Tell us which pages are causing concern, and we can discuss the technical SEO work needed to make their visibility match their purpose.

Sources and review date

Guidance checked October 30, 2026. The examples above are illustrative. Search removal timing depends on Google revisiting and processing the affected content.