What does a robots.txt file do?

A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot crawl. It is placed in the root directory of your website.

Can I block Google from crawling a page?

Yes — use Disallow directives in robots.txt to block specific pages or directories. However, this does not prevent already-indexed pages from appearing in search results.

Does robots.txt affect SEO?

Yes — incorrect robots.txt rules can accidentally block important pages from being crawled. Always test your robots.txt file before deploying changes to production.

What is the difference between robots.txt and noindex?

robots.txt prevents crawling. A noindex meta tag prevents indexing. For complete removal from search results, you need noindex — a blocked page can still appear in search from external links.

SEO Tool

Robots.txt Checker

View and validate any website's robots.txt file. See all crawl rules, test whether a specific URL is blocked for Googlebot or other crawlers, and check sitemap declarations.

Website URL

Try: github.com · bbc.com · reddit.com

What is robots.txt?+

robots.txt is a file placed in the root of a website that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It is part of the Robots Exclusion Protocol and is checked by Googlebot, Bingbot and other crawlers before accessing any URL.

Does robots.txt prevent Google from indexing a page?+

robots.txt prevents crawling — Googlebot will not visit the page. But it does not prevent indexing entirely. Google can still index a URL it has seen in links even without crawling it. To prevent indexing, use a noindex meta tag or X-Robots-Tag header, which requires the page to be crawlable.

What is the difference between Allow and Disallow?+

Disallow: /path blocks a crawler from accessing any URL starting with that path. Allow: /path explicitly permits access to a path that would otherwise be blocked by a broader Disallow rule. Allow rules take precedence over Disallow rules of equal specificity.

Can robots.txt block all crawlers?+

Disallow: / under User-agent: * blocks all compliant crawlers from the entire site. Note: robots.txt is advisory — malicious bots and scrapers ignore it. It only affects well-behaved crawlers like Googlebot that voluntarily follow the protocol.

What is robots.txt?

robots.txt is a plain-text file at the root of a domain (like anonymiz.com/robots.txt) that tells search engine crawlers which parts of a site they're allowed to visit. It's a voluntary convention, not a security mechanism — well-behaved bots like Googlebot respect it, but it can't actually block access to anything.

How to Write robots.txt Rules

A robots.txt file is built from simple rules: `User-agent` specifies which bot the rule applies to (or `*` for all), `Disallow` blocks a path, and `Allow` can carve out an exception within a blocked path. A `Sitemap` line pointing to your sitemap.xml is also commonly included to help crawlers discover your pages faster.

Common robots.txt Mistakes

The most common mistakes are disallowing a path that's actually meant to rank in search results, forgetting that robots.txt is publicly viewable (so it shouldn't be used to “hide” sensitive paths — anyone can read it), and syntax errors like a missing colon or incorrect path formatting that cause the intended rule to be silently ignored.