Fetch any site's robots.txt and test whether a URL is allowed or blocked for Googlebot, Bingbot and other crawlers.
robots.txt is a plain text file placed at the root of your domain that tells search engine crawlers which parts of your site they may request. It follows the Robots Exclusion Protocol, which every major crawler — Googlebot, Bingbot, DuckDuckBot and others — reads before fetching your pages. The file is public: anyone can view it at yoursite.com/robots.txt, so it should never be used to hide sensitive URLs.
User-agent selects which crawler the following rules apply to, with * meaning all of them. Disallow blocks a path prefix, while Allow carves out an exception inside a blocked directory. An empty Disallow: permits everything. Rules are matched by prefix, not by exact URL, so Disallow: /admin also blocks /administrator — a common and costly mistake. Where rules conflict, most crawlers apply the most specific match rather than the first one listed. Adding a Sitemap: line pointing at your XML sitemap helps crawlers discover your pages faster.
This is the single most misunderstood point about robots.txt. Disallowing a URL stops crawlers from reading it, but the URL can still be indexed if other sites link to it — appearing in results with no description. Worse, if a page is blocked in robots.txt, Google cannot see a noindex tag on it, so blocking actively prevents removal. To keep a page out of search results, allow crawling and use a noindex meta tag or an X-Robots-Tag header instead. Use robots.txt to manage crawl budget, not to control indexing.