What is a robots.txt file and why do I need one?

A robots.txt file tells search engine crawlers which pages or sections of your site they should and should not index. Without one, crawlers may index admin pages, duplicate content, or private URLs. Having a well-configured robots.txt helps control how your site appears in search results.

Does robots.txt actually stop crawlers from accessing my pages?

Robots.txt is a convention that well-behaved crawlers like Googlebot follow voluntarily. It does not provide security — it only signals intent. Malicious bots and scrapers may ignore it entirely. To truly restrict access to sensitive pages, use authentication or server-level access controls.

Should I block CSS and JavaScript files from crawlers?

No. Google recommends allowing crawlers to access CSS and JavaScript files so they can render your pages accurately. Blocking these files can prevent Google from understanding your page layout and may negatively affect your search rankings.

Robots.txt Generator

Create a properly formatted robots.txt file for your website. Control which pages search engines can index.

Basic Settings

Sitemap URL

Default crawl policy

Crawl delay (seconds, optional)

Block These Paths

Common paths to block

/admin/ — Admin panel /login/ /logout/ — Auth pages /api/ — API endpoints /private/ /secret/ — Private directories WordPress paths (wp-admin, wp-login) ?page= ?sort= ?filter= (prevent duplicate content) /cdn-cgi/ (Cloudflare paths)

Block Specific Bad Bots

AhrefsBot (SEO scraper) SemrushBot (SEO scraper) MJ12bot (SEO crawler) DotBot (SEO crawler) Baiduspider (Chinese search) YandexBot (Russian search) Mail.RU_Bot PetalBot (Huawei)

Your robots.txt

Upload this file to your website root: https://yoursite.com/robots.txt

🤖

What is robots.txt?

A text file that tells search engine bots which pages they can and cannot crawl and index.

📍

Where to upload it

Upload to your website root directory so it is accessible at yoursite.com/robots.txt

⚠️

Important note

robots.txt is publicly visible and does not prevent access — it only gives instructions to cooperative bots.

Frequently Asked Questions

Does robots.txt stop hackers from accessing my admin?

No. robots.txt only gives instructions to cooperative search engine bots. It does not block access from browsers, hackers, or non-compliant bots. For security, protect sensitive pages with authentication.

What happens if I block Googlebot?

If you block Googlebot from crawling your site, your pages will not be indexed in Google search results. Your site will effectively disappear from Google over time as cached pages expire.

How quickly does Google respect robots.txt changes?

Google re-crawls robots.txt frequently — usually within 24 hours. However, pages that were already indexed may take days or weeks to be removed from search results after being blocked.