Free Robots.txt Generator

Robots.txt Generator
Control Search Engine Crawling

Create professional robots.txt files to control how search engines crawl your website. Protect sensitive directories, manage crawl budget, and improve SEO.

Allow/Disallow Rules
Crawl Delay
Sitemap Submission

Configuration

Allow all bots to crawl everything

Use / to allow everything

Use / to block everything

Seconds between requests (recommended: 1-5)

📖 Common Directives

User-agent

Specifies which search engine crawler the rules apply to. Use * for all crawlers.

Disallow

Tells crawlers not to access specific paths or directories.

Allow

Overrides disallow rules for specific paths.

Crawl-delay

Specifies the number of seconds between requests.

Sitemap

Points crawlers to your XML sitemap location.

⚠️ Common Mistakes

  • Using uppercase in paths (use lowercase)
  • Forgetting trailing slashes
  • Blocking CSS/JS files (prevents proper rendering)
  • Multiple sitemap declarations
  • Conflicting Allow/Disallow rules

✅ Best Practices

  • Always include a Sitemap directive
  • Test your robots.txt in Google Search Console
  • Block admin areas and private directories
  • Don't block important resources
  • Keep the file clean and well-commented