W3webschool Blog

Robots.txt Generator

FREE SEO TOOL

1. Website Information

Enter your website's main URL.
Your XML sitemap URL.

2. Choose a Robots.txt Preset

3. Search Engine Crawlers

Select the crawler groups for which you want to create rules. Selecting a specific bot adds a separate user-agent group.

4. Configure Allow & Disallow Rules

DirectivePath

5. Additional Settings

Optional. Not supported by every crawler.
Optional comment or additional directive.

6. Live Robots.txt Preview

robots.txt TEXT FILE · UTF-8

   
Important SEO Note: A robots.txt file controls crawler access; it does not guarantee that a URL will be removed from Google's index. Do not block important CSS, JavaScript or content resources required to render your pages. Review all rules before publishing.