Robots.txt Generator
FREE SEO TOOL1. Website Information
Enter your website's main URL.
Your XML sitemap URL.
2. Choose a Robots.txt Preset
3. Search Engine Crawlers
Select the crawler groups for which you want to create rules. Selecting a specific bot adds a separate user-agent group.
4. Configure Allow & Disallow Rules
DirectivePath
5. Additional Settings
Optional. Not supported by every crawler.
Optional comment or additional directive.
6. Live Robots.txt Preview
robots.txt
TEXT FILE · UTF-8
Important SEO Note: A robots.txt file controls crawler access; it does not guarantee that a URL will be removed from Google's index. Do not block important CSS, JavaScript or content resources required to render your pages. Review all rules before publishing.