Robots.txt Generator
Generate robots.txt crawl directives files for web crawlers.
Your inputs are processed locally in your browser. Nothing is uploaded to any server.
Robots.txt Configuration
How to Use
- 1Specify your User Agent (use '*' for all search engines).
- 2Set an optional crawl delay to slow down aggressive bots.
- 3Paste your XML sitemap URL.
- 4List the directory paths you want to block (Disallow) from search engines.
- 5Click 'Generate' and save the output as robots.txt in your root folder.