robots.txt Generator

What this page does

A robots.txt tells a crawler what it may fetch. This page writes the lines from the fields on the page. Title and Open Graph tags stay on meta tags.

How to use it

1. User agents starts as *. Allow, Disallow and Sitemaps take one path or URL per line. Crawl delay is optional.
2. Convert runs once. Show Example sets User agents to *, Allow to /, Disallow to /admin and /search, leaves Crawl delay empty, and sets Sitemaps to https://example.com/sitemap.xml.
3. The result is User-agent: *, Allow: /, Disallow: /admin, Disallow: /search, Sitemap: https://example.com/sitemap.xml.
4. Clear All puts User agents back to * and empties the other fields and the result. Copy Result copies the text.

Rules and edges

Blank lines in a box are skipped. Several user agents are written first, then every Allow, then every Disallow, then Crawl-delay if the field has digits, then every Sitemap. A crawl delay that is not a whole number stops with an error. User agents alone, with no allow, disallow, delay or sitemap, also stops with an error. The field names in the output stay User-agent, Allow, Disallow, Crawl-delay and Sitemap.

Where people use it

• Blocking /admin and /search while leaving the rest open.
• Adding one sitemap URL.
• Copying the five lines into a robots.txt file.

Questions