Robots.txt & Sitemap Generator
Build a correct robots.txt — including AI crawler blocking — and a valid XML sitemap in seconds.
100% Free • No Signup • Copy or Download Instantly
User-agent: *
Allow: /
Upload to your site's root folder so it's reachable at yoursite.com/robots.txt.
How It Works
Pick Your Rules
Choose what crawlers may access, block AI training bots with one click, and list your pages for the sitemap.
Preview Live
See the exact robots.txt and sitemap.xml output update as you type — no guesswork, no syntax errors.
Download & Upload
Copy or download both files, put them in your site's root folder, and submit the sitemap in Search Console.
Why Use This SEO Tool?
100% Free
No fees, no signup, unlimited files.
AI Bot Blocking
One-click rules for GPTBot, CCBot, PerplexityBot and 6 more AI crawlers.
Valid Syntax
Correct robots.txt directives and schema-valid XML sitemaps, every time.
Private
Everything is generated in your browser — nothing is uploaded.
Frequently Asked Questions
Common questions about robots.txt and sitemaps
What is a robots.txt file?
robots.txt is a plain-text file at the root of your website (yoursite.com/robots.txt) that tells search engine crawlers which pages they may or may not crawl. It's the first thing well-behaved bots check before crawling your site.
Can robots.txt block AI crawlers like GPTBot?
Yes. AI companies publish crawler names — GPTBot (OpenAI), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot and others — that respect robots.txt. Tick the AI bots you want to block and the tool adds a Disallow rule for each. Note this only stops compliant crawlers; it is a request, not enforcement.
What is an XML sitemap and why do I need one?
A sitemap.xml lists your site's URLs so search engines can discover and index them faster — especially useful for new sites, large sites, or pages with few internal links. Submit it in Google Search Console and reference it in robots.txt.
What do changefreq and priority mean?
changefreq hints how often a page changes (daily, weekly…) and priority (0.0–1.0) hints its relative importance. Google largely ignores both today, but they're harmless and other crawlers may use them. lastmod is the one field Google actually reads — keep it accurate.
How many URLs can one sitemap contain?
Up to 50,000 URLs and 50 MB uncompressed per file. Beyond that, split into multiple sitemaps and link them with a sitemap index file.
Where do I upload these files?
Both files go in your website's root folder so they're reachable at yoursite.com/robots.txt and yoursite.com/sitemap.xml. Then submit the sitemap in Google Search Console and Bing Webmaster Tools.
Generate robots.txt and XML Sitemaps the Right Way
Two small files control how search engines see your site: robots.txt decides what crawlers may fetch, and sitemap.xml tells them what exists. Getting either wrong can keep pages out of Google entirely — a stray Disallow: / has taken down more than one site's rankings. This free generator produces both files with correct syntax and a live preview, so you always know exactly what you're publishing.
Blocking AI Training Crawlers
A growing number of site owners choose to keep their content out of AI training datasets. Crawlers like GPTBot (OpenAI), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot and Bytespider publish their user-agent names and respect robots.txt rules. Tick the bots you want to exclude and the generator writes a separate Disallow: / group for each — while leaving normal search indexing (Googlebot, Bingbot) untouched, so your SEO is unaffected.
Sitemap Rules Worth Knowing
- One sitemap holds up to 50,000 URLs / 50 MB; beyond that, use a sitemap index.
- lastmod is the field Google actually uses — keep it truthful; changefreq and priority are mostly ignored.
- Reference your sitemap in robots.txt (
Sitemap: https://yoursite.com/sitemap.xml) and submit it in Google Search Console. - Only include canonical, indexable URLs — no redirects, 404s or noindex pages.
Building a website? Pair this with our Favicon Generator for a complete site setup, or explore the rest of our free developer tools.