Robots.txt Generator

Robots.txt Generator

Tell search engines which parts of your site they may visit. Fill in the boxes, check any page you're unsure about, then press Copy.

Start from a template:
We brought back what you typed here last time.
1

Website address

The site these rules belong to.
2

What may crawlers open?

Type / to let them see the whole site.
3

What should they skip?

Private areas such as /admin/ or /search.
Sitemap — helps crawlers find new pages faster
Add more crawlers — give one bot its own rules
More settings — you can safely skip all of this

Bing and Yandex respect this. Google ignores it — change the crawl rate in Search Console instead.

Add a header comment
A short note at the top naming your site
What this means

Paste any page address to see whether your rules let crawlers reach it.

Blocking a path only stops crawling. A page linked from elsewhere can still appear in search, just without a description.

Anything left to do
    Your robots.txt 
    Add at least one rule, or a sitemap, to see your file here.

    Upload this as robots.txt at the root of your site, or paste it into Blogger's crawler settings.

    What does this tool do?

    Every website can carry a small text file called robots.txt, sitting at the root of the domain, that tells crawlers such as Googlebot and Bingbot which parts of the site they are welcome to visit and which parts to leave alone.

    Writing it by hand means remembering exact syntax, and one stray slash can hide your entire site from search. This tool asks plain questions, lets you test any page against your rules, and hands you a finished file.

    How to use it

    • Pick a template at the top if one matches your site — it fills everything in for you.
    • Otherwise type your website address in box 1.
    • Add the paths crawlers may open in box 2. A single / covers the whole site.
    • Add anything they should skip in box 3, such as /admin/.
    • Use Check one of your pages to confirm nothing important got blocked by accident.
    • Press Copy robots.txt and upload it to your site's root.

    Allow and Disallow, in plain terms

    Allow gives a crawler permission to fetch a path. It is mostly used to carve out one exception inside a folder you have otherwise blocked. Disallow tells it to stay away from a path entirely.

    When both match the same page, the more specific rule wins — and if they are equally specific, Allow wins. The page checker in this tool follows exactly that logic, so it agrees with what Google will do.

    Neither controls whether a page appears in search results — only whether crawlers may fetch it. To keep a page out of search, use a noindex meta tag on the page instead.

    Wildcards you can use

    • * stands for any run of characters. Disallow: /*? blocks every address containing a question mark.
    • $ marks the end of the address. Disallow: /*.pdf$ blocks PDF files but not /report.pdf.html.
    • Paths are case sensitive, so /Admin/ and /admin/ are two different rules.

    Mistakes worth avoiding

    • A leftover Disallow: / from a staging site hides everything from search. This tool warns you loudly if you add it.
    • Paths must be relative. Write /admin/, not the full https://example.com/admin/.
    • Blocking your CSS or JavaScript can make crawlers see a broken-looking page, since they render pages much like a browser.
    • Disallowing a page does not remove it from search — use a noindex tag if that is the goal.
    • The file only works from your domain root. Anywhere else, it is ignored.

    Common questions

    Do I have to fill in every box?

    No. One rule or a sitemap address is enough for a working file. The folded sections are there for anyone who wants finer control.

    Where does the finished file go?

    At your domain root, as https://www.example.com/robots.txt. On Blogger, go to Settings, then Crawlers and indexing, turn on custom robots.txt, and paste it in.

    Can I list more than one sitemap?

    Yes. Press "Add another sitemap" — many sites split theirs into posts and pages, and every address you add is listed in the file.

    Is my text sent anywhere?

    No. Everything happens inside your own browser. Your work is saved only on your own device so a refresh does not lose it, and Clear everything wipes that copy.

    Can I add rules for more than one crawler?

    Yes. Open "Add more crawlers" to give a specific bot, such as Googlebot-Image, its own rules separate from the rest.

    Can I stop AI bots from training on my content?

    To a point. Press Block AI bots at the top and the tool adds a blocking rule for the crawlers that AI companies publish, such as GPTBot, Google-Extended, ClaudeBot and CCBot — each in its own group, so your normal search ranking on Google and Bing is untouched. Well-behaved bots honour it, but robots.txt is a request rather than a lock, so it cannot force a crawler that ignores the rules to stay away.

    What happened to the Host directive?

    Yandex retired it in 2018 and no other search engine ever supported it, so this tool leaves it out. Use a redirect to set your preferred domain instead.