Robots.txt Generator

What the robots.txt Generator does

robots.txt is a plain text file at your site root that tells crawlers which paths they may fetch. It controls crawling, not indexing, and confusing the two causes most of the damage this file is responsible for.

The syntax is small. User-agent: names the crawler a block applies to, with * meaning all of them. Disallow: lists paths not to fetch; Allow: carves an exception out of a broader disallow. Paths are prefixes, so Disallow: /cart blocks /cart/, /cart-summary and anything else starting the same way — add the trailing slash if you mean the directory. Google and Bing support * as a wildcard and $ to anchor the end of a URL.

The mistake worth guarding against is Disallow: /, which blocks the entire site. It is correct on a staging server and catastrophic on a live one, and it reaches production regularly because the staging file gets deployed with everything else. This generator shows a prominent warning whenever your rules would produce it.

The second common mistake is using robots.txt to hide a page from search results. Blocking a URL stops it being crawled, which means any noindex on the page is never seen — and a blocked URL with inbound links can still appear in results as a bare link with no description. To remove a page, let it be crawled and use noindex; block it in robots.txt only afterwards, if at all.

How to use it

1. Pick a preset that matches your platform, or start from Allow everything.
2. Add paths that genuinely should not be crawled — cart, checkout, internal search results, faceted filter URLs.
3. Add your sitemap URL.
4. Read any warning carefully, then save the file as robots.txt in your site root.

A worked example

A sensible robots.txt for an online shop:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search
Disallow: /*?orderby=

Sitemap: https://example.com/sitemap.xml

Every rule there has a reason. Cart and checkout are per-visitor and worthless in an index. Search result pages generate infinite low-value URLs. /*?orderby= catches sort-order variants of category pages, which are the same products in a different sequence — a classic way to waste crawl budget on thousands of near-duplicate URLs.

What is not blocked matters just as much: no Disallow: /wp-content/ or similar blocking of CSS and JavaScript. Google renders pages to judge them, and blocking the assets it needs to render means it sees a broken page.

Frequently asked questions