Robots.txt Generator
Build a valid robots.txt file to control how search engine crawlers access your site.
Use * to target all crawlers
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
How to use this tool
Pick Allow all crawlers, Block all crawlers or Custom rules, add the paths to block or allow, and your sitemap address. Paths are tidied as you type: a missing leading slash is added, and a full address such as https://example.com/admin/ is reduced to /admin/, since rules only ever contain paths. Save the result as robots.txt at the root of your domain.
How crawlers read the rules
The rules follow RFC 9309, the robots.txt standard that Google follows. Four details catch people out:
- A rule is a prefix.
Disallow: /adminalso blocks/administrationand/admin-guide.pdf. Use/admin/to block only the folder. - The longest matching rule wins. With
Disallow: /shop/andAllow: /shop/sale/, the sale pages can be crawled and the rest of the shop can't. If an Allow and a Disallow match equally, Allow wins. - Matching is case-sensitive.
Disallow: /Private/doesn't cover/private/. *matches anything and$marks the end.Disallow: /*.pdf$blocks every URL that ends in .pdf, in any folder.
Blocking a page won't take it out of search
robots.txt stops crawling, not listing. A blocked page that other sites link to can still appear in results, shown without a description. To keep a page out of search, let it be crawled and add <meta name="robots" content="noindex">; if robots.txt blocks it, the crawler never sees that tag. Also check that your server returns the file correctly: under the standard, a server error on robots.txt means crawlers must treat the whole site as blocked, while a "not found" lets them crawl everything.
Frequently Asked Questions
Does Google obey the Crawl-delay line?
No. Google documents support for only user-agent, allow, disallow and sitemap in robots.txt, and ignores crawl-delay; it adjusts its crawl rate automatically. Bing and some other crawlers do read crawl-delay, so it is still worth including if one of them is putting too much load on your server.
Is robots.txt a security measure?
No, and this is a common and risky misconception. robots.txt is a publicly readable file that only asks well-behaved crawlers to stay away — it does nothing to actually restrict access, and malicious bots simply ignore it. Never rely on robots.txt to hide sensitive pages or data; use real authentication or access controls instead.
Where does robots.txt need to be placed?
At the root of each host: https://example.com/robots.txt covers example.com only. A subdomain such as shop.example.com, or the same site on http instead of https, needs its own file at its own root. A robots.txt in a subfolder is ignored.
Related Tools
Related Blog Posts
On-Page SEO Basics: Meta Tags, Robots.txt, and Favicons
The technical basics that decide how a page appears in search and link previews: title, description and canonical tags, what robots.txt can and can't do, and the favicon sizes search engines actually use.
Read moreHow to Use QR Codes for Marketing, Docs, and Events
How much a QR code can hold, why a shorter or uppercase link gives a code that scans more easily, what error correction actually protects against, and how to size one for print.
Read moreHEX, RGB and HSL: How the Conversions Work and Where They Round
#4F46E5 converted to RGB and HSL step by step, why 89.4% of all colours change when copied as whole-number HSL, why HSL lightness is not brightness (yellow and blue at 50% differ eightfold in contrast), and the alpha that hex can't write exactly.
Read more