Robots.txt Generator
Build a valid robots.txt file to control how search engine crawlers access your site.
Use * to target all crawlers
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
How to use this tool
Choose Allow all crawlers to open your whole site to indexing, Block all crawlers to keep it out of search results (handy for staging sites), or Custom rules to disallow specific paths with optional allow exceptions. Set User-agent to * to target every crawler, or name a specific bot, and optionally set a Crawl-delay in seconds. Add your Sitemap URL so crawlers can discover it automatically. The generated robots.txt on the right updates live — copy it and upload it to the root of your domain.
Related Tools
Related Blog Posts
Basic On-Page SEO Setup: Meta Tags, Robots.txt, and Favicons
Three small but easy-to-overlook technical details that affect how a site looks in search results and to crawlers.
Read moreBest Free Tools Every Developer Should Bookmark
Five browser-based utilities worth bookmarking for everyday development tasks, and what each one is actually good for.
Read moreJSON vs XML: Which Data Format Should You Use in 2026?
A practical comparison of JSON and XML for APIs, config files, and data exchange — including where each format still wins.
Read more