A robots.txt generator builds the robots.txt file that tells search engine crawlers which parts of your site they may crawl. Choose whether crawlers are allowed by default, list the paths to block and any to allow, name crawlers to block completely, and add your sitemap URLs, and this robots.txt generator writes a correctly formatted file you can copy or download and upload to the root of your site.
What robots.txt does
Google explains that you can control which files crawlers may access on your site with a robots.txt file. It's a plain text file that follows the Robots Exclusion Standard and lives at the root of the site: for www.example.com, it's at www.example.com/robots.txt. The file is made of rules, and each rule blocks or allows access for all crawlers or a specific one to a file path on that domain or subdomain. Unless you say otherwise, all files are allowed for crawling.
The rules
- User-agent: the crawler a group of rules applies to. It's the first line of each group, and * matches all crawlers (Google notes its AdsBot crawlers must be named explicitly).
- Disallow: a directory or page, relative to the root, that the crawler shouldn't crawl. It must start with /, and a directory should end with /.
- Allow: a directory or page that may be crawled, used to make an exception inside a disallowed directory.
- Sitemap: the full URL of a sitemap. It's optional, and you can list more than one.
All rules except Sitemap support the * wildcard. Rules are case-sensitive, so Disallow: /file.asp applies to /file.asp but not /FILE.asp. The # character starts a comment.
Example
Google's own example blocks one crawler from one folder and lets everyone else crawl everything:
User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Googlebot isn't allowed to crawl any URL starting with /nogooglebot/, all other crawlers may crawl the whole site, and the sitemap's location is given.
How to use the robots.txt generator
- Choose the default: allow crawling, or block the whole site.
- List paths to block, such as /admin/ or /cart/, one per line.
- Add allowed paths to make exceptions inside blocked folders.
- Name crawlers to block completely, if any.
- Add your sitemap URLs.
- Copy or download the file and upload it to your site's root.
Where to put the file
Google's format and location rules: the file must be named robots.txt, a site can have only one, and it must be at the root of the host it applies to; it can't be placed in a subdirectory. It applies only to the protocol, host and port where it's posted, so a robots.txt on https://example.com doesn't cover https://m.example.com or http://example.com. It must be a UTF-8 text file. Use a plain text editor, not a word processor, which can add unexpected characters such as curly quotes. If you use a hosting service such as a website builder, it may provide a search settings page instead.
What robots.txt can't do
Robots.txt is a set of instructions for crawlers about which parts of a site to crawl. It isn't a lock: the file itself is public, anyone can read it, and it doesn't stop people from visiting a URL. Don't list secret folders in it, and protect private content with a password instead. If you can't edit the root of your site, Google suggests using another blocking method, such as meta tags.
Testing
After uploading, open yoursite.com/robots.txt in a browser to check it's there, and use your search engine's testing tools to see how crawlers read it. Check that important pages, CSS and JavaScript files aren't blocked by mistake.
Blocking AI and other crawlers
Every crawler identifies itself with a user agent name. To stop a particular crawler from crawling your whole site, add its user agent name to the list of crawlers to block completely, one per line; the generator gives it its own group with Disallow: /. Crawler operators publish their user agent names in their documentation, so check the exact spelling there. Remember that robots.txt depends on crawlers choosing to follow it: well-behaved crawlers do, but it can't force a crawler to stay away.
Common paths to block
- Admin and login areas, such as /admin/ or /wp-admin/.
- Shopping cart and checkout pages, such as /cart/ and /checkout/.
- Internal search results, such as /search/.
- Test or staging folders that shouldn't be crawled.
Don't block the CSS and JavaScript files your pages need, because search engines use them to render and understand your pages.
Wildcards
All rules except Sitemap support the * wildcard, which matches any sequence of characters. For example, Disallow: /*?sort= blocks any URL containing ?sort=, which is useful for filtered or sorted versions of the same page. Use wildcards carefully and test them, because a broad pattern can block more than you intended.
Common mistakes
- Blocking the whole site by accident with Disallow: /.
- Paths without a leading /.
- Wrong case; rules are case-sensitive.
- Relative sitemap URLs. Use the full address.
What next
Use this often? Add Robots.txt to your Home screen.
Frequently asked questions
How does the robots.txt generator work?
You choose a default, list paths to block or allow, crawlers to block and sitemap URLs, and it writes a correctly formatted robots.txt to copy or download.
Where do I put robots.txt?
At the root of your site, such as www.example.com/robots.txt. It can't be in a subfolder, and each host needs its own.
Does robots.txt hide pages from Google?
It controls crawling, not privacy. Use noindex to keep a page out of results and a password for private content.
Are robots.txt rules case-sensitive?
Yes. Disallow: /file.asp doesn't apply to /FILE.asp.
Can I add my sitemap?
Yes. Add one or more full sitemap URLs and they're listed at the end of the file.
Was this tool helpful?
What went wrong?
We only receive the reason you pick, never what you typed into the tool.
Thanks for your feedback.