Robots.txt Generator

Generate and validate robots.txt files

Site URL

Preset Template

User-agent

Allow paths

Disallow paths

Crawl-delay (optional)

Host (optional)

Sitemap URL (optional)

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/

Actions:

How to Use

What is robots.txt?

robots.txt is a file that tells search engine crawlers which pages to access and which to skip on your website.

Where should I place the file?

The robots.txt file should be placed in the root directory of your website (e.g., example.com/robots.txt).

Does robots.txt block all crawlers?

No. robots.txt is a voluntary standard. Legitimate crawlers respect it, but it does not prevent access by all bots.