- Now Offering SEO Audits and Website Analysis! - Specializing in custom WordPress themes and plugins. - Get 20% off when you order 2 services! - Let's build something amazing together!
- Now Offering SEO Audits and Website Analysis! - Specializing in custom WordPress themes and plugins. - Get 20% off when you order 2 services! - Let's build something amazing together!
Configure, preview & download in seconds
Quick Presets
Block Common Areas
Block Specific Bots
Your Website URL
Sitemap URL will be set to https://yourdomain.com/sitemap.xml automatically.
Custom Rules
About This Tool
A robots.txt file tells search engine crawlers which pages they can and cannot visit. It is the very first file Google checks when it visits your site, placed at the root: yourdomain.com/robots.txt.
Upload it to the root of your domain — the same folder as your index.php or index.html. For WordPress, that is public_html. For Shopify, go to Online Store > Themes > Edit Code and find robots.txt.liquid.
Never, unless you intentionally want to be removed from Google search. Only block specific admin paths like /wp-admin/ or /wp-login.php. A misconfigured robots.txt is one of the most common SEO mistakes.
Add User-agent: GPTBot with Disallow: / to block OpenAI, and User-agent: CCBot with Disallow: / to block Common Crawl. Use the "Strict" preset above to add all known AI bots at once.
Yes. Blocking important pages prevents them from being indexed. Not including your sitemap URL misses an opportunity to help Google discover content faster. Always include Sitemap: https://yourdomain.com/sitemap.xml.
No. Robots.txt is publicly visible — anyone can read it. Malicious bots often ignore it. Use it to guide well-behaved crawlers. For real security, use server-level access controls and password protection.