robots.txt Generator

🤖

robots.txt Generator

// generate a robots.txt file for your website

FREE TOOL

Sitemap URL (optional)

▶ Crawl Settings

Apply rules to

All crawlers ( * ) Googlebot only Bingbot only GPTBot (OpenAI) only CCBot (Common Crawl) only

▶ Block Common Directories

/admin/ — admin panel /login/ — login pages /api/ — API endpoints /private/ — private files /wp-admin/ — WordPress admin /cgi-bin/ — CGI scripts /?s= — search result pages /tag/ — tag pages

▶ Custom Disallow Paths

One path per line (e.g. /secret-page/)

▶ Allow Paths (override disallow)

✔ Copied to clipboard!

Generated robots.txt

# click Generate to create your robots.txt

ⓘ Save this file as robots.txt and upload it to the root of your domain (e.g. https://yourdomain.com/robots.txt).

VERXIO · Code. Secure. Innovate.

verxio.site