Robots.txt Generator

A robots.txt file is a simple text file placed in your website's root directory that tells search engine crawlers (like Googlebot) which pages or files they can or cannot request from your site. It is the first thing a crawler checks before accessing your content.

No! A robots.txt file prevents crawling, but it does NOT guarantee a page won't be indexed. If other sites link to your disallowed page, Google might still index the URL (though it won't know the content). To prevent indexing, use a "noindex" meta tag instead.

The User-agent directive specifies which crawler the rules apply to. An asterisk (*) means the rules apply to all web crawlers. You can specify "Googlebot" to target only Google, or "Bingbot" for Bing.

Including the absolute URL to your XML sitemap in your robots.txt file is a best practice. It acts as a beacon, immediately showing any visiting crawler exactly where to find your site map of all important pages.

The crawl delay directive tells search engines to wait a certain number of seconds between requests. This is useful for large sites on slow servers to prevent the crawler from overloading the server. Note: Googlebot largely ignores the crawl-delay directive (they use Search Console for rate limiting), but other bots like Bingbot respect it.

It must be placed in the top-level root directory of your website domain. For example, it must be accessible at https://www.yourdomain.com/robots.txt. If you put it in a subdirectory, crawlers will not find it.