SEO & Web

Robots.txt Generator

Build a properly formatted robots.txt file with a guided builder. Control which search engines can crawl your site and which pages to block.

Blocked Paths (Disallow)

Allowed Paths (Allow)

Sitemap URL

Crawl-delay (seconds)

Note: Google does not support crawl-delay. It is mainly used by Bing and Yandex.

# robots.txt generated by OnOur
# https://onour.co.uk/tools/robots-txt

User-agent: *
Disallow:

User-agent

Specifies which web crawlers the rules apply to. Use * for all crawlers.

Disallow

Tells crawlers not to access the specified path. Leave empty to allow everything.

Allow

Explicitly allows access to a path, even if a parent directory is disallowed.

Sitemap

Points crawlers to your XML sitemap so they can discover all your pages.

Crawl-delay

Requests a delay (in seconds) between crawler requests. Not supported by Google.