These tools are provided to help you create a new robots.txt file for your site or to find errors in them.

1. Creator Tool

Simply select a robot (or all) and enter the path or paths you wish to exclude the bot from.

To completely ban a spider from your web site, select it and enter / in one of the path/directory boxes.

To exclude different spiders from different areas, use this tool several times, once for each spider. Simply copy/paste all results, one after another, into one file.

2. Validation Tool

Copy and paste your robots.txt file in the below box and click the button to check for errors.

This tool attempts to validate syntax and will point out errors to you if it finds any. It also shows you how to correct the error, when possible.