Logo Your Web Hosting Solution
Line
robots.txt Guide
Usage Info
List Of Bots
Creation Tool
Use Meta Tags
Block Bad Bots
Tutorials Home
Home Page
robots.txt Tutorial - Easy Tools

These tools are provided to help you create a new robots.txt file for your site or to find errors in them.

1. Creator Tool

Simply select a robot (or all) and enter the path or paths you wish to exclude the bot from.

Create a custom robots.txt file
Robot to exclude
Paths/Directories to exclude

To completely ban a spider from your web site, select it and enter / in one of the path/directory boxes.

To exclude different spiders from different areas, use this tool several times, once for each spider. Simply copy/paste all results, one after another, into one file.

2. Validation Tool

Copy and paste your robots.txt file in the below box and click the button to check for errors.

This tool attempts to validate syntax and will point out errors to you if it finds any. It also shows you how to correct the error, when possible.

Top Of Page 
Line
Copyright© 1996 - 2024 Clockwatchers, Inc.