robots.txt Tutorial
Search engine spiders and similar robots will look for a robots.txt file,
located in your main web directory.
This is a plain text file. Create or modify it with a text editor
and be sure to upload (FTP) it in ASCII mode.
Documentation sections
If your web site doesn't work properly after adding or modifying this file,
remove it and check it for errors.
