Building your robots.txt File
What is a robots.txt file?
This is a very important file that tells the search engine robots what
pages of your site you want searched.
It is super simple to create and you will want to do it right away.
Open a new page on your HTML editor and name it robots.txt
Copy and paste the following information into this file:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /_private/
Disallow: /template.html/
If you have pages that you are working on
after you have gone live, and they are not ready for
indexing, add them to this file. Don't forget to
remove the disallow after you finish the page.
Save it to your server.
The first line, User-agent: *, gives all robots
permission to index your site. If you have a certain
robot that you want to exclude from indexing your
site, just add them to a disallow:
The next lines stop the spiders and robots from
indexing any page that you disallow. This is real
handy for any pages that aren't ready for viewing or
members only areas.
If you are interested learning more about
robots.txt files, you will have to do a search on
Google and find someone who devoted more time to
this item. There is a Google search box at the
top of this page for your convenience.
|
Azoogle
Want
someone to help you with your affiliate links?
