#2 2006-09-19 04:48:45
Re: robot.txt - Do I need them?
Hi,
Well Robot.txt is generally used if you do not want any of the files in your server not to be listed or indexed. Like if you have any password protected folders you can have it hidden. It is a very powerful but simple at the same time and every website is recommended to have one.
The two conventions used in robots.txt file are User-agent: and Disallow: /
User-agent: * By using the * or wild card you are addressing ALL robots. If you wish to address individual robots you need to list each robot separately with an individual User-agent: statement. They must be listed by their specific name or IP Address, along with a separate Disallow: / statement listing the folders and files you DO NOT want the specified robot to index.
Tip: Use the * wild card to address all robots..... it is the safest way
Disallow: / List any folders that you do not want to have indexed by robots.
Warning: Disallow: / used without any folder name tells the robot do not index ANY page of the website.
Sample Code
###############################
#
# sample robots.txt file for your website
#
# addresses all robots by using wild card *
#
User-agent: *
# list folders robots are not allowed to index
Disallow: /downloads/meta/
Disallow: /downloads/images/
Disallow: /downloads/mp3/
Disallow: /downloads/videos/
Disallow: /downloads/404redirect/
#
# list specific files robots are not allowed to index
#
Disallow: /downloads/meta_tags.html
Disallow: /downloads/custom_error_page.html
#
# End of robots.txt file
#
###############################
NOTE : Once you have created the robots.txt file, you should upload it to the root directory of your domain. Uploading it to any sub-directory won't work - the robots.txt file needs to be in the root directory.
Offline
#3 2006-10-07 03:01:06
Re: robot.txt - Do I need them?
Its worth having a robots.txt - even if its just to tell the spiders they can index all the pages of your site.
Of course its more important to use if you have files you don't want search engines to index.
Offline
#4 2006-11-25 08:06:50
Re: robot.txt - Do I need them?
Just an additional advice: Robots can help you avoiding files to be fetched by search engines. But it will not 'hide' them any way.
Unpolited search engines can simply ignore the robot file, and find your content anyway.
So do not use this way to 'hide' contents from internet... do not work...
Quotes Manager - Trading Software
Offline
