|
Software to help you create a robots.txt file
By Sumantra Roy
Robots.txt - Software to help you create a robots.txt file: Find out how this
software will automate the task of creating a robots.txt file
Robogen is a software for Windows 95, Windows
98 and Windows NT/2k/XP based computers which can help you create robots.txt files without
having to worry about the correct syntax of a robots.txt file. It comes in two editions:
the free Limited edition and the Standard edition which costs $12.95. You can get Robogen
from http://www.rietta.com/robogen/.
In this article, I'll tell how you can use
Robogen to simplify the process of creating a robots.txt file. This article assumes that
you have read my
article on what a robots.txt file is and how you can manually create it.
If you missed the article, it is available
Here
I would recommend that you download the trial
version of the Standard Edition. The Limited edition does not support as many search
engines as the Standard Edition.
Once you have downloaded Robogen and installed
it in your hard drive, start the program. Here are the steps that you should follow to
create a robots.txt file:
Step 1: Click on the Logon FTP Server button.
In the dialog box that appears, type in the address of the FTP server for your web site,
make sure that the "This server requires me to log in" option is checked, and
then type in the username and the password which you use to access the server in the
appropriate boxes.
Now, depending on how your web host has set up
the server, the files for your web site may be present in the root directory of the server
itself. Or, it may be present in a sub-directory of the root directory. In the latter
case, click on the "Use specified root" option and type the relevant
sub-directory in the text box besides the option. If you don't know in which sub-directory
the files for your web site are present, ask your web host to give it to you.
Done? Now click on OK. After a while, Robogen
will display the directories and files which are present in your site in the right pane of
the main Robogen program window. If you find that Robogen has not displayed the files in
your site, it means that you did not enter the correct sub-directory in which the files of
your web site are present. In this case, ask you web host to give the correct
sub-directory to you.
Step 2: In the left pane of the main Robogen
window, click on the small button present to the right of the drop-down combo box at the
top. In the menu that appears, select "Popular Search Engines".
Step 3: Now, click on the drop-down combo box
and choose the search engine that you are targeting (say,
AltaVista). In the right pane, select a page which is
NOT meant for AltaVista. Now click on the "<<<" button. You will find
that the file which you had selected in the right pane is now present in the left pane. In
this way, click on each of the pages which are not meant for AltaVista and press the
"<<<" button.
Repeat step 3 for each of the search engines
that you are targeting. Just select the search engine that you are targeting from the
drop-down combo box, select the pages which are not meant for the search engine one by one
from the right pane, and click on the "<<<" button after clicking on
each age. Repeat this process for all the search engines that you are targeting.
Step 4: Upload the file to your server using
the "Upload Robots.Txt" button. If you want, you can also use the Save menu item
from the File menu to save a copy of the robots.txt file in your hard drive.
Article by Sumantra Roy.
Sumantra is one of the most respected search engine positioning
specialists on the Internet. To have Sumantra's company place your site at the top of the
search engines, go to
1stSearchEngineRanking.
For more advice on how you can take your web site to the top of the search engines,
subscribe to his FREE newsletter by going to:
FREE Newsletter Subscription !
|
