Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


Introduction to this tool;

Robot text generator is tool made by A TO Z website. This website generated in order to help the search engines to index your website more efficiently. A text file contains one or more rules. Each rule has the ability to block or allows to path that is specified in the website. All the websites are allowed for crawling unless you specify robot txt file.

Here are certain rules that should be followed to creating a robot. Txt file;

  1. Firstly you have to create a file with the name of robots.txt
  2. Now after creating the file add rules to robot. Txt file.
  3. Now on your website upload the robots.txt file.
  4. Now its time to test the robots.txt file

How to create a robots.txt file;

There are following steps to create the rbots.txt file.

You can use this tool to create a robots.txt file. Never use a word processer because it saves the file in format name as proprietary by saving this file in this format there would be addition of unexpected characters. Like quotes that can create problems to crawlers .you should save this file with the encoding of UTF-8.

In SEO;

The robot txt.file is used, in order to prevent the access of web crawlers to all the parts of website that you don’t want to show to crawlers. This file is used in SEO, Which contains search engines command and indexing robots that describes to pages that are indexes or not to be indexes.

Importance of robot.txt file;

This file is important in a sense  because it tells the robots of web pages which pages they would not crawl.

How to make robots.txt file by using this tool

This tool is very simple to use .these simple steps are following;

  1. First of all you have to open this website and then open this tool.
  2. Now you can see some options, carefully read all the options. If you want to keep crawl delay then there are default values of robots in the first row.
  3. When you see the second row which is about sitemap, and don’t forget to mention it in your file.
  4. Now you can choose the parts of website. which part of website should be crawled by robots
  5. Now the last option is for those which you want to select disallowing.

 


LATEST BLOGS

Word Counter and Plagiarism

Word Counter and Plagiarism

13 Sep  / 247 views  /  by Admin
Get response vs mailchimp

Get response vs mailchimp

23 Aug  / 265 views  /  by Admin

CONTACT US

ADDRESS

Sheikh Zahid Road Dubai Silicon Oasis Dubai - United Arab Emirates

You may like
our most popular tools & apps