First of all one thing you may know is robots.txt is nothing it is only a text file which is created at the time of creating a website. This name is case sensitive, maintain all the letters to be lower case.
In the robots .text We give the instruction to the search engine through this file robots.txt file.We place this file robots.txt in the root directory of the site. That means we place this file besides the index file of our website.
Example:
How to create a robots.txt file :-
User-agent: *
Disallow: /on-pageseo_folder/on-pageseo.html
user agent: * means that the instruction is for all the search engine (crawlers).
Disallow: instructs the crawler not to index the file /on-pageseo_folder/on-pageseo.html
If u want that my all file are not index then use this Coading
In the robots .text We give the instruction to the search engine through this file robots.txt file.We place this file robots.txt in the root directory of the site. That means we place this file besides the index file of our website.
Example:
How to create a robots.txt file :-
User-agent: *
Disallow: /on-pageseo_folder/on-pageseo.html
user agent: * means that the instruction is for all the search engine (crawlers).
Disallow: instructs the crawler not to index the file /on-pageseo_folder/on-pageseo.html
If u want that my all file are not index then use this Coading
