What are the robot.txt files and how can we use it?

robot.txt

Robot.txt:

Robots.txt is common name of a text file that's uploaded to an internet website's root directory and linked among the HTML code of the net site. The robots.txt file is employed to provide directions regarding the net website to net robots and spiders. net authors can use robots.txt to stay cooperating net robots from accessing all or components of an internet website merely that you just simply need to stay personal.
Website house owners use the /robots.txt file to present directions concerning their web site to internet robots; this is often referred to as The Robots Exclusion Protocol.  

It works almost like this: a robot wish to go to a Web-site URL, ex: http://www.abc.com/index.html. Before it do thus, it initial check for http://www.abc.com/robots.txt, and finds:
User-agent: *
Disallow: /

The "User-agent: *" means that this half apply to any or all robots. The "Disallow: /" tell the robot that it's purported to not visit any pages on the location.

code for robot.txt:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">