Daily Archives: March 19, 2013

Basic terms of web robot.txt

What is a Web robot: In basic, web robot is simply a program. Robot is used to automatically and recursively traverses a Web site to retrieve document content and information. Search engine spiders is the most common types of Web robots. These robots visit Web sites and follow the links to add more information to […]