Search engine spiders visit the site will be the first to visit robots.txt file, robots.txt is used to guide the search engine spiders to prohibit the capture of certain content on the site or only allowed to capture the content, placed in the root directory of the site. User-agent indicates that the following rules apply to which spider, * table ...