Robots.txt
Robots.txt file is stored in the root directory of the website. In robots.txt file you can mention which part of your website can be crawled or you can prevent some parts of your website from being crawled by search engines. You can use a robots.txt file to request that search engines remove your site and prevent robots from crawling it in the future. To prevent robots from crawling your site, place the robots.txt file in the root directory with specific mention of which Crawlers or Bots are allowed or disallowed or which part of the site is disallowed
The robots.txt file can be placed at the time of web design or during your search engine optimization (SEO) process.
