Crawler or Spider
When you submit your website to search engines, they visit your website after submission or by following links from other sites and then start crawling or spidering pages in your website. You can prevent crawling some or all of your pages by placing a robots.txt file in root directory of your web server that most of the search engines recognize. During search engine optimization (SEO) process this aspect is carefully analyzed whether all the pages are being crawled or not. Search engines crawl pages, gather the information, analyze the content, rank the pages and include these in their index to show results for user queries in the search engine results pages (SERPs). A web crawler or spider is also referred as Bot that usually starts crawling sites and as these crawlers or spiders visit sites they identify all the hyperlinks and add them to their repository, to visit these later as the time permits. Crawling or spidering is done periodically on continuous basis for the existing or new pages so that their index is updated for user queries. For your website you should provide high quality and rich content on pages as this gets indexed by search engines. Provide complete information about products and services as this will help to attract visitors on your website and entice other webmasters to cast a vote to your site.
