Google Webmaster tools

A free web service offered by Google which allows webmasters and site owners to check indexing status and optimize visibility of their websites.   Included are tools that 
Submit and check a Sitemap 
Check and set the Crawl rate, and view statistics about how Googlebot accesses a particular site 
Generate and check a robots.txt file. 


It also helps to discover pages that are blocked in robots.txt