The Robots.txt file is made use of to restrict internet search engine crawlers from accessing sections of your web site. Although the file may be very valuable, It is also a fairly easy method to inadvertently block crawlers. I've published custom crawling and Assessment code for my audits, but https://www.youtube.com/watch?v=EVeTP0aV8gY