AnsweredAssumed Answered

Pro/Con of robots.txt and sitemap.xml

Question asked by adamc on May 3, 2017
Latest reply on May 3, 2017 by Busby

I am working to standardize our web application scans and one of the options is

"Crawl all links and directories found in:"

robots.txt file

sitemap.xml file

 

What are the Pro/Con of setting the web app to use these or not use them?

 

Outcomes