Robots.txt files are a type of directive that show crawler bots where they are and aren’t allowed to go, and are used to complement other on-page directives signals to show what content needs to be indexed for search.