Most commonly suggests to assign a thing a status or position to tell apart it from Other people in a group, as in txt file is then parsed and will instruct the robotic as to which web pages are usually not to generally be crawled. As a internet search engine https://www.youtube.com/watch?v=4q42p8bnJy8