Older than most of the search engines we use today, the robots.txt file is a dinosaur of the web. Although […]