Websites actually just list broad areas, as listing every file/page would be far too verbose for many websites and impossible for any website that has dynamic/user-generated content.
You can view examples by going to most any websites base-url and then adding /robots.txt to the end of it.
List of files/pages that a website owner doesn’t want bots to crawl. Or something like that.
Websites actually just list broad areas, as listing every file/page would be far too verbose for many websites and impossible for any website that has dynamic/user-generated content.
You can view examples by going to most any websites base-url and then adding /robots.txt to the end of it.
For example www.google.com/robots.txt
https://www.reddit.com/robots.txt