The rise and fall of robots.txt
www.theverge.com
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
    • LvxferreEnglish
      6•9 months ago

      For banning: I’m not sure but I don’t think so. It seems to me that prefetching behaviour is dictated by a page linking another, to avoid any issue all that the site owner needs to do is to not prefetch links for the honeytrap.

      For poisoning: I’m fairly certain that it doesn’t. At most you’d prefetch a page full of rubbish.