cross-posted from: https://lemmy.world/post/16841877

The world’s top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.

OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.

TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.

OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.

However, according to TollBit’s findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to “bypass” robots.txt in order to retrieve or scrape all of the content from a given website or page.

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.” A spokesperson for Anthropic did not respond to emails seeking comment.

Robots.txt is a single bit of code that’s been used since the late 1990s as a way for websites to tell bot crawlers they don’t want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.

    • @[email protected]
      2•5 months ago

      On the one hand the genie is out of the bottle. On the other hand this doesn’t stop the moral debate of “is this the right path even?”.

      My gut tells me those making these choices driving “AI” forward right now are on the wrong side of history. Only time will tell.

      • 🇨🇦🇩🇪🇨🇳张殿李🇨🇳🇩🇪🇨🇦
        2•5 months ago

        The genie is not out of the bottle. It’s stuck in the neck as people realize how limited and useless the “AI” really is. The seeming ubiquity of it is a concerted marketing campaign pushed by people trying desperately to cash out before the bubble bursts.

        It’s following the hype curve of cryptocurrency, then NFTs, only in fast-forward. For a while the “true believers” (just like with crypto and NFTs) will keep it going forward as they delude themselves into thinking that it will turn around “any time now” … but in reality AI is already collapsing.

        See the problem is that while crypto and NFTs were wrapped in a whole metric fuckton of complicated jargon and terminology that laypeople couldn’t understand, the various “AI” projects had free, public-facing services that were easy enough for even technophobes like me to be able to use …

        … and find out what utter shit they are.