The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

  • @TreczoksEnglish
    19•1 year ago

    The question is: Does that crawler adhere to robot.txt policies?

    • @[email protected]English
      3•1 year ago

      They made a flag specifically for their crawler, so they can say that they do but in the most annoying way possible.