The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

  • SerotoninSwellsEnglish
    7•1 year ago

    NYT also uses a third party bot identification and mitigation service.