• Veraxus
    link
    239 months ago

    If it is publicly accessible, then yes, they scrape the data.

    This is literally how search engines have worked since damn near the inception of the internet.

      • @SacrificedBeans
        link
        129 months ago

        Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.

      • @kaffiene
        link
        English
        49 months ago

        There was, and still is