Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

  • @[email protected]
    link
    fedilink
    English
    426
    edit-2
    2 months ago

    Honestly, any platforms hosting user-generated content who use the legal argument that they only provide hosting and aren’t responsible for what their user post shouldn’t also be able to sell the same data and claim owning any of it.

    Otherwise, take away their legal immunity. Nazis or pedophiles post something awful? You get in front of the judge.

    edit: typo

    • Justin
      link
      fedilink
      English
      1812 months ago

      Exactly this. You can claim that their scraping is abusing your servers, but the moment you claim copyright for the content of the site, then you give up your Section 230 rights.

      • @[email protected]
        link
        fedilink
        English
        362 months ago

        You’d also probably lose a whole lot more processing power trying to stop the crawlers vs just letting them have API access with some sort of limit to queries.

        • Admiral Patrick
          link
          fedilink
          English
          29
          edit-2
          2 months ago

          Eh, not really.

          I block bot user agents to my Lemmy instance, and the overhead is pretty negligible for that (it’s all handled in my web firewall/load balancer).

          Granted, those are bots that correctly identify themselves via user agent and don’t spoof a browser’s.

          It’s also cheaper and easier to add another load balancer than size up or scale out my DB server to handle the bot traffic.

        • AlexanderESmith
          link
          fedilink
          10
          edit-2
          2 months ago

          Systems admin here: You’ll lose almost the same amount of processing time looking up limits and providing a “you’re over your credits / rate limit” as you would by just providing the data.

          Also, everyone will game the system with multiple accounts (cost be damned, the entities who want the data have cash to burn).

        • @[email protected]
          link
          fedilink
          English
          62 months ago

          I don’t think they actually block malicious bots, the change they’ve made is just to the robots.txt, they don’t have to do anything.

    • @givesomefucks
      link
      English
      382 months ago

      Can’t sell something you don’t own.

      So if they’re selling the parts people want, they need to own the parts no one wants.

      • Justin
        link
        fedilink
        English
        3
        edit-2
        2 months ago

        Well, you can give money to Reddit for a piece of paper, but unless Reddit is claiming copyright to the content posted there, then they can’t sue anyone for not paying. It would be very interesting to see the text of these “licensing agreements”.

        • lemmyvore
          link
          fedilink
          English
          42 months ago

          They’re not claiming copyright. They have a perpetual, non-revokable license to the content, granted by the people who use their site when they post the content.

  • @[email protected]
    link
    fedilink
    English
    102
    edit-2
    2 months ago

    Robots.txt isn’t a binding agreement, this isn’t stopping anyone for whom their drive for profit outweighs their ethics.

    Also, Fuck Spez.

  • @Sordid
    link
    English
    662 months ago

    The enshittification cycle:

    Phase one, attract users by providing a good service.
    Phase two, once the users are locked in, squeeze them for all they’re worth by selling them to business customers (advertisers and/or data buyers).
    Phase three, once the business customers are locked in, squeeze them for all they’re worth by threatening to deny them access to the users on whom they now depend.

    Spez seems to think Reddit has the pull to make phase 3 happen. I rather doubt it, but we’ll see.

    • lemmyvore
      link
      fedilink
      English
      142 months ago

      If he really had balls he’d restrict access to the site and improve the built-in search engine.

      If reddit’s own search worked well nobody would care. Engines like DDG even have bang codes that send you to a site’s own engine. So instead of having to add “site:reddit.com” to the search on DDG I’d just add “r!” and it would end up being the same thing. IF the internal search didn’t suck.

    • _haha_oh_wow_
      link
      fedilink
      English
      9
      edit-2
      2 months ago

      Yeah, as soon as the API thing happened I switched to Lemmy for mobile browsing and like it more than Reddit (Connect is pretty good, but even the mobile browser site is solid).

      The more they squeeze, the more popular alternatives like Lemmy, Kbin/Mbin, Tildes, etc. will become.

    • Boozilla
      link
      English
      62 months ago

      My guess is that phase three will work for a while. But I think you’re right that eventually they are going to drive that thing into the ground. Because it’s never enough pure profit for rent-seeking scum, and there is no lower limit to the abuse they’ll inflict on their content creators (who they call users but think of as products).

  • @[email protected]
    link
    fedilink
    English
    582 months ago

    Fuck Spez. He’s probably editing the comments anyway, he literally can’t help himself.

  • @[email protected]
    link
    fedilink
    English
    462 months ago

    I’m sure plenty of others join me in the sentiment of thinking “Who the fuck are you to restrict MY free content that I contributed?”

    God, fuck reddit so fucking hard

  • @spongebue
    link
    English
    462 months ago

    Honestly, my biggest issue with LLMs is how they source their training data to create “their own” stuff. A meme calling it a plagiarism machine struck a chord with me. Almost anyone else I’d sympathize with, but fuck Spez.

    • @[email protected]
      link
      fedilink
      English
      62 months ago

      What resonated with me is people calling LLMs and Stable Diffusion “copyright laundering”. If copyright ever swung in AI’s favor it would be super easy to train an AI on stuff you want to steal, add in some generic training, and now you have a “new” piece of art.

      LLMs and Stable Diffusion are just compression algorithms for abstract patterns, only one level above data.

      • Echo Dot
        link
        fedilink
        English
        6
        edit-2
        2 months ago

        The real takeaway of all of this is that copyright law is massively out of date and not fit for purpose in the 21st century or frankly the late 20th.

        The current state of copyright law cannot deal with the internet, let alone AI

    • @markon
      link
      English
      52 months ago

      Yep they now get paid for the data we have them. I have no sympathy lol. At least these models can’t actually store it all losslessly by any stretch of the imagination. The compression factors would have to be like 100-200X+ anything we’ve ever been able to achieve before. The numbers don’t work out. The models do encode a lot though and some of it is going to include actual full text data etc but it’ll still be kinda fuzzy.

      I think we do need ALL OPEN SOURCE. Not just for AI, but I know on that point I’m preaching to the choir here lol

  • Bappity
    link
    English
    412 months ago

    profiting off of user generated content 😒

  • @[email protected]
    link
    fedilink
    English
    412 months ago

    And yet reddit is happy to make money off our content for free.

    Or at least it did. Personally I overwrote and deleted all my content a while back.

      • m-p{3}
        link
        fedilink
        English
        192 months ago

        Bingo, the only winning move is not to play at all and stop using Reddit.

      • dual_sport_dork 🐧🗡️
        link
        English
        92 months ago

        Everyone always says this like it’s some kind of gotcha, but all of my nuked posts still have my “fuck you, reddit” content and haven’t been reverted. It’s been nearly exactly a year.

        Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they’d be handing that directly over to whoever under some kind of agreement – that certainly wouldn’t be the subject of any kind of site crawling which is the crux of the issue here.

        • @Womble
          link
          English
          62 months ago

          it never was deleted, all that happened is that an extra line was added to a database that said “comment 65432426542654 now should be displayed as “fuck you, reddit” rather than the original text”. The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.

        • @[email protected]
          link
          fedilink
          English
          52 months ago

          You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

          Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price

          IMO, you can win by jamming your “transmissions” with noise. It’s easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were

          • finley
            link
            fedilink
            English
            32 months ago

            You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

            there’s no evidence to suggest this, though.

            • @[email protected]
              link
              fedilink
              English
              12 months ago

              Content is absolutely archived and they have financial incentive to restore the quality of their “knowledge base”

              That’s a fair amount of circumstance and motivation to support my idea, regardless of tangible evidence

              • finley
                link
                fedilink
                English
                02 months ago

                Motivation and circumstance, absent actual evidence, does not make for a convincing argument.

      • finley
        link
        fedilink
        English
        3
        edit-2
        2 months ago

        i went looking for old comments and posts i had made after i overwrote then wiped them. They’re still gone. i looked again several months later, and they were still gone.

        so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.

        • @[email protected]
          link
          fedilink
          English
          62 months ago

          On the other side of the same coin: When I mass edited my comments before quitting Reddit, I got site-banned. Basically, my first account’s automated edit got me auto-banned from several subs with pro-spez mods. Some subs had set their automod to detect when people were using the more popular methods of auto-editing, and set the automod to ban for using them. Then when I did the same with my second (and third, and fourth, and fifth, etc…) account, it almost immediately got site-banned for ban evasion.

          Basically, account 1 was banned from a sub, so when account 2 started doing the same thing on the same IP address, it was flagged as ban evasion. And ban evasion is one of the few things that will get you banned site-wide instead of just from a specific sub.

          I went back and checked a few months ago, and all of those site bans were lifted and the edits were undone. Likely because a site ban prevents the comments from showing up (which hurts Reddit’s bottom line, because they show up as a bunch of [removed] comments instead,) but also prevented any of the edits from actually being published. So when they lifted the site ban (to get those old comments to show back up again) it was as if I had never edited them at all. I had probably a million karma spread across my various accounts. I was extremely active at one point, so Reddit had a direct incentive to unban those accounts with literal thousands of comments.

        • @[email protected]
          link
          fedilink
          English
          52 months ago

          As I said in another comment, I was not suggesting that Reddit would restore your comments to public view.

        • @[email protected]
          link
          fedilink
          English
          52 months ago

          You are assuming edits overwrite existing content. Instead of overwriting, they could just store the edited post as a new entry in the database with a higher version number. Then, you only show the latest version of each post to the end users while keeping the older versions available die Reddit’s own use.

          In fact, it is extremely likely they do this. It is basically a necessity if you want to be able to properly moderate a site like Reddit. Otherwise you could simply post spam or unsavory content, and then overwrite it with something benign an hour or so later, before there were enough reports and a moderator would have gotten a chance to review it.

          • finley
            link
            fedilink
            English
            -32 months ago

            You are assuming edits overwrite existing content

            i have seen no evidence to suggest otherwise, but thanks for sharing your theories

            In fact, it is extremely likely they do this

            based on what evidence? your baseless speculation?

            • @[email protected]
              link
              fedilink
              English
              52 months ago

              The fact that they managed to restore overwritten posts after people started to delete their history.

              • finley
                link
                fedilink
                English
                0
                edit-2
                2 months ago

                this could also be explained by sketchy scripts failing to completely delete posts/comments, which i even noticed myself when checking that they had done their jobs properly. as i mentioned in another comment, i had to run the shredder scripts several times for complete overwrite/deletion. or it could be database errors failing to register edits/deletions due to extremely heavy loads at the time. it could be a lot of things.

                the point is that we don’t have any direct evidence of what it actually was, just a lot of circumstantial evidence and a lot of speculation. nothing definitive.

                • @[email protected]
                  link
                  fedilink
                  English
                  72 months ago

                  Reddit used to be open source. There is still a copy of that source available on github. It’s 7 years old so it’s probably significantly different from what they are running now. Still, it gives some insight into the design.

                  For example, deleted comments aren’t deleted, it just sets a deleted flag. Example code that shows this.

                  I haven’t dug around the code enough to figure out how editing works, it’s Python code so an unreadable mess. The database design also seems very strange. It’s like they built a database system on top of a database.

        • _haha_oh_wow_
          link
          fedilink
          English
          42 months ago

          It seemed to happen to some people but I wouldn’t be surprised it it was just some sort of coincidental database fuck-up

          • finley
            link
            fedilink
            English
            2
            edit-2
            2 months ago

            i suspect that was more likely incomplete deletions than reddit restoring content. those scripts were pretty janky. i had to run mine several times to get everything, as it didn’t work fully the first couple of times. same with the overwrites. took a few times for those to get everything, especially on older accounts with lots of posts and comments.

            • _haha_oh_wow_
              link
              fedilink
              English
              32 months ago

              I’d read some claims that posts appeared to be deleted but then later came back. Could’ve even been some sort of caching shenanigans with their local browser though I guess.

              • finley
                link
                fedilink
                English
                1
                edit-2
                2 months ago

                oh, right. i suppose that’s possible. i’ve seen similar browser cache fuckery on other sites before.

                • finley
                  link
                  fedilink
                  English
                  0
                  edit-2
                  2 months ago

                  as i said in a previous comment:

                  so, unless reddit did a massive restore of everyone’s comments/posts except for my 4 accounts, then i don’t believe they did it at all except for a select number of top contributors who deleted their content.

                  but there’s no evidence they’re keeping everyone’s deleted-but-restored comments from public view or whatever it is you’re suggesting. or even anything past whaat this one person found. in fact, there isn’t even any evidence that what happened to this user was intentional and not a bug or some other fluke.

                  sure, reddit would have a vested interest in doing this, and what you’ve presented is suspicious, but it’s hardly conclusive of anything. all it does is raise more questions. but it doesn’t provide answers.

      • @Animoscity
        link
        English
        22 months ago

        Yep that’s how it works. Older content past a certain date is cached which is why you can’t comment or post on some old posts.

    • AlexanderESmith
      link
      fedilink
      -12 months ago

      All you did was screw over people looking for potentially useful content while searching online. Reddit still has all your content.

      • dual_sport_dork 🐧🗡️
        link
        English
        22 months ago

        Sure, but if they’re not displaying it what good is it doing them or their advertisers?

        • AlexanderESmith
          link
          fedilink
          02 months ago

          They’re not selling the ability to scrape their publicly viewable site, they’re selling API access to their database. Deleted posts aren’t removed from that database.

  • @Brkdncr
    link
    English
    292 months ago

    What if I had an agreement with MS that they can scrape my data and anything I post online?

    • Admiral Patrick
      link
      fedilink
      English
      252 months ago

      In all the ways that matter, it’s already dead. Once something enshittifies beyond a certain point, is its zombified, shambling corpse really considered “alive” anymore?

      • @[email protected]
        link
        fedilink
        English
        32 months ago

        It’s Digg v…5 6? v5 would be when they inflated all post scores and stopped showing upvotes and downvotes separately.

        v6 is… gestures to the all of it

    • @[email protected]
      link
      fedilink
      English
      132 months ago

      It’s easy to say this as someone who is “on the other side”. But the data doesn’t really back up that statement.

    • @[email protected]
      link
      fedilink
      English
      72 months ago

      It’s awful. Politics is unavoidable at this point, and the amount of general anger on the platform is crazy.

      People love watching their videos of people getting TBIs… Or getting too excited about a “justice served” post where a woman gets hit.

      It’s kinda nice to see someone get their comeuppance, but then you look in the comments and there are just weirdos saying stuff like “glad that bitch got hit”, like… wtf?

  • @[email protected]
    link
    fedilink
    English
    152 months ago

    I never bothered to go edit or delete my comments after the API drama that caused me to move here, but now I might just go do that because the entire point of keeping old comments up was that maybe someone will find one from a search engine and find it useful. If reddit is going to monetize THAT, they can fuck right off.

    • palordrolap
      link
      fedilink
      152 months ago

      Save your effort. What’s already there is there forever. They can just roll back your comments, or even, if they’re in the mood for it, make it appear under an entirely different username.

      The only way to win is not give them any more. And that fight is already under way. They’ve already started recommending old comments after new ones because the quality isn’t as high any more.

      Think about it: The only people who contribute to Reddit now are the clueless and the sort of people who have willingly stayed.

      I like to imagine Spez stomping around saying “Hmph! Hmph! It’s not fair! Why did they all leave?! They’re stealing my revenue by not giving me anything for free!”. I mean, he’s probably not doing that, but I do like to imagine it.

  • downpunxx
    link
    fedilink
    132 months ago

    If you’re still donating your content to the Facebook, Twitter and Reddit data stores, then I don’t know what to tell you man, you’re literally enriching the worst people, who will do the worst things, with your information, your stories, your answers, your comments, your labor, your effort. You are giving yourselves to them. Literally.

    • AlexanderESmith
      link
      fedilink
      22 months ago

      Whenever this comes up, I immediately think of how easy it would be to scrape the threadiverse.

      • @FierySpectre
        link
        English
        32 months ago

        They’re already doing that, if only just for search.

        Googling my username used to not give any results up until a few months ago… Since I’m on Lemmy you find my comments by throwing my username into a search engine

  • Michael Ten
    link
    English
    72 months ago

    Minimum royalty laws should exist!

    • @Womble
      link
      English
      2
      edit-2
      2 months ago

      Because if there’s one thing this world needs more its more rights for property.