- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.
Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.
The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).
In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.
Huffman said:
Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.
“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.
Removed by mod
On the other side of the same coin: When I mass edited my comments before quitting Reddit, I got site-banned. Basically, my first account’s automated edit got me auto-banned from several subs with pro-spez mods. Some subs had set their automod to detect when people were using the more popular methods of auto-editing, and set the automod to ban for using them. Then when I did the same with my second (and third, and fourth, and fifth, etc…) account, it almost immediately got site-banned for ban evasion.
Basically, account 1 was banned from a sub, so when account 2 started doing the same thing on the same IP address, it was flagged as ban evasion. And ban evasion is one of the few things that will get you banned site-wide instead of just from a specific sub.
I went back and checked a few months ago, and all of those site bans were lifted and the edits were undone. Likely because a site ban prevents the comments from showing up (which hurts Reddit’s bottom line, because they show up as a bunch of [removed] comments instead,) but also prevented any of the edits from actually being published. So when they lifted the site ban (to get those old comments to show back up again) it was as if I had never edited them at all. I had probably a million karma spread across my various accounts. I was extremely active at one point, so Reddit had a direct incentive to unban those accounts with literal thousands of comments.
As I said in another comment, I was not suggesting that Reddit would restore your comments to public view.
Removed by mod
Except for incidents like This
There’s no evidence of the contrary either.
You are assuming edits overwrite existing content. Instead of overwriting, they could just store the edited post as a new entry in the database with a higher version number. Then, you only show the latest version of each post to the end users while keeping the older versions available die Reddit’s own use.
In fact, it is extremely likely they do this. It is basically a necessity if you want to be able to properly moderate a site like Reddit. Otherwise you could simply post spam or unsavory content, and then overwrite it with something benign an hour or so later, before there were enough reports and a moderator would have gotten a chance to review it.
Removed by mod
The fact that they managed to restore overwritten posts after people started to delete their history.
Removed by mod
Reddit used to be open source. There is still a copy of that source available on github. It’s 7 years old so it’s probably significantly different from what they are running now. Still, it gives some insight into the design.
For example, deleted comments aren’t deleted, it just sets a deleted flag. Example code that shows this.
I haven’t dug around the code enough to figure out how editing works, it’s Python code so an unreadable mess. The database design also seems very strange. It’s like they built a database system on top of a database.
FWIW even when you properly delete something from a database table, the deleted row can be reconstructed from the audit tables. And even if that weren’t the case, databases are regularly backed up to tape drives or whatever - when people delete or munge all their comments, Reddit doesn’t go back into all the backups and make the same changes there. In fact, I would imagine that when they sell their shit to companies for AI training, they sell old pre-AI backups rather than a latest copy.
Removed by mod
I never claimed it was evidence of how it currently works, only that it gives some insight into how Reddit was designed. I would be very surprised if they changed this aspect of the design. It makes sense to not delete comments or edits for reasons I mentioned before. Unfortunately we won’t know for sure unless Reddit confirms it.
It seemed to happen to some people but I wouldn’t be surprised it it was just some sort of coincidental database fuck-up
Removed by mod
I’d read some claims that posts appeared to be deleted but then later came back. Could’ve even been some sort of caching shenanigans with their local browser though I guess.
Removed by mod
Removed by mod