This is a question that comes to mind every time I spend a few days focusing on the fediverse. Normally I’m on the microblogging side, but now I have a Lemmy account it might start a proper discussion.
So, to the point, pretty much every fedi platform has similar problems with small servers taking a beating whenever a post goes viral. This ends up costing the server owner a bunch of money trying to keep their server alive while thousands of instances attempt to pull large static files from the original host’s post. This recently instigated this call to action on this forum.
I’ve never seen the question of torrents answered and it feels like a lot of effort and a bit self entitled to get the ear of fedi software devs to implement torrents as a solution, so I’m putting this here.
If media files were made into torrents when a post was being created, an extra object could be added to post objects like
'torrentcdn': {
'https://imagePathAsKey.jpg': {
'infohash': 'ba618eab...',
'torrentLocation': 'https://directlinkto.torrent',
'webseed': 'https://imagePathAsKey.jpg',
...
}
}
This would not break compatibility as it would just be ignored by anything not looking for a ‘torrentcdn’ object, yet up to date instances could use this instead of directly pulling the static files.
This would benefit instances as when a post goes viral, the load would be distributed amongst all instances attempting to download the file.
This could also benefit clients and instances as larger files like short videos could be distributed using webtorrent, massively reducing the load on server when many people are watching the same video.
Thoughts?
How torrents validate the files being served?
Recently I read a post where OP said they were transcoding torrents in place and still seeding them, so their question was if this was possible since the files were actually not the same anymore.
A comment said yes, the torrent was being seeded with the new files and they were “poisoning” the torrent.
So, how this can be prevented if torrents were implemented as a CDN?
An in general, how is this possible? I thought torrents could only use the original files, maybe with a hash, and prevent any other data being sent.
I don’t know what that post is about. It’s not possible to change the contents of a torrent. The torrent file itself is a list of checksums which validate byte ranges within the files being downloaded. If a client downloads a poisoned piece, it discards it and deprioritises the seed it got it from. Perhaps they’re transcoding a file, whilst still seeding the original.
Torrents can work as a CDN for static files, because the downloader has to validate that the file is the same one as on the server using the checksums in the torrent file.
Yeah, I just searched a bit and found this https://stackoverflow.com/questions/28348678/what-exactly-is-the-info-hash-in-a-torrent-file
The torrent file contains the hashes of each piece and the hash of the information about the files with the hashes of the pieces, so they have complete validation of the content and amount of files being received.
I wonder if the clients only validate when receiving or also when sending the data, this way maybe the seeding can be stopped if the file has been corrupted instead of relaying on the tracker or other clients to distrust someone that made a mistake like the OP of that post.