I download lots of media files. So far I have been storing these files after I am done with them on a 2TB hard disk. I have been copying over the files with rsync. This has so far worked fairly well. However the hard disk I am using is starting to get close to full. Now I am needing to find a solution so I can span my files over multiple disks. If I were to continue to do it as I do now, I would end up copying over files that would already be on the other disk. Does the datahoading community have any solutions to this?

For more information, my system is using Linux. The 2TB drive is formatted with ext4. When I make the backup to the drive I use ’rsync -rutp’. I don’t use multiple disks at the same time due to having only one usb sata enclosure for 3 1/2 inch disks. I don’t keep the drive connected all the time due to not needing it all the time. I keep local copies until I am done with the files (and they are backed up).

  • DigitalDilemmaEnglish
    1•1 month ago

    storing these files after I am done with them

    If you’re done with them, then move them onto a backup disk rather than keep them live and have a backup?

    I’ve been doing this for a long time. I move files locally to a “To-Archive” directory and once in a while, move them to several disks based on content. Films, tv, apps, games, books - that sort of thing.

    Once one disk is full, I use another old hdd in a disk caddy and label it “Books #2” and so on.

    I use a windows program called Cathy which indexes the files, making it easy to locate a file on whatever disk it’s on. Looks like there’s a linux version available too

    This works okay for me, and gives a use for old spinny hard drives. It’s not infallible, but for stuff that I could replace (ie, I downloaded it) then I consider it an acceptable risk. All media has a risk of becoming unreadable, but do be realistic about how much bother it would be to replace stuff.

    For data that’s unique (ie, I made it, plus OS backups) then I use an offline grandfather/father/son rotation once a month and once a year turn the oldest into an annual backup. (Fully explanation of my setup is here if you’re interested.