update 2: The Linux community has suggested that I use a tar file to backup, as this preserves symlinks. With that, the home directory now takes up just 290 ish GiB, as it should. Now I will be distro hopping, wish me luck!
update: I was able to copy it! There are still some folders that are really big (as many have said, it is probably because symlinks aren’t supported in exFAT. When I transfer these files over to btrfs, will the symlinks come back or are they permanently gone?) but, with the uninstallation of Steam and Kdenlive (each taking a ridiculous amount of storage), removing a couple games I don’t really play, and removing old folders that lingered around from already uninstalled programs means I now have enough space to fit my home folder in the SSD (like 23 GiB left, so the lack of symlinks still hurts, but still, it fits!)
When running
rsync -Paz /home/sbird "/run/media/sbird/My Passport/sbird"
As told by someone, I run into a ran out of storage error midway. Why is this? My disk usage is about 385 GiB for my home folder, and there is around 780 GiB of space in the external SSD (which already has stuff like photos and documents). Does rsync make doubly copies of it or something? That would be kind of silly. Or is it some other issue?
Note that the SSD is from a reputable brand (Western Digital) so it is unlikely that it is reporting a fake amount of storage.
EDIT: Wait, is it because my laptop SSD is BTRFS and the external SSD is exFAT? Could that be the issue? That would be kind of weird, why would files become so much more bigger with the external SSD?
Thanks everyone for your help to troubleshoot! It was super helpful! Now I need to go to bed, since I’ve been up so late it’s already tomorrow!
Maybe you have hard links or sparse files in your source directory. Try with -H for hard links first. You can try with --sparse but I think hard links are more likely.
Using -H throws an error as symlinks aren’t supported in exFAT it seems.
By the way, do you have lots of torrents downloaded or large virtual machines installed? Both torrent clients and virtual machine managers use ‘sparse files’ to save space until you actually download the whole torrent or write a lot to the VM’s disk. Those files would be copied at full un-sparse size to exFAT.
If you have folders with such content, you can use e.g. Double Commander to check the actual used size of those folders (with ctrl-L in Doublecmd). Idk which terminal utils might give you those numbers in place, but aforementioned
ncducan calculate them and present as a tree.Edit: silly me, of course
duis the util to use, typically asdu -hsc dirname.using du -hsc returns 384G with /home/sbird, and 150G inside the external SSD (when it does not have any of the files transferred with rsync)
Well, that’s not what I meant. If you have directories with torrents or VMs,
dumight report different size for those directories on the source and target disks. Then that might mean that those are the culprits, depending on how much difference there is.With just the source disk, you can check
du -hsc dirnameversusdu -hsc --apparent-size dirnameto check if the disk space used is much smaller than the ‘apparent size’, which would mean there are sparse files in the directory, i.e. not fully written to disk. rsync would copy those files to full ‘apparent size’.As mentioned elsewhere, btrfs might also save space on the source disk by not writing duplicate files multiple times — but idk if
duwould report that, since this feature is specific to btrfs internals.For a typical user, hard links would be mostly employed by git for its internal structures, and it’s difficult to accumulate over 300 GB of git repos.
Sparse files would actually be more believable, since they’re used by both torrent clients and virtual machines.