Becoming A Digital Hoarder

>>58697489
I've been doing that for 15 years. Just always buy two harddrives instead of one, and keep one as backup. Every 5 year or so you can consolidate your smaller drives on one bigger one + backup. You can dump your smaller drives at your friends/parents house as an offsite third backup.

Your second challenge after ensuring you have proper backup is to account for bit flips. Over the span of 60 years, and with storage media density increasing, you will experience bit rot. You need to have a system in place to 1) be able to detect it and 2) be able to recover from it. Personally I split-rar my files with parity and recovery records, which also automatically generate checksums for the files. I also use a machine with ECC ram. It's much easier than dicking around with zfs etc.

>>58697599

Would you mind walking me through your anti-bitrot precautions a little more? I have a lot of stuff to learn I think.

>>58697698

1/?

When you put trillions of bits on a hard drive, they won't stay like that forever. Despite S.M.A.R.T., if you're unlucky, bits can change in such a way that the the error won't even be detected. Over the span of a life time, this will happen a lot of times with your data.

So how would you even know if it had happened? One way you would notice is if it happened in the header of your file, then it would be corrupt and what ever program you're trying to open the file in, will tell you it's corrupt. But if you have a lot of files, you can't manually check that each of your files are not damaged by opening them. Further more, it could happen in the data part of your file, in which case it could only partially damage the file and in such a way that it wouldn't be detectable to the naked eye, for instance a pixel in an image could change color, or a number or letter could change from 0 to 1 of from a to k, etc.

So obviously you need away to know if your files get corrupted. One way you could do that is to generate a checksum for each file. Then, when you want to make sure your files haven't changed, you run a program that will generate a new checksum for all your files, and compare the old checksum with the new one. If the checksums don't match, the file have been corrupted.

But this is a pain in the butt to do.

When you archive files in a WinRar archive, the program creates checksums for all the blocks of data using something called CRC, and if the archive gets damaged by say a bit flip, you will get a CRC error when you try to extract the archive. So in this case, an archived file is a good way to detect bit rot so you can do something about it.

So let's say you have an old dustry winrar archive that gives you a CRC error when you try to extract the files inside. If you have a backup of the archive which haven't been corrupted, you can extract the files from backup.

>>58698434
But if you don't want to take the gamble that the backups you have won't be damaged as well, WinRar has a feature called "recovery record". Basically, what this does is to perform a logical exclusive OR operation (bitwise XOR) on all your data. What this means is that even if you loose some of the data, it can be calculated what was lost based on what remains. Sounds like wizardry, but it's real. The Caveat is that if you loose to much data, at some point it won't be able to recover everything. So this is good for recovering from small data loss, such as due to bit flips etc.

But imagine if one of your harddrives starts to malfunction, and in a panic you try to transfer your archive from the drive onto another one before it dies. But because your archive is 500GB in size, you only manage to transfer 200GB before the drive dies. Well fuck, now what do you do? The answer is split archiving. Insted of making one big ass archive, you split it up into chunks. That way, in the scenario above, you got 200GB worth of chunks off the drive before it died. Now you only need to get 300GB of chunks from your backups to be in the clear. If all your backups are in such a shape that you can only get partial reads out of them, chances are still big that you can get enough chunks off them combined that you are able to get a complete set of chunks. And as a bonus, if you have a big enough recovery record, say 25%, you would only need to get 75% of the chunks to be able to extract the files from the archive!

So there you have it: a split WinRar archive, with a recovery record of say 25%, backed up to one or more storage mediums.

If you use optical media, you can loose 25% of the discs and still be able to recover your data. And it will stay intact as long as you have enough redundancy.

>>58698559
I should mention that adding a recovery record increase the archive size. Roughly speaking, a 10% recovery record will make it 10% larger, a 100% recovery record will increase the size 100%. But when you use winrar, you can compress the files , and often see a reduction in total size that will account for a modest recovery record. At any rate, you can look the recovery record as a form of backup, a 100% recovery record is like storing your files twice on the same storage media.

>>58697489
Build a zfs array and expand any time you get an additional drive. I currently buy drives in pairs for my mdadm raid6. All I have to do is extend the array, extend lvm, and extend xfs.

>>58698938
RAID != backup

Go with some cluster fuck btrfs array and backup that to le cloud
Crashplan and Blackblaze offer unlimited backups