Long-term file storage solutions

[DownhillGoat]

So without going into details, I have about 100GB of files I want to keep long-term, but I don't want to keep them on my laptop drive.

What's the best method of storing these? I was thinking either a small external drive or 2 64GB SD cards. Not sure how long either of these would keep the data (data will rarely, if ever, be accessed - so there won't be a whole lot of read/write cycles).

Help/opinions would be greatly appreciated.

[Rathji]

A couple thoughts.

- You should have the data in more than one location. That means if you don't have it on your laptop drive, you should have 2 (or 3) other locations where it is stored. One of those locations should be off-site, if possible.

- Since this isn't storage that will be not be accessed, it doesn't need to be a standard external HDD, however that option would likely less hassle and possibly cheapest long term.

- SD cards would be a good option. As would USB ($35 each) drives. Small, easy to store and not going to make you cry if they get dropped.

- Another, kind of outside the box solution, get a Hard Drive Dock and buy a couple 500GB drives (2.5" or 3.5", it holds both) to store the copies of your data on, then you can get a nice 3TB drive to dock in it permanently. This would allow you to store the drives, have easy access later but still have the dock for use in the mean time. The biggest downside to this is safely storing a naked mechanical HDD is a bit more of a hassle, as you would at minimum want some type of case. (Could even put one together with the static bag and some foam in a small box)

- All those options are going to cost you a good chunk of change up front and be a bit of a hassle. Why not consider online storage? 100GB is $5 a month on Google Drive. Don't need to worry about physically securing the hard drives, if you ever need it, just a simple log in and download, if you want to add more at a future date it is flexible, plus you could use the extra for storing other files 'in the cloud' for other reasons (music etc). Biggest downside would be upload times, especially if you only have a 500kb upload speed from your ISP. If security is a concern, you could encrypt it with something like Truecrypt if you wanted.

[Milt Schmidt]

What Rathji said. If you want to keep files long-term you absolutely need to have multiple copies, preferably one of them offsite. Hard drives are way too fragile to only have one version of important files.

Milt

[PsYcNeT]

2 120 or 128GB SSD drives would be my recommendation. Same storage types as USB sticks (Solid-State Memory), but more reliable in the long term. Floating gate transistors mean that it's resistant to all but the most ridiculous (read: MRI) magnetic fields, and you can even drop them!

[photon]

Amazon Glacier is another newer option for online storage of infrequently accessed files (retrieval times for files can be hours rather than instantly available), not instantly accessible but cost is 1 cent per GB per month.

[photon]

The only downside I could see of an SSD is usually their failure mode involves "oops, drive doesn't work anymore and isn't even visible", whereas with a mechanical drive you usually get some indication that things are going downhill (though not always).

[psyang]

Depending on how long term you are considering, the other factor is whether the storage interface will still be around when you do want to access the data. Floppies and zip disks were prevalent twenty years ago, but not so much now. Cloud storage like Amazon Glacier is safer in that sense, as long as the hosting company doesn't go belly-up. Hard to see interfaces like USB, firewire, or SATA going away anytime soon, but still something you'll want to revisit every few years.

[sclitheroe]

Originally Posted by PsYcNeT

2 120 or 128GB SSD drives would be my recommendation. Same storage types as USB sticks (Solid-State Memory), but more reliable in the long term. Floating gate transistors mean that it's resistant to all but the most ridiculous (read: MRI) magnetic fields, and you can even drop them!

I was just writing something similar, but with a different twist - floating gate transistors means they are literally, physically, susceptible to leakage - this is in fact why SSD's wear out - the electro-physical barrier that keeps the electrons trapped in the floating gate literally loses its ability to hold on to the trapped electrons. I don't think anyone really knows how long the gate can hold a charge on a idle cell at this point, and the read/write usage of an SSD actually protects against this to a degree (it's reasonable to assume every cell will get touched at some point and the charge refreshed, especially in MLC designs)

You can't beat magnetic media for long term storage - the individual magnetic 1's and 0's are tiny, but incredibly high coercivity (magnetic "charge"). Controllers are also incredibly sophisticated in their ability to read less than perfect magnetic charges and do the signal analysis required to separate the data from the noise. In fact, data density is so high now that all reads involve sophisticated signal processing to retrieve the actual data - you never get a clear 1 or 0 off the raw platter.

So I'd say buy the least dense magnetic drive that will hold the data in question - don't use a 2 TB drive to hold 100 GB of data. Lower capacity drives have larger sectors, which means larger magnetic fields, and less susceptibility to magnetic bit flips from high energy particles (yes, this is a real thing - high energy interstellar particles can cause highly dense magnetic media to bit flip. This is why we layer other logical techniques on top like checksumming, etc)

And keep two copies, in two separate locations.

[sclitheroe]

Originally Posted by psyang

Depending on how long term you are considering, the other factor is whether the storage interface will still be around when you do want to access the data. Floppies and zip disks were prevalent twenty years ago, but not so much now. Cloud storage like Amazon Glacier is safer in that sense, as long as the hosting company doesn't go belly-up. Hard to see interfaces like USB, firewire, or SATA going away anytime soon, but still something you'll want to revisit every few years.

Proper data retention techniques would suggest that you need to manage your media lifecycle as well - in the case of floppies or zip drives, for example, there was a window of time where most people relying on those technologies had access to both the old media and a new media format.

File formats are another important consideration - many formats are documented and could be re-implemented on current systems, but many others are a black hole of information about specifications, and if you can't make sense of the file format or run a legacy app on a future system, you are a bit hooped.

[Rathji]

Originally Posted by sclitheroe

Proper data retention techniques would suggest that you need to manage your media lifecycle as well - in the case of floppies or zip drives, for example, there was a window of time where most people relying on those technologies had access to both the old media and a new media format.

File formats are another important consideration - many formats are documented and could be re-implemented on current systems, but many others are a black hole of information about specifications, and if you can't make sense of the file format or run a legacy app on a future system, you are a bit hooped.

That's why for a backup, we print a copy every document we have.

Never need to upgrade a box of files!

[psyang]

Originally Posted by Rathji

That's why for a backup, we print a copy every document we have.

Never need to upgrade a box of files!

Until it's Fahrenheit 451 time. I say memorize it all, and pass it down verbally to your children.

[DownhillGoat]

Originally Posted by photon

Amazon Glacier is another newer option for online storage of infrequently accessed files (retrieval times for files can be hours rather than instantly available), not instantly accessible but cost is 1 cent per GB per month.

I like the pricing and the idea of it, but unfortunately if I read the FAQs right it requires programming knowledge to upload/download anything.

[sclitheroe]

Originally Posted by kunkstyle

I like the pricing and the idea of it, but unfortunately if I read the FAQs right it requires programming knowledge to upload/download anything.

That was the conclusion I came to as well - hopefully some enterprising soul creates an easy to use client, or integrates it into an existing app (Transmit on the Mac, I'm looking at you)

[photon]

Yeah I'm sure there'll be client apps for it eventually, heck my NAS automatically backs up to Amazon S3.