Why is data storage in the home so difficult?

I’ve been struggling with an efficient and cost effective method of personal data storage for years. How to organise my files, what to keep in my electronic archives, how best to make sure my backup strategy has everything I need?

After reaching maximum capacity on my last external disk drive I’ve written this blog post as a thought process to help me decide on a course of action in the medium to long term.

Issues

Increasing data storage needs
The growth of personal data is massive. Even my parents have over 0.5TB of family photographs on their home computer. A family with a desktop, kids laptops, media servers and mp3 players and mobile phones could well have 4TB+ of combined storage capacity. The biggest issue however is high definition video and the ease at which it can consume Gigabytes of storage.

Ease of administration
I could build my own data storage hardware solution and run an open source platform on top, there are loads out there like FreeNas, OpenMediaVault, OpenFiler, ZFSGuru and others.

However, all these require a step up in learning so large that I can’t commit to and so cannot commit my data to their systems. I travel a lot with work and need a solution that is also easy for other family members to administer while I’m away.

Organisation
People like to organise files in different ways, just as people work best in different ways. Personally I find that a fairly flat directory structure is best combined with a search that allows for advanced syntax and filter options. Microsoft search looks rather basic from the outside but has a host of advanced syntax options to help you find your files. Details at MSDN http://msdn.microsoft.com/en-us/library/aa965711(VS.85).aspx  An example of the search to find all the music by the Chemical Brothers could be:

kind:music artist:"Chemical Brothers"

Of course by having a shallow file structure it makes is easy to navigate to the music folder and simply start typing Chemi… to jump to the folder I’m looking for.

Root -> Music -> Chemical Brothers – Singles 93 - 03

Duplication
Other that running a CMS system that governs unique files it’s nearly impossible to avoid creating duplicate files at some point. The only true way of minimising their occurrence is to use a simple organisation structure and logical naming convention so duplicates easily stand out. Some things are better left to people rather than processes.

As an example I’ve started a personal rule that I only ever copy one way to my mp3 player. Copying from storage to the player means that I know when I need more space on the player I can safely delete music and if I want it back on just re-copy from storage.

Backups
Backing up data has gotten much easier. Windows 7 backup is much improved (although still has some huge issues), Apple has their iCloud and Time machine system and there are 3rd party online systems for a monthly amount you get all you can eat data.

However, it can get quite costly if you start paying by device and also quite hard to make sure you’ve got everything that’s important to you.

For me, personally, some things just aren’t worth the time backing up. Rips of DVDs and BluRay films for one. Sure it takes a while to rip the files, but the encoding methods are getting better and the downside of having to back up 8GB+ per rip are prohibitive in both time and cost.

On the other hand family photos and family video footage is like gold dust, it’s simply irreplaceable and so gets top priority for both storage and backup.

Documents, saved emails, important pdf files and encrypted document volumes are important but can usually be recreated, it’s just going to be a pain to do or a challenge to re-request documents from the originating 3rd party.

You don’t need to backup what you can re-create if you have enough time and you can spare that time in their re-creation. It’s just up to you deciding what you can afford (time / money / effort) to lose.

Possible Solutions

Simplify
Rather than worrying about all this data, all these devices, all these problems; just simplify and delete it. It can be hard to delete things, especially that collection of funny pictures you downloaded from the Internet in 1998 but do you really need to go buy a new 4TB hard disk to store them on? No, of course you don’t, and you’ll feel better with the weight of worrying about data off your mind.

Alternate
Too many family photos on a hard disk never looked at? Do what a lot of people are going back to and print out the albums. There are loads of online printing services and by having photos up on the wall or in albums chances are you’ll look at the memories that matter more.

Diversify
You can always shift part of the problem to a 3rd party if you trust them. Pictures can be uploaded to Flickr and Facebook if you’re willing to accept their terms and privacy policies. Google, Microsoft and Apple all have Free and reasonable paid Cloud storage, let the corporate giants shoulder the infrastructure. You could find yourself locked in and committed to continue a service longer term though.

Buy a NAS
This is the conclusion I came too, but my wife informs me I’m a Geek and shouldn’t all self-respecting Geeks own a NAS?

I could write a whole blog post on just choosing a NAS. I’ve deliberated over which NAS, which disks, which RAID level for almost 18 months. I am very picky about my technology and when it comes to precious data, 18 months almost feels like a snap decision!

It also feels somewhat like a cop out too, but after so long of having half baked solutions, single external disks, huge towers to administer and manage having a nice NAS solution with good support and great features makes the cost aspect worth it for me.

I’m pleased to say I’m very happy with my Synology DS1812+ and I’ll be posting a review of it.