Why is data storage in the home so difficult?

I’ve been struggling with an efficient and cost effective method of personal data storage for years. How to organise my files, what to keep in my electronic archives, how best to make sure my backup strategy has everything I need?

After reaching maximum capacity on my last external disk drive I’ve written this blog post as a thought process to help me decide on a course of action in the medium to long term.


Increasing data storage needs
The growth of personal data is massive. Even my parents have over 0.5TB of family photographs on their home computer. A family with a desktop, kids laptops, media servers and mp3 players and mobile phones could well have 4TB+ of combined storage capacity. The biggest issue however is high definition video and the ease at which it can consume Gigabytes of storage.

Ease of administration
I could build my own data storage hardware solution and run an open source platform on top, there are loads out there like FreeNas, OpenMediaVault, OpenFiler, ZFSGuru and others.

However, all these require a step up in learning so large that I can’t commit to and so cannot commit my data to their systems. I travel a lot with work and need a solution that is also easy for other family members to administer while I’m away.

People like to organise files in different ways, just as people work best in different ways. Personally I find that a fairly flat directory structure is best combined with a search that allows for advanced syntax and filter options. Microsoft search looks rather basic from the outside but has a host of advanced syntax options to help you find your files. Details at MSDN http://msdn.microsoft.com/en-us/library/aa965711(VS.85).aspx  An example of the search to find all the music by the Chemical Brothers could be:

kind:music artist:"Chemical Brothers"

Of course by having a shallow file structure it makes is easy to navigate to the music folder and simply start typing Chemi… to jump to the folder I’m looking for.

Root -> Music -> Chemical Brothers – Singles 93 - 03

Other that running a CMS system that governs unique files it’s nearly impossible to avoid creating duplicate files at some point. The only true way of minimising their occurrence is to use a simple organisation structure and logical naming convention so duplicates easily stand out. Some things are better left to people rather than processes.

As an example I’ve started a personal rule that I only ever copy one way to my mp3 player. Copying from storage to the player means that I know when I need more space on the player I can safely delete music and if I want it back on just re-copy from storage.

Backing up data has gotten much easier. Windows 7 backup is much improved (although still has some huge issues), Apple has their iCloud and Time machine system and there are 3rd party online systems for a monthly amount you get all you can eat data.

However, it can get quite costly if you start paying by device and also quite hard to make sure you’ve got everything that’s important to you.

For me, personally, some things just aren’t worth the time backing up. Rips of DVDs and BluRay films for one. Sure it takes a while to rip the files, but the encoding methods are getting better and the downside of having to back up 8GB+ per rip are prohibitive in both time and cost.

On the other hand family photos and family video footage is like gold dust, it’s simply irreplaceable and so gets top priority for both storage and backup.

Documents, saved emails, important pdf files and encrypted document volumes are important but can usually be recreated, it’s just going to be a pain to do or a challenge to re-request documents from the originating 3rd party.

You don’t need to backup what you can re-create if you have enough time and you can spare that time in their re-creation. It’s just up to you deciding what you can afford (time / money / effort) to lose.

Possible Solutions

Rather than worrying about all this data, all these devices, all these problems; just simplify and delete it. It can be hard to delete things, especially that collection of funny pictures you downloaded from the Internet in 1998 but do you really need to go buy a new 4TB hard disk to store them on? No, of course you don’t, and you’ll feel better with the weight of worrying about data off your mind.

Too many family photos on a hard disk never looked at? Do what a lot of people are going back to and print out the albums. There are loads of online printing services and by having photos up on the wall or in albums chances are you’ll look at the memories that matter more.

You can always shift part of the problem to a 3rd party if you trust them. Pictures can be uploaded to Flickr and Facebook if you’re willing to accept their terms and privacy policies. Google, Microsoft and Apple all have Free and reasonable paid Cloud storage, let the corporate giants shoulder the infrastructure. You could find yourself locked in and committed to continue a service longer term though.

Buy a NAS
This is the conclusion I came too, but my wife informs me I’m a Geek and shouldn’t all self-respecting Geeks own a NAS?

I could write a whole blog post on just choosing a NAS. I’ve deliberated over which NAS, which disks, which RAID level for almost 18 months. I am very picky about my technology and when it comes to precious data, 18 months almost feels like a snap decision!

It also feels somewhat like a cop out too, but after so long of having half baked solutions, single external disks, huge towers to administer and manage having a nice NAS solution with good support and great features makes the cost aspect worth it for me.

I’m pleased to say I’m very happy with my Synology DS1812+ and I’ll be posting a review of it.

3 thoughts on “Why is data storage in the home so difficult?

  1. Hi,

    I’ve struggled with the very same thing so your post here mirrored my thoughts.

    The problem with NAS is that it’s online storage, and it really is ‘on’ all the time. I’d really prefer offline storage or storage that I can bring online only when I need to do something with it. Saving energy is the primary reason. I feel guilty about running a massive NAS system 24×7 when I’ll only be accessing it say once a week on average when I do something with our family photos and videos. So did you explore offline storage options at all?

    Also, how much power in kWh does your Synology unit end up using over say a month or so? I’m sure the cost in $ isn’t much, but I’m curious how much it amounts to?

    Also, why’d you end up going with the DS in the end?


    1. Hi Arjun,

      I am glad I’m not alone in my thoughts on this subject of data storage.

      I understand how you feel about things being ‘on’ all the time as my wife has similar views on turning things off. The reality is that with modern low power modes and sleep states the DS1812+ is very power efficient for what it is. I actually have a second Netgear ReadyNAS that I backup to which only powers on Sunday night for the backup then powers off for the rest of the week. You can do the same power on / off scheduling with the Synology devices if you like, although ours is used a lot more than we thought for streaming music and movies daily.

      In real terms my DS1812+ with four 3TB disks uses 45w in use and 18w standby. The calculation below shows the average use of our device to be very reasonable in terms of costs.

      Daily power use
      2h @ 45 watt = 0.09 kWh
      22h @ 18 watt = 0.39 kWh

      0.48 kWh @ 11.8p (1 kWh of electric for me) = 5.7p per day

      I went with a Synology device in the end as I had administered a previous device at work and it was very reliable and I was impressed with the quality of their DSM software that manages the device. I chose the DS1812+ in particular because it allows me to host two 4x RAID 5 arrays.

      Hope that helps


Leave a Reply

Your email address will not be published. Required fields are marked *