Sound Library Storage Solutions

I had a three way coincidence the other day: first my 2 x 2TB library drives finally filled to the point of overflowing, and they were already my backup laptop storage. Secondly a friend Tom emailed asking what do I reccomend for library storage, and then I just saw this post on twitter: ‘How to store 5TB of data for 50 years, untouched?’
The latter is an extreme and aimed at archiving, which is a different problem than the first two, but rather than give Tom my ideas I thought it far wiser to make it a discussion and see what works for you, and also to see what might be the best solutions in the future…..

10MB drive

My library is currently stored on 2 x 2TB G-tech RAIDs which have worked well, but I’m not sure they are my next solution. Firstly I don’t really see the need for RAID as the performance required of a library drive is not like that of work drives playing big ProTools sessions. All I need to do is audition mono, stereo or occasionally 5.1 files.
Secondly I need to have three copies of all media regardless: one at my work studio, one at my home studio & one offline for safety. So having a RAID as a form of data safety seems unnecessary. If a drive went down, I’d simply grab a new drive and go clone the missing data from one of the other two locations.

A fried Ray has done a fair bit of research on this topic & hopefully he might chime in but an option I think he was looking at was this sort of enclosure;

Hydra drive

The MacPower Hydra has FW400, FW800, USB2, eSATA and can hold four drives, so if you loaded it with four 2TB drives that would provide 8TB of storage. Locally I’ve seen it loaded with 6TB shipping for approx NZ$1,400 = US$1,000…

Of course if you are a facility with lots of sound editors accessing a central sound library server then your needs are different than mine, but what are you using?

13 thoughts on “Sound Library Storage Solutions

  1. Tim Crossley

    Hi Tim,

    First of all, I was just introduced to your site a few days ago and I’ve eagerly been lapping up all your previous posts. Great stuff!!!

    Secondly, I’m glad you brought this topic up. I’m currently looking for a reasonably affordable solution for this exact same scenario for my home setup. At the various studios I work out of, it seems as though no one really has one good solution. Everyone has their own way of doing it. At “Dig It Audio”, they’re using a RAID drive that is accessible to any computer on their network via a Soundminer DB. At “Duotone Audio” which is more of a jingle house, they just have a portable drive that gets tossed around whenever an effect is needed. This works fine for them seeing as how they rarely ever do any sound design, since their focus is on composition. Another studio I work out of, whose name I’m not going to mention, just has a few Glyph GT Key drives that get shared around the studio when they’re needed. While the Glyph drives are great, and while this scenario works, it still seems less practical than the way “Dig It Audio” does it, while still being just as expensive. Not to mention, I’m pretty sure that they don’t have a good back up of their entire library. For shame!

    So, looking in to this myself, for my purposes, I’m thinking of going for something similar to the MacPower Hydra. I think it’s a great choice for the money. I am a little upset that there are less rackmount options out there that allow you to hot swap drives like the Glyphs do. There’s really no other company (that I know of) making anything similar. I’m definitely looking forward to hearing what other people are doing.

    Sorry for the long post. Thanks again!

    Tim

  2. Robert Catto

    Hey, Tim – I’m not a sound librarian myself, but a photographer with about 16TB of storage (including backups, that is).

    I had a custom PC built a few years ago off an Intel motherboard which included a RAID controller, so I got my first Raid 5 terabyte storage that way. BUT, and it’s a big but, about a month out of the three year warranty the motherboard fried itself, and replacements were unavailable; so the array couldn’t be rebuilt as I couldn’t replace the RAID controller with another of the same chipset. So, while RAID 5 sounds great in theory, there is still that potential for a single point of failure to take it all down…

    My new philosophy (at the moment) is to only use good quality consumer-grade – ie. readily available – single external drives, and back them up daily, with a weekly off-site swap. That way, if one fails, I can simple go out an buy a replacement the same day – no waiting for parts to be shipped from overseas, or worse, finding out they’re not available at all.

    Just a thought – lord knows I’m a sucker for flash electronics, but I’m learning to restrain myself!
    R

  3. John-Paul McCarthy

    Hi Tim,

    I had been running a MacPower Taurus NAS that, sadly, failed earlier this year after 2 years of operation. Enclosed were 2 x 1TB drives in a RAID 0. Data redundancy wasn’t an issue as I was backing up its contents. I did corrupt a good chunk of my music collection due to synchronisation of corrupt files prior to realising a failure was imminent.

    With our shift to Wellington being quite a large disruption I haven’t taken the time to determine whether the problem was with the enclosure/controller or a drive malfunction.

    The unit did have a few things going against it:
    – clunky printer sharing, would operate roughly 50% of the time
    – noisy fan
    – configuration browser was poor
    – some translation errors were evident in the software
    – although it advertises a gigabit LAN connection, the performance wasn’t even near that of a USB 2.0 drive. I’d attribute this to the controller allocating a maximum bandwidth to the number of allowable simultaneous users. I tried changing configuration settings but was unable to improve performance.
    – what I would call a ‘deal breaker’ is that connection dropouts would often occur on backup. An occasionally during general use.

    On a brighter note, I have recently purchased a QNAP 219P NAS
    http://www.qnap.com/pro_detail_feature.asp?p_id=122

    They’re a nice UK-based outfit as well.

    I purchase the item in July while in Canada. NZ street price puts it at $940 with 2 x 2TB 7200 rpm SATA drives in RAID 1.
    I chose redundancy this time around!

    Have a browse of the products available on their website (listed above). They have a vast range of NAS products; all the way up to 8-bay desktop or rack mount.

    Also, you’re more than welcome to stop by our Newtown HQ if you want to look at the drive personally. The 2-bay is quite small & quiet and the web-based configuration is much nicer to use than the MacPower product.

  4. Pingback: Tweets that mention Music of Sound » Sound Library Storage Solutions -- Topsy.com

  5. Paul Virostek

    I’ve been wanting a Drobo for some time:
    http://www.drobo.com/products/index.php

    The cool thing is that the drives don’t need to have the same capacity, which is handy for scraping together older, smaller capacity drives that on their own aren’t useful any longer.

    I plan to buy internal drives separately at NewEgg (and avoid the markup the comes with case+drive) and then stock up the drobo…

  6. Tom Heuzenroeder

    Hi Tim and everyone else.

    Before I put in my thoughts about Library Storage, here’s a bit of back story:

    One of the things that triggered my question to you Tim about data storage ideas is the fact that the hard drive in my MacBook Pro died over a period of about 12 hours, though I wasn’t aware that it was happening at the time. I merely got a lot of lethargic behaviour (ie: the beachball/colour wheel of doom kept popping up and freezing the machine for 10 seconds at a time every 10 seconds or so. I used all the onboard diagnostic tools to find the problem (Activity Monitor, Disk Utility) but non of them showed anything wrong. Finally when I suspected the drive was failing, it was too late, and attempts to copy the data off failed as a result of data corruption.

    An I.T. friend of mine gave me this explanation:
    “Disk Utility is fine for what it is, but can only look as far as the filesystem, not the hardware. Unfortunately modern hard disks have a great technique of masking underlying pending-failure from the file system through internal recovery processes that are transparent to the file system (with the exception of massive performance loss whilst they internally re-arrange themselves). Only something like SMARTReporter, that can interrogate the internal SMART diagnostics will give you visibility of this (though even then it’s sometimes too late). Arguably apple should have build SMART reporting into the O/S, and I never understand why they don’t!…”

    The issue I have found is that unless an external drive enclosure incorporates it into its design, SMART diagnostics data is ignored by external drive enclosures and don’t get relayed to the host computer, though I think it would be really helpful if it did as a preventative measure for potential data loss, should a drive begin to fail. You can actually see it’s about to fail, as opposed to one day powering it up and discovering it’s just about dead.

    So perhaps a NAS type solution with RAID? But then I think again, and wonder if that is too limiting because it’s a commitment to hardware that will date, as opposed to a slow continual replacement of off the shelf drives as and when is needed – and again as pointed out above, the drives are not being powered up all the time and used only for archival purposes. Though I must say, I do like the idea of attaching drives over a network, but perhaps I only like that for the sake of neatness and the ability to put it out of the way…for of course, eSATA would be a lot faster, no?

    I go round in circles with these thoughts… hence this discussion thread is of keen interest to me. Thanks Tim.

    Tom

    1. jeff p

      Well I don’t think NAS solutions will age badly so to speak. Ethernet is a protocol that will be with us for a long, long time and anything based on this protocol I would feel confident in purchasing.

      Again any data that is mission critical or has any value should be on ATLEAST 2 different drives. Not just a raid volume – what if the raid controller dies? as far as I know you need the exact raid controller to rebuild the data. What if there was a fire on site? I think your data would be compromised.

      1. John-Paul McCarthy

        @jeff p

        Good idea to bring up the concept of backing up data to more than one volume; not just the NAS.

        Being PC-based, I use a little app called SyncBackPro. Works a treat for creating backups (including FTP) & syncs. It’s easy to use with a lot of powerful features for filtering. I have two external drives; one for audio app data & libraries, the other for design & programming projects. These syncronise with my NAS drive to provide some redundancy. Lastly, I am planning to pick up a larger external USB drive that will be plugged into the NAS for one-button backups of the entire drives contents (or specified directories).

        Man alive, there’s certainly a great deal of choice when it comes to data storage and integrity.
        Cheers for your input everyone; and thanks Tim for posting this topic up.

  7. Tom Heuzenroeder

    Good points posted here. Yes, NAS is not something that will disappear in a hurry – I was thinking more the firewire usb aspects there. And again, off-site backups are a must! My uncertainty was more about the RAID side of things which certainly could become an achilles heel.
    I am PC-based as well as Mac, and so will check out SyncBackPro. Thanks for the suggestion. Mac’s Rsync program is powerful for that sort of stuff, so much so that one has to be careful with it!

    1. John-Paul McCarthy

      Tom, Initially I was skeptical of purchasing a third party backup / sync utility. Of course, Windows has one. But I found it to be terribly flaky; missing files, missing folders, it wouldn’t sync additional or new folder properly, etc. I highly recommend it.

      A ‘lite’ version trial is also available that has reduced capabilities but gives you an idea of how the software operates.

      SyncBackPRO
      http://www.2brightsparks.com/syncback/sbpro.html

      SyncBackSE
      http://www.2brightsparks.com/syncback/sbse.html

      Not to intentially derail, but I purchased the SyncBackPro software with this handy utilities package. The undelete saved a relations bacon when their SD card full of photos was accidentally wiped.

      OnClickUtilities
      http://www.2brightsparks.com/onclick/index.html
      • FindOnClick performs lightning fast file searches.
      • UndeleteOnClick recovers files after deleting.
      • DeleteOnClick securely deletes data.
      • HashOnClick helps guarantee files are identical.
      • EncryptOnClick delivers military grade encryption.
      • ScrambleOnClick encrypt/decrypt files & text.
      • PatchOnClick easily updates large files.
      • TrayOnClick manage Windows easily.

Leave a Reply

Your email address will not be published.