Digital archiving

[Guest noah_addis]

Advertisement (gone after registration)

I know it's not the most exciting topic, but I was wondering how people are archiving their work these days.

 

I work for a newspaper now but that's likely to end soon, and I've always shot film for personal projects up to this point. But now I'm shooting almost exclusively with my M8 now and my hard drives are filling up fast.

 

Do you generally save every frame or do you edit and save only the selects? How do people feel about optical storage like dvd's vs. hard drives?

 

Just wanted to get a feeling for what people are doing, as I need to figure something out fast as my drives are filling up with raw files. I'd appreciate any advice but particularly those who do this profesionally or at least those who shoot lots of frames.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[mat_mcdermott]

I only edit and save selects most of the time, but if two or three version of the same image have different merit I might save those too (though that it rare). I'm pretty ruthless.

 

When I was shooting film and saving entire neg. strips occassionally I would find an image that didn't make the cut the first time, when going back and looking months or years after the fact, but that was rare. I figure it's the same with digital files. No need to save every frame.

 

Everything gets backed up to a RAID system, though probably not as frequently as I should.

[wparsonsgisnet]

Noah, there have been some threads on this topic if you want to search.

 

I try (try!) to eliminate pix from shoots so that I don't archive junk.

 

I keep my pix online, in directories named for the shoot, using external drives. These directories are "mirrored" to another disk beside the first. In addition, I make another "mirror" on a disk I keep offsite.

 

When I started doing digital, I kept a second copy of my negs and was very pleased with myself -- until one of the disks died and I had no backup in place. That event caused me to add the offsite disk.

 

Many recommend CD or DVD as archive tools. There are 2 problems with this: (1) no one knows what the life of these media is, and (2) if you scratch one, you're finished.

 

I do not happen to use a RAID solution. The reason for this is that I am the sole user of my disks, and I mirror after each job. Using RAID causes disk to be exercised all the time, adding a wear factor to their usage. If I were sharing disks, RAID would be the sensible solution.

 

The package I use for mirroring is Second Copy. It's inexpensive and can be run manually as well as under a schedule. It's very nice software.

 

Regards,

[charlesphoto99]

I just purchased a Firmtek 5 disk array. Not cheap, but in retrospect one of those purchases I should have made years ago instead of yet another lens.

 

FirmTek SeriTek/5PM

 

Really, well built, quiet - very impressed. It only does Raid 1, 0, or 1+0 and of course JBOD. I will set it up so two bays are Raid 1, and the other three JBOD. Reliability vs speed most important to me. The nice thing is to be able to hot swap drives out and then take just the individual drive itself off site. Or load stuff onto a drive you may not need to access very often (but now and then) and just leave it in a spare tray ready to go.

 

It's a port multiplier array, so there is just one eSATA cable and of course one power cord. Really clears up the clutter of having lots of different external hard discs.

 

I did a lot of research before buying an array, and this one appears to be the best for the $ on the market. Of course there are better, but they can run lots and lots of $! The best hard drives I've found are the WD 640gb Caviar, the WD caviar Black 1TB and the Samsung Spinpoint 1TB. I went with 5 WD 640gb to begin with, mostly because they are $77 each (from OWC) and I'll be doing a lot of backing up for offsite. Will then go with some of the WD Blacks 1TB.

[bbodine9]

I just this week started to use an online storage site run by Hewlett-Packard known as "Upload". They can and will upload and store any type of file on your computer. Raw, tiff, jpeg etc it does not matter. The cost is only $5.00 per month for unlimited storage. My mind is now at ease as my raid starts to reach capacity. I am not associated with them in any way just a satisfied customer. They also have business accounts available.

[wparsonsgisnet]

I just this week started to use an online storage site run by Hewlett-Packard known as "Upload". They can and will upload and store any type of file on your computer. Raw, tiff, jpeg etc it does not matter. The cost is only $5.00 per month for unlimited storage. My mind is now at ease as my raid starts to reach capacity. I am not associated with them in any way just a satisfied customer. They also have business accounts available.

 

Your dependency on this solution is one-way. If HP changes or terminates something, you could lose your backup.

 

You get what you pay for; you pay for what you get.

[jaapv]

Advertisement (gone after registration)

I try to eliminate real junk, but keep most. But then I am not a typical digital machine gun

I run a nifty little program called second copy that automatically backups all image files onto another harddisk whenever I shut down the computer.

Once a week I backup that harddisk onto an external harddisk that I keep at work.

 

It is wasteful of harddisk space, I am halfway those 500 Mb disks, but well, these things get bigger and cheaper all the time.

 

I tried an on-line backup, I use one at work, but found it painfully slow for the large amount of data images generate.

[wparsonsgisnet]

I try to eliminate real junk, but keep most. But then I am not a typical digital machine gun

I run a nifty little program called second copy that automatically backups all image files onto another harddisk whenever I shut down the computer.

Once a week I backup that harddisk onto an external harddisk that I keep at work.

 

It is wasteful of harddisk space, I am halfway those 500 Mb disks, but well, these things get bigger and cheaper all the time.

 

I tried an on-line backup, I use one at work, but found it painfully slow for the large amount of data images generate.

 

Brilliant!

[charlesphoto99]

What amazes me (and really about myself, up to this point) is how many photographers shoot with thousands and thousands of dollars worth of gear and then back up to crappy little drives.

 

Forego that extra lens and buy some state of the art enclosures. It's often the cheap enclosure electronics that corrupt or shorten the hard disc life and not the disc(s) itself (I've had that happen to me).

 

The online thing seems like a pain in the ass. Fine if you shoot a few jpegs now and then, but really slow if you shoot jobs (which can be 8gbs and up a go, for me at least).

[miami91]

Aperture has a built in backup utility (I'm at work and not in front of my Mac --- can't recall what it's called). You can direct your Aperture library to be backed up to multiple disks/locations using this utility if you wish. I have mine backing up to a RAID 1 connected via eSATA to my Mac, and also backing up to a wireless disk elsewhere in my home (in case a thief were to take everything in my study!). Not sure that this is a "state of the art" backup solution, but like most things Apple, it's hella easy.

 

Jeff.

[jaapv]

What amazes me (and really about myself, up to this point) is how many photographers shoot with thousands and thousands of dollars worth of gear and then back up to crappy little drives.

 

Forego that extra lens and buy some state of the art enclosures. It's often the cheap enclosure electronics that corrupt or shorten the hard disc life and not the disc(s) itself (I've had that happen to me).

 

The online thing seems like a pain in the ass. Fine if you shoot a few jpegs now and then, but really slow if you shoot jobs (which can be 8gbs and up a go, for me at least).

 

That is true. I started out now with 500 Gb drives, but I will upgrade to disks in the Tb class. It simply fills up too fast. And good quality disks are really not that expensive any more. I tried to upload my archive to one of this services, but after running overnight it hadn't even reached half. And not because my internet is slow, but because the accepting server was slow....

[charlesphoto99]

That is true. I started out now with 500 Gb drives, but I will upgrade to disks in the Tb class. It simply fills up too fast. And good quality disks are really not that expensive any more. I tried to upload my archive to one of this services, but after running overnight it hadn't even reached half. And not because my internet is slow, but because the accepting server was slow....

 

Jaap,

 

Be sure if you get an enclosed 1TB external drive it consists of 1 drive and not two 500gb daisy chained together. I had one of those (a Fantom) and it went bust on me. Not because of drive failure (I'm now using those drives after removing them and reformatting in my new array) but because the firewire architecture went corrupt. The drives were good - the electronics inputting to then crap.

 

Best to buy separate enclosures of high quality and put your own drive(s) in.

[martinb]

I only backup my work to a portable drive once in a while, but I'm going to buy a RAID soon. One thing I like about the M8 DNG files is that they're small, especially considering how well they print at most sizes and how well they interpolate.. But 30-50mb RAW files would probably give me a headache..

[wparsonsgisnet]

Martin, a RAID array, while transparently safe, is more expensive and exercises the disk more.

 

If you're the only user, you can achieve "mirroring" a number of other ways at lower cost.

[martinb]

wparson,

I will not work from my laptops harddrive in the future. That's what I do and mirror the stuff I want to backup to a portable drive. My agency also holds a bit of my work of course..

I'm going to work with RAID but only two discs and also a portable that I backup to once in a while and store in another location..

[miami91]

Martin, a RAID array, while transparently safe, is more expensive and exercises the disk more.

 

If you're the only user, you can achieve "mirroring" a number of other ways at lower cost.

 

Bill,

 

I'm not sure this is necessarily the case. I'm no expert, but it seems to me that RAID is just a set of methodologies for addressing storage devices (internal or external) by using a particular chipset/controller in conjunction with HDDs. As an example, I have a RAID card in my MacPro that allows me to configure the HDDs in the four internal bays in any number of ways. Two of these I have on a RAID 0 arrangment (makes two separate 1TB drives appear as a single 2 TB drive), and the two others I have in a RAID 1 "mirrored" arrangement, where data is written to both drives simultnaeously. I haven't noticed the discs churning or otherwise showing signs of unusual levels of read/write activity compared to a standard single HDD. Granted, there is extra overhead --- something is performing computation to support the structure, but I think that this is done by a separate hardware solution (RAID card), the host machine's processor, or the relatively simple controller built in to many external solutions. It's not taxing the disks themselves.

 

Similarly, I can (and sometimes do) completely turn off the external RAID I have if I'm not actively using it for archiving. In this scenario, there clearly is no extra disk activity! I turn it on when I'm ready to run the Aperture backup, wait till the icon appears on my desktop, run the backup, then turn it off as soon as it's finished. This isn't an option if you use timed backups in the middle of the night, but I've never found this necessary. Unlike, say, an enterprise-wide database application, my photo library only changes a handful of times a week, and only as a result of my activity, so I manually engage the backup only when I've downloaded and/or scanned new images. And since it's incremental (in Aperture anyway), this takes all of a couple of minutes.

 

My experience, for what it's worth.

 

Jeff.

[digger1914]

Aperture has a built in backup utility (I'm at work and not in front of my Mac --- can't recall what it's called). You can direct your Aperture library to be backed up to multiple disks/locations using this utility if you wish. I have mine backing up to a RAID 1 connected via eSATA to my Mac, and also backing up to a wireless disk elsewhere in my home (in case a thief were to take everything in my study!). Not sure that this is a "state of the art" backup solution, but like most things Apple, it's hella easy.

 

Jeff.

 

I think you are referring to their Vault system

 

You can't back up too much, so i use SuperDuper! to clone my MacBook Pro at home to an external drive, and when I take it to work I do the same on a drive I keep there. I also use Aperture's Valut and Apple's time Machine as well.

 

I think it is essential to have copies of your data in at least two separate locations.

[haris]

Try this for digital arciving

 

Film as a means of storage of digital data - APUG

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[wparsonsgisnet]

Jeff, I think I agree with your procedures. Turning the RAID array off, keeps it from going to the gym every time you touch one of the disks.

 

The important thing, of course, is Backup, Backup.

 

I look at the notebooks of analog negatives in the corner of my office and wonder how I'd get that shelf full of books out of the house in a fire, while at the same time herding the bride, HER valuables, the teddy bears, and so on.

 

Through all of this, I managed to delete some files last year -- erroneously thinking they were backed up. S'ok, tho. they weren't much and it made me more careful.

[xjr]

I only use films : slides & BW.

 

a. I scan everything in my Coolscan 5000 and save them in directories whose names reflect the slide boxes and film sleeves by occasion,place and date as well as position in the slide box or film sleeves.Everything is scanned at basic settings. They are saved on a 2nd hard drive on the Mac Pro and also on an xternal drive. This is where the speed of the 5000 comes really handy.

 

b. Therefore to locate a slide or negative takes little time.

Once a negative or slide is to be processed for a print or presentation or whatever only then it is scanned at 16bit with all the required settings turned on. Then it is saved again in specific directories.

 

The first process can be boring and tedious but necessary if precise records are to be kept.

After all even the same subject at different times looks and feels different...the beauty of this passion !