Archiving Digital Images

[mustafasoleiman]

Advertisement (gone after registration)

Back to the original impetus for this thread -- one problem with weeding out the not-ready-for-prime-time images that that this takes time. That's a commodity I would willingly pay for.

 

That is a big problem... the time it takes to weed them out... That is also why I want to keep the rejects easily accessible, just in case I change my mind, or someone else changes it for me (usually my wife or son).

 

I was thinking to have two databases in Aperture, separate but both accessible, do an initial quick weeding out as Guy recommended, splitting the choices between the two dbs.

 

In terms of hardware I can only recommend LaCie drives. I have two 1Tb drives, one backup of the other.

They are totally silent, which is extremely important for me as I have a silent computer, and consequently I can hear myself think in my office!

 

The issue with LaCie drives is that the power supplies tend to overheat and some eventually fail, giving the intermittent not mounting issues some experience... it's the power supply not the drive. Just be aware of that and you will enjoy what I think are the best drives around.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[Guest guy_mancuso]

Well editing is one of the hardest things to do . You really have to be ruthless in many ways. I find this so hard i can't tell you with my own work, with others work it is very easy but emotion gets in the way with your own.

[Shootist]

To go with my other post.

I have used every make drive that is available over the years. WD, Seagate, Maxtor, IBM/Hitachi, Fujitsu?? (SP) and have had every brand fail at one point or another. Some within 1-2 weeks of putting it in a system. Others years after I startd using them and some never in it's time I used them, which could be 5-7 years or 1-2 years. It's just the nature of the beast.

I now use Seagate if I'm buying a new drive. A year or so ago it was Maxtor and a year before that it was WD. I stopped using WD when I ordered a 80GB 16MB cache drive 4 years ago for a new build and the drive I got had a 80GB label on it but was only a 60GB drive and only 8MB cache. I got a replacement unit for that one and that replacement drive failed within 2 weeks of startup. The new drive that replaced the original replacement has been running 24/7 for the last 3 years + with NO problems. But since WD couldn't get the label coerrect on one drive, or a series/lot of drives, and then a replacement that failed I figured it was time to change brand. That is when I went with Maxtor which 8-10 years ago I wouldn't even consider. Now as I said I use Seagate, for internal drives, and the big reason for that is they have a 5 year warranty.

My 2 WD external drive were chosen on price for size as I do not use them as a add-on to the internal storage of my PC's so they only run for short periods of time. They are just for back ups and even then my most important files are on at least 2 other PC along with my main desktop.

Back up and back up and back up.

At this point I would need to lose 3 PC's and the external drives all at the same time to lose all my data.

[Eoin]

Mustafa,

 

I use Aperture a lot and it can offer a range of storage management options.

 

You can used referenced images, ie images which reside on a drive but are not necessarily within the actual aperture database. Aperture creates full sized previews for these images so even if they are actually not on line they are still visible within the application.

 

You can of course import all your images into the data base. This is the option I tend to use. Then after import I tend to rate images and reject ones I don't want. From time to time I create smart albums and filter using the rating to isolate the rejects, I have a last browse of these for any keepers but then delete the ones I'm sure I don't want from the library.

 

If I have a project I'm finished with, I'll export the whole project after copying my 4 or 5 star images into a master Best of project. I just select the project I'm finished with and export it to an external HDD or DVD, Then I can delete the project within the library. I can always re import this project from the external source at a later stage if I so wish while keeping my adjustments keywords and ratings.

 

I do use the Vault function as a master back up to an external FW drive and from time to time.

 

The key to this is to understand the flexability ratings, keywords and other image tags give you to hone down your search using smart albums. Keep copies of your picks, 5 stars in the library by all means and export the junk or even if you have lots of hard drive space just use stacks to hide and manage the clutter.

 

If you need any help with any aspect of this just let me know and I'll help if I can.

 

[pete_g]

The problems had by many with external drives are more often than not due to the circuitry involved with the USB or Firewire interface, known as the bridge chipset, rather than the hard disk itself. Most maufacturers of off the shelf external drives put in cheap and unreliable bridges and the drives will unmount, fail to mount or corrupt data, LaCie in this respect are no better than others, they just have sexier casings. If you can determine who manufactures the bridge on your intended drive purchase, you can improve the situation but most companies do not publish this. The bridges to steer clear of are made by Prolific and Cypress, they are pretty horrible, if you go with the Oxford chipset you will have a more reliable drive, but these are in the minority. It is better to have only a Firewire or a USB2 enclosure, the dual standard chipsets are the most unreliable, between USB2 and Firewire I have found firewire to be a little more reliable and it is a better protocol.

 

External drives designed specifically for video edting generally have the Oxford chipset, they are a little more expensive but it's peanuts compared to the price of Leica gear, G-Tech is one source.

 

The hard drives themselves are generally reliable, Seagate are recommended, although it should be noted that IBM themselves have said no consumer HD should be expected to work for more than 18 months.

 

It should be noted that if your drive with all your pix fails, it may only be the chipset not the disk that is dead, fitting the disk into a new enclosure would enable you to get your stuff back.

 

Backup to two separate HD's is the best plan, forget optical disks for archiving, they deteriorate in a few years.

 

All magnetic forms of storage are inherently unstable over the years, this is something media producers are ignoring at the moment, the more enlightened are considering copying there Hi Def movies back onto 35mm film, which is good for more than 50 years at least.

 

Ironic and amusing.

[pete_g]

The problems had by many with external drives are more often than not due to the circuitry involved with the USB or Firewire interface, known as the bridge chipset, rather than the hard disk itself. Most maufacturers of off the shelf external drives put in cheap and unreliable bridges and the drives will unmount, fail to mount or corrupt data, LaCie in this respect are no better than others, they just have sexier casings. If you can determine who manufactures the bridge on your intended drive purchase, you can improve the situation but most companies do not publish this. The bridges to steer clear of are made by Prolific and Cypress, they are pretty horrible, if you go with the Oxford chipset you will have a more reliable drive, but these are in the minority. It is better to have only a Firewire or a USB2 enclosure, the dual standard chipsets are the most unreliable, between USB2 and Firewire I have found firewire to be a little more reliable and it is a better protocol.

 

External drives designed specifically for video edting generally have the Oxford chipset, they are a little more expensive but it's peanuts compared to the price of Leica gear, G-Tech is one source.

 

The hard drives themselves are generally reliable, Seagate are recommended, although it should be noted that IBM themselves have said no consumer HD should be expected to work for more than 18 months.

 

It should be noted that if your drive with all your pix fails, it may only be the chipset not the disk that is dead, fitting the disk into a new enclosure would enable you to get your stuff back.

 

Backup to two separate HD's is the best plan, forget optical disks for archiving, they deteriorate in a few years.

 

All magnetic forms of storage are inherently unstable over the years, this is something media producers are ignoring at the moment, the more enlightened are considering copying there Hi Def movies back onto 35mm film, which is good for more than 50 years at least.

 

Ironic and amusing.

[ericperlberg]

Advertisement (gone after registration)

I've found a neat little tool WiebeTech Micro Storage Solutions - SATADock™ v4 - Drive Docking Station, SATA Hard Drive Dock, FireWire and USB 2.0 Hard Drive imaging which allows one to connect any 3.5" internal SATA hard drive (or IDE or 2.5" notebook drives depending on version) to your computer. For me, its going to be a quick and cheap way to back up my internal drives but if a bridge chipset dies in a LaCie or Maxtor or Seagate enclosure, take the drive out, hook it up to the SATA Dock and off you go! It's ugly but not expensive (relatively speaking).

[heninger]

External HD's are ok, but maintaining duplicate backups is a pain that never seems to work as smoothy as I would have liked.

 

I'd highly recommend a Drobo drive - plug in any size of SATA drives and it keeps everything redundant. It is expandable as drives get bigger/cheaper, and has none of the hassle of RAID arrays. Its really plug/play.

 

http://drobo.com/

 

Costs a bit up front, but you'll save money/headache in the long run.

[ericperlberg]

External HD's are ok, but maintaining duplicate backups is a pain that never seems to work as smoothy as I would have liked.

 

I'd highly recommend a Drobo drive - plug in any size of SATA drives and it keeps everything redundant. It is expandable as drives get bigger/cheaper, and has none of the hassle of RAID arrays. Its really plug/play.

 

Data Robotics, Inc.

 

Costs a bit up front, but you'll save money/headache in the long run.

 

The Drobo has been interesting to me since I heard about it a few months ago (not sold outside of North America yet). But I just looked at their product page which says:

 

Mix 'n match capacities, brands or speeds. No downtime, data migration, or waiting to access new capacity. Drobo works the way you do.

 

Which makes it sound as if you can just pick up any of a variety of internal drives and plug them in, anyway, that's what I've always assumed. But i had a look at their store and while they sell various drives of different brands and sizes, they do so at a huge markup on what are really commercial level drives. And ultimately there is very little choice.

 

I"m not sure if I'm reading this right but lots of companies like LaCie market interchangeable/hot swappable devices but you're forced to buy hard drives from them because that's the only way to get the special sled that the drives sit in needed to make the whole thing work. And unfortunately they (eg LaCie) charge like 3 times the price of the very best drives available but only use commercial grade drives leaving LaCie with a nice profit and the user being really ripped off for a bit of convenience. Do you know if this is the situation here? Because if it is, the initial price is cheap (it is anyway for a drive like this), they'll be getting you on the replacement drives.

[Guest guy_mancuso]

Looks like you buy the Drobo for 499.00 with no drives than you can add your own. You don't have to buy there drives, which is good because you can do much better . The same thing Apple does with the Mac pro they offer high prices on the drives that you can buy for 150 dollars less at a electronic store. When i bought my Mac pro i bout just 2 drives from Apple than bought 2 WD raptor 10 k's than Raid O them than put the OS on that and took the Apple purchased drives and used as backups. Looks like the Drobo will do the same thing except not sure if you need at least one drive with them with there software , have to figure that part out

[gdi]

Wow - at first I thought that Drobo was just what I needed - but no ethernet? What were they thinking?

[ericperlberg]

Looks like you buy the Drobo for 499.00 with no drives than you can add your own. You don't have to buy there drives, which is good because you can do much better . The same thing Apple does with the Mac pro they offer high prices on the drives that you can buy for 150 dollars less at a electronic store. When i bought my Mac pro i bout just 2 drives from Apple than bought 2 WD raptor 10 k's than Raid O them than put the OS on that and took the Apple purchased drives and used as backups. Looks like the Drobo will do the same thing except not sure if you need at least one drive with them with there software , have to figure that part out

 

That's good, I couldn't find anything definitive on the site. Their price for a Western Digital Cavier drive is $60 more than Newegg.

[ericperlberg]

Wow - at first I thought that Drobo was just what I needed - but no ethernet? What were they thinking?

 

It's also strange that its only USB 2.0. I've been moving large amounts of data around the past few days and when I had to use drives with firewire 400 or even worse USB 2.0, it took forever. I would have expected an e-sata connection or at least Firewire 800. No point getting drives with 16 or 32 mb caches and fast seek times, etc, if the data is moving in and out over USB 2.0. Oh well, there's always version 2.0 to look forward to.

[fotografr]

You can buy backup external hard drives until you're blue in the face. Or you can use an online storage like Carbonite, which has gotten consistently high reviews from everyone in the business.

 

Online Backup Service: Automatic Backups

[wlaidlaw]

You can buy backup external hard drives until you're blue in the face. Or you can use an online storage like Carbonite, which has gotten consistently high reviews from everyone in the business.

 

Online Backup Service: Automatic Backups

 

At British Telecom broadband speeds where my 8Mb/sec connection is sometimes running at a glacial bandwidth of 40Kb/sec due to the 50+ year old local pipeline from the exchange to our T3 roadside green box, that's just not an option.

 

Wilson

[ericperlberg]

You can buy backup external hard drives until you're blue in the face. Or you can use an online storage like Carbonite, which has gotten consistently high reviews from everyone in the business.

 

Online Backup Service: Automatic Backups

 

There's a review of online services published just recently here: Online backup solutions: a review: Page 1

 

These services are fine if you only have small amounts of data to back up.

 

Anyone with significant data to backup is not going to find web upload and download times even remotely acceptable unless you have massive bandwidth and the accompanying data costs. It depends on how much data you have to move across the internet. For me, Carbonite, Mozy, Iron Mountain or S3 are not feasable solutions. If they work for you, fantastic.

[ericperlberg]

At British Telecom broadband speeds where my 8Mb/sec connection is sometimes running at a glacial bandwidth of 40Kb/sec due to the 50+ year old local pipeline from the exchange to our T3 roadside green box, that's just not an option.

 

Wilson

 

And Wilson, that's not including the fact that your (and my) BT 8mb/sec connection is your download speed. Your best upload speed is 512k, at night, when England sleeps. At 512 kilobits (not kilobytes 8bits in a byte so divide by 8) per second uploading a few gigs of data (that's gigabytes not gigabits) would take, errr, a very long time! Or as you say, just not an option.

[kajuku]

External HD's are ok, but maintaining duplicate backups is a pain that never seems to work as smoothy as I would have liked.

 

I'd highly recommend a Drobo drive - plug in any size of SATA drives and it keeps everything redundant. It is expandable as drives get bigger/cheaper, and has none of the hassle of RAID arrays. Its really plug/play.

 

http://drobo.com/

 

Costs a bit up front, but you'll save money/headache in the long run.

 

Hello everyone-

 

This is my first post to this community, and I am sorry to say it is a tale of woe.

 

I am a drobo early adopter. I have had the drobo since June 9, and had been pretty happy with it. I set mine up with 4x 500GB drives. I was formerly using the acclaimed Seritek 4-drive external SATA RAID setup. This was much faster than the drobo, but I was weary of nightly rotating backups, worrying about drive integrity, moving weeklies offsite, etc. I was willing to sacrifice speed for safety and convenience.

 

It was an illusion.

 

My drobo failed catastrophically on Sunday 15 July. When I booted up, I got the "disk you inserted is not readable by this computer" message.

 

Disk Utility, under Mac OS X 10.4.10, recognized the drobo hardware as 'TRUSTED USB Mass Storage media, but the disk shows no volume name and cannot be mounted. Verifying the disk with DU reveals a major but not well described error that cannot be fixed. Disk Warrior reports 'no volume wrapper' and cannot rebuild or create a new directory.

 

The diagnostic report generated by the drobo indicates that a file contained in flash memory could not be read. I suspect but cannot prove that volume header information is stored in flash memory. (My understanding is that the drobo masquerades some of its volume information to overcome the USB limits on volume size.)

 

All the while, the drobo utility program is telling me that my 680GB of "data is protected." It still says that to this moment. All lights are green on the unit. Tragically, I can't get to my beautifully protected digital assets. It seems proprietary nature of the drobo system makes its ailments refractory to the tools with which one would normally handle problems such as this.

 

Data Robotics blames the OS. Their reasoning is that since the OS writes the volume header information, the drobo could not be responsible for the corruption. I can tell you from my phone calls and email exchanges with DR that they seem to be deeply in CYA mode.

 

I am angry and frustrated. In my final conversation with them, DR contended that my data was protected, so the drobo had lived up to its billing. All I had to do, it was suggested, was to use Disk Utility to rename the drobo. A more cynical person might wonder if this was some kind of trick: reinitializing the drive might absolve them of the responsibility for loss of data. I think they were also frustrated and not thinking clearly.

 

I let DR know that I was going to look into a cause of action for a liability claim. After all, drobo positions itself as a fail-safe system. I am not a litigious person, and I don't know yet if there is a basis for a suit, but I would love to have access to their engineer's notes on the review of my problem. At the very least, I would like others to know that the drobo can fail catastrophically and without warning.

 

Here's the bottom line: if you have a drobo, you have to back it up in accordance with your needs for asset protection. For me, that's every day that I work on my mac. In that case, there's no point in using the painfully slow drobo.

 

I'm going back to the big fast stripe and chronosyncing it nightly to 1TB WD FW externals.

 

Regards,

 

Jonathan Reiss

[Advertisement]

Hallo,

Look here: Leica M

[Sandokan]

As well as worrying about disc failures and CD/DVD longevity, have you also thought about the problems in 20 years time of not being able to attach the devices? Who has a 51/4 inch or 31/2 floppy?

 

In the days before "beamers" I remember the days when you could produce a Powerpoint (or Persuassion) slide show on a computer and then sent it away to be put onto slides...

 

Why not resurrect this technology and print your good images to film? After all, we know how long film will last and we know hot to view it.

[ericperlberg]

Hello everyone-

 

This is my first post to this community, and I am sorry to say it is a tale of woe. snip...

 

I am a drobo early adopter.

I'm going back to the big fast stripe and chronosyncing it nightly to 1TB WD FW externals.

 

Regards,

 

Jonathan Reiss

 

Jonathan, what a nightmare. It seems obvious to me now, reading your story, that DROBO's "fooling the OS" is a recipe for knock on problems once something goes wrong. Thanks for the warning.

 

Have you tried Prosoft's Data Rescue II or SubRosaSoft's File Recovery which go in at the file level rather than trying to fix the directory? You'll wind up with a bunch of unsorted files but one time for me it found and recovered files on a dead drive when DiskWarrior was unable to fix the drive's directory.