Archiving Digital Images

[ericperlberg]

Advertisement (gone after registration)

As well as worrying about disc failures and CD/DVD longevity, have you also thought about the problems in 20 years time of not being able to attach the devices? Who has a 51/4 inch or 31/2 floppy?

 

In the days before "beamers" I remember the days when you could produce a Powerpoint (or Persuassion) slide show on a computer and then sent it away to be put onto slides...

 

Why not resurrect this technology and print your good images to film? After all, we know how long film will last and we know hot to view it.

 

I think most people are not expecting their magnetic media to even last 5 years. It's an ugly situation but I view my current digital storage as an interim method pending advancements in digital archiving. It's not just photographers who face this problem. It's pretty much the whole of the developed world wherever digital data is stored from corporate records to the military to google, etc etc. I would be amazed if there's not some serious research being done because who ever comes up with long term stable archiving for digital works is going to make a serious amount of money.

 

I also did corporate presentations which were saved to slides many years ago. But powerpoint presentations and digital photographs are night and day in terms of image complexity. I haven't heard of anyone doing this. Perhaps it would be viable but I suspect that there would be a host of complicating factors not the least of which would be colour management issues. I can't even imagine how this process you're suggesting would be done. As I remember the powerpoint slide thing was done with a dense tiny screen in a darkened housing that was then photographed by a built in camera. With powerpoint slides nobody was too critical about the nature of the orange in a slice on a pie chart. But mass production of archiving would involve a lot of decision making IMO. It would be torturous I think. And of course film becomes more and more problematic the older it gets. Personally I'm not so bothered by waiting for a more stable technology but that's me...

[wlaidlaw]

I guessing that in 5 to 10 years time we will be using non-volatile, solid state, flash memory. It is a much better solution that having to have tiny disks whizzing round at thousands of RPM with read heads accurate to 10 microns. Given the unbelievably low cost of disks, the complexity and the very tight tolerances, I am surprised they don't fail more often.

 

Wilson

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[kajuku]

Have you tried Prosoft's Data Rescue II or SubRosaSoft's File Recovery which go in at the file level rather than trying to fix the directory?

 

Thanks for the advice. I have been working my way through every tool in the arsenal. (I used to do mac tech work so I have many.) Subrosasoft's FileSalvage couldn't do anything with the drobo. Data Rescue II is currently doing a block scan, but the projected time for completion was 54 hours. I am about a third of the way there.

 

I wonder how this compares to a RAID 5 or RAID 6 rebuild after drive loss.

 

~ Jonathan

[fourfa]

there are companies writing digital files to slides. Slides.com Order 35 mm Slides Online is one I'd bookmarked (4096 X 2730 fixed resolution and they claim a color-managed process) at $2.50 apiece. These companies mostly came up doing powerpoints and such but have branched out into photography as well. It's still probably the best option for a large projected slideshow presentation from high quality digital files, given the miserable state of digital projectors for stills (the most expensive digital projector B&H carries is 1600x1200, for $34,000)

 

As for backing up a whole digital archive...at $2.50 each I'd have to be in a much higher tax bracket to embrace that. besides, it isn't a backup until the image is recoverable, which means for most users the recovered file will be a scan of the slide. What is the quality of that third-generation copy compared to the original? For $2.50 it would be an interesting experiment.

[Shootist]

Thanks for the advice. I have been working my way through every tool in the arsenal. (I used to do mac tech work so I have many.) Subrosasoft's FileSalvage couldn't do anything with the drobo. Data Rescue II is currently doing a block scan, but the projected time for completion was 54 hours. I am about a third of the way there.

 

I wonder how this compares to a RAID 5 or RAID 6 rebuild after drive loss.

 

~ Jonathan

 

Have you tried disconnectiing the drive, after shutting down the system, then reboot and reconnect the drive?

 

Does the OS even see the drive attached? In all honesty I do think it is a OS problem more then the external units problem. Unless the drobo is dead.

I have heard about Mac's losing the connection with external drives before.

[pete_g]

Thanks for the advice. I have been working my way through every tool in the arsenal. (I used to do mac tech work so I have many.) Subrosasoft's FileSalvage couldn't do anything with the drobo. Data Rescue II is currently doing a block scan, but the projected time for completion was 54 hours. I am about a third of the way there.

 

I wonder how this compares to a RAID 5 or RAID 6 rebuild after drive loss.

 

~ Jonathan

 

I've had drives go on my 2TB RAID arrays (RAID 5) and they rebuild pretty quickly, although it's a few hours, but with RAID you just keep working as if nothing had happened, the hardware simply looks after itself. RAID (I'm talking hardware RAID arrays, not RAID in software) is not cheap though and is not necessarily what people want "at home". I still think a couple of external drives would suit most. As to the previous remark that it is difficult to keep track of the backups, any backup software will usually keep a couple of directorys on separate drives syncronised. I have used Syncback SE, which is freeware, and it's great.

 

I don't believe your problem with the Drobo is a fault of the OS either. I'm no great fan of Macs but BSD 4.xx is a damn good basis for any OS, it's highly unlikely that it's broken and more likely that the non standard device is at fault. My comiserations, we all end up in these situations from time to time.

[ericperlberg]

Advertisement (gone after registration)

there are companies writing digital files to slides. Slides.com Order 35 mm Slides Online is one I'd bookmarked (4096 X 2730 fixed resolution and they claim a color-managed process) at $2.50 apiece. These companies mostly came up doing powerpoints and such but have branched out into photography as well. It's still probably the best option for a large projected slideshow presentation from high quality digital files, given the miserable state of digital projectors for stills (the most expensive digital projector B&H carries is 1600x1200, for $34,000)

 

As for backing up a whole digital archive...at $2.50 each I'd have to be in a much higher tax bracket to embrace that. besides, it isn't a backup until the image is recoverable, which means for most users the recovered file will be a scan of the slide. What is the quality of that third-generation copy compared to the original? For $2.50 it would be an interesting experiment.

 

What you said interested me, not because I think archiving work on slides is a good idea but because I wanted to make sure that my hunch that it isn't a good idea was right. I read through Slides.com and the technology (very briefly mentioned) sounded the same as what I remember, a film shot of a high density CRT. Even if the process is colour managed, the CRT is only capable of, at best, adobe RGB colour gamut (and that's questionable), far smaller than what many digital cameras can produce in raw. So on that point alone printing to slides wouldn't be archiving.

 

But a review that I found kind of puts the question to rest I think ProPallette FilmRecTest

[ericperlberg]

I have heard about Mac's losing the connection with external drives before.

 

I haven't heard this for awhile, since the early OS X days. And the references were related to OS updates and a particular Oxford chipset and firmware. It wasn't the OS losing connection, it was a chip set firmware designed for earlier times that didn't get along with the OS. I remember LaCie issuing a patch to correct the problem (lots of manufacturers used the chipset, but as I had LaCie drives at the time, I know about the LaCie patch). Anyway, like Peter, I personally don't see any reason to suspect the OS based on what Jonathan reported.

[fourfa]

But a review that I found kind of puts the question to rest I think ProPallette FilmRecTest

 

hm? a pocket P&S 5mp camera with a tiny 1/2.5" sensor, uprezzed to 6mp, written to slide with a recorder capable of 6500x4880 (32mp) and compared to the same scene on Velvia (presumably shot with an SLR though it doesn't say). Both then scanned on a consumer desktop scanner. I don't see how you could conclude from that test that the slide writing process was the weak link, not the P&S. I'd say the slide scan looks pretty good compared to some of the 5mp P&S cameras I've used...

[Guest sirvine]

I'm a big fan of hardware RAID. In fact, from a total cost of ownership perspective, I think it's the only truly reliable method today (paired of course with a local and remote backup). My feeling is that you need at least four disks in a RAID 5 array to genuinely forget about it.

[ericperlberg]

hm? a pocket P&S 5mp camera with a tiny 1/2.5" sensor, uprezzed to 6mp, written to slide with a recorder capable of 6500x4880 (32mp) and compared to the same scene on Velvia (presumably shot with an SLR though it doesn't say). Both then scanned on a consumer desktop scanner. I don't see how you could conclude from that test that the slide writing process was the weak link, not the P&S. I'd say the slide scan looks pretty good compared to some of the 5mp P&S cameras I've used...

 

I accept your criticism, I overstated the case in terms of the article though I find it more compelling than you. In regards to any CRT based film recorder I still see severe colour gamut limitations, esp in saturated colours, gradient smoothness and dynamic range that would keep it from being considered an archive of a digital file. The Lightjet seems to use a laser technology different from the CRT technology used at Slides.com and I don't know what their colour gamut ability is though on Lightjets for photo printing you're stuck in sRGB... back to the point and shoots.

[fourfa]

browsing the first thirty results in google for "digital to slide," they're all using very similar machines - monochrome CRT, three color filters, 4096x2732 - all around the same cost of $2-2.50 per slide. I'm not sure what the color filter wheel and monochrome CRT implies about gamut, but judging by reviews it seems that the contrast range is less than ideal. Most seem to be shooting onto Kodak E100G, which has its own gamut and contrast that may or may not suit the subject. This must be a relatively inexpensive machine as I'm not finding any shops advertising a higher-res or more capable machine like the Lightjet. If someone came across such a shop, perhaps we'd find that more compelling.

 

some interesting reading: Making your Own Slides

[kajuku]

Have you tried disconnectiing the drive, after shutting down the system, then reboot and reconnect the drive?

 

Does the OS even see the drive attached? In all honesty I do think it is a OS problem more then the external units problem. Unless the drobo is dead.

I have heard about Mac's losing the connection with external drives before.

 

Hello-

 

Yeah, I did the reconnect/reboot thing many times. On different systems with different versions of OS X (10.4.8 to 10.4.10). No go.

 

The mac sees the drobo just fine. It just doesn't see the Volume it contains. The drobo dashboard (the software utility that supplements the diagnostic information provided by the lights on the unit itself) also sees the unit, says everything is healthy and protected, but calls the volume 'unlabelled,' which of course it is not. Or should I say was not.

 

Worst part is that Data Robotics is refusing to take any responsibility for this failure. I can understand why. There is no point to the drobo if it can fail suddenly, without warning, in ordinary use, presumably without hardware failure.

 

I say presumably because the obvious experiment is to determine whether the problem is on the data written to the drives, in the flash memory of the unit, in the bridgeboard, or controller. I suggested that DR send me a drobo to do this test. They would have none of that, contending that the diagnostic file produced by the drobo indicates everything is functioning as it should.

 

The logical error here is that the diagnostic tests are accounting for scenarios anticipated by the drobo engineers. The two days it took them to analyze the diagnostics, and then to suggest reinitializing the drobo, indicates they don't really have all the data they need. They have not asked me to send the unit to them for examination or testing either. I think they don't really want to know what happened.

 

This is strange to me. If it's a hardware failure, I'd hang with them. Any hardware can fail sporadically. We just fix or replace the broken component and move on. As long as the data is protected and remains accessible, what more could you ask from an electromechanical storage device? On the other hand, I have little tolerance for a system that is based on a flawed design or operating principle, especially one which causes (or allows) it to fail unrecoverably in normal use.

 

I believe, but cannot prove, that DR is making a calculated decision to protect their market position. If failures are rare, potential buyers may be more likely to give the benefit of the doubt to the drobo.

 

Last night I found a blog post, dated 27 June, from photographer David Plummer who had the same failure to read the volume on reboot. His unit was one of the first 500. If I had encountered his post in May, I may have shrugged it off, but I would probably have adopted a wait and see posture. Had I seen two failures in less than two months, I'd have stayed away.

 

~ Jonathan

[dlc43]

Hardware aside, let's get back to the main question: the management of huge numbers of digital images.

 

I'm surprised that no one has yet mentioned Peter Krogh and his book The DAM Book. (DAM=Digital Asset Management). Here's a link:

http://www.iview-multimedia.com/dam/whatisdam.php

 

While his solutions may be complex, his analysis of the problem is cogent. I recommend reading his book just to stretch your brain. He also offers courses. His preferred software is iView, now renamed Expression Media after it was purchased by (yes) Microsoft. At least one can expect that if Microsoft invested in it, they will probably continue to refine it and will go all out to support it .

 

How to manage huge numbers of photos makes for a really hard choice at this stage of things: iView (Expression Media) or Aperature or Lightroom or what?? The learning curve (then integrating any of these into your workflow), is so steep that dollar cost by comparison is trivial. Then once committed, are you stuck? Which of these avenues will survive the test of time? I am not sure.

 

Krogh examines the issues, raises good questions, and also explains different hardware strategies.

[pete_g]

While his solutions may be complex, his analysis of the problem is cogent. I recommend reading his book just to stretch your brain. He also offers courses. His preferred software is iView, now renamed Expression Media after it was purchased by (yes) Microsoft. At least one can expect that if Microsoft invested in it, they will probably continue to refine it and will go all out to support it .

 

How to manage huge numbers of photos makes for a really hard choice at this stage of things: iView (Expression Media) or Aperature or Lightroom or what?? The learning curve (then integrating any of these into your workflow), is so steep that dollar cost by comparison is trivial. Then once committed, are you stuck? Which of these avenues will survive the test of time? I am not sure.

 

Thanks for this info as I'm in the market for some kind of asset management software, but I don't think it will be Expression. I checked the price for the UK and it's GBP 254 whereas the price in the US is USD299. I can't abide this kind of piracy and will have to steer clear of it, so I guess it's probably Lightroom although I don't really want all the image editing stuff as I'm happy with PShop. I'm on a PC so Mac sw is not an option.

[ericperlberg]

Adobe Forums - How to organize Folders/Collections: best practices ?

[vanhulsenbeek]

His preferred software is iView, now renamed Expression Media after it was purchased by (yes) Microsoft. At least one can expect that if Microsoft invested in it, they will probably continue to refine it and will go all out to support it.

 

 

Well, hmmm....

 

I was an Iview afficionado, and even had the Iview people contemplating including DNG M8 files, but then, after Microsoft budged in....... :-(

 

Noting happened for 6 months. The finaly released version was buggy to a hame, and still no Leica M8 DNG. No program support, except for a rather dodgy Forum, where - seemingly Microsoft guys, asked me to give Microsoft a break. And wait. They had already done so much work ....

 

I bought Lightroom the next day and haven't looked back since.