Long term archival image data storage?

[wlaidlaw]

Advertisement (gone after registration)

A museum for whom I am going to be doing some archival imaging in November, has asked me to write them a letter with suggestions and commentary on long term image data storage. My normal practice when I do these jobs, is to give the museum 2 copies of the images, one on Mitsui Gold Archival DVD-R (or +R if requested) discs and a second copy on Kingston 4GB SDHC cards. Each SD card will match each DVD. I only ever write 4GB to the DVD's even though in theory, they can hold 4.7GB. I have looked at the SanDisk WORM SD cards (write once read many times), which are claimed to have a 100 year life but they are expensive and at the moment, only 1GB.

 

I then normally tell the museum or other institution that what they do with the data and how they store it, is totally their responsibility but suggest that they are stored in static resistant bags and then sealed into air tight containers, which if possible, should be flushed with nitrogen or even better, inert gas. I also suggest that these are kept off-site for obvious reasons (I suspect they are often just put in the basement) and at least one digital copy stored with a reputable (i.e. expensive) off-site data storage company.

 

Notwithstanding these precautions, I am very dubious if much if any data will be recoverable from either of the 2 media I supply, after 50 years or maybe considerably less. I am wondering if I should suggest that I revert to silver halide film, which is OK for black and white images but colour seems to be much less stable. Looking at my older slides, Ektachrome has faded very badly, Perutz has gone greenish and Agfa has gone brownish. The only slides which have lasted very well are Kodachrome but obviously (and sadly), that is no longer an option. I am also not sure if these museums, who are now very short of money, would be prepared to pay the considerably higher cost involved in using film. When I quote them, I very often get the answer that it would have to be done from next year's budget and do they mind if I don't get paid for 6 or 9 months. No pay = no data is my answer.

 

Any suggestions and commentary from members would be very welcome, as I have to write this letter within a couple of weeks.

 

Wilson

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[sandymc]

Well, I'd be inclined to tell them that it isn't about what the image is on - even film is subject to deterioration, as you point out. It's about having a process for preservation. Emphasis on process.

 

This is especially the case for digital media - not only does the media itself fail, but the media readers become unavailable, and decoders for the file content become unavailable.

 

So any organization that's serious about keeping digital artifacts in the form of files or whatever needs to stop thinking about what they are stored on, and start thinking about what process they have in place to preserve the actual data itself. So of course they need to store stuff on two (at least) different forms of media, but every year (or whatever), they need to go through all their digital assets, and check that they are still readable, and will be for the next several years. AKA, do a risk analysis. If there are signs that the media is failing, or reader's hardware is becoming obsolete, or the reader software needs an OS that's no longer supported, or whatever, they need to migrate the asset.

 

BTW, as the author of the only current software that properly decodes Kodak Photo CD files, I see this play out often. Lots of museums, collections, etc digitized images onto Photo CD, based on its "archival" properties back in the early 90's. Now they want to retrieve the images, and they find that nobody even remembers what they originally used to read the CDs. And if they're unlucky, they find the CD's are corrupted.

 

Sandy

[wlaidlaw]

Thanks for the info Sandy. The problem is that the sort of museums I deal with are usually small, poor and with very limited knowledge of IT. The procedures you outline make total sense but with the sort of institutions I deal with, are just never going to happen. At the moment, I give them the images as TIFF files but as you say, how much longer is this format going to be available. In that it is in effect just a coded bitmap, I hope that it may be readable for say at least 20 to 40 years.

 

The main purposes of the archival images for smaller museums is firstly to have a visual inventory for insurance purposes and secondly to have a decent record in the event of damage, deterioration or destruction of any item and lastly to have as a digital resource on websites, as and when they can afford the bandwidth to offer this and someone to write and manage such a website. There were government grants for this but these have now virtually disappeared.

 

I know what you mean about the Kodak Photo CD format. I had to fire up an old 1997 vintage Sony laptop running Windows 98, on which there was a copy of the Kodak software and exported the files as GiF's but they were poor. I have recently been using ImageMagick and have now converted all my old Kodak files (not very many given the price of even a 16 Base CD in the 90's).

 

Wilson

[Jeff S]

Well, for a reasonable fee you might offer them a renewable extended service whereby you agree to check the files after some stated period, say 3 years. At the end of the time period, they could renew for another term, or not, or choose to seek other help. Of course the wording would need to protect you in the event of loss.

 

Jeff

[sandymc]

I know what you mean about the Kodak Photo CD format. I had to fire up an old 1997 vintage Sony laptop running Windows 98, on which there was a copy of the Kodak software and exported the files as GiF's but they were poor. I have recently been using ImageMagick and have now converted all my old Kodak files (not very many given the price of even a 16 Base CD in the 90's).

 

Wilson

 

You want to try pcdMagic. Even the unlicensed trial version will do a better job than ImageMagick

 

pcdmagicsite"]pcdMagic for the Mac

pcdMagic for Windows

 

Regards,

 

Sandy

[giordano]

Wilson,

 

Give them a copy of Caring for CDs and DVDs and make sure they are aware of the British Library Preservation Advisory Centre, which is a national resource for the preservation of library and archive collections.

 

For digital data, planned migration is essential and should be costed into the project. That minimises the problem of media going obsolete.

 

Avoid lossy compression if at all possible. I reckon that the main flavours of TIFF are going to be round for a long time, and that if another "standard" takes over there'll be plenty of time for a lossless migration.

 

For film, I get the impression that modern E6 on a polyester, not acetate, base does pretty well if kept cold in the dark, but that if you really want archival colour you still can't beat colour separations. But will there be suitable scanners around when someone needs to digitise a 35mm film image in 50 years time?

[wlaidlaw]

Advertisement (gone after registration)

Well, for a reasonable fee you might offer them a renewable extended service whereby you agree to check the files after some stated period, say 3 years. At the end of the time period, they could renew for another term, or not, or choose to seek other help. Of course the wording would need to protect you in the event of loss.

 

Jeff

 

Jeff,

 

Given my age and state of health, I will probably peg out before the DVD's do ;-}}

 

All the museums seem to have got hold of the same agreement for me to sign, before I do the archive imaging. It says that I must not keep copies of the images and must confirm after the museum has accepted the images that I have deleted all copies. It is all written in the finest "civil-servant-ese" with articles and sub articles referring to sub-clause 17.2.5.6 etc, etc, so I am guessing it was written for them by some government department.

 

The other issue is that I am only in the UK for about 4 months every year and could not provide the sort of service that I would want to buy. At the moment I point all clients in the direction of IBM-Tivoli Storage Manager. It is not cheap but I think is the most flexible and secure solution for smaller users. One of the larger museums I did some work for about 18 months ago, is using it after I suggested and are so delighted with the service, that they are moving all their data archives to it. I should arrange to get commission from IBM!

 

Wilson

[wlaidlaw]

You want to try pcdMagic. Even the unlicensed trial version will do a better job than ImageMagick

 

pcdmagicsite"]pcdMagic for the Mac

pcdMagic for Windows

 

Regards,

 

Sandy

 

Sandy,

 

On most of the films, now that I have a decent Epson V700 scanner, I am going back to the original films, where I think the image is worth keeping and rescanning. Scanning has moved on a fair bit since 1993/4. If I had got 64 Base scans, it would be more worthwhile converting but I only have 16 base scans.

 

Wilson

[KM-25]

Good topic, one that we can all relate too on many levels. I would try contacting a larger institution like the Library of Congress or the National Geographic Society to see what they are doing, I am sure it is large budget stuff, but it might give you ideas on what can be done in optimum settings.

[sandymc]

Sandy,

 

On most of the films, now that I have a decent Epson V700 scanner, I am going back to the original films, where I think the image is worth keeping and rescanning. Scanning has moved on a fair bit since 1993/4. If I had got 64 Base scans, it would be more worthwhile converting but I only have 16 base scans.

 

Wilson

 

If the original films are in good condition, that's probably a good idea. Photo CD had good pixel resolution, but the dMax rating, dynamic range basically, wasn't anywhere near what a scanner today would deliver.

 

Regards,

 

Sandy

[Jeff S]

 

Given my age and state of health, I will probably peg out before the DVD's do ;-}}

 

All the museums seem to have got hold of the same agreement for me to sign, before I do the archive imaging. It says that I must not keep copies of the images and must confirm after the museum has accepted the images that I have deleted all copies.

 

I can certainly relate to the age issue.

 

My thought on the servicing after the fact was not to suggest that you keep copies; rather that the museum bring you back in to inspect (and update if necessary) theirs.

 

But, probably not a great solution in any event.

 

Jeff

[giordano]

Maybe it's time to go back to carving our data on stone.

[wlaidlaw]

Maybe it's time to go back to carving our data on stone.

 

I know technically the M disc is a bit silly but it might be a good selling point for me. Most of my jobs go on one or two DVD's and it would give the curators bragging rights. "our data is on discs that last 1000 years."

 

A friend of mine, who was one of the people amongst the earliest designers of integrated circuits, at Ferranti Research in Edinburgh, had this theory. He thought that ghosts might just be replays direct to the human brain, of extreme events recorded in the silicon content of stones of buildings. This is why they are often seen, when old buildings are being rebuilt or rewired. Well maybe. I personally always thought the best way to see a ghost was through the bottom of an empty whisky bottle.

 

Wilson

[adan]

Realistically, the best advice has already been given. Archival storage is a process and a strategy - not any particular medium.

 

Back about 2004 when my digital images began to really pile up, I asked the newsroom Technology Editor (i.e. the woman charged with building and maintaining the newspaper's 100s of thousands of image files in perpetuity - 3 terabytes at the time) what the best approach was.

 

She said (in paraphrase) "Have 3 copies of everything, and assume that no matter what medium you use, you will need to migrate it to new disks every 5 years or so. No only will ANY medium fail eventually, but the connectivity will change fairly fast, so today's USB or Firewire 400 drive will be a paperweight within a decade. Given that - compare copying 500 gigabytes of images on DVDs to new DVDs (a week-long project - times 3) to copying them from hard-drive to hard-drive (3-6 hours).

 

Her summation of DVD/CD ROMs - "They are equivalent to Post-It notes - great for passing information around, lousy for keeping anything important safe." That included the "100-year" Gold CDs.

 

Shortly thereafter, in migrating my earliest image files from CD to hard disk, I found out that some of my CDs had corrupted or totally unreadable pictures only 3 years after I had burned them - fortunately, those were film scans (thus backed up by film originals) and not digital originals.

 

Conversely, I had a total, fatal, unrecoverable internal hard-drive crash 18 months ago - it took only 4 hours to restore my 300+ gigabytes of images flawlessly from my external hard-drive copies to my new internal disk.

[wlaidlaw]

Andy,

 

The connectivity is a very good point - thank you. Who has SCSI these days, which was the professional standard for high speed data transfer just 10 to 15 years ago. When I wanted to use my Polaroid scanner on my Powermac, I could not find an internal Mac PCI card which would work and had to use a horrible, homemade looking external Firewire to SCSI dongle with separate power supply and I am sure it was this that made my Polaroid go on fire. Not funny if you had the same thing happen on vital data on an external hard drive.

 

I will therefore, emphasize the process in my letter and also my current favourite solution of using a reputable external agency as a back up to the internal process. If you use a big enough agency e.g. IBM, even if they go bust, someone will pick up the business as a whole or in parts. Their Tivoli system is simple and very easy to use. OK it is not particularly cheap but nothing good ever is cheap.

 

Wilson

[giordano]

I know technically the M disc is a bit silly but it might be a good selling point for me. Most of my jobs go on one or two DVD's and it would give the curators bragging rights. "our data is on discs that last 1000 years."

 

So long as they don't actually believe it! Also, the M disc might survive longer in a "fireproof" safe than a conventional one.

 

I've kept backups on external hard drives for a few years now. Just the other day, however, I tried a couple of 10 year old CD-Rs, one Kodak Gold and one name-brand non-gold. They'd been stored in Microsoft-supplied plastic CD sleeves in a Microsoft binder with no temperature or humidity control, and were still in working order. But I'm not counting on them!

[roguewave]

I use a Drobo & also have mirrors on 2 of the largest Web hosting companies. Cloud storage is VERY cheap & with redundancy on different platforms, I haven't had any issues. As an example: Network Solutions has an unlimited package for less than $40/month. When I say unlimited, I have over 5 terabytes of assorted data on one of my accounts. At home, I like the small Drobo which gives me about 7-8 TB's on a redundant array. That means even if 2 of the 5 drives fail, my files are still there. REDUNDANCY - that's key for me. I'm sure we will see SSD in similar formats at a fraction of the price soon.

[nick.edwards1]

Film has to be a serious contender. I know of one organisation who is using film to archive digital originals.

Using the 3 copies of everything argument - you could always keep 1 film and 2 digital copies.

I am sure the film manufacturers will provide the optimum storage conditions to best retain colours. Even so, I have been amazed how old colour film colours can be restored by scanning.

In terms of cost - it cannot be that cost prohibitive if you start with a film image and generate good quality scans at the time of processing as oppose to starting digitally and then producing a film copy.

 

Best regards,

 

Nick

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[Jase]

As museums are very used to the storage of paper under optimum conditions, why not archive on paper?

 

Really though as has been said before, it's the retrieval and re-use that is important and so it is a strategy. Even if you use cloud storage for electronic files which is regularly backed up and eliminates the need to worry about the hardware being used you still need to ensure you stay on top of the format itself. In 100 years time you may find that it is difficult to read the DNG/JPG/TIFF on software that runs on the computers that are in use at that time.

 

The time capsules created in the 80s on laserdisc are now effectively frisbees.....

[wlaidlaw]

Film has to be a serious contender. I know of one organisation who is using film to archive digital originals.

Using the 3 copies of everything argument - you could always keep 1 film and 2 digital copies.

I am sure the film manufacturers will provide the optimum storage conditions to best retain colours. Even so, I have been amazed how old colour film colours can be restored by scanning.

In terms of cost - it cannot be that cost prohibitive if you start with a film image and generate good quality scans at the time of processing as oppose to starting digitally and then producing a film copy.

 

Best regards,

 

Nick

 

Nick,

 

I think for the sort of museums I do work for, it would be prohibitively expensive. For a start, I would have to re-aquire an MF or LF camera outfit. as the only MF I have at present is a Rolleiflex TLR.

 

For the MF I would need a full macro set up with bellows, adapters and macro/shift lenses as well as standard and wide angle lenses. Say I went for a used Contax 645 or Hassy 503 outfit, I would not get much change out of £2,000 for such a kit. OK it would be cheaper if I went with Mamiya or Bronica but I have had bad reliability/body alignment issues with both those in the past, so would not want to go that route. I would have to look to recover that capital outlay in my costings over say 10 to 15 jobs.

 

Also if say I take 200 images in a day, the professionally processed and high rez scans cost of that in 120 film adds at least £480 to the cost (16 films at £30 each and that is probably too low per film).

 

These two factors would be close to doubling what I currently charge and I just don't think I would get the business. The larger museums which could afford it, generally have their own very professional archiving departments. in fact they are my main competition, as they usually don't have enough to do and their management are looking to make them into a cash cow.

 

These museums are very strapped for cash. I was doing a small job last year, copying a huge Victorian pressed flower book that the museum had on loan, to make 36" x 24" wall posters. One of the curators took it upon herself to bring a party of school children into my work area, which was supposed to be off limits. My V2 Elmar 65 Macro lens was pocketed by one of the children. I can no longer get insurance for my equipment. as I split my time 50/50 UK/France and two separate policies would be over £1,000. The museum refused to call the police to search the kids, saying they would sort it out. I still have not received payment for the lens and sadly have not been able to find another V2 65 Elmar at a reasonable price since, so I am currently using the inferior V1 65 Elmar.

 

Wilson