What is your long term backup strategy?

[dpitt]

Advertisement (gone after registration)

@Jaap @Jeff S As promised:

As a software engineer I have been using computers as long as PC's are around. My first one was a PC/XT with two floppy drives, 360K each.
Over the years I have been using Windows and Mac daily for work as well as privately. My first steps in digital images were scans in the mid 90's and the Nikon Coolpix 995 in 2001.
I am proud to say that I did not loose more than a few days of work over 40 years. Making backups was essential from day one. Now, it is one thing to make it, I found out that restoring the backup, even if you have made a backup it in time is not always easy:

1. It can get physically lost, misplacing, crashing drives, defect CD/DVD, fire, theft... it all happened
2. The hardware will get obsolete. I still have floppies, ZIP disks, JAZ disks, IDE drives around, but it gets hard to find hardware that can still read them.
3. If you manage to read the files, or copied them over to modern hard drives like I did, you will find that the file formats can not be read by current software. Try opening a Filemaker 1.0 database, or a Word Perfect 1.0 text, with todays software. The only way could be to run the old software on a museum PC or Mac running a contemporary OS.
4. The same goes for my PP software. My iPhoto archives probably can not be upgraded anymore to current Apple Fotos. With some tricks and effort I managed to keep Aperture 3.6 running on my Mac OS 10.14, even if it would not install on higher than Mac OS 10.11 Even then, when I opened the old 3.1 catalogs it said on all RAWs that the RAW engine was updated, so RAW processing needed to be redone. And worse... All my X2 files appeared as black rectangles (unsupported camera!) all of a sudden. I had to wait and upgrade to Mac OS for 10.14.6 to fix this.
   I considered myself lucky that I switched to Lightroom 5.x the moment Apple dropped Aperture for any new content.
   If they do this with a 'recent' model like the X2, what will happen to my 2004 DMR files with next Mac OS system updates?
5. Last question is solved the easy way. Neither Aperture or Lightroom will run on my new Macbook Pro Mac OS Monterey OS 12.6! Again I need to make a decision, and I do not like the Adobe subscription system. I am fed up with this constant forced upgrading. The Mac OS forces you to upgrade because of your iPhone or because you buy a newer Mac and then the rest stops working. I now have 20 years of catalogs to upgrade/move, and all options use incompatible RAW engines that can not import crops correctly let alone sharpening, de-noise, exposure... Even the ratings and keywords are often an issue. I will take some time before I put my RAW eggs in an other basket.

So, whatever I migrate next to (Capture One, Darktable, On1...) I will never trust it for the long run.
I am lucky, I still have an old mac mini running Mac OS 10.14.6 to open my Aperture and Lightroom catalogs.
That is why I will start by systematically exporting all my processed keepers as JPGs so that I can at least save the effort of 'developing' the RAWs in my current PP system. Any upgrade and certainly every move to an other brand will throw this processing effort away. Technology got me in the end... I lost days, weeks of processing work because of forced upgrades. Or not exactly, the effort got stuck in the past.
The JPG archive can also serve as off-site backup or cloud backup as last resort.

For the time being I use Darktable on my Macbook Pro for any new content.

And yes, there is a silver lining. I can probably get a better result from my old RAWs (as long as they support my older camera's) with newer PP tools, but I would prefer to do this for maybe for 1 in 100 files instead of doing them all.

Any comments, similar experiences, advice are welcome...

Edited December 24, 2022 by dpitt

[hepcat]

Although I emphasize that I am NOT a software, network OR hardware engineer, I pre-date you.  I used TRSDOS and CP/M long before DOS 1.0 and later Windows 1.0,  and the 8086-series chips came about.  I used luggables from Kaypro and Corona.  I built 8086 and 80286 machines as a side gig.  My first "work" machine purchased by my employer was an $18,000 PS2 Mod 50 with a 20mb hard drive and 20mb tape streaming backup.  I used Netware 3 and 4 predating Windows networking.  Although those were also the days before digital photography,  I still have years of outdated custom work on dBase II, III and III+ and Lotus spreadsheets backed up on 3.5" floppies in what are now probably unreadable files on an unreadable storage medium.  I haven't seen a 3.5" floppy drive in many years.

I am currently on a 64GB I7 Mac Mini with a 6 TB raid 1 mirrored array for my photo and file storage.  I used Lightroom 5 and PS 6 until they no longer upgraded Camera RAW and my latest cameras couldn't be read.  I switched to Capture One and Affinity.   I have upgraded and copied from hard drive to hard drive over the years, and managed to stay a generation or two behind the bleeding edge, but far enough ahead that I haven't lost anything.   I also have about 35k .jpgs warehoused on Flickr.   I began transitioning to digital in about 1999.  So far, all of my RAW files can still be opened in Capture One and Affinity Photo, so I haven't bumped up against loss because of file format.  Like you, I save the highest quality  processed .jpgs as archival work.  I think that's about the best we can do until some better, "universal" RAW format comes to the photo world.

Edited December 24, 2022 by hepcat

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[LocalHero1953]

What I would like to have as a long term back up system is not what my actual practice is. IMO the best back up of my images/photos is as prints, not digital files. The content of my images will be far more important in 10-20(+) years time than whether I can marginally improve how they look by future software. Thus I decided some time ago to create photo books of everything I ought or wanted to keep. I scanned all family photos (back to 1860) and created several photo books for different periods. These have been distributed as real copies and PDFs to our offspring.

The problem has come once I reached the digital era: the quantity is hard to deal with. Even though common sense tells me there is a lot of repetition and redundancy, it still takes a lot of effort to decide what should be preserved and what should take its chance as a digital file. I have recently concluded that I should just produce 'annals' - photo books by the year, which does limit the number of photos to triage at a time, even though the demand for shelf space increases. Decision made - now I need to implement it!

Edited December 24, 2022 by LocalHero1953

[dpitt]

1 hour ago, hepcat said:

 So far, all of my RAW files can still be opened in Capture One and Affinity Photo, so I haven't bumped up against loss because of file format.  Like you, I save the highest quality  processed .jpgs as archival work.  I think that's about the best we can do until some better, "universal" RAW format comes to the photo world.

All of my RAW's can still be opened in any of today's software, but I do not get the original changes with it and also not the ratings and keyword system behind them without manual effort. Unlike you I did not have JPGs from all my old work and just relied on exporting it to JPG from the catalog when I would need them. That is why I now have to try and export them from the old catalogs first, which in some cases means processing them again.Yes, from now on, JPGs  on google photo's is my last resort backup.

52 minutes ago, LocalHero1953 said:

What I would like to have as a long term back up system is not what my actual practice is. IMO the best back up of my images/photos is as prints, not digital files. The content of my images will be far more important in 10-20(+) years time than whether I can marginally improve how they look by future software. Thus I decided some time ago to create photo books of everything I ought or wanted to keep. I scanned all family photos (back to 1860) and created several photo books for different periods. These have been distributed as real copies and PDFs to our offspring.

The problem has come once I reached the digital era: the quantity is hard to deal with. Even though common sense tells me there is a lot of repetition and redundancy, it still takes a lot of effort to decide what should be preserved and what should take its chance as a digital file. I have recently concluded that I should just produce 'annals' - photo books by the year, which does limit the number of photos to triage at a time, even though the demand for shelf space increases. Decision made - now I need to implement it!

For myself I was always pleased with the results on screen and occasionally a small percentage ended up in a book or as poster on the wall. Indeed, I should make more books, but these take even more time and the results sometimes make you want to print yourself, taking even more time...
Great for you and your offspring, apart from fire hazard, they will probably survive your digital files.

[Jeff S]

This topic has been covered many times.  In short, I make prints of all my worthy photos; that’s all that any interested person will ever likely access.  For digital files, I backup everything to multiple disks, at least one of which is stored offsite.  Lightroom has been my software for file management since going digital.  The latest LR Classic (not cloud storage) is a no-brainer for me.  Adobe’s Photographer Plan includes LR Classic and Photoshop for a mere 10 bucks a month, less than I used to spend on upgrades, and far less than many of my former darkroom expenses.  And LR Classic has significantly better features and controls… especially for local adjustments… than prior iterations.  If Adobe ever mandates cloud storage, however, I will probably consider migrating to other software; but not before I’ve thoroughly investigated and assessed all the implications, including file/editing history.  
 

Jeff

[ianforber]

Like others, I output the photographs I like as jpegs (occasionally TIFFs as well if I think I’ll end up ordering a large print), usually into Apple Photos so that they automatically back up. Unlike many, I don’t care that much about keeping the RAWs. I’ve used Photoshop Express, Aperture, Lightroom and now Capture One and lost plenty of RAW over the years. No big deal. Yes, some of the jpegs I would process differently now and I’m sure modern RAW processing would get me a better result but life’s to short to fret about it.

More broadly, my Mac is backed up using Time Machine on a NAS drive and I have 2 additional backups of my photo library on separate hard disks. 

[Ray Vonn]

Advertisement (gone after registration)

As a Mac user, this works for me:

1) Automatically set it and forget it backup to Amazon AWS through the Arq software client.  This happens hourly.

2) Backup to external hard drive.  My discipline here has dropped somewhat in this regard but I'm covered by 1).  The software program Chronosync has proven itself to be reliable in terms of a local backup client in terms of cloning the HD and file backup.

3) Retire aged hard drives.  Only use up to 70% of their capacity.

[M11 for me]

vor 1 Stunde schrieb Ray Vonn:

Retire aged hard drives

What does this mean in practice?

vor 1 Stunde schrieb Ray Vonn:

Only use up to 70% of their capacity

Why do you suggest that. Not that I fill my 8TB disc more than that. But just curious.

[Ray Vonn]

56 minutes ago, M11 for me said:

What does this mean in practice?

It means an old HD will eventually die in time as all machines do, so keep replacing them every few years or so.

 

57 minutes ago, M11 for me said:

Why do you suggest that. Not that I fill my 8TB disc more than that. But just curious.

I remember reading many years ago that it's a very wise thing not to go over 70% storage capacity.  Why going over that could cause wear and tear or damage a hard drive, I don't honestly know.

[evikne]

On 12/24/2022 at 12:25 PM, dpitt said:

Any comments, similar experiences, advice are welcome...

I have a similar experience. When Apple dropped further support for Aperture, I migrated to Lightroom. But first I exported all Aperture photos to high quality JPEGs and RAW files. When I imported them to LR, I stacked all corresponding JPEGs and RAW files to keep them together. I also tried Capture One for a short time, and did the same export thing there.

Since then, I have sometimes opened old RAW files in LR and tried to make them as similar to the original JPEGs as possible (or, preferably, even better). This has proven to be very difficult. In most cases I prefer the Aperture edit, so I really miss that application.

Especially the white balance has been difficult to copy. I often use a WhiBal or ExpoDisc white balance card/filter to achieve a perfect WB. But even if I have the WB values from the original editor, it doesn't look similar in another, because they interpret it differently. So one thing I have learned, is to never throw away the original WB reference shots. If I keep them, I can always get the right WB with a new editor.

Edited December 25, 2022 by evikne

[dpitt]

On 12/25/2022 at 9:03 PM, Ray Vonn said:

I remember reading many years ago that it's a very wise thing not to go over 70% storage capacity.  Why going over that could cause wear and tear or damage a hard drive, I don't honestly know.

Only reason for 70% I can think of has to do with the speed of th OS, to leave it plenty of headroom for caches, memory swapping and such. Currently on Mac OS leaving about the amounnt of RAM memory installed free on the disk makes sense.
This does not apply to Data disks, 100% full should not damage them but if you use them for prolonged periods this full they will wear down faster because the files get fragmented more and so the heads need to move more for reading and writing your files. An occasional read/write like with backups should not matter AFAIK

On 12/25/2022 at 5:50 PM, Ray Vonn said:

 Retire aged hard drives.  Only use up to 70% of their capacity.

That is easy, hard drives keep getting cheaper and bigger. Every year I back up all my files on a bigger drive, usually one I retired from my active system.
But that does not solve the issue I am having now, see below.

On 12/25/2022 at 9:41 PM, evikne said:

Since then, I have sometimes opened old RAW files in LR and tried to make them as similar to the original JPEGs as possible (or, preferably, even better). This has proven to be very difficult. In most cases I prefer the Aperture edit, so I really miss that application.

Especially the white balance has been difficult to copy. I often use a WhiBal or ExpoDisc white balance card/filter to achieve a perfect WB. But even if I have the WB values from the original editor, it doesn't look similar in another, because they interpret it differently. So one thing I have learned, is to never throw away the original WB reference shots. If I keep them, I can always get the right WB with a new editor.

If I upgrade to the current Mac OS, my Aperture catalogs will only be readable by Apple Fotos and a lot of processing and meta data get lost, so I would never get the same foto output or searchability in Fotos, or any of the alternative Raw processors without considerable manual input. And even then the result might be better or worse, but different  from the original.

I also miss the simpeler Aperture UI, compared to Capture One and LR. All in al Apple did what it could do by upgrading Aperture for compatibility for 5 years, and even merging the file structure for catalogs of Fotos and the old Aperture app. Makes you wonder what will happen when Adobe (LR) or Phase One (Capture One) stops development or goes out of business. 😱

[Ray Vonn]

19 minutes ago, dpitt said:

But that does not solve the issue I am having now, see below.

Don't quite follow that, sorry.

[dpitt]

9 hours ago, Ray Vonn said:

Don't quite follow that, sorry.

I mean that, even if my catalogs are preserved perfectly. If the software to open them will not run anymore on any of my systems, I have lost the data anyway. There will not always be an upgrade path to current software without any quality or data loss...

[Alberti]

On 12/27/2022 at 9:20 AM, dpitt said:

Only reason for 70% I can think of has to do with the speed of th OS,

From my time working with IBM minis the 70% was there as a limit because of the extensive indexing it used. 
In my opinion, just backing up files can go to 100 - say 95%. You just read it once in a while.

[Alberti]

ah yes to add to the problems of the OP - I note that some files as read by LRc are signalled as corrupted. While Finder sees them DNGs and can read them without a problem.

I moved files around bit. Now on 2TB SSD and now I have 50% space left for a next two years I think. Growth is crazy. 

 

[dpitt]

1 hour ago, Alberti said:

ah yes to add to the problems of the OP - I note that some files as read by LRc are signalled as corrupted. While Finder sees them DNGs and can read them without a problem.

I moved files around bit. Now on 2TB SSD and now I have 50% space left for a next two years I think. Growth is crazy. 

 

Oh bummer! We call this software rot 😀.
So you increased your working space to 2 TB SSD?
Not sure what I would trust more as backup device. Hard disks that are not connected all the time and start up only when you need to read them have not failed me for the last 25 years. Much better than CD's or DVD's and a lot better than floppy disks! SSD's have crashed on me more than that already. I had 3 crashes caused by failing SSD's on my active systems in the last 8 years or so. Of course if you do not actively use them, SSD's will probably stay even better than normal hard drives because there are no moving parts. Not sure what my next backup disk(s) will be (given size needed) HD or Solid state. Even USB sticks and SD cards come in the picture, price/GB is dropping fast. Not sure if I would trust them as much as SSD's for long term. Of course my daily backup disk will go over 4TB soon, need to buy an other 4 or 8 TB HDD soon. They are dirt cheap anyway.

[Alberti]

I have not had the file errors for some time now. It has to do with some old folders probably.

Yes I have 2TB SSD for cities (2007-202now) for 70% filled and another 2TB WD Blue 550 for 40% filled for holidays such as in my case of Switserland. The first one is on Thunderbolt (2.300 read), the latter a USB-C drive at 1.000 Mbs.

Then I have CCC that grabs all these files/folders and compares these to its backup, on a 6TB WD HDD, once a week, &  it is set to only add to the folders.

& When I import, I backup to a local SSD quickly, just because something could happen along the way, I throw these away every year. You know how we work sloppily. Make a selection. Throw things out. Gone. &Later . .  In C1 they have a better strategy: a wastebin that gets filled with all discards. 

There are 4TB USB thumb drives. I tried one. A disaster to write to, slow, not very comforting. I threw it away.