Rethinking my digital photo storage - how do others do it?

[jhluxton]

Advertisement (gone after registration)

I am reaching the point where my 2TB hard drive on my main PC will be full by or before the end of 2021.

Space started shrinking rapidly once I started shooting in DNG at the end of 2016 and using Lightroom after getting an M262 and then switched my other Leicas to shooting in DNG/RAW. 

As well as the DNG/RAW images I also have the outputted jpgs stored on the hard drive as well, along with most of my other jpg images since I bought a Digilux Zoom in 2000! 

I seriously need to rethink my storage strategy within the next couple of months as I have some trips planned for the autumn. 

Yes I could I delete the DNG files and solve the issue - but then all I would have are the JPGs which although outputted at highest quality will never be as good as the original DNGs which I sometimes go back to reedit. 

In addition as the collection has grown backing up takes longer.

I keep three backups on each on 2TB external drives.

I used to just reformat one external then move everything across as that would include any re-edits as well as new images, now I just copy odd the odd directory as and when.

Obviously  I am going to have to move to an external solution but not sure which way to go.

I want to ensure what I have is backed up as well. Thus when I go away I can take copies with me just in case burglars or disaster strike at home when I am not present. At present I just take one of the 2TB drives with me, lock one in the safe and usually hide the other one. 

There must be quite a few out there who have faced a similar dilemma and I wonder what storage / back up solutions others have adopted?

Appreciate any feedback / suggestions.

Thanks

John

 

 

 

 

 

[zeitz]

With today's technology 2TB is tiny.  Much larger HDDs are available at very reasonable prices.  If you are serious about your files, you may want to consider a RAID.  I am fond of Drobo 5D3 software controlled RAIDs, but the unit itself is pricey.  The advantages are you can use any drives you have on hand (up to 5) because they don't have to match, you can hot-swap drives, and you can easily set the unit up for triple redundancy.  Triple redundancy means that three drives have to fail at the same time to lose any data.

[wda]

Your valuable files are your RAW files and any worked derivatives.  There is less need to store associated jpeg files. Sacrificing those would be my first step. If keeping jpegs is important, consider migrating them off-line. After all, I doubt they are referenced very often.

Second, start to re-edit your masters. I have an ongoing operation doing this, which has two benefits. One, it weeds out duplicates and stock of little value. Second,  I rediscover long-forgotten gems which gain a new life. These two actions regain valuable disk space. However,  you probably need to consider adding more primary disk space. Every few years, I have my internal drives upgraded to increase access speed and size. But that strategy is not viable if you change your computer fairly often. I hope these thoughts help you.

 

[Jeff S]

More in various other discussions…

Jeff

[ianforber]

I store all my photo files on external drives: one master, two separate copies which I update regularly and a third copy on an NAS (two disks running as RAID 1) that I don’t update that often. I also have jpg versions of my better/most treasured photographs on Apple Photos for easy viewing with family and friends. 

I haven’t bothered going as far as cloud backup storage but I seem to remember that last month’s Black &White Photography magazine had a review of the main providers if that’s of interest

[ianman]

16TB QNAP NAS with RAID, backed up to Amazon Glacier.

[hansvons]

Advertisement (gone after registration)

I own a small film post-production boutique (ADSK Flame mainly). In an average year, I amass 20-50 TB of video files depending on the size of the projects. Generally, there are two streams of data that need backup. 

1. First is the original camera data,
2. second, she projects files that include selected shots, the post-production data and the master versions. 

In photography land, typically the selected DNGs, the LR or C1 data and the edited master images. The original camera files end up on two separate HDs (4-6TB these days) or, in my case, on LTOs. The project files end up on another backup set, the master files additionally on my cloud storage (in my case, Nextcloud) for convenient access. 

In the last ten years, about 12 disks died on me, the majority in video raids. Only a stringent backup strategy to two mirrored disks provides sufficient safety. I'd buy only enterprise-grade disks; they have a significant better failure rate.

That being said, for the enthusiast or small commercial photography, I'd get a 4TB cloud storage for the 2nd backup stream (selects, edit data, master files) which is mirrored on an HD of the same size in my computer and two HDs for the original camera data. 

 

 

 

[convexferret]

@jhluxton it's not clear from your ortiginal post if you keep all of your DNGs or not. My experience is that only a small % of the images I take are actually useful. So I'll outline how I keep my archives small, and therefore easy to backup in multiple places

1. Import all DNGs from card to Lightroom

2. Mark the "good ones", add them to Lightroom albums. I have an album-per-month for general shots and albums for particular events

3. At the end of the year list all images that aren't in any album. Scan through them for anything of interest and then delete everything else.

I end up with about 1000 images a year saved and in albums and so 15 years of digital images use up less than 1TB. 

4. Export the albums, both original DNGs and output JPEGS into Dropbox as my secondary backup.

 

Edited August 5, 2021 by convexferret

[Tailwagger]

I have a pair of Synology NAS servers,  RAID 1 (mirrored and hot swap-able), one with a pair of 3TB drives, the other with a pair of 6TB for a total of 9TB.  Both are backed up to external USB drives.  Every month I offload the previous month's work to the current NAS.  Once that fills, I'll buy a third. 

[jhluxton]

Thanks for the various ideas - there does appear to be quite diverse strategies in use.

Someone suggested deleting the JPGs and just retaining the DNGs but I wont be going down that route as all my images pre October 2016 are jpgs. 

Besides JPGS are viewable quickly in the viewer I use they are all neatly organised in date then category folders and sub folders which makes finding things a doddle. 

I never used this method with Lightroom thus the DNGs are stored in the Lightroom linear timeline.

What I am probably going to do is move all the JPGs off the hard drive to an external and leave the DNGs on the main hard drive. Both of course backed up as now. 

That will free up around 0.75TB on the PC hard drive,  besides I can then store jpgs together with older image JPGs created from slide and film scans. 

The Raid back up option is interesting I actually have an old raid drive for keeping of documents and downloads on - but the problem is with Raid drives is they tend to be one unit - so lose the unit to a thief or household disaster - lose the data. l would be looking at two Raid drives and they are bulky.

My present one is the size and weight of a house brick and images I have seen of current models don't suggest they have shrunk much and some multi drive ones appear even bigger.

Not exactly the sort of thing one wants to cart off on holiday. I always take a full set of backups away with me - just in case. 

Thanks

John

  

 

[250swb]

I don't throw negatives in the dustbin and I don't delete digital files, it's madness. Memory has always been relatively cheap, buy more.

[EricC]

2TB LaCie plus a 6TB LaCie for images plus three other external LaCie drives as back-up.

I also have never deleted a digital file, neither RAW or JPEG and still also have all of my negatives too.

[Doug A]

My digital images are in Apple Photos backed up in iCloud. I discard about as many in a year as I add. The number has held steady at about 7K-8K for a long time. 

My negative scans are in a desktop folder on my MacBook Air backed up in iCloud. I shoot RAW+JPG, use the JPG's to make contact sheets, convert the RAW's to DNG's, and discard both the RAW's and JPG's. I don't discard any of the DNG's. I also keep all of my negatives. 

[leica dream]

At the risk of being fasecious is it worth going back a step and ask "WHY", then think through likely demand or when you would actually want to access such a huge volume of images. I understand why commercial units will need to maintain an archive, just like they do with email exhanges,  but it seems to me that without a clearly defined need and purpose of storing such vast volumes of images might not be productrive. In short, why store huge volumes of images if they never get looked at again. For me, photography is about capturing the moment for near term satisfaction or review, but as time moves on those moments fade and rarely see thge light of day again. 

[250swb]

37 minutes ago, leica dream said:

At the risk of being fasecious is it worth going back a step and ask "WHY", then think through likely demand or when you would actually want to access such a huge volume of images. I understand why commercial units will need to maintain an archive, just like they do with email exhanges,  but it seems to me that without a clearly defined need and purpose of storing such vast volumes of images might not be productrive. In short, why store huge volumes of images if they never get looked at again. For me, photography is about capturing the moment for near term satisfaction or review, but as time moves on those moments fade and rarely see thge light of day again. 

Yes, what you describe is a millennial thing of living in the moment and spewing out what's in that moment, and yesterday can go hang. Probably better saved for social media and other vacuous outlets for photographers. Photographers can actually learn from mistakes, and sometimes mistakes or shots that don't fit now turn out to be the presage of a later idea. It's kind of like 'why there are so many Picasso preparatory sketches that still exist', it's because he didn't bin the things that taught him something. So you were wherever you were, you pressed the shutter, and if nothing comes from it so you bin the picture, is that a way or working to recommend for other people, or a wake up call for yourself? 

[MelBuckpitt]

 I convert the scanned negs to DNGs.

Store each roll in a folder named with a unique id number, camera, lens, film and date unloaded.

These get replicated in three places

1. Host computer on which I process and print

2. Separate SSD storage

3. The cloud. I use Backblaze

[LocalHero1953]

11 hours ago, leica dream said:

At the risk of being fasecious is it worth going back a step and ask "WHY", then think through likely demand or when you would actually want to access such a huge volume of images. I understand why commercial units will need to maintain an archive, just like they do with email exhanges,  but it seems to me that without a clearly defined need and purpose of storing such vast volumes of images might not be productrive. In short, why store huge volumes of images if they never get looked at again. For me, photography is about capturing the moment for near term satisfaction or review, but as time moves on those moments fade and rarely see thge light of day again. 

The trouble is that actually asking that question poses an existential threat: "why have I spent my life taking photos that no one else will ever find interesting?"😉. I doubt you'll get much support for your idea here!

 

[iRandom]

I've been fortunate that mine continue to fit on the internal drive. I also do a backup to a local Time Machine drive and BackBlaze. The photo library itself is iCloud Photo Library which gives me a second offsite backup (including the RAWs). I used to do a weekly backup of the Photos folder to a LaCie Rugged RAID but recently repurposed the drive.

I keep kicking the can down the road when it comes to the question of what will I do if I outgrow the internal storage.

[leica dream]

OK, I accept fair comment but it never hurts to float the unthinkable. A millennial thing, well perhaps at 80 I should be proud! I do support the view that after a lifetime of taking pictures, keeping them is futile for those who follow but will have no interest in the subjects. Times, tastes and attitudes move on.

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[ianman]

On 8/12/2021 at 3:29 PM, jhluxton said:

the problem is with Raid drives is they tend to be one unit - so lose the unit to a thief or household disaster - lose the data.

That is why mine is backed up to Amazon Glacier. It is astonishingly cheap. The point with this system is that it really is a last resort. You only really pay when you download. Storage is frankly peanuts. I pay something like €2,50 per month for storage. IIRC if you do need to download you can do so at very slow speeds to reduce cost.