Shooting in DNG and how people deal with storage

[Graham H]

Advertisement (gone after registration)

Hi my wife is the keen amateur photographer (although I am getting into it from a nerd / tech perspective) and she has only just contemplated shooting in DNG rather than JPEG.  Currently she uses Icloud to keep her pics, and I am about to download Lightroom.  She takes a lot of pics for an amateur (on her Q2), recently in Venice about 2200 over a few days.  Clearly now shooting in DNG is going to present storage problems of its own.  We also have a desktop Mac as well as ipad and iphone as well as quite slow wifi (despite advertised speeds).  In the same scenario, how would members proceed?  We have been struggling with data transfer speeds.  How do people normally shoot?  Straight to memory card and then memory card into computer?  Sorry for multi-faceted beginners question!  Thanks Graham

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[jaapv]

Shoot to card, import in LR by a card reader and have a couple of external disk s USB 3 or USB-C) to store and backup your data. I have 12 TB external. 

[250swb]

As @jaapv says, do it old school and download to your own external drive, it will outlast the vagaries of future computers or any glitches in uploading to the cloud. Upload to the Cloud as a backup, but even then your genuine and important backups should be to another hard disk run in tandem. I'm not a fan of massive TB external drives, but I don't think the risk is as high as needing to work on your photos if your internet goes down and everything is locked away on the iCloud.

[Ouroboros]

WD external drives here.  I don't go above 4TB drives, I buy them in pairs for back up and when they fill up I store them away and buy two more.

I dig into the archive occasionally to make sure the older drives are ok,  I've only ever had one external drive go down.  The main caveat about archiving this way is to be disciplined with regular backing-up to both drives.      

[Graham H]

Thank you so much for the replies so far. I quite like the idea of holding multiple hard drives. We will see. Enjoying the forum. G

[Jeff S]

Keep at least one copy offsite, in case of fire, etc.

Jeff

[Ko.Fe.]

Advertisement (gone after registration)

Get rid of not worth pictures before transferring. Review in camera and delete.

Do not upload anything to the cloud before final edit is done. 

 

I can't comment on how shoot it normally. But to me 2K pictures over few days is something I was doing at early stage of using digital. Later, I have grown into Leica photography with film M. I observe, wait and take. No retakes.

If I miss it, just like GW was saying "nothing happened". 

[roydonian]

19 hours ago, Ouroboros said:

WD external drives here.  I don't go above 4TB drives,

WD external drives are what I use for backup. But why do you stay with 4TB drives? I've started using 6TB and have had no problems so far.

[Ouroboros]

5 hours ago, roydonian said:

WD external drives are what I use for backup. But why do you stay with 4TB drives? I've started using 6TB and have had no problems so far.

No reason why you should have a problem with any size drive.  I prefer smaller capacity drives mainly because I keep my paid work seperate from  my personal work.  I also archive different types of work on seperate pairs of drives, ie weddings, portraits, events etc.  

[djmay]

On 5/28/2023 at 9:14 PM, Graham H said:

Hi my wife is the keen amateur photographer (although I am getting into it from a nerd / tech perspective) and she has only just contemplated shooting in DNG rather than JPEG.  Currently she uses Icloud to keep her pics, and I am about to download Lightroom.  She takes a lot of pics for an amateur (on her Q2), recently in Venice about 2200 over a few days.  Clearly now shooting in DNG is going to present storage problems of its own.  We also have a desktop Mac as well as ipad and iphone as well as quite slow wifi (despite advertised speeds).  In the same scenario, how would members proceed?  We have been struggling with data transfer speeds.  How do people normally shoot?  Straight to memory card and then memory card into computer?  Sorry for multi-faceted beginners question!  Thanks Graham

Network Attached Storage (NAS). It is simpler than it sounds. There is lots of information available for beginners. Two good manufacturers are QNAP and Synology; I use QNAP.

Basic minimum setup is a box with two drives arranged in RAID 1. This means all your data are mirrored on two drives, so that if one drive fails you can continue working with the other drive. All your devices can connect to your NAS through your home network, as well as remotely. Your main computer and NAS need to be on the wired network, otherwise it will be probably be too slow. You still need to make at least two backups of your NAS for data safety. The NAS box has USB ports for connecting external drives.

Installation and backup is almost automatic with the provided wizards.

Once set up, your software, such as Lightroom, will treat your NAS as a drive.

As your data grow in size you can upgrade the size of the NAS drives if you need more space.

[roydonian]

26 minutes ago, djmay said:

Network Attached Storage (NAS). It is simpler than it sounds.

But what happens in the failure is not in one of the drives, but in the NAS box itself?  Will a replacement NAS box built to a NAS standard that has moved along in the meantime, and is perhaps made by a different manufacturer, recognise your existing drives and recreate the array?

[zeitz]

2 hours ago, roydonian said:

But what happens in the failure is not in one of the drives, but in the NAS box itself?  Will a replacement NAS box built to a NAS standard that has moved along in the meantime, and is perhaps made by a different manufacturer, recognise your existing drives and recreate the array?

Drive system failure, fire, theft and drive system manufacturer doing out of business (eg Drobo) - all have the same impact.  So as Jeff said in post #6, you need a duplicate storage system, one of which is off site.

[ianforber]

4 hours ago, roydonian said:

But what happens in the failure is not in one of the drives, but in the NAS box itself?  Will a replacement NAS box built to a NAS standard that has moved along in the meantime, and is perhaps made by a different manufacturer, recognise your existing drives and recreate the array?

My QNAP box eventually failed once. I bought a new one (different model) slotted the old drives in and it all worked perfectly. I had the data all backed up on separate external hard drives though so I was covered either way. 

If the OP decides to invest in a NAS, remember that RAID is designed primarily for business continuity. I also have mine set up as RAID 1, which means the second drive mirrors the first one which, in turn means that if you delete something, it is deleted off both drives.

To the OP, like others here, I import using an SD card reader with the images going straight into an external hard drive, which is then backed up several times but with the Library, as called in Capture One  (ie the interface through which you view, modify and organise your images) held on the computer and backed up online to my Dropbox account. 
 

Worth remembering that RAW/DNGs are just that, Digital Negatives. The final pictures that you have edited to your satisfaction should be saved as jpeg or TIFF files - I save a copy in my Capture One file structure but also export a high quality jpeg version of the ones I really like to my iPhoto app. I learned this the hard way when I was forced to move from Aperture to Lightroom and then again when I moved from using Lightroom to Capture One. JPEGs and TIFF files transfer seamlessly between different programmes but DNGs with their edits rarely do so - all you get is the DNG without any of the adjustments you’ve made. 

[LocalHero1953]

I had a QNAP RAID NAS for a while, with the intention to access it from both my PC (wired) and my laptop (wifi). In the end gave it away to my s-i-l. Three reasons:

• Too slow for use as a Lightroom drive for image storage. There was an irritating difference in speed between the NAS and an internal HDD, whether connected by wire or by wifi.
• I have a thing about equipment that has to be left on all the time. I prefer stuff that is only on when I want to use it.
• Accessing the same image store with Lightroom on two different devices is a recipe for confusing Lightroom.

My image store is now on a 10Tb internal HDD on the desktop PC, backed up to an external RAID 1 drive, which is backed up to the cloud (Backblaze). My Lightroom catalogue is on an internal m.2 drive for speed.

Edited May 30, 2023 by LocalHero1953

[marchyman]

3 hours ago, ianforber said:

The final pictures that you have edited to your satisfaction should be saved as jpeg or TIFF files

Minor disagreement.   I prefer to save the instructions needed to create the edited file on demand.  That might be in a Lightroom catalog, written to the DNG file (possible with Lightroom, not Capture One), or Capture One .cos and .comask files.  In my case the odds are that I'll tweak the edit before re-creating an image, anyway.  That's not to say I don't have some edited copies.  If I prepare an image for a web page the copy will live on that web page, but it is unlikely it will be at full resolution.

[ianforber]

22 hours ago, marchyman said:

Minor disagreement.   I prefer to save the instructions needed to create the edited file on demand.  That might be in a Lightroom catalog, written to the DNG file (possible with Lightroom, not Capture One), or Capture One .cos and .comask files.  In my case the odds are that I'll tweak the edit before re-creating an image, anyway.  That's not to say I don't have some edited copies.  If I prepare an image for a web page the copy will live on that web page, but it is unlikely it will be at full resolution.

Fair point. I just to make life simple for myself and I know I’ll forget to do something down the line so I just make sure I have an edited jpeg as well as the DNG file. When I die (and 30 mins ago while walking down the street I narrowly avoided a motorbike flying through the air in my direction following a nasty crash with a car) my wife won’t be using photo software to look at our photographs. She’ll be using iPhoto. 

[Jeff S]

Best way to preserve photos IMO is by making prints.  I can’t imagine my heirs going through hard drives, even if still functional. I use drives for my benefit while still around, not for others.  
 

Jeff

[dpitt]

On 5/30/2023 at 3:35 PM, ianforber said:

Worth remembering that RAW/DNGs are just that, Digital Negatives. The final pictures that you have edited to your satisfaction should be saved as jpeg or TIFF files - I save a copy in my Capture One file structure but also export a high quality jpeg version of the ones I really like to my iPhoto app. I learned this the hard way when I was forced to move from Aperture to Lightroom and then again when I moved from using Lightroom to Capture One. JPEGs and TIFF files transfer seamlessly between different programmes but DNGs with their edits rarely do so - all you get is the DNG without any of the adjustments you’ve made. 

I very much agree. And came to that conclusion the hard way (twice!) There is one constant in software and that is that everything changes very fast. Your DNG is to be seen as a film negative. To see the result you need some extra steps and these steps are made by your PP software with a TIFF or JPEG as a final result. What if your future PC or Mac refuses to run your current PP software? All you can do is try to import your project files into the future version... And what if there is no upgrade available, and no migration possible without loss of information? I found out that all edits that I made in Apple Aperture where completely lost once I tried to move to Adobe  LightRoom. All it could import from my catalogs were keywords and ratings (and not even that in full). This is because every PP uses a different engine behind each slider. Exposure +1 does not mean the same in LR or C1 or Aperture, and clarity +1 can not even be translated to some of the PP software packages out there.... And the 'future' is sometimes closer than you think. Lots of things happen in 5 years, 10 years is a lot and 20 years is an eternity.

So it is very important to save the time spent in editing your DNGs by exporting them as full resolution JPGS or TIFFS and have a backup of those in a simple, logical file structure. Keep this and your original DNGs and you can decide later to use your current result or redo the work you did today with more advanced PP software later.

On 5/30/2023 at 7:29 PM, marchyman said:

Minor disagreement.   I prefer to save the instructions needed to create the edited file on demand.  That might be in a Lightroom catalog, written to the DNG file (possible with Lightroom, not Capture One), or Capture One .cos and .comask files.  In my case the odds are that I'll tweak the edit before re-creating an image, anyway.  That's not to say I don't have some edited copies.  If I prepare an image for a web page the copy will live on that web page, but it is unlikely it will be at full resolution.

As said above, this works for a few years, but not over decades because even upgrading within one brand will become impossible without loss over time. All software we now use will become incompatible one day. Even reading current hardware and even JPEG or TIFF will become hard one day. If you do not believe me... Try to read a 5 1/4" floppy disk on any computer now, or try to open a Word Perfect 1.0 file or a Lotus notes file from the 80s.

5 hours ago, ianforber said:

Fair point. I just to make life simple for myself and I know I’ll forget to do something down the line so I just make sure I have an edited jpeg as well as the DNG file. When I die (and 30 mins ago while walking down the street I narrowly avoided a motorbike flying through the air in my direction following a nasty crash with a car) my wife won’t be using photo software to look at our photographs. She’ll be using iPhoto. 

Exactly. Any backup system depending on anything more than a basic file standard like JPEG or TIFF and a common file system is doomed to fail by incompatibility rather soon. A nice backup of organized JPEGS is the best bet for the future.

@Graham H My long term backup strategy is to keep several copies (2 at least of everything important). Each year I use an old hard drive to copy all of it and give it to a friend or family member to keep safe as offsite storage. When disks grow larger, I copy over the small disks to a larger one. I can go back this way to about 1990 in files, some of which might now be very hard to read because the original software is not around anymore. The original hardware would be impossible to read now because hardware to do that would be extremely hard to find, and the disk itself would probably not work anymore anyway. But because it is always copied over to the next computer, the files are intact.
Most of my data is copied in the cloud, but my photos are only stored in the cloud as reduced size JPGs, not as DNG. Offline storage is only used as worst case backup scenario.

Price of storage and size of disks have improved more than my needs for data over the years. In 1990 my computer had only 100MB of disk storage and all important files only took up about 10 MB of disk space. Now I can store everything including my DNGs on one 2TB disk (50K+ images). I have about 10 TB in use now (more if you count every disk in the house), most of it is redundant or for backups. My largest disk is now 5TB. So I can still keep this up for the foresee-able future. Having a yearly copy of everything, kept off site is almost completely safe. Over more than 30 years of computer usage (started in 1987), I can say the hardest issues were human error and software incompatibility rather than hardware failures.

Some photographers will have more need for storage than I do. Some take 10 times the number of pictures I do, and some use much larger files because I often use older camera systems that produce smaller files than the 60MP sensors of today. But even then you can make this work. Storage hardware is relatively cheap, so even if you have 20TB to deal with now, it would not be an issue and it could be fitted on 2 disks. If you shoot lots of video files you might get into trouble, but not as an amateur with DNGs IMO.

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[roydonian]

12 hours ago, dpitt said:

What if your future PC or Mac refuses to run your current PP software?

 

You run it in a virtual machine. That's what I had to do a decade ago then I switched to a 64 bit computer and operating system, but the secure datalink software used by a major client was 32 bit and would not talk to a 64 bit system. This solution added less than a minute to the duration of each comms session with the client.

Real soon now I plan to install a Win98 virtual machine so that I can play some of my classic games from the 1990s.

Faced with possible file-incompatibility problems caused by freelancers using some of the many word-processing programs that were around in the late eighties and early nineties I bought a file-conversion program that could handle a wide range of formats. Early versions of one of the open-source office suites (probably OpenOffice) could also read a huge range of obsolete file formats.

Reading a 5.25 inch floppy is a more difficult task, but there are at least two companies offering adaptor boards that allow this to be done on a modern PC.

[dpitt]

14 hours ago, roydonian said:

 

You run it in a virtual machine. That's what I had to do a decade ago then I switched to a 64 bit computer and operating system, but the secure datalink software used by a major client was 32 bit and would not talk to a 64 bit system. This solution added less than a minute to the duration of each comms session with the client.

Real soon now I plan to install a Win98 virtual machine so that I can play some of my classic games from the 1990s.

Faced with possible file-incompatibility problems caused by freelancers using some of the many word-processing programs that were around in the late eighties and early nineties I bought a file-conversion program that could handle a wide range of formats. Early versions of one of the open-source office suites (probably OpenOffice) could also read a huge range of obsolete file formats.

Reading a 5.25 inch floppy is a more difficult task, but there are at least two companies offering adaptor boards that allow this to be done on a modern PC.

Reading the 5.25 inch floppy:

Yes, I did not say it was impossible, but very hard and also very costly. And this is talking about PC stuff only.

Let's try the same with Apple systems. Apple II is easy compared to the early Macintosh classic systems. AFAIK there are no easy ways to run a VM for Mac OS classic. Trying to read your disks on a PC VM will not help either because Mac and PC were incompatible at the time, and even if you succeed getting the data file, finding and running the right version of the Mac OS software is an other issue. Reading other Mac hardware is getting hard too. Try reading a SCSI drive from the 80s or ZIP and Jazz disks... Eventually only museums will have the hardware to do that.

My point is that you should try not to rely on any hardware or software as much as possible to archive your stuff. I hope my children will be able to open a JPEG file 30 years from now but even that is unsure.

Edited June 2, 2023 by dpitt