DNG compressed VS uncompressed?

[mjh]

Advertisement (gone after registration)

There is no good reason not to use lossless compression. For one thing (as others already said), lossless compression is lossless, period. Decompressing losslessly compressed data yields a bit by bit identical copy of the original. And I don’t think there is a reason to worry about an increased chance of data loss either. Yes, a given number of corrupted bits will do more harm to a compressed than to an uncompressed file. On the other hand, as the compressed file is smaller, there is less of a chance that some given corruption affects it at all. A bullet of a given size may do more harm to a small prey than to a larger one, but the smaller one is also more easy to miss.

 

Older software is a non-issue. Using older software that cannot deal with all variations of you camera’s raw files means you are using software that wasn’t really written with your camera in mind. Don’t do that.

[bencoyote]

I agree with you. This is not a good reason for not using compressed DNG files.

 

Yes, a given number of corrupted bits will do more harm to a compressed than to an uncompressed file. On the other hand, as the compressed file is smaller, there is less of a chance that some given corruption affects it at all. A bullet of a given size may do more harm to a small prey than to a larger one, but the smaller one is also more easy to miss.

 

Good way of putting it. I like the analogy. I wish that I'd thought of it when I was working in this field. Just keep in mind that this is less like an elephant and a mouse than it is like a sheep and a goat. 

 

Having better internal consistency checks to facilitate error detection and correction is an area where DNG could be improved. If I worked for a camera vendor or Adobe, this would be a fun project but I'm just an amateur taking personal shots and I'm not losing any sleep over it now.

[menos I M6]

…

Older software is a non-issue. Using older software that cannot deal with all variations of you camera’s raw files means you are using software that wasn’t really written with your camera in mind. Don’t do that.

 

It is not the older software to worry about compatibility of data. Is is the software we will be using in 20, 30 or even 50 years to display or work with the images we took in our youth.

 

The general implication is that an uncompressed file does pose a slightly smaller risk of being incompatible with future software.

For the very same reason I for example do not use any DNG repackaging algorithms but store and work with the original camera raw files exclusively (there has been some great debate about the actual standards and it's flexible use by Adobe some years ago).

 

Hard drive space is cheap. I have no issues even with the (now considered) small 16GB cards I use in my Leica M digitals.

Hard drive space gets higher and higher performing from generation to generation too, so does my archive.

 

In my view the answer to this question is exclusively an answer about keeping it simple. The advantage of smaller files is such a short one and even with only current computer technology working with hundreds or even thousands of shots at a time is reasonably fast (I use a 2015 MBP and 2013 nMP with Adobe software and image files on OWC SSD drives and slow Seagate storage drives.)

 

Picking single files at random from the archive (on slow HDDs), files that have no previews built and taken from external drive arrays, at sleep - even this is reasonably fast.

Replacing those files with slightly smaller compressed files has zero practical benefits to my workflow, but it attaches an additional risk.

 

And yes, even when ridiculed as rays from space, data corruption is absolutely real and DOES happen.

I had over the course of the last few years a handful of cases where certain files were corrupted (image files could not be rebuilt any longer) and I had to replace those corrupted files from backups.

 

It is therefore also absolutely essential to run regular hashtag checks (I use a Lightroom plugin and on top of that an additional standalone software that runs over the entire archive for that).

These software solutions btw where the only means for me to find out about the corrupted files in the first place.

 

Now that has happened only over a time span of the last 2-3 years - think about the upcoming 30 years. I really WANT to keep those photographs until then.

I don't want to loose them to a real risk just because I saved a few meaningless megabytes on a SD card in an ancient tech camera I had for just a few years when I was young - and no, I do not print all of my photographs.

 

The very reason why there even is a discussion about this compressed vs uncompressed issue is proof enough that it isn't that simple a matter of lossless is lossless. Sure the mathematical algorithm is, but it doesn't stop at the mathematical algorithm, does it?

 

If there would have been The Internet 50 years ago and all amateur photographers and camera geeks would have been active on forums like this, the discussion would probably have been: Should I store my negatives bare in cartoon boxes in the attic to save space or should I store them in archival sleeves in folders in the studio? My wife complains about all the folders.

The attic has the occasional critter and if that roof should leak at some point, it may or may not leak on those bare negatives in the cartoon boxes, rotting them away, … I'd keep 'em in the studio.

[pop]

....

For the very same reason I for example do not use any DNG repackaging algorithms but store and work with the original camera raw files exclusively

....

I do not recommend doing this. The documentation of the DNG format is publicly available; it's just a variant of the TIFF format, which is well documented as well. The documentation for a camera specific raw format may or may not be publicly available. However, changes of it remaining available are better if the publication is done by a body whose purpose is the publication of standards.

 

Software which can process those camera specific files may become hard to find, once the camera is no longer sold and the old software will not be run any more by more recent operating systems. That might not be an issue within a time span of a few years. It certainly will become one when thinking in decades, as you do.

 

Archives become increasingly aware of this and recognize that the only way around "progress" lies in regularly verifying the digital resources and in converting them to more recent formats, once the old ones are no longer actively maintained.

 

Your argument about having to have backups and about scanning the files for errors is quite correct. However, this will take longer for larger files, and that is not always irrelevant. However, if you feel the security of your files is enhanced by using larger files, the price of doing so would not be a great deal.

[jaapv]

However, so much essential data has been stored in TIFF and JPG by now that I think the digital world cannot afford to let those formats become obsolete.

[pop]

Both TIFF and JPG are well documented; supporting those formats does not present any particular difficulties. For archival purposes they are just as good as  DNG, I think. 

[Exodies]

Advertisement (gone after registration)

Friday's XKCD was about this topic. Not compression, but digital degradation.

[adli]

As Michael Hußmann pointed out,since it's lossless you don't lose anything using the compression, and you gain space in your card and speed in transfering files to your computer, so its no point not using compressed files.

 

But then, I use uncompressed files, so do as I say, not as I do

[mjh]

Both TIFF and JPG are well documented; supporting those formats does not present any particular difficulties. For archival purposes they are just as good as  DNG, I think.

As long as a file format is well documented and there is sufficient interest in its support then it will be supported. There is no reasonable doubt about that. Compressed DNG is nothing exotic; it is as vanilla as it gets.

[mjh]

Friday's XKCD was about this topic. Not compression, but digital degradation.

Strangely the guy is talking about copying data whereas what the pictures show is something quite different (accumulating JPEG artifacts, watermarks etc.).

[Adam]

It is therefore also absolutely essential to run regular hashtag checks (I use a Lightroom plugin and on top of that an additional standalone software that runs over the entire archive for that).

These software solutions btw where the only means for me to find out about the corrupted files in the first place.

 

Now that has happened only over a time span of the last 2-3 years - think about the upcoming 30 years. I really WANT to keep those photographs until then.

I don't want to loose them to a real risk just because I saved a few meaningless megabytes on a SD card in an ancient tech camera I had for just a few years when I was young - and no, I do not print all of my photographs.

 

The very reason why there even is a discussion about this compressed vs uncompressed issue is proof enough that it isn't that simple a matter of lossless is lossless. Sure the mathematical algorithm is, but it doesn't stop at the mathematical algorithm, does it?

 

If there would have been The Internet 50 years ago and all amateur photographers and camera geeks would have been active on forums like this, the discussion would probably have been: Should I store my negatives bare in cartoon boxes in the attic to save space or should I store them in archival sleeves in folders in the studio? My wife complains about all the folders.

The attic has the occasional critter and if that roof should leak at some point, it may or may not leak on those bare negatives in the cartoon boxes, rotting them away, … I'd keep 'em in the studio.

 

Yes it all boils down to an algorithm, compressed or not data on your machine is a bunch of 0/1 scattered on a drive, even the most artistic photos. In order to translate it into a picture you need software regardless whether your picture is compressed or not. From a programmers point of view, adding a routine that translates lossless compression into an uncompressed data set is trivial. DNG lossless compression will be read as long as DNG files are read. There is no need to be concerned about lossless compression.

 

Degradation will happen. Magnetic media deteriorates even without cosmic rays, etc.  It is a matter of probabilities, the more pics one has, the more likely your data will be corrupted, running hashtags just tells you that it happened. The only way to combat it is to build redundancies. Multiple copies.  In that sense it is easier than archiving actual prints.

[pico]

I do not recommend doing this. The documentation of the DNG format is publicly available; it's just a variant of the TIFF format,

 

It certainly is. Just for the heck of it, I just renamed a .DNG file to .TIF extension. (Mac OS X).

The desktop preview was maintained, it opened in both Preview and ACR and the metadata was retained.

[Adam]

Note sure if that says anything. Preview and ACR read both DNG and TIFF.

 

For the fun of it I renamed a .rtf document as a .doc and successfully opened the file in word. It reads both formats, it did not care about the extension.

[pico]

That is just smart programming.

[Exodies]

Strangely the guy is talking about copying data whereas what the pictures show is something quite different (accumulating JPEG artifacts, watermarks etc.).

 

 

That's the joke - what he says is correct the data copying can be bit perfect but stuff you are not controlling for will happen.

[pop]

Most applications "sign" their data files. RTF and Word files can be told from each other by inspecting the first four bytes, although you need a program which displays the hexadecimal values to be certain. You can not tell DNG from TIFF files that way; both have the same signature.

 

The specifications of the DNG format clearly state that DNG is an application of the TIFF format. The specifications of both the TIFF and the DNG format can be downloaded from Adobe.

 

There's not much room for guessing.

[bencoyote]

It is not the older software to worry about compatibility of data. Is is the software we will be using in 20, 30 or even 50 years to display or work with the images we took in our youth.

 

The general implication is that an uncompressed file does pose a slightly smaller risk of being incompatible with future software.

For the very same reason I for example do not use any DNG repackaging algorithms but store and work with the original camera raw files exclusively (there has been some great debate about the actual standards and it's flexible use by Adobe some years ago).

 

 

The DNG file format http://www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf specifies Huffman Compression https://en.wikipedia.org/wiki/Huffman_coding for raw data. The original paper describing it was written in 1952 and so it is not exactly one of those popular today quickly lost in the churn of silicon valley. Huffman compression while not the most effective lossless compression algorithm for any given kind of data is EXTREMELY well known and documented and there are MANY free and open source implementations of it. It is not going away. It kind of one of those things that is in the canon of fundamental computer science and used all over the place. You're not going to need to worry about it being incomprehensible in 20-30-50 years. 

 

The biggest problem with DNG 1.4.0 and its specification is really the lack of intrinsic error detection and correction metadata within the file. It is my belief that any long term archival storage format like a Digital Negative purports to be. It should include some sort of internal block code https://en.wikipedia.org/wiki/Block_code which would allow the integrity of the file and the data to be verified and if corruption does occur, to be recovered in at least some cases.

 

If we really wanted to be fancy and make something that would stand up to journalistic and legal standards, the ideal would be that there would be an intrinsic block code for error detection and corruption, then a digital signature https://en.wikipedia.org/wiki/Digital_signature identifying the camera another digital signature identifying the photographer if the copyright is set and all of that would be wrapped in a digital signature for the file. This would allow a news organization could verify that a particular picture was taken with a specific camera by a specific photographer and that the photograph has not been altered in any way since it was captured. That is the kind of thing that would need to be designed into a new version of the DNG specification like 1.5 and would take some careful engineering. Like I said before, that would be a fun project.

[pico]

Should it not be the role of the OS to guarantee a bit-by-bit perfect copy? It does, does it not?

 

And regarding the opening of files - a few programs, and more to follow use a known construction. For example, JAVA .jar files and MS Word .DOCX files are ZIP files containing separate components of such things as format, text, images, macros and so-forth. During Microsoft's early use of .DOCX with their bugs I would sometimes decompress them and manually patch the components, re-zip/rename and return the file. There is a lot more of that to be done.

[jmahto]

About digital degradation... I have approximately 60k files in LR repository and I have noticed only 2 files (DNG) that OS could not read after some times. The corruption was detected during backup when copy failed for those two files. I don't have any explanation why it happened but now I have became more careful about taking original backup and doing periodic exports (processed jpegs) that are stored separately.

 

Even then I feel better about digital longevity vs film. Once I lived for year and half in a humid place and noticed my slides and negatives degrading due to growing fungus. I had to send them to non humid place to stop fungus growth.

 

Digital, just like any other media, needs appropriate precautions.

[Advertisement]

Hallo,

Look here: Leica M

[Adam]

Most applications "sign" their data files. RTF and Word files can be told from each other by inspecting the first four bytes, although you need a program which displays the hexadecimal values to be certain. You can not tell DNG from TIFF files that way; both have the same signature.

 

The specifications of the DNG format clearly state that DNG is an application of the TIFF format. The specifications of both the TIFF and the DNG format can be downloaded from Adobe.

 

There's not much room for guessing.

 

My point was that changing file extensions and opening a file with a program that opens both types of files does not tell you much about the files. I just changed a compressed jpg extension to a DNG one and opened the file in preview (not in Photoshop though). That is because DNG can 'contain' compressed jpg, or TIFF for that matter.

 

 

When talking about DNG -Tiff structure similarities, it depends on which Tiff format you are talking about. DNG from 2012 is TIFF 6  (de facto Adobe TIFF). Open file TIFF-EP standard is different, not much, but enough. According to Adobe: 

 

"DNG is an extension of the TIFF 6.0 format, and is compatible with the TIFF-EP standard. It is possible (but not required) for a DNG file to simultaneously comply with both the Digital Negative specification and the TIFF-EP standard"

 

That is from: http://www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

 

The text clearly shows that a DNG file can be NOT compatible with latest TIFF-EP standard, and still be a DNG file. Something to keep in mind since DNG is just a file container specification and Leica does its own proprietary stuff within the ramifications of DNG. Plus, there are different versions of DNG. The format keeps evolving. Plus there is a difference between DNG containing raw data, and DNG containers with non raw data. Many people agree that converting NEF raw files files to DNG is not such a  great idea, the same applies to TIFF -> DNG conversion. See:

 

http://petapixel.com/2015/07/16/why-i-stopped-using-the-dng-file-format/

 

By the way, Nikon NEF format is based on TIFF-EP. But not open.

 

I am not criticizing DNG, just pointing out the details. DNG is not the open standard it advertises to be. It is open, but not a standard yet. One day, I hope, we will get an open ISO file format standard.

 

But I think the Leica implementation of DNG is excellent. Native raw DNG is the best solution out there for now, and Leica raw files are wonderful to work with. And I am sure once there is a standard, DNG raw files will be easy to convert.