Compressed vs uncompressed M9 DNG

[JonPB]

Advertisement (gone after registration)

Over the last 16 months, I have taken almost no photos -- the longest such period for me in decades. I've found my skills are rusty ... the fundamental intentions are there but some of the habits have been lost. So I'm treating my M9 like a new-to-me camera. Here's one interesting thing I've found that I wasn't even aware of beforehand: the difference between compressed and uncompressed raw files.

Background: the M9 offers raw and JPEG formats; raw files can be either compressed or uncompressed. The compressed files are half the size of the uncompressed files, so this is not a data-oriented lossless algorithm (like .zip files) nor a visually oriented lossy algorithm (like .jpg files). My understanding is that the uncompressed values are bit-shifted to fit 8-bit containers, which is a very fast operation for computers to perform (and bit-shifting a 16-bit value to an 8-bit value is mathematically the same as taking a square root). Then, when processing the raw file, those 8-bit values are expanded to 16 bits. A raw value of 255 (clipped white) becomes 255^2 = 65,025, a value of 254 becomes 254^2 = 64,516, and so on until a raw value of 2 becomes 4, with 1 and 0 unchanged. Thus there are gaps between the possible values that the raw photo processor can interpret, much like tines in a comb: it simply isn't possible for an 8-bit raw value to tell the image processor that the captured value was 3 or 65,026. If you then turn around and output a .jpg file, the image will be re-compressed to 8 bits and the image is, as far as this goes, unchanged. However, if you manipulate the photo, you're theoretically losing out on a fair amount of precision that might cause visible artifacts.

(Note that the sensor package outputs 14-bit values, so the range of values actually lost is not as large as going from 16 to 8, but remains numerically substantial. Yet is it photographically significant?)

Previously, I had assumed that truncating each datum meant that lost data was taken from the shadows, so I tested by pushing images and looking for artifacts. I never found any, so I always shot compressed images. Upon this reanalysis, I am convinced that I was wrong: information is lost evenly throughout the entire range of values.

To test for visual effects, I took several photos of a corner of a room with a white wall with gradual tonal transitions. That wasn't useful. But, the photos did include a few wall hangings, and while not in precise focus there is enough detail for me to draw conclusions adequate for my purpose. Comparing two shots with the same exposure and similar framing, one compressed and one uncompressed, I used Rawtherapee to apply absolutely minimal adjustments -- no sharpening, no noise reduction, and the "fast" and simple demosaicing method. I took white balance off the same spot in each image. Then, to simulate the kind of image processing that could theoretically cause issues, I applied a fairly steep, but not unreasonable, curve. I examined the images on a high-DPI but not photo-specific monitor. Then, to simulate more realistic conditions, I applied my usual demosaicing settings but did nothing else, e.g. with sharpening, noise, etc. Finally, I desaturated the images because I almost exclusively work in monotone.

There are seven attached images (I'm not sure how to inline images I'm uploading to the site):

1. The curve used.
2. Basic raw processing, uncompressed file. This has the tall, flat histogram.
3. Basic raw processing, compressed file. This has the jagged histogram.
4. Normal raw processing, uncompressed file. This image is rotated slightly clockwise of its counterpart.
5. Normal raw processing, compressed file. In addition to rotation, this image's histogram has a green bump just before going hitting the right edge.
6. Normal raw processing, desaturated, uncompressed file.
7. Normal raw processing, desaturated, compressed file.

Note 1: the "basic raw" images have a raw-value histogram that does not change with image processing; the "normal raw" images have histograms that reflect the full image.

Note 2: in the crop view, the blue and brown fields are under the steep parts of the curve, while the wall and shadows areas are not.

The raw histogram makes it obvious that the compression does indeed take place throughout the entire range of possible values: the bottoms of those comb-like jaggies indicate where there are no values at all, whereas the uncompressed image indicates the presence of a relatively continuous range of values.

On inspection of the basic images, the differences are far subtler than the histogram might suggest. What I see are smoother gradients and less noise. This was predictable: when the compressed value drops just one increment in one channel while the other channels stay the same, the image processor sees a fairly large change that indicates a meaningful change in the color. Something I did not predict was that continuous lines of moire -- optically inevitable aliases along edges -- are more pronounced due to the increased precision of the uncompressed file. Furthermore, the compressed image appears slightly sharpened; I triple checked that this was not a Rawtherapee setting. Still, the raw compression has visual artifacts that I do not like -- when viewed at 400%.

Inspecting the normally processed color images, the differences become yet more subtle. I use the LMMSE (linear minimum mean-square estimation) demosaicing method due to how well it handles aliasing on these anti-alias-filter-free cameras, and LMMSE also removes a lot of the compression-induced aliasing. The compressed image still appears lightly sharpened. Notably, the highest resolution features in the compressed image lose most of their color saturation. My sense is that the compressed image also has slightly more impulse noise, but I learned a lot time ago that comparing noise from two crops like this is absolutely unreliable; true noise is random, so you need a large sample size to get a sense of the actual distribution. My takeaway is that the uncompressed image retains better color fidelity at high resolution, and retains more consistent edge transitions. (Remember, this part of the frame is slightly out of focus.) There's just more magenta in the compressed image.. this might be worth inspecting in relation to high ISO traits.

Turning to the final pair, the monochrome images, the difference ... remains about the same. At this point, I'm wondering if the sharpness difference isn't due to change in focus precision between shots. (Perhaps now you see why subscribing to Sean Reid's reviews is money well spent. He has these issues locked down.) But that hypothesis does not explain the unsharp-mask-like behavior that I see: a slight overcompensation to lightness when transitioning from dark to light structures. Regardless, the compressed file still seems to yield a slightly noisier tonal fields and has jaggier aliasing; the highest resolution structures have more consistent contrast in the uncompressed file.

Now, the penultimate test: zoom out and see if I notice any differences. At full-screen, roughly the size of my prints, I can tell that there's no need for the ultimate test -- prints. I won't be able to tell them apart. At 100%, though, I'm compelled to say that there is a difference. I haven't found a way to illustrate it, but my impression is that the uncompressed file is more Monochrom-like: it looks less like a crowd of pixels and more like a coherent photo. Of course, I've spent 3 hours with these images and so my psychology training tells me that I'm reliably unreliable at this point: I've made my interpretation and my brain will see those for me whether or not they actually exist.

Hopefully I've shown you enough to either recognize it as a non-issue (or obvious issue) for your purposes, or perhaps I've offered an indication of how you can effectively conduct your own tests to make your own determination.

But storage is cheap, I shoot slowly, and I can no longer pretend that there's no difference at all between compressed and uncompressed. Crop a bit, draw a steep curve over part of the range, and these issues, which look like moire or chromatic aberration, will start showing up. If I could reduce a lens issue with a larger memory card, I certainly would; that's effectively what's happening here. If I was shooting an event and running into buffer limitations, I'd absolutely start using compressed DNGs: half the size means half the write time. Or, if I needed to conserve space for some reason, the compressed DNGs are still excellent (they've served me well this long!) and far superior to JPGs. But consider me converted: uncompressed from here on. At least until I get a more advanced camera with a reasonably modern (lossless) compression method.

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[jaapv]

Hmm... According to Leica, the DNG compression is lossless, so I wonder what you have found here.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[pop]

vor 35 Minuten schrieb jaapv:

Hmm... According to Leica, the DNG compression is lossless, so I wonder what you have found here.

The data sheet for the M9 mentions these storage formats: DNG (raw data), choice of uncompressed or slightly compressed (by non-linear reduction of color depth),
2 JPEG compression levels. A 'non-linear reduction of color depth' seems to imply possible losses.

The user manual says: The Compressed format includes a slight compres­sion, which  (...) causes only a negligible deterioration in quality

Edited August 3, 2021 by pop

[jaapv]

That explains the findings in the OP then.  

[darylgo]

Can we generalize these findings to other Leica cameras that use DNG compression?

[pop]

vor 16 Minuten schrieb darylgo:

Can we generalize these findings to other Leica cameras that use DNG compression?

No, we can't. You have to consult the technical data sheet for each model. Luckily, they're easily available on Leica's web site.

I think it's only the M9 (and derivates) which use this particular kind of compression, and I think all the more recent models use more sensible procedures, i.e. lossless ones. RTFM.

Edit: I just did a spot check on Leica's web site. It appears that the Niner (M9) is the only variant with lossy compression, and I could not see anything about compressing for the M8. They explicitly state both for the M and the M10 that the compression is optional and lossless. Perhaps they couldn't stand the derision their lossy compression fetched them.

Edited August 3, 2021 by pop

[darylgo]

Advertisement (gone after registration)

37 minutes ago, pop said:

No, we can't. You have to consult the technical data sheet for each model. Luckily, they're easily available on Leica's web site.

I think it's only the M9 (and derivates) which use this particular kind of compression, and I think all the more recent models use more sensible procedures, i.e. lossless ones. RTFM.

Edit: I just did a spot check on Leica's web site. It appears that the Niner (M9) is the only variant with lossy compression, and I could not see anything about compressing for the M8. They explicitly state both for the M and the M10 that the compression is optional and lossless. Perhaps they couldn't stand the derision their lossy compression fetched them.

@pop  Thanks, good to know, I compress whenever available always thinking it is lossless.  re: RTFM, the English version is better than other camera manufacturer manuals but if I had to give it a grade, they're solid F's.  

[JonPB]

I think the M8 uses the same compression, and, at least on release, there was no way to turn it off.

As far as manuals go, well, I think they assume that users will find their way here eventually. 🙂 Actually, looking at tech manuals over the years is fascinating. In the 1980's, manuals were often graphically designed and expertly typeset, and didn't just tell the basics but discussed why and provided meaningful examples that invited study. In the 1990's, the profusion of features meant that documenting everything displaced discussions beyond the basics. By the 2000's, manuals were cobbled together feature by feature, often by many different writers with varying styles and fluency, and dumped into a Word document. Admittedly, internationalization practices started improved dramatically, so more people gained access to those basic feature descriptions. In the 2010's, most people weren't reading the docs -- wonder why? -- and so it became unusual to even provide a printed copy. And the high turnover of new models killed off the third-party manual business.

I think this trend might particularly affect Leica. The little things that make those last little improvements but yield Leica's premium value are often hidden from view. Having more accessible documentation -- in customer support, in marketing, even in production and development -- that clearly articulates these details would benefit everybody because Leica is one of the few brands whose products can stand up to such examination. And I have enough faith in humanity that we'll eventually come to the collective realization that most blogs are nothing more than late night infomercials with a modern polish, and then having a trusted, reliable, and authoritative voice will be an invaluable component of brand identity.

[pop]

vor 59 Minuten schrieb JonPB:

think the M8 uses the same compression,

The M8 manual dedicates more than one page to the topic of the image file formats. According to that manual, the DNG format was never compressed; at least, that was strongly implied. However, there were two levels of compression available for JPEG format images.

[Ko.Fe.]

Quote

when viewed at 400%.

Wow! 

I'm not sure if it makes any practical reason to not use compression for M9 sensor's dngs.

[jaapv]

20 hours ago, pop said:

The M8 manual dedicates more than one page to the topic of the image file formats. According to that manual, the DNG format was never compressed; at least, that was strongly implied. However, there were two levels of compression available for JPEG format images.

There was an extensive article on LFI in 2007. The M8 uses a 12 bit LUT. 

[Jean-Michel]

I could never see any reason to use compression of any sort for raw (DNG for my Leicas, or CR2 for my Canons). Why bother with compressing to then having to decompress, just seems like an unnecessary exercise. I just save as uncompressed.