8bit DNG issue explained

[Max Penson]

Advertisement (gone after registration)

Hi guys,

 

I've written an article about the 8 bit DNG issue. I hope you'll find it interesting. As it turns out, Leica is placing a LUT inside the DNG file which can be reversed. I have also concluded that the M8 is really a 14 bit camera. I know there were a lot of discussions about that, but after seeing this "Linearization table" inside the DNG file, I am now more sure than ever.

 

KammaGamma » Articles » Solving the Leica M8 DNG riddle

[blakley]

Very interesting. I'd love to see the last few paragraphs illustrated with pictures showing the phenomena described. I've experienced the problems with blown highlights when photographing with a very saturated red channel as you describe.

[jaapv]

Very interesting. Thanks. What I do not understand is that you postulate problems with underexposed files, whereas practice shows that the M8 has the best shadow recovery of all digital cameras. Could you please explain what I see wrong here?

[blakley]

I'm trying to work that out too. The naive conclusion seems to be that if the shadow stops hold more information than the highlight stops, one should underexpose in order to put more of the image into the high-info region. On the other hand, increasing the exposure in post-processing will certainly quickly blow any detail in the highlight region because there's so little to work with in terms of information at stop 7....

 

I'll have to play with this in the camera & lightroom and see what happens.

[RainerJ]

Hi,

the problem with the over saturated red channel can be seen in my picture in this (German) post:

 

http://www.leica-camera-user.com/forum-zur-leica-m8/23134-farbsaum.html

[Jamie Roberts]

Hi,

the problem with the over saturated red channel can be seen in my picture in this (German) post:

 

http://www.leica-camera-user.com/forum-zur-leica-m8/23134-farbsaum.html

 

Rainer--that's not an oversaturated red channel at all That's a contrast artifact common to many digital cameras.

[Jamie Roberts]

Advertisement (gone after registration)

I'm trying to work that out too. The naive conclusion seems to be that if the shadow stops hold more information than the highlight stops, one should underexpose in order to put more of the image into the high-info region. On the other hand, increasing the exposure in post-processing will certainly quickly blow any detail in the highlight region because there's so little to work with in terms of information at stop 7....

 

I'll have to play with this in the camera & lightroom and see what happens.

 

I think the key to the M8's spectacular performance here is noise.

 

The experiment done in the article, to predict the inability to recover shadows and lack of highlight detail in the M8, was done with a D30--hardly equivalent to an M8 in terms of the noise floor.

 

Yes--the image will break up quickly in the shadows *if the noise level* results in errors due to compression.

 

However, IMO, the incredible low-noise of the M8 sensor at low ISOs means you can actually underexpose, take advantage of the less-compressed 14-bit shadow detail, and easily recover in the RAW converter with spectacular results.

 

And underexposing slightly pushes the highlights into areas of more details. As long as the noise is low enough, you're going to like the results. I've found I can do this up to ISO 1250. At 2500, you have to push the exposure to the right to avoid break up in the shadows, as we all know.

 

As for the highlights themselves--particularly red channel nuttiness, I haven't seen this in the M8 shots so far at all. But then, I try NOT to overexpose significant highlights. Specular highlights, such as they are, I could care less about--however it's possible that the lack of detail there is actually a bonus since the M8 blows out to a nice white much more agreeably than many other cameras!

[dfarkas]

Max,

 

I'd be curious how M8 DNGs compare to DMR DNGs. The DMR files are 20MB each. Are they uncompressed 16bit as we are led to believe?

 

Thanks and great article.

 

David

[Max Penson]

Very interesting. Thanks. What I do not understand is that you postulate problems with underexposed files, whereas practice shows that the M8 has the best shadow recovery of all digital cameras. Could you please explain what I see wrong here?

 

It's not that you won't see any details after shadow recovery, but it's that you'll have "jumps" in tonal range. It won't be smooth....

[Max Penson]

I think the key to the M8's spectacular performance here is noise.

 

The experiment done in the article, to predict the inability to recover shadows and lack of highlight detail in the M8, was done with a D30--hardly equivalent to an M8 in terms of the noise floor.

 

Yes--the image will break up quickly in the shadows *if the noise level* results in errors due to compression.

 

However, IMO, the incredible low-noise of the M8 sensor at low ISOs means you can actually underexpose, take advantage of the less-compressed 14-bit shadow detail, and easily recover in the RAW converter with spectacular results.

 

And underexposing slightly pushes the highlights into areas of more details. As long as the noise is low enough, you're going to like the results. I've found I can do this up to ISO 1250. At 2500, you have to push the exposure to the right to avoid break up in the shadows, as we all know.

 

As for the highlights themselves--particularly red channel nuttiness, I haven't seen this in the M8 shots so far at all. But then, I try NOT to overexpose significant highlights. Specular highlights, such as they are, I could care less about--however it's possible that the lack of detail there is actually a bonus since the M8 blows out to a nice white much more agreeably than many other cameras!

 

 

The problem is not with how low noise the sensor is, the problem is that you average noise without considering where the pixel is. So you may bland noise in the DC (flat) with noise in the AC (edges).

[Max Penson]

Max,

 

I'd be curious how M8 DNGs compare to DMR DNGs. The DMR files are 20MB each. Are they uncompressed 16bit as we are led to believe?

 

Thanks and great article.

 

David

Thanks

 

I don't have any RAW images from the DMR, would you like to send me a sample?

[dfarkas]

Thanks

 

I don't have any RAW images from the DMR, would you like to send me a sample?

 

Sure. You can grab a DNG here:

 

http://www.dalelabs.com/David/L9470705.dng

 

It was a shot taken during one of my lighting workshops with a borrowed 35-70 f/2.8. I only took one shot, but hey... beggars can't be choosers. I was there to teach, not shoot. I'm attaching the processed JPG file for referrence as well.

 

Thanks again Max. I look forward to your analysis.

 

David

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[Jamie Roberts]

Max, with all due respect, isn't the amount of noise directly related to the effective DR here?

 

IOW, with low noise, it doesn't matter if the results are averaged because there isn't much noise to begin with.

 

Besides, again with all due respect, my M8 apparently does none of the things your description suggests.

 

I find I can recover a lot of highlight information that is marked as blown by the histogram (of course, it's reading the JPEG, not the RAW file), but I just don't see a visual deficit in the highlights on the M8.

 

Quite the contrary--upper midtones are in fact among the best I've seen. I can also push the shadows in post and they are anything but posterised.

 

Instead they're almost uncannily smooth; something I'd expect from a less-compressed shadow range.

 

So I don't understand why absolute sensor noise wouldn't affect your predicted results. All I know is that I get much more usable DR from my M8 than from my DMR, 1ds2, or 5d.

 

But it's all very interesting! I just don't know how to understand the practical results from the the theory--which says there should be lots of information loss.

 

PS--lovely shot David! But of course the whole point of a shot like that is that you're working with controlled" (well, wrestled;)) light...

[Max Penson]

David, the file does not exist... could you check?

 

Thanks, lovely lighting...

 

Max, with all due respect, isn't the amount of noise directly related to the effective DR here?

 

IOW, with low noise, it doesn't matter if the results are averaged because there isn't much noise to begin with.

 

You are not wrong about and DR and noise, but the 8 bit thing has noting to do with the sensor. The fact is that when you take 12 or 14 bit of data, and map it to 8 bit, you average values globally. If the image would have a lot of fine detail or noise frequency (and all image do have noise), your value resolution is now not enough to express all those variations. So you have to round, up or down or by some kind of threshold. If your image is an high ISO image, you are now doing some amount of averaging, that averaging is not by where the pixel is, but what value it has.

 

So back to the M8, if the Noise reduction would have worked on the full 14 bit data the result in high ISO images might have been better. Well, we may never know how much better....

 

Besides, again with all due respect, my M8 apparently does none of the things your description suggests.

 

I find I can recover a lot of highlight information that is marked as blown by the histogram (of course, it's reading the JPEG, not the RAW file), but I just don't see a visual deficit in the highlights on the M8.

Please note the histograms on cameras usually start to mark pixels as blown from around 245 to 250...

 

 

 

But it's all very interesting! I just don't know how to understand the practical results from the the theory--which says there should be lots of information loss.

 

 

Well, our (humans) biggest problem is that we each see differently the same thing, where pixel values are always the same. I am not saying with this article that you will SEE all of those problems, but they are there. And from the image processing point of view, 8 bit non-linear RAW files are just wrong! I could have lived with 9 bits or so....

[Jamie Roberts]

Max, thanks for your explanation.

 

I completely agree that high ISO M8 images are quite brittle. No doubt about that whatsoever, and high ISO shooting is one reason I own a 5d still.

 

I suspect though that the tradeoff here might still be noise, but in a different sense.

 

The M8 sensor is less noisy at high ISOs than the DMR is (which uses an uncompressed DNG).

 

But even with the lack of compression in the DMR anything above ISO 800 is so noisy it's not worth processing.

 

Perhaps the M8's sensor is also so noisy above ISO 640 that there is nothing practical to be gained by greater (more accurate) bit depth?

 

Just wondering, again.

 

Practically, of course, the ISO 1250 M8 files are pretty good; not great, but pretty good.

 

I'd still like an uncompressed option (or a lossless compression option) for the M8. I hope they can do that someday.

 

As for post processing, I'd like to agree with you; it certainly seems correct: I like as much data as possible (within reason, of course)! But, again, at least with low-ISO shots I've found the M8 shots to be quite amenable to a lot of post processing, interestingly enough--even in areas that act like gradients, like skies, where you should see the problem clearly.

[Max Penson]

There is a little story I like to tell people about RAW. I used to use the Kodak 760 back then, fantastic camera, but had a lot of image processing problems. Then the 14n and the SLRn cameras released, kodak improved their image processing and the results from my 760 were good as new (The 760 was RAW only). Noise reduction was getting there, false colors were under control, color profiles were improved. The same old 760 improved with time! Can you imagine that?

 

The same is going to happen to the M8.... Adobe is working on new noise reduction algorithm, apple will improve, C1 will improve. Noise reduction in DIGIC III is far ahead from any raw software in the market, there is A LOT (believe me I know) to improve. One day the M8 at 1250ISO will show better results from DNG files. So, your only problem is that your going to be improved M8 DNG files may not take FULL advantage of those new noise reduction systems coming as it would have at full 14 bits. My 760 will....

[Jamie Roberts]

There is a little story I like to tell people about RAW. I used to use the Kodak 760 back then, fantastic camera, but had a lot of image processing problems. Then the 14n and the SLRn cameras released, kodak improved their image processing and the results from my 760 were good as new (The 760 was RAW only). Noise reduction was getting there, false colors were under control, color profiles were improved. The same old 760 improved with time! Can you imagine that?

 

The same is going to happen to the M8.... Adobe is working on new noise reduction algorithm, apple will improve, C1 will improve. Noise reduction in DIGIC III is far ahead from any raw software in the market, there is A LOT (believe me I know) to improve. One day the M8 at 1250ISO will show better results from DNG files. So, your only problem is that your going to be improved M8 DNG files may not take FULL advantage of those new noise reduction systems coming as it would have at full 14 bits. My 760 will....

 

Max--you're no doubt correct here. I saw images and prints from the Canon 1d3 (DIGIC III) at ISO 6400 and I was duly impressed as far as noise goes... "what noise?" I was thinking.

 

Ok, there was some. But not much, it's true.

 

And future processing is why I'd like to still have the option of an uncompressed file...

After all, I don't shoot RAW to throw data away

[dfarkas]

Max,

 

Sorry, try the same file location, but with the "dng" part capitalized. UNC paths are case sensitive....

 

http://www.dalelabs.com/David/L9470705.DNG

 

Try it again.

 

Thanks,

 

David

[Max Penson]

Max,

 

Sorry, try the same file location, but with the "dng" part capitalized. UNC paths are case sensitive....

 

http://www.dalelabs.com/David/L9470705.DNG

 

Try it again.

 

Thanks,

 

David

 

OK, checked, this file is not compressed in any way, and those seem to be full 16 values. But sometimes camera makers put 14 bit or 12 bit of data inside 16 bit, to the right side or the left side. So there is a small chance those are 14 bits shifted. I need a black image from a DMR to be sure. Although those are most likely be true 16 bits. If i'll had a white (overexposed) and a black image I could be really sure

 

Thanks for the image, the R lenses are really something else....

[Advertisement]

Hallo,

Look here: Leica M

[dfarkas]

OK, checked, this file is not compressed in any way, and those seem to be full 16 values. But sometimes camera makers put 14 bit or 12 bit of data inside 16 bit, to the right side or the left side. So there is a small chance those are 14 bits shifted. I need a black image from a DMR to be sure. Although those are most likely be true 16 bits. If i'll had a white (overexposed) and a black image I could be really sure

 

Thanks for the image, the R lenses are really something else....

 

I'll take a black frame and a white one and post the link here tomorrow when I get a chance.

 

Yeah, the R glass is quite something, isn't it?

 

Thanks again,

 

David