Jump to content
Sign in to follow this  
Julius Bjornsson

8-bit DNGs-how is this done?

Recommended Posts

Advertisement (gone after registration)

Mark - Many thanks for your concise explanation of the 14 Bit to 8 Bit mapping with the DNG file. I have struggled to understand this in the past, but it makes greater sense to me now.

 

................. Chris

Share this post


Link to post
Share on other sites
That's in the eye of the beholder. Highlight rendering is important to many of us.

 

Indeed it may be, but I'd suggest to you that reserving over 32000 different brightness levels for the highlights (let's asssume that's the areas of the scene in the brightest single EV value range) and only 2 for the lowest in not the best use of resources. I doubt that even a "professional photographer" can distinguish 32000 different light levels in those highlights. I know this "amateur photographer" cannot.

 

Instead, Leica's algorithm reduces the proportion of brightness levels used by the highlights which frees some up for use further down the EV scale.

 

So, every pixel in the camera is represented by one of 256 different light levels and for best image quality and best image gradation, as many of those levels should be used as possible; in exposure terms, that means exposing to the right up to the point of highlight clipping. If you leave a safety margin and under expose, you risk reducing the number of different brightness levels used to render your image. As Julius' graph shows, under-exposing by 1 f-stop will mean that a quarter of the available 256 light levels will never get used; under expose by two f-stops and only about a half of them will get used.

Share this post


Link to post
Share on other sites
Mark - Many thanks for your concise explanation of the 14 Bit to 8 Bit mapping with the DNG file. I have struggled to understand this in the past, but it makes greater sense to me now.

 

................. Chris

 

Thanks Chris!

Share this post


Link to post
Share on other sites
I belong to those that may not be not entirely happy with Leica's decision to use raw file compression.

 

That being said, Mark Nortons simple and beutiful explanation made me think whether it would be possible to keep the good part of Leicas compression program and leave out the rest, ie. to let the firmware do a partial compression of the dark tones (good for the shadows) and leave the brighter tones unchanged (from which they would, indeed benefit).

 

Is it possible for a programme specialist to find the relevant codes in Leica's firmware as someone did with the 6-bit coding table?

 

Would be fine the idea to put the hands in the original 14 bit map and managing it "as you like"... but undoable in practice, and not only for the obvious "legal" issues : as Mark has well explained, the 14-to-8 bit mapping is strongly embedded in the firmware : I have not direct experience on "photo" files, but years ago I was professionaly engaged in similar issues on audio files, and can tell by sure that is a VERY VERY delicate question: firstly, Leica has decided that its RAW IS DNG COMPLIANT, that means that your "in-house software" had to take into account this fact (otherwise, it would be almost impossible to manage the file into the final part of the electronics pipeline, i.e., to be WRITTEN in the Card or delivered via USB...). I suspect thet if one would make all the work to obtain a DNG compliant file... well....the result would be no better than Leica's. Second : you could even find the way to manage a non-DNG file with some trick, but (99% sure) you couldn't definitely manage a more-than-8 bit file: this many times becomes infact also an hardware issue: the data flow from processor to i/o units (memory card and USB) is very probably strictly 8 bit: to conclude... if you haven't the capability to manage more than 8 bit, I'll simply would say that working on the firmware probably cannot give you anything more than you can do playing on exposure setting and postprocessing Software.

Share this post


Link to post
Share on other sites
Indeed it may be, but I'd suggest to you that reserving over 32000 different brightness levels for the highlights (let's asssume that's the areas of the scene in the brightest single EV value range) and only 2 for the lowest in not the best use of resources. I doubt that even a "professional photographer" can distinguish 32000 different light levels in those highlights. I know this "amateur photographer" cannot.

 

Instead, Leica's algorithm reduces the proportion of brightness levels used by the highlights which frees some up for use further down the EV scale.

 

So, every pixel in the camera is represented by one of 256 different light levels and for best image quality and best image gradation, as many of those levels should be used as possible; in exposure terms, that means exposing to the right up to the point of highlight clipping. If you leave a safety margin and under expose, you risk reducing the number of different brightness levels used to render your image. As Julius' graph shows, under-exposing by 1 f-stop will mean that a quarter of the available 256 light levels will never get used; under expose by two f-stops and only about a half of them will get used.

 

Fascinating in theory. In practice, however, the M8 is more abrupt in the highlights than other cameras I use, such as the Canon 5D. I know this because I routinely convert thousands of pictures made, with both cameras, of the same event. And, yes, I do indeed know how to expose for the highlights.

Share this post


Link to post
Share on other sites

Thanks everyone for the very informative discussion. I did not know what I was stepping into when I asked the (I thought) simple question at the beginning of the thread. But this is great and I have certainly learned a lot. Keep the discussion going.

 

Thanks again.

 

JKB

Share this post


Link to post
Share on other sites

Advertisement (gone after registration)

That's in the eye of the beholder. Highlight rendering is important to many of us.

 

Agreed. Highlight blowout is a big, big problem (particularly with a camera whose meter does not see small extremely bright areas). Having the gradation info in at least one channel helps immensely with pulling things back to the left. It would also help with manually-metered shots. Recovery from M8 DNGs doesn't really seem to work that well.

Share this post


Link to post
Share on other sites

It seems to me that there is a degree of confusion in this thread.

 

1) Some contributors, me included, think that the highlight rendering in Leica M8 DNG files is less flexible than it should be resulting in difficulties, such as posterisation and banding, when some images are modified in programs like CS3.

 

2) Others argue that there is no problem and to support this they point out that even after the “equalisation” applied by the Leica DNG 8-Bit compression / decompression the highlight region still has more “information” to work with than the other, darker, parts of the image.

 

3) There is no clarity regarding issues such as whether the data taken from the sensor is actually 16 Bit or, as some imply, actually only 14 Bit and the sensor’s dynamic range seems to vary from 10EV to 14EV depending on the data being presented.

 

4) There are some highly simplified explanations of how the “14 Bit to 8 Bit” transformation, (compression), is carried out. If one takes these at face value they do not yield the same results as authoritative people such as Michael J. Hußmann have published.

 

5) Some argue that underexposure is “throwing away” data at the rate which would pertain if one were using 16 Bit linear files, but we are not – that is the whole point of the equalisation applied in the 8 Bit conversion.

 

What are we who believe there is an issue here to make of all this?

 

Firstly I fully accept that it makes little sense to devote half of the available signal levels to just the brightest EV value, whether this is half of a 16, 14 or 8 Bit signal makes no difference. The question posed was whether if this data was retained at the editing and modification point in the image management work flow the problems some of us encounter with some images might be eliminated. There is as yet no satisfactory answer to this question though many have pointed out the considerable implications and problems that the retention of such data would have in practice.

 

If one accepts that higher resolution data, i.e. data which allows the intensity level of each pixel to be determined more precisely, (which is not the same as differentiating between intensity levels!), is not a factor in the performance of the Leica DNG files then what is?

 

Michael J. Hußmann’s graph demonstrates that the resolution of the output file data at the various signal levels varies by a factor of at least 4 Bits, is this what some of us are detecting?

 

I’ve started setting the camera permanently on -1/3 EV and it seems to be giving some benefit. Does this imply that if the metering were more accurate and reflected the true sensitivity of the sensor, which seems to be 200ISO rather than the stated 160ISO, that it would diminish the highlight rendering problems? We don’t know and nobody with the appropriate technical knowledge and expertise seems prepared to engage in this topic.

Share this post


Link to post
Share on other sites
We don’t know and nobody with the appropriate technical knowledge and expertise seems prepared to engage in this topic.

 

Peter, Michael has written up 4 or 5 (I can't remember exactly) full pages about this in the Feb./2007 issue of LFI magazine. Just buy one from their web site and all your questions will get answered.

Share this post


Link to post
Share on other sites

Given the power of the .dng file to quickly map 14 bits into 8 bits giving attention in those bits to the most perceptually significant part of the image, I have to wonder why the table wasn't drawn to to populate the .dng with the logarithm of the original pixel electron counts. It's just a table after all, and only gets computed when the firmware is compiiled. A logarithmic encoding gives the same number of bits to every "zone" in the image. A slightly better code would not try to generate bits that aren't there in the 14-bit image, so would still give more bits to the highlights than to the shadows. I bet Leica tried this, and concluded that the easier-to-explain square root code also gave flexibility to rescue and reconstruct highlight contrasts that was more important to most users than comparable shadow recovery.

 

scott

Share this post


Link to post
Share on other sites
Peter, Michael has written up 4 or 5 (I can't remember exactly) full pages about this in the Feb./2007 issue of LFI magazine. Just buy one from their web site and all your questions will get answered.

 

Michael says there is no problem and that those of us who think there is are deluded - that does not help or advance the debate.

 

If all the answers are in LFI, something I gave up years ago as it was pure marketing puff with just the occasional technical tit-bit, can someone please post a brief summary.

 

Come to think of it; why do people like Sean think there is an issue and continue to seek an explanation when they are totally familiar with LFI's content?

Share this post


Link to post
Share on other sites
Michael says there is no problem and that those of us who think there is are deluded - that does not help or advance the debate.

 

In fact, Michael has given analysis from both the pro and con sides ... and his conclusion is very well balanced. The original article is absolutely worth a careful read. You only have to buy one single issue instead of paying annual subscription anyways.

Share this post


Link to post
Share on other sites
Fascinating in theory. In practice, however, the M8 is more abrupt in the highlights than other cameras I use, such as the Canon 5D. I know this because I routinely convert thousands of pictures made, with both cameras, of the same event. And, yes, I do indeed know how to expose for the highlights.

 

Sean, as you know, I also do the same thing, and there are two points I want to make...

 

First, and FWIW, I much prefer the way the M8 blows highlights out to the way Canon (in C1, anyway) seems to "band out" in extreme highlights (we're talking on-purpose overexposure here).

 

Next, I'm not at all sure that the compression scheme has anything to do with the fact that beyond a certain point the 5d's files seem a touch more recoverable in the highlights.

 

Instead, I think it may just be that the red channel is a wee bit more sensitive in the m8. I'm not sure why, but often I find the red channel on the m8 clipped when an 'identical' pushed to the right exposure on the 5d wouldn't be.

 

I wonder why that is? Is it just good old extra IR sensitivity?

Share this post


Link to post
Share on other sites
{snipped}

I’ve started setting the camera permanently on -1/3 EV and it seems to be giving some benefit. Does this imply that if the metering were more accurate and reflected the true sensitivity of the sensor, which seems to be 200ISO rather than the stated 160ISO, that it would diminish the highlight rendering problems? We don’t know and nobody with the appropriate technical knowledge and expertise seems prepared to engage in this topic.

 

I think, practically speaking, that if the EV compensation works for you, then use it. However, I've found no such "benefit" to properly exposed shots, and the meter doesn't "overexpose" in my opinion.

 

If you want to retain highlight detail (not speculars--which should blow to 255/255/255) then by all means treat this as slide film and expose appropriately.

 

But you are losing shadow detail then. Just like film

 

And when you then switch to high ISOs, and you lose another stop in the shadows due to noise, and you put your upper midtones around 130/130/130, well, don't be surprised if it looks really, really bad

 

I personally don't think the compression scheme has anything to do with lost highlights, but that's just me. I don't see a significant difference between DMR and M8 shots in this regard (shot at the same event) and the DMR is completely uncompressed 16/14bpp AD.

Share this post


Link to post
Share on other sites

Sorry that you think the contributions here are worthless Peter, but we are necessarily working without all the information available to us. Given open access, I'm quite sure Carsten, Carl, Scott and Sandy would be able to enlighten us fully.

 

If you dig into a DNG, you can find the linearization table, which is used to convert the 8 bit DNG values back to 14 bit pixel values.

 

I'll publish the full table just as soon as I have time to write a program to extract the full data. In the example I looked at, DNG values of 0 and 1 are converted back to 0, suggesting at these low levels, they can't see the signal for the noise, so just blank it out. A DNG value of 255, representing the brightest level, has a value of 3F80 which has a decimal value of 16256.

 

A full 14 bit system would yield data in the range 0-16383 and I expect the difference is accounted for by the need to provide headroom for black reference level correction or maybe the hardware simply falls apart as it saturates.

 

Further, if you browse Analog Devices chip range, you'll find 14 bit analog to digital converters which can handle the required data rate but not 16 bit. All in all, I think the Leica M8 is a 14 bit system, not a 12 or 16 bit one.

Share this post


Link to post
Share on other sites

Sorry, Peter, but I do not see confusion, but simply different views of a BASIC problem that is intrinsic to the DNG 8-bit color table, which, in turn, is a specific case of the generalized problem of analog-to-digital conversion, that is present in any digital information acquired from a sinusoidal source (be it wavelenghts of light or wavelenghts of sounds). For people who have some basic knowledge of the math that stays behind this issue, things are rather simply to understand: anyway, I think that anyone can understand that if you decide to assign 256 (and no more) values to the (theorically) infinte values of light levels of a scene you have to make some CHOICE, and choices can be always questionable... if you would have to assign not 256 but 65536 values (16 bit) your choices are maybe more difficult to be tuned and computed, but are SURELY less questionable. Leica's choice (dealing with a 8 bit DNG map) has been to privilege the details in shadows: this is sure and well illustrated in the graph we saw hereby and in the LFI article. This means that if you have a scene with some highlight areas you would like to have rendered at best, a certain underexposure can be of help (your rule of "-1/3 EV" can be a good one, indeed); and the fact that the 8-bit is remapped into a 16 bit space means that you COULD, after, "regain" details in highlights in post-processing... but this is theory, for it depends on HOW the Software works...

BTW, Leica choice and the above (theorical, repeat) capability to regain higlights detail in PP, resembles to me an old rule of some famous photographer of the past :

"Expose for the shadows, develop for the highlights" (Weston ? I'm not sure)

I think their choice for M8 has not been wrong and... I'm very curious to see if and how and when they will implement a real 16 bit (or 14) processing.

Share this post


Link to post
Share on other sites
I bet Leica tried this, and concluded that the easier-to-explain square root code also gave flexibility to rescue and reconstruct highlight contrasts that was more important to most users than comparable shadow recovery.

 

scott

 

AGREE, this is another way to expose the same conclusion I made in my above post...they have followed the rule of "expose for the shadow, develop for the highlights"

Share this post


Link to post
Share on other sites

 

If you want to retain highlight detail (not speculars--which should blow to 255/255/255) then by all means treat this as slide film and expose appropriately.

 

 

I've done some testing for myself as a way to better understand what "proper exposure" means for this camera and for the lens I use and I've come to the conclusion that a -1/3 EV setting can give the impression of helping prevent a blown highlight but I think that this comes with a risk of having less tolerance to overall latitude.

 

I'm not an engineer, so I'm reluctant to use specific terms for fear of using them incorrectly, but what I'm finding is that adjusting the EV is really adjusting the gain of the sensor and a change in gain appears to result in a change in the tolerance of the overall curve.

 

So, by setting the camera to -1/3 EV I see a sharper differentiation in gradation from highlights to midtones. This is more pronounced as I adjust EV down.

 

I think it's also important to note the I live in Southern California, the sun here is generally very bright and there is very little cloud cover, so the separation between highlight and shadow can sometimes be very severe.

Share this post


Link to post
Share on other sites
I’ve started setting the camera permanently on -1/3 EV and it seems to be giving some benefit. Does this imply that if the metering were more accurate and reflected the true sensitivity of the sensor, which seems to be 200ISO rather than the stated 160ISO, that it would diminish the highlight rendering problems?

I would say the metering is actually accurate, only that doesn’t imply it will necessarily lead to optimum results. Center-weighted average metering (or in fact any kind of average metering) works on the assumption that the average reflectivity in the scene will be equivalent to a middle, typically 18 percent gray. The camera will then settle for a shutter speed that ensures that a middle gray in the scene will get rendered as a middle gray in the image.

 

Unfortunately, in practice you don’t really care about middle gray being reproduced as middle gray. Rather, you want the contrast within the scene to be preserved, i.e. you want the camera to choose a shutter speed that prevents highlights from hitting the sensor’s saturation level and shadow detail from drowning in noise. Only average metering, by its very nature, doesn’t know about contrast, highlights, or shadows – all of that gets lost in the averaging. If you are fortunate, then highlights and shadows will observe an equal distance from middle gray and the camera’s average metering will do the right thing. But more often than not, there will be no such neat symmetry in the tonal values and the images will turn out under- or overexposed.

 

Your standard -1/3 EV correction setting ensures that the camera will tend to err on the side of underexposure. This is a useful strategy insofar as with sensors, there is a graceful degradation of image quality with underexposure whereas overexposure leads to clipping – noisy shadows can be recovered to some extent, while clipped highlights are gone for good. Still, there’s no alternative to getting the exposure exactly right, and the only way to make sure is to check the histogram.

Share this post


Link to post
Share on other sites
I would say the metering is actually accurate, only that doesn’t imply it will necessarily lead to optimum results. Center-weighted average metering (or in fact any kind of average metering) works on the assumption that the average reflectivity in the scene will be equivalent to a middle, typically 18 percent gray. The camera will then settle for a shutter speed that ensures that a middle gray in the scene will get rendered as a middle gray in the image.

{snipped}.

 

Precisely and well put.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...