8-bit DNGs-how is this done?

[Julius Bjornsson]

HI all. I just stumbled across an explanation of how Leica manage to keep the DNG size in the M8 down to 10 megs. I had been wondering about this as I had calculated that the file should be twice this size. So I assumed lossless compression, but have now found out that the camera downsamples to 8 bits before storing the data. Now this surprised me, to put it mildly. My small D-lux3 saves raw files that are 20 megs, and it is also 10 megapixels.

Now let me say before going further that I am extremely happy with tonality, (dynamic range compared to other digitals I have used-mostly Nikons) to say nothing about the sharpness, contrast etc which is the best I have used so far. Let me repeat I am extremely happy with the M8 but would like to understand this.

I still do not understand why Leica decided to do this downsampling. Memory is cheap,prices going down all the time and I would think that adding some SD cards would be a small issue for us (relatively seen compared to leica prices). But the basic question is, are we loosing some quality because of this? How would the files look if we got the whole thing? And the downsampling, what method are they using, as the photos manage to keep up to 7 stops of dynamic range they must be doing something very special before going down to 8 bits. The brochure from Leica states:

"DNG™ file information 16 bit-color resolution, 10.2 Mbyte file size per picture"

How on earth can we get 16-bit color resolution when the data is downsampled to 8 bits and a 10 megapixel color file is only 10 megabytes?

Please someone explain this to me, or point out some material to read, as I really want to understand this.

Sorry for the long post, but this issue has been bothering me a little bit.

J

[marknorton]

First, the individual samples are not 16 bit, they are at most 14 bit, so take the value of 0 - 16383. To map to an 8 bit space, they use a table in the firmware to map each value in this range to a value in the range 0 - 255. The DNG includes a table of 256 values to map them back to a 16 bit space.

Roughly, this is equivalent to multiplying by 4 and taking the square root. So, it's not a linear mapping, giving greater emphasis (more of those 256 values) over to lower sample values, that is, increased precision in the shadows.

Intuitively, it sounds all wrong to do this but if a full 16 bit linear representation was used, half of the available 65536 values would be used to represent brightness in the brightest f-stop of a 16 bit dynamic range where it really has no sensible use and only 1 value in the darkest f-stop. If your goal is to be able to distinguish between different brightness levels, spreading the 256 values in this way increases the precision towards the darker areas where it's required.

That said, many of us are not entirely happy with this being imposed on us and would like an option which is not so aggressive. We'd like an option to generate unmapped DNGs so that we, not Leica, can make the decision which is best.

By halving the file size, the camera benefits from better storage usage, faster write times and longer battery life.

[jaapv]

As for literature: the 2007/1 issue of LFI has an extensive article on this.

[Julius Bjornsson]

Thanks you for these explanations, I have ordered LFI but have not yet gotten it so I will try to be patient and wait for the issue containing the explanation. But I looked at the numbers and I think I understand the conversion, and the nonlinearity of the correspondance from 14 bits to 8. I made up a graph to help me visualize this and it is attached here. this shows clearly how the information is distributed in the conversion and why it might be ok to throw away information from the top with the help of a conversion table. But I will wait for the article and if I am still in the dark after that, I will post about this again.

But thanks again for pointing me in the right direction.


J

[marknorton]

It's also interesting to plot this graph with the "input" log to base of 2 which then shows you how many of the 256 values are assigned to each successive f-stop of exposure. As it is, you can see that, roughly, 40% of the 256 values are used to represent the darkest 16% of light levels, but only 20% of the 256 values are used to represent the the brightest 40% of light levels.

[gwelland]

One side effect of the non-linear compression would be that there's less benefit in 'exposing to the right' with the M8 compared to other digital platforms. There should be less need to try to cram the image into the brighter tonal range to take advantage of the increased sampling bit density since more data is retained in the darker levels.

Nice - I think this maps RF photographic reality better when you consider how often you'll be lifting an image's brightness vs toning it down from over exposure.

[michael_b_elmer]

I belong to those that may not be not entirely happy with Leica's decision to use raw file compression.

That being said, Mark Nortons simple and beutiful explanation made me think whether it would be possible to keep the good part of Leicas compression program and leave out the rest, ie. to let the firmware do a partial compression of the dark tones (good for the shadows) and leave the brighter tones unchanged (from which they would, indeed benefit).

Is it possible for a programme specialist to find the relevant codes in Leica's firmware as someone did with the 6-bit coding table?

[ho_co]

Julius--
Your graph is similar to the one in LFI, although differently plotted.

There was a horrendously long thread on the topic as we gradually came to the conclusion that these were indeed 8-bit files.

In addition, Leica was the first company to implement one of the two versions of DNG allowed by the DNG specification. That's one reason it took so long for some RAW converters to be able to work with the M8 file.

As a matter of fact, the M8 writes the same look-up table into each file just because of the DNG specification. The files could be made slightly smaller if the camera didn't go through the step of making them DNG compatible.

Very interesting and very sophisticated, gives very good results--but still leaves a lot of us wondering if a full, uncompressed file wouldn't be preferable.

--HC

[adan]

Personally, I have had no trouble with highlights from M8 files...

But, in theory, if one begins making extensive tonal changes that affect highlights in Photoshop, after the RAW has already been converted, or if the original is a jpeg - then the M8 files might begin to show posterization and banding in the brightest areas.

I got banding from manipulation of DMR jpegs in a scene where a white wall was sidelit by a window, and also partially shaded by a bookcase, such that there were very subtle shadings of near-white to gray across the wall. Even a fairly slight adjustment of overall image brightness created arcing bands around the "hot spot" on the wall that were quite visible in prints.

Not quite as blatant as the sunset "rainbow posterization" that showed up in the original Viking pictures from the surface of Mars - http://nssdc.gsfc.nasa.gov/imgcat/midres/vl1_12a240.gif - but a similar effect.

[wlaidlaw]

Originally Posted by adan

Personally, I have had no trouble with highlights from M8 files...

But, in theory, if one begins making extensive tonal changes that affect highlights in Photoshop, after the RAW has already been converted, or if the original is a jpeg - then the M8 files might begin to show posterization and banding in the brightest areas.

I got banding from manipulation of DMR jpegs in a scene where a white wall was sidelit by a window, and also partially shaded by a bookcase, such that there were very subtle shadings of near-white to gray across the wall. Even a fairly slight adjustment of overall image brightness created arcing bands around the "hot spot" on the wall that were quite visible in prints.

Not quite as blatant as the sunset "rainbow posterization" that showed up in the original Viking pictures from the surface of Mars - http://nssdc.gsfc.nasa.gov/imgcat/midres/vl1_12a240.gif - but a similar effect.

Andy,

I don't think it is my imagination but this effect does not seem to happen to the same extent if you convert DNG's to 16 bit TIFF's rather than 8 bit ones (I use C1). This is the main reason I upgraded from PSE4 on my Macs to CS3, so that I would have the full range of manipulative tools for 16 bit files - on PSE4 only about half the tools/filters work on 16 bit files and to get all them to work, you have to downgrade the mode to 8 bit.

Wilson

[jaapv]

It is not your imagination, Wilson. I find the same. PSE5 has 16 bit compatability for those filters that need it. I suppose they cannot make it fully 16-bit without eating into the sales of their more expensive products.....

[Piers]

Originally Posted by wlaidlaw

Andy,

I don't think it is my imagination but this effect does not seem to happen to the same extent if you convert DNG's to 16 bit TIFF's rather than 8 bit ones (I use C1). This is the main reason I upgraded from PSE4 on my Macs to CS3, so that I would have the full range of manipulative tools for 16 bit files - on PSE4 only about half the tools/filters work on 16 bit files and to get all them to work, you have to downgrade the mode to 8 bit.

Wilson

I agree Wilson--unless you are going to work in 16bit space for editing images, I cant see much point in shooting raw in the first place, given the tricks Photoshop can perform with a basic jpeg or tiff...

Best,

[wlaidlaw]

One irritating thing is that CS3 seems to be a single machine licence. Previously you could install Adobe products on all your owned machines. I have three Macs, an iBook, which travels around with me, an iMac in France and a Powermac in the UK. I only ever use one at a time, so I don't really see why I should not have CS3 in all of them for one purchase. Given CS3's high price, it would be a very expensive exercise to install CS3 in all of them. When I am traveling, I now try to resist the urge to fiddle with photos on my iBook as the screen does not hold its calibration and is very ambient condition dependent. I have in the past, fiddled with photos, sitting in my hotel in the evening and once I get back home and look on a calibrated screen at what I have perpetrated on my iBook, it is OMG! At least I now always have the DNG's to fall back on and start again.

Wilson

[Tim B]

Originally Posted by wlaidlaw

One irritating thing is that CS3 seems to be a single machine licence. Previously you could install Adobe products on all your owned machines. I have three Macs, an iBook, which travels around with me, an iMac in France and a Powermac in the UK.

Wilson

Wilson,

I have just upgraded my CS2 to CS3 on my PC and as before I have repeated this process on my laptop. There has been no problem activating both installations. It is my understanding that Adobe allows 2 installations on a two machine setup such as mine. Whether this extends to your multi unit system I'm not sure; no doubt someone else has the answer.

Tim

[stunsworth]

Tim, it's certainly at least two as I have CS3 on a PC laptop and desktop.

[guy_mancuso]

Your allowed two installs and if you try on a third machine it won't work unless you deactivate one of the other two machines. Just did this on a new Mac laptop and the same is true for Quark. i forgot to do this when I installed on my new machine and had to go into Adobe and deactivate the old machine and Quark I had to call. I wiped the second machine out so did not have that chance. may want to remember this if you get a new machine but two installs is the Max.

[stunsworth]

Originally Posted by Piers

I agree Wilson--unless you are going to work in 16bit space for editing images, I cant see much point in shooting raw in the first place, given the tricks Photoshop can perform with a basic jpeg or tiff...

I think that's the point, many people (ok, me at least :-) do all the editing in 16 bit and then convert to 8 bit for web output.

[stunsworth]

So if Wilson de-activates the machine in the UK he'll be able to use both of the French ones. Then if he de-activates the French one before returning to the UK he'll be able to re-activate the UK one. Bingo!

[wlaidlaw]

That is good news for me if I can do two installs of CS3. The wording of Adobe's licence was in "lawyerspeak" and it seemed to imply if you wanted to install it on a second machine, you had to uninstall it on the first one. I think I will not bother on my iBook, which even though it has max memory of 768K installed and an upgraded 1.3 Ghz G4 chip, would struggle. I do however want it on both my Duo-Core2 iMac and Dual 2.3 G5 PowerMac machines. It will also remove the temptation to 'fiddle' with images on the iBook.

Wilson

[sean_reid]

Originally Posted by marknorton

...but if a full 16 bit linear representation was used, half of the available 65536 values would be used to brightness in the brightest f-stop of a 16 bit dynamic range where it really has no sensible use...

That's in the eye of the beholder. Highlight rendering is important to many of us.