Jump to content

8-bit DNGs-how is this done?


Recommended Posts

Advertisement (gone after registration)

HI all. I just stumbled across an explanation of how Leica manage to keep the DNG size in the M8 down to 10 megs. I had been wondering about this as I had calculated that the file should be twice this size. So I assumed lossless compression, but have now found out that the camera downsamples to 8 bits before storing the data. Now this surprised me, to put it mildly. My small D-lux3 saves raw files that are 20 megs, and it is also 10 megapixels.

 

Now let me say before going further that I am extremely happy with tonality, (dynamic range compared to other digitals I have used-mostly Nikons) to say nothing about the sharpness, contrast etc which is the best I have used so far. Let me repeat I am extremely happy with the M8 but would like to understand this.

 

I still do not understand why Leica decided to do this downsampling. Memory is cheap,prices going down all the time and I would think that adding some SD cards would be a small issue for us (relatively seen compared to leica prices). But the basic question is, are we loosing some quality because of this? How would the files look if we got the whole thing? And the downsampling, what method are they using, as the photos manage to keep up to 7 stops of dynamic range they must be doing something very special before going down to 8 bits. The brochure from Leica states:

 

"DNG™ file information 16 bit-color resolution, 10.2 Mbyte file size per picture"

 

How on earth can we get 16-bit color resolution when the data is downsampled to 8 bits and a 10 megapixel color file is only 10 megabytes?

 

Please someone explain this to me, or point out some material to read, as I really want to understand this.

 

Sorry for the long post, but this issue has been bothering me a little bit.

 

J

Link to post
Share on other sites

First, the individual samples are not 16 bit, they are at most 14 bit, so take the value of 0 - 16383. To map to an 8 bit space, they use a table in the firmware to map each value in this range to a value in the range 0 - 255. The DNG includes a table of 256 values to map them back to a 16 bit space.

 

Roughly, this is equivalent to multiplying by 4 and taking the square root. So, it's not a linear mapping, giving greater emphasis (more of those 256 values) over to lower sample values, that is, increased precision in the shadows.

 

Intuitively, it sounds all wrong to do this but if a full 16 bit linear representation was used, half of the available 65536 values would be used to represent brightness in the brightest f-stop of a 16 bit dynamic range where it really has no sensible use and only 1 value in the darkest f-stop. If your goal is to be able to distinguish between different brightness levels, spreading the 256 values in this way increases the precision towards the darker areas where it's required.

 

That said, many of us are not entirely happy with this being imposed on us and would like an option which is not so aggressive. We'd like an option to generate unmapped DNGs so that we, not Leica, can make the decision which is best.

 

By halving the file size, the camera benefits from better storage usage, faster write times and longer battery life.

Link to post
Share on other sites

Thanks you for these explanations, I have ordered LFI but have not yet gotten it so I will try to be patient and wait for the issue containing the explanation. But I looked at the numbers and I think I understand the conversion, and the nonlinearity of the correspondance from 14 bits to 8. I made up a graph to help me visualize this and it is attached here. this shows clearly how the information is distributed in the conversion and why it might be ok to throw away information from the top with the help of a conversion table. But I will wait for the article and if I am still in the dark after that, I will post about this again.

 

But thanks again for pointing me in the right direction.

 

 

J

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

Link to post
Share on other sites

It's also interesting to plot this graph with the "input" log to base of 2 which then shows you how many of the 256 values are assigned to each successive f-stop of exposure. As it is, you can see that, roughly, 40% of the 256 values are used to represent the darkest 16% of light levels, but only 20% of the 256 values are used to represent the the brightest 40% of light levels.

Link to post
Share on other sites

One side effect of the non-linear compression would be that there's less benefit in 'exposing to the right' with the M8 compared to other digital platforms. There should be less need to try to cram the image into the brighter tonal range to take advantage of the increased sampling bit density since more data is retained in the darker levels.

 

Nice - I think this maps RF photographic reality better when you consider how often you'll be lifting an image's brightness vs toning it down from over exposure.

Link to post
Share on other sites

Advertisement (gone after registration)

I belong to those that may not be not entirely happy with Leica's decision to use raw file compression.

 

That being said, Mark Nortons simple and beutiful explanation made me think whether it would be possible to keep the good part of Leicas compression program and leave out the rest, ie. to let the firmware do a partial compression of the dark tones (good for the shadows) and leave the brighter tones unchanged (from which they would, indeed benefit).

 

Is it possible for a programme specialist to find the relevant codes in Leica's firmware as someone did with the 6-bit coding table?

Link to post
Share on other sites

Julius--

Your graph is similar to the one in LFI, although differently plotted.

 

There was a horrendously long thread on the topic as we gradually came to the conclusion that these were indeed 8-bit files.

 

In addition, Leica was the first company to implement one of the two versions of DNG allowed by the DNG specification. That's one reason it took so long for some RAW converters to be able to work with the M8 file.

 

As a matter of fact, the M8 writes the same look-up table into each file just because of the DNG specification. The files could be made slightly smaller if the camera didn't go through the step of making them DNG compatible.

 

Very interesting and very sophisticated, gives very good results--but still leaves a lot of us wondering if a full, uncompressed file wouldn't be preferable.

 

--HC

Link to post
Share on other sites

Personally, I have had no trouble with highlights from M8 files...

 

But, in theory, if one begins making extensive tonal changes that affect highlights in Photoshop, after the RAW has already been converted, or if the original is a jpeg - then the M8 files might begin to show posterization and banding in the brightest areas.

 

I got banding from manipulation of DMR jpegs in a scene where a white wall was sidelit by a window, and also partially shaded by a bookcase, such that there were very subtle shadings of near-white to gray across the wall. Even a fairly slight adjustment of overall image brightness created arcing bands around the "hot spot" on the wall that were quite visible in prints.

 

Not quite as blatant as the sunset "rainbow posterization" that showed up in the original Viking pictures from the surface of Mars - http://nssdc.gsfc.nasa.gov/imgcat/midres/vl1_12a240.gif - but a similar effect.

Link to post
Share on other sites

Personally, I have had no trouble with highlights from M8 files...

 

But, in theory, if one begins making extensive tonal changes that affect highlights in Photoshop, after the RAW has already been converted, or if the original is a jpeg - then the M8 files might begin to show posterization and banding in the brightest areas.

 

I got banding from manipulation of DMR jpegs in a scene where a white wall was sidelit by a window, and also partially shaded by a bookcase, such that there were very subtle shadings of near-white to gray across the wall. Even a fairly slight adjustment of overall image brightness created arcing bands around the "hot spot" on the wall that were quite visible in prints.

 

Not quite as blatant as the sunset "rainbow posterization" that showed up in the original Viking pictures from the surface of Mars - http://nssdc.gsfc.nasa.gov/imgcat/midres/vl1_12a240.gif - but a similar effect.

 

Andy,

 

I don't think it is my imagination but this effect does not seem to happen to the same extent if you convert DNG's to 16 bit TIFF's rather than 8 bit ones (I use C1). This is the main reason I upgraded from PSE4 on my Macs to CS3, so that I would have the full range of manipulative tools for 16 bit files - on PSE4 only about half the tools/filters work on 16 bit files and to get all them to work, you have to downgrade the mode to 8 bit.

 

Wilson

Link to post
Share on other sites

It is not your imagination, Wilson. I find the same. PSE5 has 16 bit compatability for those filters that need it. I suppose they cannot make it fully 16-bit without eating into the sales of their more expensive products.....

Link to post
Share on other sites

Andy,

 

I don't think it is my imagination but this effect does not seem to happen to the same extent if you convert DNG's to 16 bit TIFF's rather than 8 bit ones (I use C1). This is the main reason I upgraded from PSE4 on my Macs to CS3, so that I would have the full range of manipulative tools for 16 bit files - on PSE4 only about half the tools/filters work on 16 bit files and to get all them to work, you have to downgrade the mode to 8 bit.

 

Wilson

 

I agree Wilson--unless you are going to work in 16bit space for editing images, I cant see much point in shooting raw in the first place, given the tricks Photoshop can perform with a basic jpeg or tiff...

 

Best,

Link to post
Share on other sites

One irritating thing is that CS3 seems to be a single machine licence. Previously you could install Adobe products on all your owned machines. I have three Macs, an iBook, which travels around with me, an iMac in France and a Powermac in the UK. I only ever use one at a time, so I don't really see why I should not have CS3 in all of them for one purchase. Given CS3's high price, it would be a very expensive exercise to install CS3 in all of them. When I am traveling, I now try to resist the urge to fiddle with photos on my iBook as the screen does not hold its calibration and is very ambient condition dependent. I have in the past, fiddled with photos, sitting in my hotel in the evening and once I get back home and look on a calibrated screen at what I have perpetrated on my iBook, it is OMG! At least I now always have the DNG's to fall back on and start again.

 

Wilson

Link to post
Share on other sites

One irritating thing is that CS3 seems to be a single machine licence. Previously you could install Adobe products on all your owned machines. I have three Macs, an iBook, which travels around with me, an iMac in France and a Powermac in the UK.

 

Wilson

 

Wilson,

 

I have just upgraded my CS2 to CS3 on my PC and as before I have repeated this process on my laptop. There has been no problem activating both installations. It is my understanding that Adobe allows 2 installations on a two machine setup such as mine. Whether this extends to your multi unit system I'm not sure; no doubt someone else has the answer.

 

Tim

Link to post
Share on other sites

Guest guy_mancuso

Your allowed two installs and if you try on a third machine it won't work unless you deactivate one of the other two machines. Just did this on a new Mac laptop and the same is true for Quark. i forgot to do this when I installed on my new machine and had to go into Adobe and deactivate the old machine and Quark I had to call. I wiped the second machine out so did not have that chance. may want to remember this if you get a new machine but two installs is the Max.

Link to post
Share on other sites

I agree Wilson--unless you are going to work in 16bit space for editing images, I cant see much point in shooting raw in the first place, given the tricks Photoshop can perform with a basic jpeg or tiff...

 

I think that's the point, many people (ok, me at least :-) do all the editing in 16 bit and then convert to 8 bit for web output.

Link to post
Share on other sites

That is good news for me if I can do two installs of CS3. The wording of Adobe's licence was in "lawyerspeak" and it seemed to imply if you wanted to install it on a second machine, you had to uninstall it on the first one. I think I will not bother on my iBook, which even though it has max memory of 768K installed and an upgraded 1.3 Ghz G4 chip, would struggle. I do however want it on both my Duo-Core2 iMac and Dual 2.3 G5 PowerMac machines. It will also remove the temptation to 'fiddle' with images on the iBook.

 

Wilson

Link to post
Share on other sites

...but if a full 16 bit linear representation was used, half of the available 65536 values would be used to brightness in the brightest f-stop of a 16 bit dynamic range where it really has no sensible use...

 

That's in the eye of the beholder. Highlight rendering is important to many of us.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...