DNG compressed!

[robofc]

Advertisement (gone after registration)

How do you set Lightroom for lossless compression?

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[lars_bergquist]

Considering the loss negligible is nonsense. To coin an old cliche, one cannot be slightly pregnant [ ... ]

 

In this case, the appropriate metaphor would be "one cannot be slightly poor". And that is of course nonsense, just as "one cannot be slightly well off".

 

Pregnant/non pregnant is a binary proposition. Tertium non datur. Rich/poor is not binary. This means that gradations matter. They do, in photography too.

 

If it doesn't matter that you can't see it, I wonder why you bother to print your images. Should it not be enough to just know that they exist, in all their pristine 14 bit glory, in Platonic cyberspace?

 

The old man from the dark, wet and smelly darkroom

[Guest roey]

How do you set Lightroom for lossless compression?

 

It does it automatically. The trick is to choose "Copy photos as DNG and add to catalog" in the File Handling pop-up when importing the files.

[adan]

1) It is perfectly easy to have a lossy compression algorithm that produces identically-sized compressed files. E.G. the sqrt function used in the M8 (and I presume M9). Which will compress a totally black or totally white shot exactly the same as a detail-filled landscape.

 

Lossy compression like the jpeg standard looks at the image and variably applies more compression to detail-free areas like open skies, so in that case the final file size varies a lot.

 

2) As someone who is unemployed and working his b*tt off to establish a business, I'm a little offended by the "let them eat cake" attitude of statements as:

 

"In any context that applies to Leica users, data storage is so cheap as to be fundamentally free."

 

"we should not sacrifice any data, (translate to image quality ) however slight, given the trouble and expense we have already gone to."

 

"The camera compression is lossy, plain and simple, and there is no need for using it."

 

Perhaps $150 per terabyte is pocket change to some, and a 16-minute download instead of 8-minute is just an opportunity for a nice Cohiba in the garden - but for me they are serious money and serious downtime.

 

I'm sure offense was not intended - but please think about the real world when posting, and don't live wrapped up in the clubby atmosphere of theoretical armchair "perfection".

 

3) That being said - can anyone actually demonstrate a difference in output from uncompressed vs. compressed DNGs? I'm perfectly willing to invest the time or money on uncompressed DNGs - if they provide a practical benefit.

[luigi bertolotti]

Well, Andy, I can understand your remarks, but the discussion started from an answer from Leica Customer Support dept. to a question - and the answer was clearly wrong, or at least un-precise : theory of information has some definitive basics postulates from Shannon's times... : if a certain alghoritm makes disappearing certain values, there IS loss of information; how much this loss does have impact in the global pipeline of transfer of information from a certain source to a certain recipient (for instance, from a RAW file to a printed paper) is another part of the problem, and the impact can have a computed value that is zero (this happens also in sound recording and even in CNC machining, to quote two very different environments).

About your 3) point: I think is very difficult that one can demonstrate in practice (and above all, posting images here) any difference between comp/uncomp DNG... and maybe it could show up "unexpected" or "random" in certain details of certain images... but, even if not having a M9, I had the opportunity to play with some 16(14) bits files : my finding is that they "support" better PP in Photoshop, expecially playing with tonal transitions... but admit that my ones were rough tests, probably with few impact in REAL reasonable processing of REAL images; anyway, the concept of "more robustness" in my opinion is right.

 

Happy Xmas to all in the forum !!!

[adan]

Oh, I agree that the response the original poster got from someone at Leica was wrong about "lossless" compression in the M9. And that compression can cause image problems (worst I ever saw were, ironically, from a DMR - but in jpgs, not DNGs!)

 

As you say, Happy Holidays! Best wishes that 2010 improves on 2009 (M9 notwithstanding).

[eritho]

Advertisement (gone after registration)

Oh, I agree that the response the original poster got from someone at Leica was wrong about "lossless" compression in the M9. And that compression can cause image problems (worst I ever saw were, ironically, from a DMR - but in jpgs, not DNGs!)

 

Excuse me but what is DMG:confused:

[pelagia]

Andy,

 

Cohiba is banned in USA, isn't it? Otherwise agree with your statements about the attitude "the ones who can afford M9 can afford anything", it can be vice versa as well...

 

But may I ask my earlier question again in more simple words. Does the 17 MB DNG out of camera means "compressed", or does it simply means "8 bit image", which is not necessarily subject to compression algorithm imho. Sure, 8 bit would record less info than 14 bit. But how is the 8-bit obtained, is it via compressing the 14 bit (like jpg or lzw compression), or is 8 bit an initial, albeit less "informative" format?

 

Tom

 

EDIT. The reason I ask it: I suppose that "14 bit compressed to 8 bit size and then uncompressed", even if lossy, has more information than initially 8 bit image.

[sandymc]

Andy,

 

Cohiba is banned in USA, isn't it? Otherwise agree with your statements about the attitude "the ones who can afford M9 can afford anything", it can be vice versa as well...

 

But may I ask my earlier question again in more simple words. Does the 17 MB DNG out of camera means "compressed", or does it simply means "8 bit image", which is not necessarily subject to compression algorithm imho. Sure, 8 bit would record less info than 14 bit. But how is the 8-bit obtained, is it via compressing the 14 bit (like jpg or lzw compression), or is 8 bit an initial, albeit less "informative" format?

 

Tom

 

EDIT. The reason I ask it: I suppose that "14 bit compressed to 8 bit size and then uncompressed", even if lossy, has more information than initially 8 bit image.

 

The 8-bit image is level compressed from 14-bits with a compression law that roughly mimics the luminance response of the eye. In a linear image (as from the A/D converter) half of all the available codes are used up on just the brightest f-stop of the image, 1/4 on the next f-stop, etc. The level compression discards some of those levels in the brighter f-stops, making the number of codes per stop more equal. Which means that it is visually a lot better than if, for example, the A/D converter on the M8 was an 8-bit converter. But that's not like either JPEG or lzw compression. What LR does is something akin to those (actually Huffman compression).

 

Sandy

[haroldp]

1) It is perfectly easy to have a lossy compression algorithm that produces identically-sized compressed files. E.G. the sqrt function used in the M8 (and I presume M9). Which will compress a totally black or totally white shot exactly the same as a detail-filled landscape.

 

 

Andy:

 

This is exactly right. One cannot however have a lossless compression that results in the same output size every time (unless the designer padded the output to some preset value, which would be beyond silly).

 

Priorities for each of us will always depend on our own needs and situation, I can only decide for myself, and share my reasons on this forum.

 

I fully understand and respect that other's priorities and situations are different from mine. If I clumsily implied otherwise, I will be more careful as this was not my intention.

 

On a recent trip I took to several mediteranean destinations, I shot about 40 GB of Leica images.

 

I took two camera bodies and 4 lenses on this trip ( cost > USD $25,000 ).

 

The trip cost more than $ 10,000.

 

Since flash memory is re-usable, I do not count it's cost here.

 

Since an external (USB, eSATA) 1TB drive is now USD $80.00 (in the US), and I use redundant storage, the cost of consumed disk storage for this trip is about USD 20.00. Assuming disk at 50% density, but also that compressed dng would still use half of that storage, the difference is stil $20.00.

 

I get a trip like this 3 or 4 times a year.

 

I personally do not wait while data uploads, I am doing something else, either on or off of the computer, so I am less sensitive to how long it takes.

 

While I was a professional, that was 35 years ago and I am now a tourist. I have no deadlines and I am 2 years behind in post-processing.

 

If I was a pro faced with regularly going through several thousand frames and getting proofs to a customer in a few days, my methods and workflow would be very different. As it is, If I get one or two dozen prints a year that are worthy of wall space, I am happy.

 

I often have my D3X set to 12 bit rather than 14 because the frame rate goes from 1.5 /sec at 14b to 5/sec at 12. If I am shooting sports or wildlife the frame rate becomes important. If I am handholding brackets for HDR, the frame rate is more important. Single frame landscapes get 14b.

 

Regards and happy holidays to all .... Harold

[adan]

Erik - I don't know what a DMG is either, but:

 

DMR = Leica's digital back for the R9

DNG = Adobe's Digital Negative Format for RAW files, which is also the format Leica's digital cameras use for RAW.

 

Harold - understood, that are good reasons to use compressed or uncompressed in different settings - and thus good reasons NOT to use compressed or uncompressed in ALL settings.

[jaapv]

It still is an interesting question how Leica keeps the uncompressed file to a constant size. It undoubtedly has to do with the DNG format and I suspect the trick is to write the values by pointers to a Lookup Table, but I would like to learn more from one of the experts on this forum.

[haroldp]

Erik - I don't know what a DMG is either, but:

 

DMR = Leica's digital back for the R9

DNG = Adobe's Digital Negative Format for RAW files, which is also the format Leica's digital cameras use for RAW.

 

Harold - understood, that are good reasons to use compressed or uncompressed in different settings - and thus good reasons NOT to use compressed or uncompressed in ALL settings.

 

The 'DMG' extension is Apple's disk image format, often used for installing software. I do not see any particular relevance to this forum and it is probably somebody's Typo.

 

I agree that there are no universal settings, as situations vary. I love having options. I believe one of the main purposes of this forum is to exchange knowledge and viewpoints of analysis so that members will better understand the trade-offs and apply them to optimize their own situation.

 

Regards ... Harold

[haroldp]

It still is an interesting question how Leica keeps the uncompressed file to a constant size. It undoubtedly has to do with the DNG format and I suspect the trick is to write the values by pointers to a Lookup Table, but I would like to learn more from one of the experts on this forum.

 

Jaap:

I am not an expert in leica firmware or software, but I am expert in lossless compression techniques and algorithm's, as well as data management.

 

I would expect uncompressed raw files of a given sensor / firmware to be the same size since they represent a fixed notation and count of photosites. In fact Leica M9 uncompressed raw files are exactly the same size as each other, and so are Olympus uncompressed raw files exactly the same size as each other.

 

Nikon uncompressed Raw files ( I just ran a test on a D200 I had handy ) have a slight variation ( from 14.9 to 15.2 mb ). I am not certain as to why but I believe it is most likely that NEF's contain an embedded jpeg thumbnail which might vary in compressibility.

 

Using pointers to a lookup table ( for non-contiguous repeating strings) is one of a type of technique used in lossless compression, most refined in what are called huffman algorithm's. Other techniques would be tokenization of contiguous repeating strings. For example, if I had some completely black space in the frame, rather than repeating thousands of photosites at 0, I might create a token that had a begin identifier, a string representation (in this case 0), an end identifier and a count (eg: next 10,000 are the represented value ).

 

Advanced Video codecs will not only do this for the frame, but reference prior frames so that parts of frames that do not change (eg: backgrounds) are not repeated.

 

This is grossly oversimplified but I hope helpful.

 

Regards ... Harold

[Guest roey]

It still is an interesting question how Leica keeps the uncompressed file to a constant size.

 

They just write out a fixed amount of bits per pixel plus the metadata. Since the number of pixels doesn't change your file size doesn't change either. For the compressed format they use a linearization table of fixed size (256 entries of 2 byte values) and write out 1 byte per pixel which encodes an index into the table. Again, since the number of pixels doesn't change the file size doesn't change either.

 

The varying file sizes of losslessley compressed DNGs stem from the compression algorithm used that is more effective on some images than on others.

[eritho]

After reading all of the information in the answers following on my original post I decided to send a link to Mr. asking for his opinion once again.

Today I received the following answer:

 

Dear Mr. Thomsen,

 

I must admit that I answered to quick and mined DNG in general (Adobe specification).

 

M9 use non-linear reduction of color depth for DNG compression. That is nearly loss-free compression of the image data in the case of files in DNG format. This doubles storage speed while requiring only half of the storage capacity. During the prototype phase, tests were performed with a 16 bit version, but they did not reveal any image quality advantages. As a result, Leica opted for the 8 bit technology.

 

Mit freundlichen Gruessen / kind regards

 

Konstantin Eisner

 

Leica Camera AG

Informationsservice Software Support

This is of course not quite true as I asked twice to be sure he really reffers to the in camera compression and not to the Adobe compression in Lightroom!

However, I must admit I don't understand his explanation fully, but I'm convinced that at least some of you do

[jaapv]

It is a bit of a mix-up between M8 and M9 decisions here, but it is basically correct. As we know Leica opted for 8-bits compression on the M8 and for choice on the M9.

Whether one needs the lossless uncompressed format now depends on the use of the file and is not really an issue. In the future? I like to keep my options open.

[jlam]

After reading all of the information in the answers following on my original post I decided to send a link to Mr. asking for his opinion once again.

Today I received the following answer:

 

Dear Mr. Thomsen,

 

I must admit that I answered to quick and mined DNG in general (Adobe specification).

 

M9 use non-linear reduction of color depth for DNG compression. That is nearly loss-free compression of the image data in the case of files in DNG format. This doubles storage speed while requiring only half of the storage capacity. During the prototype phase, tests were performed with a 16 bit version, but they did not reveal any image quality advantages. As a result, Leica opted for the 8 bit technology.

 

Mit freundlichen Gruessen / kind regards

 

Konstantin Eisner

 

Leica Camera AG

Informationsservice Software Support

This is of course not quite true as I asked twice to be sure he really reffers to the in camera compression and not to the Adobe compression in Lightroom!

However, I must admit I don't understand his explanation fully, but I'm convinced that at least some of you do

 

I don't know what tests were done for the 8bit decision but I suspect that the lack of perceivable quality difference in those tests is because:

1. Most output devices are limited to 8bit per channel

2. The tests didn't involve significant image adjustments (e.g. shadow/highlight recovery)

3. RAW processing software is doing a good job at smoothing out the gradient in highlights and shadows

 

I think the best approach would be a lossless compression on the 16bit (or 14bit which is what you actually get from the AD) data. But that would require more processing power than simply dumping the 16bit (with two wasted zero bits) file to the SD card or a simple mapping to 8bit. The processor in the M9 may not be that powerful enough so Leica might have made a compromise.

 

Since storage is getting cheaper everyday I can accept such compromise but would really hope that newer firmware would bring us either faster writing speed or, better yet 16(14) bit lossless compression option.

 

Joseph

[Advertisement]

Hallo,

Look here: Leica M