Compressed or uncompressed?

[tdtaylor]

Advertisement (gone after registration)

Guess the question remains, who can verify that Importing in LightRoom and directing it to convert to DNG actually results in a Lossless copy. JAAP brings up a good point, how do we really know. I for one don't have the patience...I mean intelligence to find out. Inquiring minds want to know......

 

Seriously, I would like it confirmed.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[Jeff S]

http://www.l-camera-forum.com/leica-forum/leica-m9-forum/110293-dng-compressed.html

 

See especially Sandy's comments.

 

Jeff

 

Gosh I like that little search box.

[diogenis]

I rest my case. Why shoot uncompressed! What am I missing? Nobody knows or can show. Shoot compressed.

Maybe not yet. Maybe later when pp programs are more powerful they will need a more solid base. Maybe monitors will get better that will eventually show differences. Maybe printing will evolve as well. Maybe there is no real difference (then why did Leica bothered with that menu option?) In short too many factors to consider when there is only one problem on the other hand: HD space which costs nothing and SD card memory which come cheap as well.

[erlingmm]

I rest my case. Why shoot uncompressed! What am I missing? Nobody knows or can show. Shoot compressed.

 

It is fun to think back on the heated discussion on compression when the M8 was released Seems Leica was right - nobody can see the difference.

 

I shoot mostly compressed myself, for time and speed reasons. But I appreciate the option of uncompressed, and use it mainly for landscapes with strong highlights.

[jaapv]

Your point was that there is "no realistic gain" in compressing. Half as many hard drives to buy, catalog and store, and half as many hours backing up to an off-site server is a realistic gain. And if your work requires the extra redundancy of optical backups, it means half as many disks to burn, catalog, and store. It adds up.

 

Therefore, the question is of real interest. If there is no benefit to shooting uncompressed and keeping the files uncompressed for storage, then those tangible benefits of compressed weigh in its favor.

 

Similarly, some shooting conditions (bad weather) weigh in favor of less frequent card changes, and thus shooting compressed.

You make a good case for storing JPGs only . I don't sit next to my computer when I back up to my office server - usuall I am in bed, sleeping, so there is no time penalty. But I'll happily amend my words to "small financial gain" Whether that is significant offset against the possibility of future quality loss is of course a wholly personal decision.

[diogenis]

Your point was that there is "no realistic gain" in compressing. Half as many hard drives to buy, catalog and store, and half as many hours backing up to an off-site server is a realistic gain. And if your work requires the extra redundancy of optical backups, it means half as many disks to burn, catalog, and store. It adds up.

 

Therefore, the question is of real interest. If there is no benefit to shooting uncompressed and keeping the files uncompressed for storage, then those tangible benefits of compressed weigh in its favor.

 

Similarly, some shooting conditions (bad weather) weigh in favor of less frequent card changes, and thus shooting compressed.

 

Totally flawed logic:

Since your shooting is gonna be an infinite job, it doesn't make any sense to say that shooting uncompressed is gonna give you infinite HD space, while shooting compressed is gonna give you infinite/2 HD space

[01af]

Advertisement (gone after registration)

It seems people keep confusing two different things: first, the in-camera DNG compression ('Leica compression'), and second, the compression that can be applied by the Adobe DNG Converter ('Adobe compression').

 

The Leica compression compresses DNG files to always exactly 50 % of the original size, and does so in a lossy way. It can be applied at shooting time only. The losses are said to be insignificant ... one can argue about that but so far no-one has been able to come up with an example where any losses were visible.

 

The Adobe compression compresses DNG files to varying degrees, perhaps 55 - 60 % of the original size on average, depending on the image content, in a lossless way. It can be applied at card download time, or at any later time.

 

Usually, one must not apply two compression cycles to the same file because compressing an already compressed file typically will inflate it. But when the two compression methods are working after different principles—as is the case with the Leica and Adobe DNG compression methods—then applying both is safe. So you can apply the Adobe DNG compression to Leica-compressed DNG files which will yield yet smaller files.

 

So Leica M9 shooters have the choice between four options:

 

1. No compression at all — lossless — file size 34.8 MB.
   
2. Leica DNG compression only — lossy — file size 17.4 MB.
   
3. Adobe DNG compression only — lossless — varying file size, approx. 20 MB on average.
   
4. Leica and Adobe DNG compression both — lossy — varying file size, approx. 10 MB on average.
   

Note that the losses of options 2 and 4 are equal, as they always come from the Leica compression only. In option 4, the Adobe compression does not add any more losses.

[jaapv]

Good summary. However, I still have my doubts about "lossless" compression. When all is said and done, any compression loses data, and in the case of "lossless" compression it loses irrelevant and/or redundant data. My doubt (as a non-expert) can be summarized as : Are these data really irrelevant/redundant and will they stay so in the future?

[diogenis]

Maybe they are both:

Both irrelevant and you can show them differences if you know where to look. Just as the case with mp3: with a high end system you are able to spot quality, but in the end it's irrelevant.

I can bet, Sandy knows where to look and spot them

[Rolo]

The losses are said to be insignificant ... one can argue about that but so far no-one has been able to come up with an example where any losses were visible.

 

They're only insignificant if you modify the image. You're discarding half of the data, the majority of which is coming from the highlight information. The camera and its decoding algorithms are deciding which tones will irretrievably be thrown away and which will remain.

 

The relevance of whether to shoot one or the other will depend upon your: subject matter, vast subtle highlight areas being the most important; the amount of pushing around in post processing you may need, where you will most notice the difference in file content; and the final product that will require the detail.

 

To explain further:

 

The M8 (because Leica have explained this), and presumably also the M9 (because the file sizes suggest that no more than 8 bits per pixel are stored), transforms the raw data before storing it. The transformation reduces the number of distinct values that might need to be stored from 4092 (or perhaps 8184) to 256 (by rounding), saving memory and speeding up the camera. The decoding algorithm applies a compensating transformation to the DNG file to restore the original values. Because of the rounding, however, there are wide gaps in the series of values in the restored data. Here's a sort of cartoon of how it would work. The first column shows the full-resolution data. The second column shows the transformed data. The third column shows the restored data.

 

1 0 1

2 1 2

3 1 2

4 2 4

5 2 4

6 2 4

7 2 4

8 3 8

9 3 8

10 3 8

11 3 8

12 3 8

13 3 8

14 3 8

15 3 8

16 4 16

etc.

 

Lots of values are missing. This wont matter unless you modify the image. If you do modify it, you will have lost subtlety because the system has lost the difference between (for instance) 9 and 15.

 

You would also lose subtlety if you were to display or print the image with a system using more than 8 bits. Those aren't common now, but they probably will be someday.

 

Would you buy a £6,000 Leica lens for your M9 that didn't capture these very subtlety of tones that are being happily discarded by shooting compressed DNG ? I suspect not.

 

Asking to illustrate it on a 900x900 pixel image reduced to 75% quality for 260k Forum purposes is hardly a fair demonstration of the differences. It's when you need the file size to manipulate, print, or indeed need to 'uprez' that it's necessary.

 

Card space has little importance to me as it's temporary and I've sufficient cards for my needs; long term storage space is valuable (not financially, but in terms of management) as I'm storing key images 3 times in total, for periods of 10 years and more. That's 500,000 images coming into the system in 10 years, say, reducing to maybe 100,000 x 3 after processing.

 

Therefore, shooting Uncompressed, then very aggressive culling of the useless and duplicated images makes a big dent in my space requirements, followed by compressing Tiff or Jpeg files depending on future use. I see no point in me saving a RAW file of a 'walkabout' snap if I will only ever use it to post on a Forum, Flickr, or include in a slideshow.

 

Hope that helps.

[jaapv]

Rolo, DMR users have been seeing the differences between compressed M8 and uncompressd DMR files for years, as you say, in postprocessing. I see the same differences between compressed and uncompressed (in the camera) on the M9, However, I didn't feel like posting the observation before, as I am not interested in the resulting argument.

[diogenis]

Basically, it's a non issue. SD memory and hard disks are soooo cheap these days, that this should never be of any concern. And as Rolo elegantly put it, you won't justify paying 6k for a lens to save in memory.

[wattsy]

I understand the theory (nicely summarised by Rolo) but I'm with Brett on this - show me an example where you can actually see the difference between uncompressed and compressed. Otherwise it's just talk about what might be.

[jaapv]

Like insurance....

[andyb]

When all is said and done, any compression loses data, and in the case of "lossless" compression it loses irrelevant and/or redundant data. My doubt (as a non-expert) can be summarize as : Are these data really irrelevant/redundant and will they stay so in the future?

 

I'm not a professional in informatics but I do know a little about how compression algorithms work. Your hypothesis is not correct: using a lossless compression algorithm doesn't produce any data loss. A simple explanation could be an algorithm that substitutes a long sequence of zeroes (frequently found in binary files) with a much shorter one. After flagging this new sequence as a compressed one, it will be possible to simply "inflate" it afterwards with no data loss at all. This happens in many different lossless compression algorhitms, for example "lzw", "zip" and so on. A zipped file will return to it's original state after uncompressing it.

 

Andreas

[wattsy]

Like insurance....

 

It's just internet mantra.

[01af]

When all is said and done, any compression loses data ...

No.

 

Lossy compression loses data. Lossless compression doesn't. That's why it's called 'lossless.'

 

 

... and in the case of "lossless" compression it loses irrelevant and/or redundant data.

Lossless compression doesn't lose redundant data but redundancy. And the removal of irrelevant data is no loss—e. g. the two zeroes in every 16-bit word storing a 14-bit value.

 

Think of a rucksack packed for a trip. It's so stuffed you can't close the ziplock. So you have to re-pack all items more intelligently in order to waste less space (lossless compression), or to remove a few items that you think you won't actually need on your trip (lossy compression), or both.

 

 

Are these data really irrelevant/redundant and will they stay so in the future?

Yes, they are, and yes, they will.

 

 

Hope that helps.

No, not really. We have re-hashed all this so many times, so why do you keep spreading misinformation?

 

 

You're discarding half of the data ...

Actually, not. The lossy Leica DNG compression discards 3/7, or 42.9 %, of the data.

 

 

Because of the rounding, however, there are wide gaps in the series of values in the restored data.

Actually, the gaps are fairly small, and they are particularly small in the shadows and wider in the highlights, in a way that matches our visual perception.

 

 

Here's a sort of cartoon of how it would work. The first column shows the full-resolution data. The second column shows the transformed data. The third column shows the restored data.

Your outline of the Leica compression method is wrong. There will be gaps but not as many as you're suggesting. It actually works along these lines: The original pixel data after A-to-D conversion are numbers between 0 and 16,383. These get multiplied by four and then the square root extracted and the results rounded to integer values. This rounding operation is where the data loss occurs. This yields integer numbers between 0 and 255 which get stored. The square root extraction is a non-linear operation which affects the high values more than the low.

 

 

You would also lose subtlety if you were to display or print the image with a system using more than 8 bits. Those aren't common now, but they probably will be someday.

That's utterly wrong!

 

8-bit systems can only display linear 8-bit values. However to fully and properly display the data stored in a Leica-compressed DNG file you still would need a 14-bit system due to the non-linear nature of the compression. After all, you wouldn't try to print or display the compressed non-linear 8-bit data but the restored linear 14-bit data.

[zlatkob]

Basically, it's a non issue. SD memory and hard disks are soooo cheap these days, that this should never be of any concern.

 

Photographers who are low-volume leisure shooters may have different needs and constraints than photographers who shoot more heavily and/or depend on photography for their income. Let me quote Andy Piper from the "DNG compressed!" thread:

 

Perhaps $150 per terabyte is pocket change to some, and a 16-minute download instead of 8-minute is just an opportunity for a nice Cohiba in the garden - but for me they are serious money and serious downtime.

 

I've been shooting Canon raw files since 2003 and have accumulated dozens of full hard drives and dozens of storage bins full of CDs, DVDs, BD-Rs. All of them bought, handled, labeled, stored, etc. And I am just one photographer, not a studio with multiple photographers. Organized, reliable and redundant storage for even a small a photography business has significant costs, both in media, physical space and time. Yes, hard drives are cheap, but they fill up, and they age, and they require power, and time spent connecting and disconnecting them, and numerous power and data cords, and shelf space, and a system for tracking what's on them, and so on. The question of compressed vs. uncompressed M9 files thus has more than trivial implications.

[Advertisement]

Hallo,

Look here: Leica M

[jaapv]

N

Yes, they are, and yes, they will.

 

 

May I borrow your crystal ball on occasion?

[01af]

May I borrow your crystal ball on occasion?

You mean, in case you need confirmation on whether 2 + 2 will be 4 tomorrow?