To compress or not to compress

[hoppyman]

Advertisement (gone after registration)

Yeah, Geoff. But for those of us who have nothing else to do, admiring the brilliance behind the thing is still a pretty big deal.

 

All good except for THIS part, Howard

Nope! Show me!

.... If you've got an 8-bit file in Photoshop and want to make a major adjustment, you will lose less data if you first convert it to 16-bit, do the adjustment, and convert back to 8-bit. (I've seen it demonstrated, but can't tell you how.)

 

By the way I should have qualified my comments about testing here. I only have one speed card, The newest Sandisk Extreme 30MB/s ones. Not warranted but maybe in some future camera/firmware they might show some advantage.

As always Henning is very well informed and accurate.

[alexmann]

There's more to it than just how data are manipulated. He covers also the fact that digital storage of photographic data by its nature offers fewer brightness steps as brightness increases. The Leica compression algorithm takes advantage of that fact and automatically tends to hold more shadow detail. .

 

Another good reason to shoot compressed then.

Alex

[chris_tribble]

+1 - it's an argument that has persuaded me. For the vast majority of my work I shoot compressed. The only times I don't (perhaps superstitiously) are when I'm doing lanscape work at 160. Here I'm working in a very considered way - tripod, delayed shutter release minimise vibration etc. - so it seems worth the overhead. Otherwise, for the style of work I do, the "fleetness of foot" that comes with compressed DNGs seems the best way to go.

[elgenper]

A pragmatic take on the matter: Lots of extremely good photographers with top-notch technical expertise have argued for more than a year about whether IQ suffers from this compression or not.

 

If it really did, there would be no argument by now, only images that proved it....

 

I´ll compress until I see proof that I actually lose IQ

[jan_brittenson]

On 'C', after 7 shots the camera can't buffer any more, and if I then remove my finger and wait for the data transfer light to stop blinking it takes 13 sec with compressed DNG and 25 sec with uncompressed DNG. This is with 16Gb class 6 Transcend cards. With class 10 cards it's slightly faster.

Ah, I didn't check the time to clear; I care more about the shot-to-shot delay when the buffer is full. And this appears the same regardless of compression.

 

I've been meaning to take a look at the compression as well; in theory it could be okay. The thing to do is shoot in difficult WB like incandescent and fluorescent using both, then correct the images identically in post and subtract them. If the image can be discerned from the difference, then something was lost.

[Lindolfi]

Here's what I found: In practice if you do not shoot more than 7 images quickly after each other, it does not matter if you shoot compressed or uncompressed. I have never run into a situation where I can not take a picture because the camera could not handle the data flow fast enough, even in uncompressed mode.

 

After having measured the noise of part of an image of a very smooth surface in various ISO settings and in compressed and uncompressed mode, I found that compressed introduces noise (due to digitization in bins) equivalent to 1/3 ISO stop.

 

So the noise at 200 ISO uncompressed is the same as 160 ISO compressed. It is extremely hard however to see the difference in normal images between compressed and uncompressed at the same ISO setting. Above 640 ISO, there is no advantage of uncompressed according to my measurements.

 

So the advice is if you usually take many exposures in a situation (and so also may run into buffer emptying time), use compressed, since the difference is very small and you can store twice the number of images on a card.

 

If you shoot film Leica M style, you can shoot uncompressed with a tiny advantage. Personally I shoot uncompressed. I carry two 16 Gbyte cards and two batteries and have never run into shortage. (32 Gbyte holds 850 images, which also happens to cover 2 batteries). When loading into the computer I load as lossless dng, which is on average 18 Mbyte per image, so harddrive space usage is not more than storing compressed from the M9.

[chris_tribble]

Advertisement (gone after registration)

When loading into the computer I load as lossless dng, which is on average 18 Mbyte per image, so harddrive space usage is not more than storing compressed from the M9.

 

The one caveat I'd offer here is that the ONLY time I've had corrupted DNGs was when I used the Lightroom compressed DNG option following a portrait shoot. It was nearly a disaster as it has been incredibly difficult to find a time when the artist in question was going to be available - reshooting would have been out of the question. Fortunately, I hadn't formatted the card before doing the post process, so was able to delete the corrupted files and reload.

[wda]

Again as I am new to the M9. What are the merits/demerits to compressed or uncompressed DNG files?

Bouic49

Bouic49, welcome to the Forum! The choice of compression depends greatly on what you intend to do with your Leica images. Had there been no advantages to 'uncompressed' files, Leica would not have given us that option. Since any of my M9 pictures could become stock pictures, I welcome the option to save my files 'uncompressed'. In that way I know that I have ultimate quality from the lens and sensor. It is one of the major reasons I upgraded from M8. If I had lesser needs, I would accept the economy of smaller files sizes.

[ho_co]

... Nope! Show me!

.... If you've got an 8-bit file in Photoshop and want to make a major adjustment, you will lose less data if you first convert it to 16-bit, do the adjustment, and convert back to 8-bit. (I've seen it demonstrated, but can't tell you how.) ...

 

I can't; probably best left alone.

 

A commercial photographer demonstrated this to my satisfaction in a Photoshop class at Rice University, but I've never been able to come up with an example on my own to illustrate the contention.

 

According to my notes, he worked three times from the same badly underexposed RAW file.

 

First, he opened it in 16 bits, then corrected the underexposure.

Next, he opened it in 8 bits, converted to 16-bit, then corrected the underexposure.

Then, he opened it in 8 bits and corrected the underexposure still in 8-bit space.

 

He then showed us the same area of all three versions at high magnification. The shadows in the all-8-bit version included what looked like grain clumping. The 8>16>8-bit version had a little of the same, but noticeably less. The all-16-bit version didn't show the clumping.

 

He stressed that this was a case of 'stretching the data.' His point was that having more buckets to hold the data means that more intermediate values will be maintained in the process, and fewer values will get lost between buckets.

 

He was pretty methodical, so I'm pretty sure that in all three cases, he saved the results as an 8-bit TIFF. But this was my first intensive exposure to Photoshop and I didn't record what method he used to improve the exposure.

 

The class was in summer of 2005. I think we used Photoshop 7. So despite what happened then, the assumptions may no longer hold.

[hoppyman]

A pragmatic take on the matter: Lots of extremely good photographers with top-notch technical expertise have argued for more than a year about whether IQ suffers from this compression or not.

 

If it really did, there would be no argument by now, only images that proved it....

 

I´ll compress until I see proof that I actually lose IQ

 

Which is perfectly reasonable. However you can also turn that around and say that I will shoot uncompressed until I see proof about whether IQ suffers from compression or not

There hasn't really been any documented rigorous comparison done that I am aware of personally. The only exception that I can think of is the work that Sandy McGuffog has shared regarding black point variation and ISO levels in compressed vs. uncompressed. Lots of opinions from users that it probably doesn't matter much in practice for most work, and I agree. That's fine but it is opinion rather than proven one way or the other.

If anyone has links or references there to quantifiable results, I'm sure that many would be interested to learn more.

[Wiggi]

No, not quite. The full so-called 16-bit space (actually 12 or 14 bits) is still recorded, but packed into a look-up table accessible to 8-bit pointers.

 

Not really. The compression algorithm of the M8 and M9 is lossy. The 16bit color depth will be compressed to 8bit.

 

The compression algorithm of LightRoom or CameraRaw is lossless (LZW compression). The S2 also will get a lossless algorithm.

 

There was a serious reason, that the M9 got the ability to safe uncompressed DNG files.

[ShotCapture]

Those who say they have tested uncompresses vs compressed haven't all specified their testing, but I would guess that it involves matching pixels and noise. There is probably no real world difference this way. However the difference, if any, would come when doing major retouching in Photoshop as mentioned by another post. This doesn't have to be a photo with major mistakes though, but rather, for instance, a beauty shot requiring major "normal" work.

[ho_co]

Not really. The compression algorithm of the M8 and M9 is lossy. The 16bit color depth will be compressed to 8bit.

"Compression of M8 and M9 is lossy." No one argues that. That's why this thread exists.

 

"16-bit color depth will be compressed to 8-bit." Ambiguous, can be taken two ways. The 16-bit color depth of the M8 and M9 is not compressed to 8-bit color depth. Rather, an 8-bit LUT is built to access the full range of data.

 

... The compression algorithm of LightRoom or CameraRaw is lossless (LZW compression).The S2 also will get a lossless algorithm....

Correct, mentioned above, and off-topic.

 

 

... There was a serious reason, that the M9 got the ability to safe uncompressed DNG files.

 

Incorrect. See the links provided above.

 

See Stefan's simple explanation 05:50 into the Reichmann interview. There was no "serious reason" for the change. It was simply user request.

[mjh]

Not really. The compression algorithm of the M8 and M9 is lossy. The 16bit color depth will be compressed to 8bit.

Howard quite rightly objected against your statement that the M9’s lossy compression did result in an 8-bit colour depth. For one thing there is no such thing as the colour depth of a raw file; colour is created (by way of interpolation) only when the raw data is processed. More importantly talking about an 8-bit depth is misleading as it suggests as loss in dynamic range that Leica’s compression method is designed to avoid. Yes, the distinction of brightness values for each pixel isn’t as fine as it is in uncompressed raw data, but then for all practical purposes a great part of these fine distinctions are irrelevant anyway.

 

There was a serious reason, that the M9 got the ability to safe uncompressed DNG files.

Customers had been asking for this feature … For four years now I’ve been waiting for someone to convincingly make a case for the superiority of uncompressed vs. compressed raw files – still no luck.

[sandymc]

The 16bit color depth will be compressed to 8bit.

 

14 bits, not 16. Not that it makes any difference to the argument.

 

Sandy

[jan_brittenson]

The one caveat I'd offer here is that the ONLY time I've had corrupted DNGs was when I used the Lightroom compressed DNG option following a portrait shoot. It was nearly a disaster as it has been incredibly difficult to find a time when the artist in question was going to be available - reshooting would have been out of the question. Fortunately, I hadn't formatted the card before doing the post process, so was able to delete the corrupted files and reload.

My workflow deletes/reformats cards after the images have been processed and stored for safekeeping on my Drobo. Even if I process on my laptop while on the road I still don't clear out cards until the contents are also on my Drobo. When I take a full card out of the camera I flip the write protect. When it's safely archived I flip it back. This way even if I accidentally reach for a card that's not safe to clear I'll get an indication I'm doing something wrong.

 

With Transcend 16GB class 10 cards going for $27 at B&H there's really no good reason to ever not have enough... I have a pile of these.

[sblitz]

perhaps its a fault in my m9, but the difference between the compressed and uncompressed versions is astounding -- even in the lcd in the back of the camera. the uncompressed is much much better showing dynamic range, etc. time and space issues aside, it does take longer to write, the uncompressed version is far superior. if rapid fire shooting was a requirement, compressed is more than good enough, but uncompressed really shows the iq of the m9

[BDX]

...When loading into the computer I load as lossless dng, which is on average 18 Mbyte per image, so harddrive space usage is not more than storing compressed from the M9.

But what if you use Lightrooms compression on an already compressed (by the M9) DNG?

In this case you get files with about 8-10MB.

 

Does anybody now for sure, that these files in the end are not worse than the uncompressed DNGs (if we originate that the loss of dynamic range from the camera compression is negligible)?

 

If so, that would mean we can compress the 36MB DNG's to 8-10MB without significant loss in image quality.

[wattsy]

perhaps its a fault in my m9

 

I think it must be.

[Advertisement]

Hallo,

Look here: Leica M

[sandymc]

Does anybody now for sure, that these files in the end are not worse than the uncompressed DNGs (if we originate that the loss of dynamic range from the camera compression is negligible)?

 

The loss from camera compression is actually mostly in tonality, rather than dynamic range - you get bigger steps over the same dynamic range. But there is no other loss from the Lightroom compression. For sure

 

Sandy