PSA: Adobe recompression of Q3 DNGs saves 15-30% storage

[haelio]

Advertisement (gone after registration)

On the bright side, the Q3 definitely uses a lossless compression format so we can get more out of our SD cards. This can be seen by shooting the same scene at different ISO values. Higher ISO results in more noise and noise is harder to compress.

However, I've noticed that if I ask Lightroom to convert the Q3 DNGs to DNG (yes that's a DNG -> DNG conversion), it replaces the compression done in-camera with their own offline lossless compression which results in a noticeable file size saving depending on the image with no quality (or functional) loss that I can ascertain.

This is probably because Adobe can do more when not contstrained to the power/thermal/latency budget inherent in a portable camera.

[Stephen_C]

Note that, for those of us who do not use Lightroom, this is also possible using the Adobe DNG Converter. I use that routinely when importing all of my RAW files from the Q3 (as, indeed, I did when using my old Q2).

Stephen

[leitna]

what is about the file size of a Q3 ?

 

[Stephen_C]

7 minutes ago, leitna said:

what is about the file size of a Q3 ?

DNGs are anything between about 60MB and 77MB after lossless compression using the Adbobe DNG converter—and, of course, dependent upon the subject matter.

Stephen

[jbripley]

1 hour ago, haelio said:

On the bright side, the Q3 definitely uses a lossless compression format so we can get more out of our SD cards. This can be seen by shooting the same scene at different ISO values. Higher ISO results in more noise and noise is harder to compress.

However, I've noticed that if I ask Lightroom to convert the Q3 DNGs to DNG (yes that's a DNG -> DNG conversion), it replaces the compression done in-camera with their own offline lossless compression which results in a noticeable file size saving depending on the image with no quality (or functional) loss that I can ascertain.

This is probably because Adobe can do more when not contstrained to the power/thermal/latency budget inherent in a portable camera.

I've heard about this, but it seems this DNG -> DNG conversion only seems to be available in Lightroom Classic, not the one just called Lightroom (gotta love Adobe's naming)? Or maybe someone can help me find the equivalent in Lightroom (non-classic)?

[hdmesa]

7 hours ago, haelio said:

On the bright side, the Q3 definitely uses a lossless compression format so we can get more out of our SD cards. This can be seen by shooting the same scene at different ISO values. Higher ISO results in more noise and noise is harder to compress.

However, I've noticed that if I ask Lightroom to convert the Q3 DNGs to DNG (yes that's a DNG -> DNG conversion), it replaces the compression done in-camera with their own offline lossless compression which results in a noticeable file size saving depending on the image with no quality (or functional) loss that I can ascertain.

This is probably because Adobe can do more when not contstrained to the power/thermal/latency budget inherent in a portable camera.

Variance in DNG file size at different ISOs is not indicative of lossless compression. If you can compress the Q3 DNGs with Adobe DNG Converter or directly in Lightroom, they were not lossless compressed to begin with.

[hdmesa]

Advertisement (gone after registration)

Since I use Capture One, which can't compress DNGs on the fly, I batch convert Leica DNGs through Adobe DNG Converter before importing them. If Adobe DNG Converter encounters DNGs that are already lossless compressed (such as those from the M10/11), it will not provide any further file size reduction.

[Stephen_C]

9 hours ago, Stephen_C said:

DNGs are anything between about 60MB and 77MB after lossless compression using the Adbobe DNG converter

Interestingly, on further checking, I note there appears not to be any difference in Q3 DNG file sizes when the files are run through the Adobe DNG Converter. I shall be interested to know of others' experience in trying losslessly to reduce Q3 DNG file size using the Converter. It seems odd that Lightroom does apparently achieve a further file size reduction but the Converter itself seems not to do so. (I should say my Converter settins are for Camera RAW 14.0 compatibility or later, to embed a medium size JPEG preview, to embed fast load data, not to use lossy compression, to preserve the pixel count and not to embed the original…all of which I was using with Q2 files that were reduced in size.)

Stephen

[haelio]

21 hours ago, hdmesa said:

Variance in DNG file size at different ISOs is not indicative of lossless compression. If you can compress the Q3 DNGs with Adobe DNG Converter or directly in Lightroom, they were not lossless compressed to begin with.

What makes you say that?

When you have uncompressed files, you end up with the same filesize for each RAW file which you can usually calculate by width x height x bit_depth/8. There is then additional storage used for metadata which includes a JPEG preview which varies in size due to JPEG compression. Taking the example of the Sony a7R IV, the uncompressed RAW files are around 110MB (9504 x 6336 x 14 / 8) and are the same regardless of the content of the image.

When you have lossy compressed files, the compression technique used tends to be a fixed rate codec. In the algorithm used by Sony, a fixed macroblock of pixels is compressed down into a fixed size using a fixed function algorithm. IIRC the Sony algorithm ends up storing 14-bit captures into RAW files with an average of 8-bits per pixel storage. It has artefacts but is pretty clever. This results in the RAW files once again being essentially the same size like in the uncompressed case.

When you get to lossless compressed files, the rate of compression depends on the inherent entropy of the source data. When you take a picture with the lens cap on, or a portrait with a lot of bokeh you will get a small file, and when you shoot high-entropy data such as foliage you will end up with a larger one. Shooting at high ISO results in higher entropy in the file (pure noise has max entropy). This was the basis of my analysis in the original post, and is a fairly standard way to evaluate entropic compression algorithms [1].

This doesn't mean that all lossless compression with result in the same compression ratio. If you have more compute resource and memory, you can usually do better. If you've used a compression program (like WinZip, or WinRAR, etc.) you will remember seeing the different options around dictionary size etc. which will affect how much compression you get.

It seems to me that Lightroom Classic uses a more compute/memory intensive lossless compression algorithm than the Leica Q3 has, which results in the savings.

HTH!

[1] For example https://diglloyd.com/blog/2022/20220120_1720-LeicaM11-DNG-compression.html

 

[hdmesa]

11 minutes ago, haelio said:

What makes you say that?

When you have uncompressed files, you end up with the same filesize for each RAW file which you can usually calculate by width x height x bit_depth/8. There is then additional storage used for metadata which includes a JPEG preview which varies in size due to JPEG compression. Taking the example of the Sony a7R IV, the uncompressed RAW files are around 110MB (9504 x 6336 x 14 / 8) and are the same regardless of the content of the image.

When you have lossy compressed files, the compression technique used tends to be a fixed rate codec. In the algorithm used by Sony, a fixed macroblock of pixels is compressed down into a fixed size using a fixed function algorithm. IIRC the Sony algorithm ends up storing 14-bit captures into RAW files with an average of 8-bits per pixel storage. It has artefacts but is pretty clever. This results in the RAW files once again being essentially the same size like in the uncompressed case.

When you get to lossless compressed files, the rate of compression depends on the inherent entropy of the source data. When you take a picture with the lens cap on, or a portrait with a lot of bokeh you will get a small file, and when you shoot high-entropy data such as foliage you will end up with a larger one. Shooting at high ISO results in higher entropy in the file (pure noise has max entropy). This was the basis of my analysis in the original post, and is a fairly standard way to evaluate entropic compression algorithms [1].

This doesn't mean that all lossless compression with result in the same compression ratio. If you have more compute resource and memory, you can usually do better. If you've used a compression program (like WinZip, or WinRAR, etc.) you will remember seeing the different options around dictionary size etc. which will affect how much compression you get.

It seems to me that Lightroom Classic uses a more compute/memory intensive lossless compression algorithm than the Leica Q3 has, which results in the savings.

HTH!

[1] For example https://diglloyd.com/blog/2022/20220120_1720-LeicaM11-DNG-compression.html

 

I was coming from the M11 point of view where the lossless compressed in camera came out the same size when I ran them through Adobe DNG Converter. That's lossless compression to me. If Lightroom can compress them further, that's likely lossy compression. So if the Q3 is giving similar file sizes to the M11, I would call that lossless and not further compress them after import. This source says the Lightroom re-save as DNG is lossy.

Edited June 2, 2023 by hdmesa

[Guest]

15 minutes ago, hdmesa said:

I was coming from the M11 point of view where the lossless compressed in camera came out the same size when I ran them through Adobe DNG Converter. That's lossless compression to me. If Lightroom can compress them further, that's likely lossy compression. So if the Q3 is giving similar file sizes to the M11, I would call that lossless and not further compress them after import. This source says the Lightroom re-save as DNG is lossy.

I'm sorry, but I don't see where the linked article states this. Lightroom allows lossy or lossless compression. The article specifically tells you to to select Lossy...

"Use Lossy Compression – Select this checkbox to convert RAW file to lossy DNG file"

[SrMi]

Downloaded raw files for Q3 and M11 (different scenes).

Leica Q3 
OOC raw: 77.2MB
LrC lossless compressed: 62.7MB
LrC lossy compressed: 14.8MB

M11
OOC raw: 67.8MB
LrC lossless compressed: 61.3MB
LrC lossy compressed: 14.6MB

The simplest way to apply lossless compress in-place (replaces original DNG) is using LrC’s Metadata>Update DNG Preview & Metadata.

[Guest]

11 minutes ago, SrMi said:

Downloaded raw files for Q3 and M11 (different scenes).

Leica Q3 
OOC raw: 77.2MB
LrC lossless compressed: 62.7MB
LrC lossy compressed: 14.8MB

M11
OOC raw: 67.8MB
LrC lossless compressed: 61.3MB
LrC lossy compressed: 14.6MB

The simplest way to apply lossless compress in-place (replaces original DNG) is using LrC’s Metadata>Update DNG Preview & Metadata.

That does work well and replaces the JPEG preview with the current LRc Edits.

An alternative is to use "Library->Convert Photo to DNG" which preserves the in camera JPEG and Metadata.

[hdmesa]

3 hours ago, SrMi said:

Downloaded raw files for Q3 and M11 (different scenes).

Leica Q3 
OOC raw: 77.2MB
LrC lossless compressed: 62.7MB
LrC lossy compressed: 14.8MB

M11
OOC raw: 67.8MB
LrC lossless compressed: 61.3MB
LrC lossy compressed: 14.6MB

The simplest way to apply lossless compress in-place (replaces original DNG) is using LrC’s Metadata>Update DNG Preview & Metadata.

I think I found the root cause of my incorrect thinking there: I didn't have the current version of Adobe DNG Converter. I used the latest version on several M11 DNGs, which we know are lossless compressed already, and it did reduce the file size on all of them! Cool. A 90mb M11 file was reduced to 62mb, and several M11 DNGs in the 63mb range were reduced to 52mb.

Apologies for the outdated info.

[Guest]

A bit of extra information. The DNG specification uses the DEFLATE (zip) algorithm for lossless compression The amount of compression can be specified as being between 0 and 9, with 0 being no compression. The greater the required compression the longer it takes to process.

I'm guessing that, for the camera, there is a balance between compression time and amount of data to be written to the SD Card in order to give maximum write bandwidth. This would maximise continuous shooting rates.

When processing later on a computer there is not the same time constraint so a higher compression value could be used. This may explain why compressing an already compressed file does lead to a reduction. I'm guessing here, so if there are any experts please chip in.

If you're a masochist the DNG Spec is here https://helpx.adobe.com/content/dam/help/en/photoshop/pdf/dng_spec_1_6_0_0.pdf

And you can find DEFLATE (ZIP) information on wikipedia.