SL vs. M10 File Size

[jhgnag]

Advertisement (gone after registration)

Hello,

 

I'm guessing it's a setting or something, but any idea why the DNG files from an SL I used earlier in the year are about 45MB each, but the DNG files from my M10 are 30MB and less?

 

Thank  you.

 

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[pico]

Check the setting for jpeg files. Are the same for each camera?

Edited September 14, 2018 by pico

[jhgnag]

Check the setting for jpeg files. Are the same for each camera?f

 

Thanks.  I checked the M10 and I don't have jpeg on.  It is on FINE though.  I don't have the SL anymore, so I can't check, but suspect it was the same.  I wouldn't think a jpeg setting would impact the DNG size, but I don't really know.

[UliWer]

When a camera applies automatic correction of lens distortion it should have an increasing effect on the file size. Other automatic functions applied by one model and missing with the other may also explain the differences.  

[thighslapper]

Hello,

 

I'm guessing it's a setting or something, but any idea why the DNG files from an SL I used earlier in the year are about 45MB each, but the DNG files from my M10 are 30MB and less?

 

Thank  you.

 SL files are always 42-45mb. The RAW files are not compressed.

 

Other Leicas have varying forms of lossless compression applied ....... presumably as in the M10.

 

My assumption has always been that using unaltered data is better from the point of view of writing it to the EVF/Screen/Card in a mirrorless camera like the the SL that has to shove a lot of data about very fast.

 

If I recall correctly the CL/TL files were always exactly the same size whereas the SL's always varied by a few mb.

[pico]

Hello,

 

I'm guessing it's a setting or something, but any idea why the DNG files from an SL I used earlier in the year are about 45MB each, but the DNG files from my M10 are 30MB and less?

 

Thank  you.

 

My M10 DNG files are from 32mb to 24mb on a 64mb card and 24.4mb on a larger disc. Differences in size can depend upon the size of the medium. Large discs can show files as larger storage sizes because they have greater cluster sizes - they round-up to fit.

Edited September 14, 2018 by pico

[jaapv]

Advertisement (gone after registration)

Leica uses lossless compression in various cameras. Lossless means that the original file can be reconstructed from the compressed file without data loss. Thus an uncompressed file is not "better".

[scott kirkpatrick]

The CL and SL files are also compressed.  Each pixel's output has 14 useable bits in it, so instead of putting those 14 bits into an easy-to-retrieve 2 bytes of storage, they are packed densely in just 14 bits apiece.  Presumably 1 or 2 low order bits that are just noise are stripped off.  There is still some variation in file size, because there are at least three jpegs included for quick viewing or thumbnails.  One of the jpegs is quite large, and can be used to check quality of focus.  They are there even if you don't request to have a jpeg saved in addition to the DNG.

[frame-it]

i shoot DNG only..the size is almost always 44.x mb

[adan]

Bottom line - the SL does not have a "compressed .DNG" option, and the M10 does not have an "uncompressed .DNG" option.

 

Therefore, M10 .DNGs will always be smaller than SL .DNGs. And NO, there is no longer a "setting" to change that (there was, in the M9 - the user could choose compressed or uncompressed .DNGs)

 

IF both cameras had both options, one could get compressed 20-30Mb files from the SL, and uncompressed 45Mb files from the M10 - but that is just not how Leica programs them.

 

https://www.l-camera-forum.com/topic/265201-compressed-dng/

 

BTW, in addition to compressing or not compressing the .DNG 1s and 0s, .DNG files (as do most raw files) contain a low-quality embedded JPEG image for reviewing the pictures in-camera (since "raw" sensor data is not a displayable "picture" in and of itself - just data to create a picture from, once brought into ACR, LR, C1, Bibble, or whatever). The preview jpeg itself can have different compression amounts from one camera to another.

 

Equally, a given camera can have a couple of different available levels of compression, so that pictures with broad swaths of detailess areas (skies, or blurry backgrounds) can be compressed a bit more, and those with little extra detail (including noise specks at high ISOs), get compressed a bit less. If the CL/TL files are always exactly the same size, Leica is not bothering to do that with those cameras.

[jhgnag]

Bottom line - the SL does not have a "compressed .DNG" option, and the M10 does not have an "uncompressed .DNG" option.

 

Therefore, M10 .DNGs will always be smaller than SL .DNGs. And NO, there is no longer a "setting" to change that (there was, in the M9 - the user could choose compressed or uncompressed .DNGs)

 

IF both cameras had both options, one could get compressed 20-30Mb files from the SL, and uncompressed 45Mb files from the M10 - but that is just not how Leica programs them.

 

https://www.l-camera-forum.com/topic/265201-compressed-dng/

 

BTW, in addition to compressing or not compressing the .DNG 1s and 0s, .DNG files (as do most raw files) contain a low-quality embedded JPEG image for reviewing the pictures in-camera (since "raw" sensor data is not a displayable "picture" in and of itself - just data to create a picture from, once brought into ACR, LR, C1, Bibble, or whatever). The preview jpeg itself can have different compression amounts from one camera to another.

 

Equally, a given camera can have a couple of different available levels of compression, so that pictures with broad swaths of detailess areas (skies, or blurry backgrounds) can be compressed a bit more, and those with little extra detail (including noise specks at high ISOs), get compressed a bit less. If the CL/TL files are always exactly the same size, Leica is not bothering to do that with those cameras.

 

Thanks Andy.  I think this answers it.  I was convinced I must have the M10 set for compressed RAW, but could not find a setting.  I guess Leica decided for me:)

[mmradman]

In my book SL601 has only 2 shortcomings; lack of ability to write compressed RAW files and lack of LENR. Probably both could be fixed in firmware.

 

Is there a reason why SL does not offer option to compress losslessly RAW file. Could it be related to 11fps performance?

 

Before anyone suggest otherwise, quality of lossless compressed equals uncompressed file.

[scott kirkpatrick]

Speed is probably a factor in not compressing the SL's files.  The bits are simply packed directly into one continuous stream on the SD chip.  Reading them out again is trickier since they begin and end on arbitrary even numbered points in the chips address space, rather than the multiples of 16 or 32 that most hardware would prefer to grab.  Compressing the same data achieves about 40-60% reduction but an adaptive code (the most efficient in use of storage) has to develop its code book ad it reads the file which it is encoding, and that takes a lot of checking.  And memory silicon is incredibly cheap.  The sweet spot (best price per bit) is currently at around 64GB per chip.  I don't see this as a problem. 

[mmradman]

Speed is probably a factor in not compressing the SL's files.  The bits are simply packed directly into one continuous stream on the SD chip.  Reading them out again is trickier since they begin and end on arbitrary even numbered points in the chips address space, rather than the multiples of 16 or 32 that most hardware would prefer to grab.  Compressing the same data achieves about 40-60% reduction but an adaptive code (the most efficient in use of storage) has to develop its code book ad it reads the file which it is encoding, and that takes a lot of checking.  And memory silicon is incredibly cheap.  The sweet spot (best price per bit) is currently at around 64GB per chip.  I don't see this as a problem.

 

Thanks for the explanation, makes sense. Perhaps Leica should provide option to compress for user happy to shoot at slower shooting rate, I use single shot mostly, it is probably very small fraction of users who machine gun at 11fps. Some ten years ago my Nikon DSLR provided all these file saving options at somewhat pedestrian rate of 5fps or 7fps with additional grip for extra battery power - D700.

 

I take it you reference to 64Gb is for memory card. That what is what I use, however I am more concerned about hard drive sizes and long term storage management in general. Storing 1000 RAW files at 45Mb equals 45Gb which can be one trigger happy session or a long holiday. Chunks of 45Gb quickly add up.

[pico]

I take it you reference to 64Gb is for memory card. That what is what I use, however I am more concerned about hard drive sizes and long term storage management in general. Storing 1000 RAW files at 45Mb equals 45Gb which can be one trigger happy session or a long holiday. Chunks of 45Gb quickly add u

 

My M10 DNG files are typically about 24mb. Of course saving them from Photoshop in compressed TIFF makes them about 41mb. I've tried six different compression methods and none shrink the DNG files significantly, usually less than 5%. Oh, saving them in native photoshop format produces files three times larger, a little over 71mb.

[frame-it]

interesting...when i save one of my processed SL DNG's as pro-photo uncompressed tiff for printing its around 120++mb

My M10 DNG files are typically about 24mb. Of course saving them from Photoshop in compressed TIFF makes them about 41mb. I've tried six different compression methods and none shrink the DNG files significantly, usually less than 5%. Oh, saving them in native photoshop format produces files three times larger, a little over 71mb.

[adan]

Raw files are a single channel - effectively, a "grayscale image" of brightness only, for each pixel. If you could see the actual raw image, it would look like a gray checkerboard - a red shirt would rendered as bright in the red-filtered pixels and dark in the blue and green pixels around it. Same for a blue sky or green tree - gray checkerboard.

 

Approximately 2 bytes (16 bits) times 24 million pixels = 48 megabytes. If the assumption is that 14-bit files are repackaged to only use 1.7 bytes of space per pixel, then 1.7 x 24x10⁶, or 40.8 megabytes.

 

Once you debayerize the image to a standard full-color RGB format, that goes up by 3 times - you now have to have three channels for the three colors, each one its own 24-megapixel picture and each pixel taking up 8 or 16 bits (1 or 2 bytes) depending on user preference. Or files that are nominally 72 megabytes (8-bit) or 144 megabytes (16-bit data).

 

Now, there are fudge-factors to that - an M10 image is not 4000 x 6000 pixels (24 Mpixels), but 3984 x 5976 pixels (23.8 Mpixels) - at least as read by Adobe Camera Raw. This is because, to debayerize an image, one has to borrow data from the surrounding, say, "red" pixels, to get the correct color in a blue or green pixel, and so on. And edge pixels have incomplete data (a pixel on the extreme left edge of the sensor has no pixels to the left of it from which to borrow data and figure out color accurately) - so those are generally thrown away or cropped off in debayerizing.

 

There may be "hidden compression" going on, in saving any particular file format. When I open an approximately 27.26 megabyte M10 .dng in Adobe Camera Raw, and save my various settings, ACR recompresses the .dng - once - to something like 23-24 megabytes. Apparently Adobe thinks Leica's compression is not as efficient as it could be, and redoes it. Saves a little disk space - but is the place where an M10 (or M240, for that matter) image occasionally crashes my Photoshop CS6.

 

Heck, I just saved uncompressed TIFFs from an M10 image, just for grins, and saved as a PC TIFF, it is still a different file size than saved as a Mac TIFF - figure that one out.

 

All computer image files also have some kind of header data or metadata, in addition to the actual image pixels. EXIF, but also more obscure "need to know" information, such as "this sensor type starts with a red (or green or blue) pixel in the top-left corner," or look-up tables, or the "As shot" WB value. or the lens profile, and on and on. And all that adds even more bits and bytes to the file size.

 

Plus, of course, a megabyte is not the same as a megapixel - the first is 1,024,000 bytes, the second is 1,000,000 pixels (oddity of base-two, 2^x computer math).

[Jeff S]

Raw files are a single channel - effectively, a "grayscale image" of brightness only, for each pixel. If you could see the actual raw image, it would look like a gray checkerboard - a red shirt would rendered as bright in the red-filtered pixels and dark in the blue and green pixels around it. Same for a blue sky or green tree - gray checkerboard.

 

Approximately 2 bytes (16 bits) times 24 million pixels = 48 megabytes. If the assumption is that 14-bit files are repackaged to only use 1.7 bytes of space per pixel, then 1.7 x 24x10⁶, or 40.8 megabytes.

 

Once you debayerize the image to a standard full-color RGB format, that goes up by 3 times - you now have to have three channels for the three colors, each one its own 24-megapixel picture and each pixel taking up 8 or 16 bits (1 or 2 bytes) depending on user preference. Or files that are nominally 72 megabytes (8-bit) or 144 megabytes (16-bit data).

 

Now, there are fudge-factors to that - an M10 image is not 4000 x 6000 pixels (24 Mpixels), but 3984 x 5976 pixels (23.8 Mpixels) - at least as read by Adobe Camera Raw. This is because, to debayerize an image, one has to borrow data from the surrounding, say, "red" pixels, to get the correct color in a blue or green pixel, and so on. And edge pixels have incomplete data (a pixel on the extreme left edge of the sensor has no pixels to the left of it from which to borrow data and figure out color accurately) - so those are generally thrown away or cropped off in debayerizing.

 

There may be "hidden compression" going on, in saving any particular file format. When I open an approximately 27.26 megabyte M10 .dng in Adobe Camera Raw, and save my various settings, ACR recompresses the .dng - once - to something like 23-24 megabytes. Apparently Adobe thinks Leica's compression is not as efficient as it could be, and redoes it. Saves a little disk space - but is the place where an M10 (or M240, for that matter) image occasionally crashes my Photoshop CS6.

 

Heck, I just saved uncompressed TIFFs from an M10 image, just for grins, and saved as a PC TIFF, it is still a different file size than saved as a Mac TIFF - figure that one out.

 

All computer image files also have some kind of header data or metadata, in addition to the actual image pixels. EXIF, but also more obscure "need to know" information, such as "this sensor type starts with a red (or green or blue) pixel in the top-left corner," or look-up tables, or the "As shot" WB value. or the lens profile, and on and on. And all that adds even more bits and bytes to the file size.

 

Plus, of course, a megabyte is not the same as a megapixel - the first is 1,024,000 bytes, the second is 1,000,000 pixels (oddity of base-two, 2^x computer math).

You took the words right out of my mouth.

 

Not.

 

Have you written a book yet, Andy? If not, you should.

 

Jeff

[Advertisement]

Hallo,

Look here: Leica L-Mount

[scott kirkpatrick]

....

Approximately 2 bytes (16 bits) times 24 million pixels = 48 megabytes. If the assumption is that 14-bit files are repackaged to only use 1.7 bytes of space per pixel, then 1.7 x 24x10⁶, or 40.8 megabytes.

....

The 14 bit data uses 1 3/4 bytes per pixel, so the image data along will occupy (7/8)* 48 MB = 42 MB.  But there is more in a DNG file than the raw data.  There are multiple small to large jpegs, lots of meta data and unreadable stuff called "Maker Notes" that only someone licensed by the manufacturer can decode.  The AWB information and parameters for distortion correction  are also in there, but they occupy only a few KB.  Adobe adds all sorts of extra metadata to record what LightRoom has done.  Watch carefully and you can see DNG files change as they are processed.

 

I keep all usable DNG files in folders by date or event, and zip-compress the folder into a single zip file once the files have been rendered.  This gives a size reduction of about 0.5X, or about the same as having the raw files in the manufacturer's compressed format (which may be proprietary or may even go away someday).  And backup storage is subject to the same declining costs as SD cards.  

 

Edit:  I was just pricing some backup storage for our department -- we have a new data science institute and if they do their job they will generate and collect lots of data.  I figure we can back up 100 TB for about $5K at current prices.  That includes all the surrounding stuff, such as frames and power supplies, but assumes you already have a computing cluster or center. 

Edited September 16, 2018 by scott kirkpatrick

[pico]

[...] there is more in a DNG file than the raw data.  There are multiple small to large jpegs, lots of meta data and unreadable stuff called "Maker Notes"  [...]

 

Sure, but for disc space all the extra data is accounted for.