DNG compressed VS uncompressed?

[uhoh7]

Advertisement (gone after registration)

I resurrect this thread as M10 does not offer uncompressed.

 

Unfortunately I haven't learned anything so far after reading it. Lossless means lossless? Really?

 

What I wonder is if anyone has actually taken the time to test the compressed vs uncompressed. You know, like editing 200 files one way and then the other?

[Exodies]

How did we get in the state that people imagine they could be losing information by lossless compression? Is this post truth information technology?

[pop]

I resurrect this thread as M10 does not offer uncompressed.

 

Unfortunately I haven't learned anything so far after reading it. Lossless means lossless? Really?

 

What I wonder is if anyone has actually taken the time to test the compressed vs uncompressed. You know, like editing 200 files one way and then the other?

"Lossless" is a technical term. It refers to a set of procedures which can shrink a file in size in a way that lets you restore the original file down to the last bit.

 

IOW, if you shrink a file and then unshrink it again and there is no difference between the original file and the un-shrunk one, then you did not lose any information and the procedure was lossless.

[uhoh7]

"Lossless" is a technical term. It refers to a set of procedures which can shrink a file in size in a way that lets you restore the original file down to the last bit.

 

IOW, if you shrink a file and then unshrink it again and there is no difference between the original file and the un-shrunk one, then you did not lose any information and the procedure was lossless.

No, "Lossless" is a claim. It is still a highly debated subject, there are numerous methods and benchmarks.

 

https://en.wikipedia.org/wiki/Lossless_compression

 

Saying a file is "lossless" without providing further information is meaningless.

 

What is the method and what are benchmark results for Leica lossless DNG vs uncompressed? If you can't answer that and further have not tested the actual files you do know what you are talking about. You are just parroting a claim which may or may not be the case.

[pop]

No, "Lossless" is a claim. 

Sorry, you're mistaken. "Lossless" is no more a claim than a "Sum" is a claim.

 

However, if you feel more comfortable "testing" compressed files for any loss, do so by all means. You should be aware that any difference you're likely to find would be caused by your testing procedure, rather than by properly decompressing a losslessly compressed file.

[Exodies]

We should also be wary of the claim that DNGs are digital. Where's the proof?

After all, they lie about the "negative" part.

[marchyman]

Advertisement (gone after registration)

No, "Lossless" is a claim.

 

 

Not a claim.  It is a computer science (and Information Theory) term of art with a specific definition.  I'll use the wikipedia version: Lossless compression is a class of data compression algorithms that allows the original data to be perfectly reconstructed from the compressed data.

 

Perfectly reconstructed.   Every bit of the uncompressed data is identical to the original before compression.

[uhoh7]

Not a claim.  It is a computer science (and Information Theory) term of art with a specific definition.  I'll use the wikipedia version: Lossless compression is a class of data compression algorithms that allows the original data to be perfectly reconstructed from the compressed data.

 

Perfectly reconstructed.   Every bit of the uncompressed data is identical to the original before compression.

BS.

 

It's a claim until verified. You want to show the verification of Leica DNG lossless, I'm all ears.

 

What benchmarks were used? Reference the tests.

 

If you can't answer that you are speaking out of your.....

 

The compression method would also be nice to know (there are multiple), but just the above would be fine.

 

I've been building computers for 16 years. At least I know some of what I don't know, which is more than most in this thread can claim.

[pop]

Of the numerous data compression algorithms, some are lossless and some are lossy. There's never any doubt about which is which. Provided they're properly coded, no testing will be needed to ascertain whether it loses information or not. In fact, no amount of testing would ever demonstrate with any certainty whether an algorithm will be lossless under all circumstances.

 

Leica state in the technical data that they use a lossless compression type. They also state for Summicron lenses that their maximum aperture is f:2.0. The say that the image contains a given number of pixels. These are all assertions. Were you to buy an M (Typ 240), you could count on it having all those properties.

 

Have you ever programmed any kind of compression?

[Exodies]

Algorithms are not made by trial and error. They are constructed mathematically to perform a specific task. The only testing you do is to catch coding errors and measure performance. Nobody writes a new sort function then tries it on lots of data to see if it comes out alphabetically. Do I need to point out that building computers isn't algorithm design?

[bencoyote]

I resurrect this thread as M10 does not offer uncompressed.Unfortunately I haven't learned anything so far after reading it. Lossless means lossless? Really?What I wonder is if anyone has actually taken the time to test the compressed vs uncompressed. You know, like editing 200 files one way and then the other?

If you look at page 19 of http://wwwimages.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

It specifies the compression algorithm as Deflate aka ZIP. This is a well-known compression algorithm very commonly used for computer binaries In packages used to distribute software. It is important to understand this because a change of even one bit in a computer binary can render it inoperable. Thus actual losslessness is very important. You can read up about Deflate compression here: https://en.m.wikipedia.org/wiki/DEFLATE

 

The thing about deflate is it is a good general purpose lossless compression. It can be used for anything but with often times very marginal results. The thing about compression algorithms is if they are designed for a specific purpose they can do a much better job. On image data deflate probably reduces the file size to 60% of its original size. While a format like JPEG could probably reduce image data down to about 5% of its original size with very little detectable difference. Of course we all know that the difference ends up being that a bunch of transformations programmed into the camera I get baked into the result making it much more difficult for us to develop an image to our liking. No such transformations are applied in the process of applying the deflate algorithm to the raw image data.

[mjh]

What benchmarks were used? Reference the tests.

You don’t need benchmarks. If the compression algorithm is lossless, then it’s lossless. There are a couple of lossless compression algorithms and I suppose you are familiar with these?

[uhoh7]

You don’t need benchmarks. If the compression algorithm is lossless, then it’s lossless. There are a couple of lossless compression algorithms and I suppose you are familiar with these?

You contend Leica DNG is Jpeg2000?

[pop]

You contend Leica DNG is Jpeg2000?

Why don't you just open a DNG file with a byte editor and use the DNG specs to determine the kind of compression applied? Surely you've heard of ZIP, GZ and similar "archive" formats used to distribute document files and software? Do also think they might be lossy? Good grief.

[pechelman]

I havent read this whole thread, but I'm confused with why it's gotten so long, so I had to click on it.

 

It's really as simple as the two statements;

1. If you value SD Card or Disk Drive space and the time to transfer files from SD to Disk Drive over maybe a little extra computing time in processing, use COMPRESSED DNG
2. If you dont care about file size and are ever concerned about some sort of file corruption during zip/unzip (highly unlikely with checks), use UNCOMPRESSED DNG.

There is no loss in quality of information as has already been stated numerous times on just this last page.

 

The one thing I am uncertain about, and have no interest to benchmark, is if there's a difference in processing speed between the two in camera.

One might think that a compressed dng would require more processing and take more time to put through the buffer and onto the card, but that may not be the case.

With the larger file size of the Uncompressed DNG, it may take a little longer.  Really, for most of Leica shooters, this really isnt an issue due to how infrequently we use high bursts. Also, there might be some parallels here one can draw between JPG vs RAW processing time in camera and how quickly the buffer fills up or the difference in number of frames per second, but the compression ratio of those 2 formats is much different than CDNG/DNG, especially since JPG is a LOSSY compression. 

 

In reality, that last paragraph is good for academic discussion. The first 2 bullet points are where the practical digital photographer lives. Pick item 1 or 2 and be done with it and move on with life

[ECohen]

I havent read this whole thread, but I'm confused with why it's gotten so long, so I had to click on it.

 

It's really as simple as the two statements;

1. If you value SD Card or Disk Drive space and the time to transfer files from SD to Disk Drive over maybe a little extra computing time in processing, use COMPRESSED DNG
2. If you dont care about file size and are ever concerned about some sort of file corruption during zip/unzip (highly unlikely with checks), use UNCOMPRESSED DNG.

There is no loss in quality of information as has already been stated numerous times on just this last page.

 

The one thing I am uncertain about, and have no interest to benchmark, is if there's a difference in processing speed between the two in camera.

One might think that a compressed dng would require more processing and take more time to put through the buffer and onto the card, but that may not be the case.

With the larger file size of the Uncompressed DNG, it may take a little longer.  Really, for most of Leica shooters, this really isnt an issue due to how infrequently we use high bursts. Also, there might be some parallels here one can draw between JPG vs RAW processing time in camera and how quickly the buffer fills up or the difference in number of frames per second, but the compression ratio of those 2 formats is much different than CDNG/DNG, especially since JPG is a LOSSY compression. 

 

In reality, that last paragraph is good for academic discussion. The first 2 bullet points are where the practical digital photographer lives. Pick item 1 or 2 and be done with it and move on with life

 

Amen

.....but I was enjoying the geeky back and forth

[uhoh7]

Why don't you just open a DNG file with a byte editor and use the DNG specs to determine the kind of compression applied? Surely you've heard of ZIP, GZ and similar "archive" formats used to distribute document files and software? Do also think they might be lossy? Good grief.

Why don't you just share your knowledge and tell us the compression method for Leica DNG, how you determined it, and how it's verifiable. Maybe you have not done it?

 

Jpeg2000?

 

Is that such a ridiculous question?

 

Or if you can't be bothered to inform an idiot like me, maybe somebody else will use their hex skills

 

I'd love to see any evidence besides pedantic statements: lossless means lossless, end of story. I don't claim to be a genius, but I need more than that to believe it, sorry. Yes I've read a number of articles on the concept. They only make the question more obvious.

[pop]

Leica has published a document called "Technical Data".

The information in that publication is binding.

The technical data says that the M (Typ 240) stores the image either as JPEG file or as DNG ; if DNG, you can store the image in uncompressed or in losslessly compressed format.

There is a number of compression algorithms known which will produce losslessly compressed images.

Leica does not say which format they chose.

It does not matter which format they chose.

The document mentioned above is legally binding.

If you want to know which algorithms are used in compressing the file you can find out as I wrote above.

Testing an image compression software in order to see whether there will be any losses is pointless. If you know how the algorithm works, you don't need a test as it can be easily shown whether the algorithm is lossy or not. If you do not know how the algorithm works, testing is pointless as you don't know what you should be looking for and which situation may demonstrate the losses.

So: when Leica state that they apply lossless compression to the image, they mean that they used an algorithm that's proven to be lossless. 

If pressed, I would guess that they might have chosen the LZW algorithm, which is used in some ZIP files as well. But then, they might have chosen another one on account of the copyright issues in connection with LZW. Look at the file.

[pico]

You contend Leica DNG is Jpeg2000? [...]

Or if you can't be bothered to inform an idiot like me, maybe somebody else will use their hex skills

 

You are clearly unfamiliar with the science of data compression. You might enjoying reading the reading literature more than you enjoy ranting about something you choose to believe is something like Alternative Fact.

 

And how do you think reading HEX even matters here?

[Advertisement]

Hallo,

Look here: Leica M

[bencoyote]

I havent read this whole thread, but I'm confused with why it's gotten so long, so I had to click on it.

 

It's really as simple as the two statements;

1. If you value SD Card or Disk Drive space and the time to transfer files from SD to Disk Drive over maybe a little extra computing time in processing, use COMPRESSED DNG
2. If you dont care about file size and are ever concerned about some sort of file corruption during zip/unzip (highly unlikely with checks), use UNCOMPRESSED DNG.

There is no loss in quality of information as has already been stated numerous times on just this last page.

 

The one thing I am uncertain about, and have no interest to benchmark, is if there's a difference in processing speed between the two in camera.

One might think that a compressed dng would require more processing and take more time to put through the buffer and onto the card, but that may not be the case.

With the larger file size of the Uncompressed DNG, it may take a little longer.  Really, for most of Leica shooters, this really isnt an issue due to how infrequently we use high bursts. Also, there might be some parallels here one can draw between JPG vs RAW processing time in camera and how quickly the buffer fills up or the difference in number of frames per second, but the compression ratio of those 2 formats is much different than CDNG/DNG, especially since JPG is a LOSSY compression. 

 

In reality, that last paragraph is good for academic discussion. The first 2 bullet points are where the practical digital photographer lives. Pick item 1 or 2 and be done with it and move on with life

I've done technical benchmarking very much like you describe. There is no value in uncompressed data. The CPUs and image processors are so good at doing the kinds of operations that are needed for compression that compression comes for free or even has a negative cost.

 

To be all geeky about it:

1) There is a cost in power to clear a block on a SD card.

2) There is a cost in power to transfer data from the CPU's buffers to the SD card.

3) There is a certain amount of processor time needed to transfer data.

on the other hand there is:

4) Processor time and power to compress the data

 

1) When you store less data you have to clear fewer blocks on the SD card. This saves power.

2) When you transfer less data you use less power

3) Processors use the least amount of power when they are asleep. When you keep the processor awake for longer to transfer more data you end up using more power than if you let the processor sleep longer. 

4) The kinds of operations to implement Deflate on image data are so quick and take so little power that the power saved by 1, 2, & 3 vastly cover the cost to Deflate the data. I don't have the stats for Leica but basically just the cost of writing to the SD card is about 4x the cost to compress the data.

 

On the bigger machines that I work with. The speed of RAM in relation to the aggregate speed of processors and their demands for data is nearing a point such that it may be more efficient to have RAM compressed and only have the processor caches uncompressed. So don't be surprised if you start seeing things like this appearing in memory controllers. The old arguments about the cost in processor cycles to do things like compress data losslessly really don't apply anymore.