Compressed or uncompressed?

[tuscland]

Advertisement (gone after registration)

Hi all,

 

I'm surprised that nobody mentioned that the DNG compression that happens in the the M9 uses a lookup table in order to maximize the dynamic range. The 14 bits pixels that the sensor measures are not rounded to 8 bits at all, rather, a list of all possible values is made, and an algorithm picks the values that are most important in terms of dynamic range. Upon decompression, the decompression algorithm uses the lookup table stored in the DNG to restore the 14 bits values.

 

Of course, there is some loos in the process but this is far from the 42.9% of data loss previously mentioned in this thread.

 

As other posters said, we need real life example that outlines the differences between a (in camera) compressed and an uncompressed DNG.

 

 

Best,

Cam

[Revdockj]

So, where are the examples?

[erlingmm]

Funny deja vu from the M8 discussion a couple of years ago, but nothing new. There was a very thorough explanation in LFI 1/2007 p. 35 for those who are interested, but this is basically what 01af already has explained correctly here.

 

Other comments:

- Lossless compression: zip has been mentioned, if unzipping did not re-construct the original file bit by bit, it would not work. Simpler example: the words "one thousand zeroes" is a lossless compression of 000000 ....(one thousand times). The first expression is 19 letters long, the second 1000. This is definitely lossless compression. I have "thrown away" 98.1 % of the data, but it is still fully reconstructable. So: General assumptions about "throwing away half of the data" are meaningless.

 

- Rolo, Leica Ambassador: If you don't understand the math, why try to teach it and just create confusion?

 

- Future proof: Yes, I think this is an argument, there MAY come software, methodologies that will REVEAL a subtle difference between in-camera compressed and un-compressed, especially for critical images with strong highlights

 

- Final verdict, especially for the mathematically challenged: Can you see a difference? I have never seen an example where a diffence has been shown.

 

With the M8 this compression discussion created a lot of havoc, partly based on the same misunderstandings that have been repeated here. But then we did not have the option of uncompressed to compare with. With the M9 we have, and nobody can see the difference....

 

But as 01af says, there are options available, it is up to the photographer to decide, based on factors like space, processing speed, future-proof-ness, how critical ultimate quality is for the type of image etc. etc.

[zlatkob]

Time, maybe.

Money, how? Give me an example. What are you using to store your client's files at?

Four backups after each shoot but before editing (2 hard drives, optical disks, and off-site), followed by additional backups after editing. The optical disks (blu-ray & dvd) take up the most space, but I don't want to give up their extra security. One terabyte = 40 blu-ray disks. I'm not saying it's a lot of money; I'm just saying file size is not a trivial question when money, time and shelf space are all considered.

[01af]

The words "one thousand zeroes" is a lossless compression of 000000... (one-thousand times). The first expression is 19 letters long, the second 1000. This is definitely lossless compression. I have "thrown away" 98.1 % of the data ...

No, you haven't. You have thrown away most of the redundancy but not one single bit of the data.

 

 

Future-proof: Yes, I think this is an argument, there MAY come software, methodologies that will REVEAL a subtle difference between in-camera compressed and un-compressed, especially for critical images with strong highlights.

This applies to lossy compression only, if at all.

 

 

But then we did not have the option of uncompressed to compare with. With the M9 we have, and nobody can see the difference ...

Jaap claims he can ... but doesn't dare to adduce any evidence.

[swamiji]

The term lossless is silly, what the term means is that there is no reduction is quality of image, which is true since all the information is accounted for, not that it is missing data, which it is. Remember a zero is data too, and the US debt is only $16 (redundant zero's removed). It is easier to fix a uncompressed file than a compressed file.

 

My understanding from Leica, is that on the M9 compressed file is 8-bit, and uncompressed is 14-bit. On 8-bit equipment, it is difficult to reproduce any difference.

[erlingmm]

Advertisement (gone after registration)

The term lossless is silly

 

The term lossless is an accepted scientific term, and is precisely defined, see for example:

Lossless data compression - Wikipedia, the free encyclopedia

Quoting the first lines: "Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data. The term lossless is in contrast to lossy data compression, which only allows an approximation of the original data to be reconstructed, in exchange for better compression rates."

 

 

My understanding from Leica, is that on the M9 compressed file is 8-bit, and uncompressed is 14-bit. On 8-bit equipment, it is difficult to reproduce any difference.

 

Your understanding is wrong. The image data are not 8-bit in the ordinary sense, the 8 bits are used in a lookup-table to reconstruct the 14-bit image, closely approximating the uncompressed 14 bit, to the extent that no-one so far have been able to show the difference in actual images.

[swamiji]

There are lots of silly scientific terms... I have been a DASD manager for 30 years, using various forms of compressions, this is just another one....

 

On Leica's "lossy" compression, The info came from them.... confirmed by support.

[jaapv]

No

 

 

 

Jaap claims he can ... but doesn't dare to adduce any evidence.

How do I produce evidence that I can see something:confused: This is precisely the type of forum arguing that holds no appeal for me. I don't care what others think they can see or cannot - I report my experience. Take it or leave it. My experience being that normally the difference is marginal, but that uncompressed files will stand more extensive post-processing before breaking up.

[jaapv]

There are lots of silly scientific terms... I have been a DASD manager for 30 years, using various forms of compressions, this is just another one....

 

On Leica's "lossy" compression, The info came from them.... confirmed by support.

Well, the LFI article that was written in conjunction with Leica claimed " equivalent to 12 bit" and supported it with a graph.

[diogenis]

....

 

- Rolo, Leica Ambassador: If you don't understand the math, why try to teach it and just create confusion?

How can you tell he doesn't understand the maths, when there aren't any here? Rolo is offering his experience as a photographer on a photographer's forum. How bad is that?

- Future proof: Yes, I think this is an argument, there MAY come software, methodologies that will REVEAL a subtle difference between in-camera compressed and un-compressed, especially for critical images with strong highlights

 

- Final verdict, especially for the mathematically challenged: Can you see a difference? I have never seen an example where a diffence has been shown.

 

With the M8 this compression discussion created a lot of havoc, partly based on the same misunderstandings that have been repeated here. But then we did not have the option of uncompressed to compare with. With the M9 we have, and nobody can see the difference....

"the mathematically challenged" know that the difference is there. Showing the difference is something else. I believe you are asking us to show you the difference, but to do this, we have to know where to look and understand how the algorithm works. If we know how the algorithm works then we will know where it is vulnerable and know where to look. Since olaf tried so hard to explain, he can tell us where to look I suppose

edit: and then setup an environment to show the differences, which is quite some work for something that -again- we know it is there

But as 01af says, there are options available, it is up to the photographer to decide, based on factors like space, processing speed, future-proof-ness, how critical ultimate quality is for the type of image etc. etc.

[diogenis]

Four backups after each shoot but before editing (2 hard drives, optical disks, and off-site), followed by additional backups after editing. The optical disks (blu-ray & dvd) take up the most space, but I don't want to give up their extra security. One terabyte = 40 blu-ray disks. I'm not saying it's a lot of money; I'm just saying file size is not a trivial question when money, time and shelf space are all considered.

Woa! 4x backups??? why??

Well in your case -yes- you both need $$$ and time, shoot compressed

edit: I'm sure you've heard of drobo. get some server grade disks, a ups and you are done Zlatkob.

[stunsworth]

I'm sure you've heard of drobo. get some server grade disks, a ups and you are done

 

No you're not. Something like a Drobo does not provide a full backup. Having several generations of back up is a common IT practice. Whether it's needed for an individual is up to them - for me it's not.

[zlatkob]

No you're not. Something like a Drobo does not provide a full backup. Having several generations of back up is a common IT practice.

 

That's correct. I haven't looked at Drobo lately, but as I recall, it is a form of RAID. RAID is not a true backup. It offers multiple disks in a unit and, if one disk fails, all of the data is still retained. But if two disks fail at the same time, you've lost a lot of data. And if the something happens to the Drobo itself (theft, fire, lightning), you can lose years' worth of photos. Not good. While a Drobo is nice, it can't be relied upon for backups by itself, imo.

[diogenis]

You got two things here:

The storage system itself (drobo, cds, dvds or other)

And your software you use for backups or even manual procedures maintained by you (archives x1, x2 etc)

I don't want to advertise drobo, but drobo can use several raid levels. Anyway, one of them is mirroring. As you create your archives you fill one hard disk and the other one acts as a duplicate. When the disk is full you take one out and this is your library. You store this into your fireproof safebox. Then you take a fresh HD reformat the array and you begin new. Drobo is just simple to use and is one of many ways to increase safety. You are not the IT director of Apple's newest data center, so I believe 4x backups are quite a lot but whatever...clickme

[jaapv]

I use in-computer backup on another disk, which could be RAID, but I use Second Copy, and for external backup I use IASO.

 

Best Online Backup Software | IP-based Backup Solutions from IASO Backup Technology

[diogenis]

Now, if you ever need a file from your old library, you can hot unplug the array you currently use for an older library, pull the file and go back again.

HDs can be so cheap: last time I bought one, you can buy 1TB for less than $80 or something(if memory serves me right) and these were server grade disks.

[diogenis]

I use in-computer backup on another disk, which could be RAID, but I use Second Copy, and for external backup I use IASO.

 

Best Online Backup Software | IP-based Backup Solutions from IASO Backup Technology

 

Jaap's solution is even more elegant. Everything is stored in the cloud -maybe in Apple's data centers, which we know who manages them -

[jaapv]

I take no credit. I use the same system professionally. It does not do to lose x-rays or patient data

[Advertisement]

Hallo,

Look here: Leica M

[diogenis]

Yea it's an overkill for a photographer, but at least other worry about keeping frequent backups