New M DNG compression

[cirke]

Advertisement (gone after registration)

I worry over future reconstruction

me too

[pop]

Lossless Compression is just that: compression.

 

So far no one can answer for me this simple question.

 

If it's a no brainer to use lossless compression then why give us the choice? Why even have the line item with 2 choices?

 

I came exactly to the same question

 

Have you seen this:

 

If you use software which does not handle files which are compressed, do not compress the files.

 

If you use software which does handle compressed files, compress the files.

[cirke]

Have you seen this:

yes but is it good for the past or the future ?

Shall I open compressed/uncompressed DNG files in the same way in 10 years

 

thanks

[jaapv]

Maybe not in the same way, but consider this. If a new format would come in to replace an existing one (and this goes for TIFF, JPEG and all others too) there will be conversion programs to convert to the new standard. It will either be a question of running the conversion program or, if that has become ancient too, consulting an IT museum and converting in their historical department.

For DNG specifically, this is open source, so the parameters can always be found to implement, even in a far future.

 

To set your mind at rest, just have a look at the list of obsolete formats that are supported by Photoshop.

[mjh]

yes but is it good for the past or the future ?

It is. The capability of dealing with compressed DNGs isn’t likely to go away, short of envisioning some post-apocalyptical scenario where the technology to handle uncompressed DNGs would probably be lost as well.

 

There isn’t much reason to offer a choice here, if it weren’t for the fact that people, as a rule, love to have a choice. (But when given a choice they complain about being forced to make a choice, and about all the agonising that entails.)

[cirke]

To set your mind at rest, just have a look at the list of obsolete formats that are supported by Photoshop.

yes of course

[algrove]

Advertisement (gone after registration)

Most pros I talk to on this subject, say simply -- disk space is cheap so they will not compress.

[kdriceman]

Most pros I talk to on this subject, say simply -- disk space is cheap so they will not compress.

 

True enough, but I wonder whether there is a performance increase in using the compressed files. I.e. can you get a burst of more shots before the buffer bogs down. Has this been discussed yet in this thread?

[digitalfx]

Most pros I talk to on this subject, say simply -- disk space is cheap so they will not compress.

 

Disk space is insignificant, what really matters is performance. If the cameras buffer fills up faster and performs slowly using uncompressed files then why?

 

The M uses a true lossless compression. Lossless is just that, lossless.

As far as the worry if opening the files in 10 years this is unwarranted. Lossless compression is standard and the process is the same on all lossless compressions. It's not going to become obsolete.

Lossless compression is very basic and has been the same since introduced.

[mjh]

Disk space is insignificant, what really matters is performance. If the cameras buffer fills up faster and performs slowly using uncompressed files then why?

While the buffer doesn’t fill up faster, it empties slower.

[digitalfx]

While the buffer doesn’t fill up faster, it empties slower.

 

This is correct. I just did a quick test and the difference isnt as significant as I expected. Im still sticking with lossless compression. The buffer empties slower and therefor performance takes a slight hit after 8 exposures or so.

 

Since the file sizes decrease from 48mb to roughy 23mb, why doesnt the buffer fill up faster?

 

If Im not mistaken, the DNG Lossless Compression employed here is based on JPEG 2000. This is truly lossless. We use a similar, if not identical compression on our feature film work. And there is absolutely no difference between the two files, but the resulting file sizes are significantly smaller.

[algrove]

So far my M has not had the dreaded M9 buffer problem and I shoot it same as before with my M9.

 

Simple, if I am in a situation and I see it is slowing, then I can compress. Simple solution.

[mjh]

Since the file sizes decrease from 48mb to roughy 23mb, why doesnt the buffer fill up faster?

Because the buffer contains raw sensor data. Image processing (most of it anyway), JPEG compression and/or DNG compression happens downstream from the buffer.

 

If Im not mistaken, the DNG Lossless Compression employed here is based on JPEG 2000.

Actually it is based on Lossless JPEG. Lossless JPEG was specified in 1993 and is the older of the two lossless JPEG compression methods, the other being JPEG-LS. And of course there is the lossless mode of JPEG2000. All these compression methods are truly lossless, but apart from that, they differ considerably.

[CheshireCat]

In 'Aperture' I have an M9P - 18MB file shot as a compressed DNG. If I export out the original from Aperture it remains an 18MB file. If I export out as a 'Full Size DNG' the file is nearly 37MB.

 

By doing this, you are basically wasting 18MB of disk space.

The M9P encoding is lossy, and you cannot restore bits of information that are gone forever.

 

Am I completely wrong in assuming I have opened a compressed DNG and saved it out in it's uncompressed state?

 

I would say so

We should define the original 18MB file produced by your M9P as quantized rather than compressed.

After quantization, you have half the bits, but the file is not actually compressed, as far as the entropy is concerned.

Quantization is destructive, therefore when you save it as "Full Size DNG" you are not saving it out in the original uncompressed state, but just in a form that uses twice the bits as needed to encode the same information, and only makes sense if you need to import the file into applications that only support the full size format.

[CheshireCat]

yes but is it good for the past or the future ?

Shall I open compressed/uncompressed DNG files in the same way in 10 years

 

Yes, don't worry. Algorithms live forever on the Internet

 

Not the same with hardware. I would be more concerned about being able to attach your storage hardware to future computers in order to read the files

 

Seriously, I have backups on SCSI tapes from 10 years ago. I don't have the hardware to read them now, and I don't think it will be easy to find the hardware to read them in 2020.

[tele_player]

The solution to that problem is to transfer important data from old storage to new storage before the old storage becomes useless or unavailable. There's always an overlap.

 

Yes, don't worry. Algorithms live forever on the Internet

 

Not the same with hardware. I would be more concerned about being able to attach your storage hardware to future computers in order to read the files

 

Seriously, I have backups on SCSI tapes from 10 years ago. I don't have the hardware to read them now, and I don't think it will be easy to find the hardware to read them in 2020.