smaller DNGs!

[Lightwrangler]

Advertisement (gone after registration)

And this is exactly the problem here.

 

Although I am not right in the mood for it, I try to explain it carefully and politely:

 

The expression "Can you provide a source/link" is very common on internet-forums. Because most topics discussed there are quite complex it is usually not possible to provide a single source that explains EVERYTHING and the world.

 

Because that is a fact, the expression "can you provide a source" usually just means "i do not believe you", but without further explanation.

 

If that is what you mean, just say it.

 

 

if you care to read one of my posts above, you will see I already have.

 

All i see is a guess, an unproved statement about lossless JPEG (which btw was never the topic) and an insult. What makes you think your guess is better then a well discussed fact on this forum since i am a member (and quite certainly already long before i was a member)? The fact being the M8/9 has a pretty weak processor.

 

 

I could be more technical, but I am convinced you do not have the technical knowledge, and especially not the willingness to understand.

 

Thank you for calling me an idiot. Maybe you can provide a link?

 

Mike

[CheshireCat]

Although I am not right in the mood for it, I try to explain it carefully and politely:

 

The expression "Can you provide a source/link" is very common on internet-forums. Because most topics discussed there are quite complex it is usually not possible to provide a single source that explains EVERYTHING and the world.

 

Because that is a fact, the expression "can you provide a source" usually just means "i do not believe you", but without further explanation.

 

If that is what you mean, just say it.

 

I have also said "this is ridiculous". Don't you think this implies "I do not believe you" ?

And thanks for explaining that the Internet is full of ridiculous unfounded myths. Like the one you are endorsing here in this forum.

 

All i see is a guess, an unproved statement about lossless JPEG (which btw was never the topic) and an insult. What makes you think your guess is better then a well discussed fact on this forum since i am a member (and quite certainly already long before i was a member)? The fact being the M8/9 has a pretty weak processor.

 

Lossless DNG contain lossless JPEG. Lossless DNG are smaller. Read the thread title. That is the topic.

 

Can you please quote my insult ? I may have different sensibility, and I am new to this forum, so please help me understand what users here take as insults.

 

"The M8/9 has a pretty weak processor" is just a generic sentence and I will agree if you completed it with "compared even to most low-end point&shoot cameras". But I will not agree if you completed it with "unable to handle DNG lossless compression".

 

Thank you for calling me an idiot. Maybe you can provide a link?

 

I never called you an idiot (and don't think you are).

 

About the link:

http://www.w3.org/Graphics/JPEG/itu-t81.pdf

 

The algorithms used by DCT-JPEG (aka JPEG, currently implemented in M8/M9) is much more computational intensive than DPCM-JPEG (aka Lossless-JPEG). One may still argue that the lossy phase outputs less data to be processed by the final Huffman encoding, but this does not compensate for the earlier lossy stages.

 

Finally, my educated guess is that Analog Devices JPEG library for the BF-5xx does not support lossless compression.

 

Hope this helps stopping Internet myths.

[ho_co]

Can you cite the source ?

 

Long discussions on the topic here when the M8 was current; at least two articles in LFI; and Mark Norton's Anatomy thread (http://www.l-camera-forum.com/leica-forum/leica-m8-forum/21331-anatomy-leica-m8.html) for starters.

 

Actually, one doesn't usually need a citation for such accepted facts, but since you're new to the forum, those pointers may help.

 

Some folks get a bit testy when what is today common knowledge is questioned, not realizing that the facts may be unfamiliar to the poster. My apologies for them.

[pico]

Actually, I believe what happens is that when you import your raw files into Lightroom and you check lossy compression, then Lightroom will automatically create a demosaiced & JPEGed DNG of your original raw file. Now, in Leica case, that original is already a DNG, but in Canon or Nikon's case it isn't. Adobe are selling this as an archive or space-saving option, but as I mentioned before it does kind of lock you into their interpretation of your raw data. [...]

 

Adobe is not selling it as a space-saving option. What they doing is trying to establish a free and clear standard. A 'lossy' compression of DNG is incomprehensible and counter-productive to me because lossless compression is perfectly feasible, but vendors can do it if they want, and in that case it is the software developer who is 'locking' you in, not Adobe.

 

Vendors who want to get out of the gate quickly and who want to avoid development of their own format and translation software can adopt DNG. (IOW - on the cheap). Is Leica making some kind of bitwise in-camera lossy compression of their own or are they just obviating some information? I hope not.

 

I never save compressed in-camera. It makes no sense because it does not save faster and large cards accommodate enough uncompressed DNGs to get anyone into trouble. (If Leica does not speed up transfers to card, then that will be a profound disappointment to many.)

[k-hawinkler]

Adobe is not selling it as a space-saving option. What they doing is trying to establish a free (but not 'open') standard. A 'lossy' compression of DNG is incomprehensible and counter-productive to me, but vendors can do it if they want, and in that case it is the software developer who is 'locking' you in, not Adobe.

 

Vendors who want to get out of the gate quickly and who want to avoid development of their own format and translation software can adopt DNG. (IOW - on the cheap). Is Leica making some kind of bitwise in-camera lossy compression of their own or are they just obviating some information? I hope not.

 

I never save compressed in-camera. It makes no sense because it does not save faster and large cards accommodate enough uncompressed DNGs to get anyone into trouble. (If Leica does not speed up transfers to card, then that will be a profound disappointment to many.)

 

 

Yes, it does on my M9. http://www.l-camera-forum.com/leica-forum/2016614-post3.html

[pico]

[...]Lossless JPEG would also halve the time to write a DNG to the SD card. It is really a no-brainer.

 

One has to consider the time required to create that compressed file (whatever the format), and THEN consider the write to card. If the compression requires too much time, then the write saving time can be insignificant.

 

In any event, this has been well covered by some very knowledgeable people here as in this excellent thread: http://www.l-camera-forum.com/leica-forum/leica-m9-forum/110293-dng-compressed-2.html

 

Best,

Pico

[pico]

Advertisement (gone after registration)

Yes, it does on my M9. http://www.l-camera-forum.com/leica-forum/2016614-post3.html

 

Thank you, K-H. I will read up.

 

In turn, I offer this news which might help others understand Adobe's changes to LR4 such as their Fast-load data option.

Adobe offering new reasons to get DNG religion | Deep Tech - CNET News

 

(I'm posting too much and will stop here. G'Nite, all.)

 

EDIT: Wow! I would like to get some Pretec SD cards. Thanks again for the help.

[Jamie Roberts]

Adobe is not selling it as a space-saving option. What they doing is trying to establish a free and clear standard. A 'lossy' compression of DNG is incomprehensible and counter-productive to me because lossless compression is perfectly feasible, but vendors can do it if they want, and in that case it is the software developer who is 'locking' you in, not Adobe.

{snipped}

I never save compressed in-camera. It makes no sense because it does not save faster and large cards accommodate enough uncompressed DNGs to get anyone into trouble. (If Leica does not speed up transfers to card, then that will be a profound disappointment to many.)

 

Pico, I think you're incorrect here on a number of items.

 

First, Adobe (no-one else, no third party) *is* selling lossy DNG in LR 4 as an archive format--which is all about saving space when you back up. They do it on their forums, and their advocates are doing it in LR and related forums. They even do it in the link you provided And don't get me wrong: as an archive format or emergency backup it has its appeal--if you use Lightroom.

 

If you're shooting, say, a Nikon D800 your raw NEF files are 75mb each What Adobe is saying is "surely all your files don't need to be NEF files that take up this much space? Only very special ones need to be kept that way. For the rest, why don't you convert and save as lossy DNG?"

 

This is, as I've said before, an 8bit curve-corrected, demosiaced JPEG rendering where you can still change the white balance

 

Those same d800 files are now about 5-8mb each

 

No-one but Adobe is doing this to date, despite the "standardization" of DNGs.

 

But even if, say, C1 supported the lossy-compressed DNG format (which I doubt they will) you will be locked into LR's demosaic algorithm and lower bit depth, which is locked in at conversion time.

 

As for the M9, Karl-Heinz already pointed out that Leica's lossy compression (which is NOT JPEG compression and not the same as what LR is doing) actually makes the M9 much more responsive, since the M9's "slow" processor processes compression faster than it writes out a large file.

[sandymc]

First, Adobe (no-one else, no third party) *is* selling lossy DNG in LR 4 as an archive format--which is all about saving space when you back up. They do it on their forums, and their advocates are doing it in LR and related forums. They even do it in the link you provided

 

In fairness, not all of Adobe is advocating lossy DNG as an archive format. Notably, the Camera Raw engineering team, who should know, are specifically on the record as NOT recommending lossy DNG in archival applications. I pointed this out in the blog post I wrote about lossy DNG back when it first emerged: ChromaSoft: Lightroom's new lossy DNG compression Take a look right at the bottom.

 

Adobe has always been cursed with semi-official advocates (without mentioning names) with more enthusiasm and attitude than actual technical knowledge :roll eyes:

 

Of course, Adobe tolerates and even encourages said advocates.

 

Sandy

[Jamie Roberts]

In fairness, not all of Adobe is advocating lossy DNG as an archive format. Notably, the Camera Raw engineering team, who should know, are specifically on the record as NOT recommending lossy DNG in archival applications. I pointed this out in the blog post I wrote about lossy DNG back when it first emerged: ChromaSoft: Lightroom's new lossy DNG compression Take a look right at the bottom.

 

Adobe has always been cursed with semi-official advocates (without mentioning names) with more enthusiasm and attitude than actual technical knowledge :roll eyes:

 

Of course, Adobe tolerates and even encourages said advocates.

 

Sandy

 

Thanks Sandy--I really have to check your blog more often (but I did actually read that one on my way to understanding this) Adobe's 'semi-official' advocates are causing quite a stir in other places on this!

 

And the team's response about not being that comfortable with it as an archive format is interesting. It begs the question "why do it at all then?" For point and shoot in-camera "raws"?

[pico]

Thanks Jamie. Apparently there are others, even within Adobe's circle, who have differing views on 'what should be'. It is no surprise to have product development and pioneers at healthy odds.

 

I'll stay tuned to learn more.

 

.

[sandymc]

And the team's response about not being that comfortable with it as an archive format is interesting. It begs the question "why do it at all then?" For point and shoot in-camera "raws"?

 

Well, good question(!). In camera "raws" does seen like a legitimate use of the technology - in my opinion, a lossy DNG could be a lot better than a JPEG. There also seems to be a point of view at Adobe that some images are "throw away", and don't need full raws.

 

But I'm dubious about the whole lossy DNG thing for two reasons: (1) who, other than Adobe, will actually support it? and (2) it undermines DNG as a raw format, in the sense that most people will no longer know whether a "DNG" file is a raw, or a fancy JPEG. Which would seem to me to be a real negative for the average photographer's perception of DNG.

 

Regards,

 

Sandy

[CheshireCat]

Long discussions on the topic here when the M8 was current; at least two articles in LFI; and Mark Norton's Anatomy thread (http://www.l-camera-forum.com/leica-forum/leica-m8-forum/21331-anatomy-leica-m8.html) for starters.

 

Actually, one doesn't usually need a citation for such accepted facts, but since you're new to the forum, those pointers may help.

 

Some folks get a bit testy when what is today common knowledge is questioned, not realizing that the facts may be unfamiliar to the poster. My apologies for them.

 

No problem, everybody is entitled to their own opinion.

 

And my opinion is that going through those 20 pages of "anatomy" posts, I have not found any real evidence... well actually any real technical comment. Can you point out the "fact" ?

 

I am sorry to question this "fact" again, but I think I have already proved it is an Internet Myth.

 

I am new to this forum, and honestly, I can understand why my favorite camera leaves a lot to be desired: nobody is questioning its issues ! This has more to do with faith and religion rather than engineering and arts.

 

And honestly, after 10 posts, I don't "feel at home"

[Jamie Roberts]

I don't know which CPU is in the M9, but my educated guess is that this is more a software issue than a hardware one. Lossless JPEG is a very simple algorithm, simpler than lossy JPEG.

Lossless JPEG would also halve the time to write a DNG to the SD card. It is really a no-brainer.

 

Ok, so I just think you're missing something here...or I am, which is fine...

 

Can you tell me if lossless JPEG is actually a JPEG? As in "gamma corrected, "demosaiced" and then written out in 8bpp colour?"

 

If the answer is "yes," then why would anyone want that over a lossy DNG with Leica's compression scheme? The DNG would have greater bit depth, would be usable in most raw converters, and would still have a lot more data intact.

 

The M9 writes JPEGs fast enough, doesn't it? It writes compressed DNGs (not using JPEG compression) fast enough. It writes uncompressed DNGs, well, more slowly

 

Lossless JPEG--if it's a JPEG--wouldn't serve to make better DNGs. While it might improve the JPEGs nobody I know shoots an M9 using JPEGs. I myself have never even bothered to see what they look like after the initial reports that they were similar to the M8

 

So your solution is technology in search of a problem, as far as I can tell. Even if the processors could processes lossless compression schemes, you have to wonder why they'd do that since we're not talking a raw file anymore--right?

 

Having said that, welcome to the forum I think you'll find, for the most part, Leica lossily compressed DNGs are very good.

[CheshireCat]

One has to consider the time required to create that compressed file (whatever the format), and THEN consider the write to card. If the compression requires too much time, then the write saving time can be insignificant.

 

Agreed. and SD write speed depends on multiple factors.

Writing an uncompressed (uncorrected, no lens detection) DNG on a fast Samsung card on the M9 seems to take years compared to equivalent file sizes on other cameras. Again, this seems a hardware issue rather than a software one, but given how poor the firmware in M9 is, I really don't know. In any case, I suspect SD writing is not properly implemented on the M9.

 

In any event, this has been well covered by some very knowledgeable people here as in this excellent thread: http://www.l-camera-forum.com/leica-forum/leica-m9-forum/110293-dng-compressed-2.html

 

Sorry, I fail to recognize how that thread may provide useful information about what we are discussing here.

[CheshireCat]

[quote name='Jamie Roberts;2062125

]Ok' date=' so I just think you're missing something here...or I am, which is fine...

[/quote']

 

It is fine

 

Can you tell me if lossless JPEG is actually a JPEG? As in "gamma corrected, "demosaiced" and then written out in 8bpp colour?"

 

If the answer is "yes," then why would anyone want that over a lossy DNG with Leica's compression scheme? The DNG would have greater bit depth, would be usable in most raw converters, and would still have a lot more data intact.

 

Answer is "no".

 

The M9 writes JPEGs fast enough, doesn't it? It writes compressed DNGs (not using JPEG compression) fast enough. It writes uncompressed DNGs, well, more slowly

 

Not by my (and what should be you guys') standards.

Note that uncompressed DNGs are the easier ones to process. What you say clearly shows that the performance bottleneck is not the DSP but SD write throughput.

This means that the M9 is most likely not DSP-bound but memory bound (which is exactly my point in this thread).

 

Lossless JPEG--if it's a JPEG--wouldn't serve to make better DNGs. While it might improve the JPEGs nobody I know shoots an M9 using JPEGs. I myself have never even bothered to see what they look like after the initial reports that they were similar to the M8

 

Lossless-JPEG is different from JPEG, more than hot-dogs are different from dogs.

As the name implies, they are totally lossless meaning they will output the same bit-exact image of an uncompressed DNG. As if you zipped a DNG, just more efficiently.

 

So your solution is technology in search of a problem, as far as I can tell. Even if the processors could processes lossless compression schemes, you have to wonder why they'd do that since we're not talking a raw file anymore--right?

 

Wrong, It is good technology. It is a raw file as much as an uncompressed DNG is. We spent lots of money for our M8/9. We deserve something better.

 

Having said that, welcome to the forum I think you'll find, for the most part, Leica lossily compressed DNGs are very good.

 

Thanks, I feel a bit more at home now

And no, I don't like Leica lossily compressed DNGs a bit (no pun intended).

[CalArts 99]

fwiw, here's Eric Chan himself explaining the details of the lossy DNG process and why he developed it for Adobe. And he also says: "To be honest, I'm not really comfortable with the idea of lossy compressed DNG for archival storage purposes."

 

Adobe Forums: Info on lossy DNG?

[k-hawinkler]

I wouldn't use lossy compressed DNG of the Adobe kind.

It is important to distinguish this kind from the lossy compression Leica uses in the M9.

I use the Leica kind as it speeds up the M9 when taking pictures.

[Jamie Roberts]

{snipped}

Wrong, It is good technology. It is a raw file as much as an uncompressed DNG is. We spent lots of money for our M8/9. We deserve something better.

 

Thanks, I feel a bit more at home now

And no, I don't like Leica lossily compressed DNGs a bit (no pun intended).

 

So what you are saying when you say "lossless JPEG" is "lossless JPEG compression"--you're not talking about the other restrictions on the format.

 

Ok--thanks for explaining that: it's the use of the term JPEG that throws me What you say about the bottleneck makes sense too. I've always wondered why the M9 couldn't write losslessly compressed raw files like every other pro camera available.

 

Oh, and I'm glad you feel a bit more welcome

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[CheshireCat]

Ok--thanks for explaining that: it's the use of the term JPEG that throws me

 

You are not the only one

The term "JPEG" just stands for "Joint Photographic Experts Group" which is the group of people who created both DCT (lossy) and DPCM (lossless) compression variants.

Since DCT is what 99.9% of applications will use, then the term JPEG alone is associated to the DCT lossy variant, and since most people don't know a lossless variant exists, they associate the word "JPEG" to "lossy".

 

In reality, JPEG DPCM is also used by some proprietary RAW formats, such as Canon's CR2.