Leica news: LFI #1/2007

[scott kirkpatrick]

Advertisement (gone after registration)

I don't see why the fact that IRs may be reflected by a coating in any angle would pose problem. Again sorry if this has been solved previously.

Do you agree with Riley, Scott?

 

I think the claim that internal IR-cut filtering could contribute to flare and ghosts dates to the statement from Solms, translated by Leica US, that Sean Reid added to his review M8-II on Nov 6, 2006. (Proper crediting of ideas in this discussion has gotten as complex as legal citation in court briefs.) Deep in that statement is the claim that the only safe place to have light bouncing off the highly reflective surface of a multilayer IR interference filter is outside the optical system. The problem isn't IR light, it's the visible light that would also reflect off the surface at angles more than 30 degrees from vertical. IR is still rejected by such a filter on its second or third bounce, but pink light could come back at an angle that permits passing through the filter. Any such internal reflections have to bounce off the walls of the camera body next, so I would think there is a solution involving additional baffling if that is a real danger. But it is not a quick fix, it is an engineering path requiring further work.

 

The Nov. 6 statement had awkward translation, some arrogant assumptions, and used terms like "astigmatism" in ways unique to Leica. The immediate responses misunderstood some of it, assumed ignorance where there was just poor communication, and jumped to the conclusion that Leica was confusing the thickness of a protective or absorbtive coverglass (mm) with the thickness of a dichroic filter (microns). It actually took several rounds of posts before it was generally realized that Leica had thrown out dichroics altogether, since in doing so they ignored well-known issues that surfaced and had to be dealt with in earlier digital cameras from Kodak and Nikon.

 

scott

[scott kirkpatrick]

German translates reasonably well to English, meaning-wise.

 

As long as you can still hear the original German echoing in the back of your head. I recently received a note from Leica explaining that M8 purchasers would "become" IR filters, which made perfect sense because "bekommen" = "recieve" in the original language. There seems to be an excess of cognates available, since English and German have such a long common path.

 

I recently finished writing a 600 page book in English with a German coauthor who speaks and writes fluent English. Extracting the final 1% of Teutonicisms that both of us had thought just added color was a very painful experience which really paid off in clarity of the text.

 

scott

[carstenw]

So that leaves the 8- vs. 16-bit issue. Does the magazine say anything about the thinking on this? You say LFI compared the 8- and the 16-bit images and found both equal, and the 8-bit file-writing faster. So the 8-bit software was already in the camera when they went back to compare the 16-bit software.

 

Certainly that doesn't mean someone had decided to go to an 8-bit DNG without telling anyone?

 

In other words, for LFI to confirm that the M8 writes luscious files is nice, but am I the only one who would like more info on how the decision came to be made in the first place? No other company has 8-bit DNG's, so someone must have had to sell the idea to Leica. Any word on how that came about? Seems to me like a major case of 'thinking outside the box,' and both whoever did it and whoever accepted the idea should be named and congratulated.

--HC

 

Quick correction (I am not sure we are saying the same thing): they had two M8s, one old test model which still wrote 16 bit files, and one newer one which writes 8 bits per pixel, like ours.

 

(I recall seeing that someone had said that Adobe recommends this way of storing data in their DNG specification, but that no one had done it up until this point.)

 

There are essentially three ways to go: uncompressed RAW (slow writing speeds, large files), compressed RAW (slow writing speeds due to extra whole-image processing, small files), and some simple-to-compute trick to make them smaller, such as this one, which yields fast writing speeds AND a small file.

 

The trick is this: the sensor reads out 14-bit files. 14 bits gives values in the range from 0 to about 16,000 (2^14). This is too large to store in an 8-bit number obviously 2^8=256, btw), but if you take the square root of the value first, then it fits (sqrt(2^14)=2^7=128). Taking the square root of a number for a limited range of input values can be implemented in a chip as a single operation, probably just a lookup table, and thus remains fast. Without this choice of compression, we would not get the battery life nor the 2 fps the M8 has in shipping form.

 

---

 

But what does this mean for the image? All values from 0-16000 are mapped to values from 0-128 in the DNG, and can re re-squared to get back values from 0-16000 on the computer, but clearly the 128 different possible values do not map back to all values from 0-16000, but rather, a limited subset of those values. All values close to each other in the 0-16000 range will get mapped to a single value in the range 0-128, and when this is reversed, the subtlety among the original numbers is gone, and the values now all look the same, whereas before they were merely similar.

 

The question is just if this loss of subtlety is visible. This is what the LFI tests established is not the case. They took pictures of white cars, for example, and viewed and examined and printed them, to try to find problems, but couldn't. I want to do some testing on this myself, but my M8 is not back yet. I am hopeful that Leica did it right, and that quality is unaffected, but I want to be sure.

 

---

 

There is another interesting point here, something which the article does not mention, and which I have mentioned before on this forum, but no one picked up on it. As background, it is necessary to understand that there is much more compression in the high values than there is in the low values. For example, 0*0=0, 1*1=1, 2*2=4, 3*3=9, 4*4=16, 5*5=25... It is easily seen that the gaps between the values become larger as the values get larger. For the last few: 126*126=15876, 127*127=16129, with a gap of 253 between those last two numbers! This means that a mapping from values read by the sensor to values we get in our RAW converter reads like this, assuming truncation:

 

0 -> 0

1 -> 1

2 -> 1

3 -> 1

4 -> 4

5 -> 4

6 -> 4

7 -> 4

8 -> 4

9 -> 9

...

 

The last 253 values all get stored as 15876 except the last one which gets stored as 16129. I hope I didn't lose anyone there. To get this sequence, I took the square root of the number on the left and threw away the fraction (stored the DNG on the camera) and then multiplied it by itself again (converted to 16-bit in the RAW converter).

 

I presume that RAW photographers here are familiar with the expose-right rule. In summary, it has to do with the fact that in the past, sensors readouts have been stored linearly. Sensors count photons, small particles of light. Twice as many photons gives us twice as large a value in the RAW. Now note also that twice as many photons, i.e.. twice the light, is a one-stop difference. Combining these two observations, we realise that the brightest stop in the scene is stored in the top half of the numerical values. The second-brightest stop is stored in the top half of the *remaining* values, i.e. the 2nd quarter of the range. And so on. Here is a small list, for a 14-bit sensor (2^14=16000):

 

top stop: 8000-16000

second stop: 4000-8000

third stop: 2000-4000

fourth stop: 1000-2000

fifth stop: 500-1000

sixth stop: 250-500

seventh stop: 125-250

eighth stop: 64-125

ninth stop: 32-64

tenth stop: 16-32

11th stop: 8-16

 

I will stop here, since the dynamic range of these cameras is around nine stops. Note that for the brightest stop, we have 8000 numerical values to express the subtlety in the colours, whereas in the darkest shadows, we have only 8, for a camera with 11 stops of dynamic range, probably more than even the M8. And this is for RAW! For JPG the situation is disastrous, because it is an 8-bit format (2^8=256):

 

top stop: 128-256

second stop: 64-128

third stop: 32-64

fourth stop: 16-32

fifth stop: 8-16

sixth stop: 4-8

seventh stop: 2-4

eighth stop: 1-2

ninth stop: 0-1

tenth stop: 0

11th stop: 0

 

JPG can't express a dynamic range of more than 9 stops! And the fifth stop and down are expressed with just 16 values in total. Horrible shadows. To be fair, if you don't want to manipulate your image much, you won't really see it most of the time. But as soon as you want to lighten some shadows or something, you will see banding in the shadows. There just is no information down there to work with.

 

Hence, we "expose to the right". What that means, is that you find the brightest part of the scene, and expose the image so that this part is not over-exposed, but only barely. Typically, I do a spot measurement of the sky, and over-expose that reading by 2 stops on my 5D. Actually, I do 1 stop over-exposure, to be safe. That way, everything in the scene is as light as possible, without blowing it out, and I have as much information to work with as possible, in case I want to adjust the exposure later.

 

Expose to the right maximises our use of the available bits.

 

It has an annoying side effect though: my images often look over-exposed and washed out on the LCD on the back of the screen. I hate that part. I like to see the image as I want it eventually, but if I expose "correctly", rather than "right", I will often have much less information to work with if I want to change something.

 

---

 

As a side-note, this is a very unfortunate state of affairs, as the human eye-brain combination is much more sensitive to minor changes in dark values than in light values. This comes down to survival of the fittest. Our enemies hide in the shadows, not in the sunlight. What this means is that our insatiable demand for more bits of information from our camera is only the case because our storage format wastes so many bits storing subtle differences in bright values which we don't care about (can't see) at the expense of the number of bits used to store subtle variations in shadows, which we do care about (can see).

 

---

 

Here is the interesting part: The M8 doesn't store the values linearly, as we see above. It takes the square root of the values, which has the side-effect of using more bits to store small values (small gaps between neighbour values), and fewer bits to store large values (large gaps between neighbour colours). Essentially, it does what we want, not what we say. In other words, we want more subtlety in our images (what we want), and therefore demand more bits (what we say).

 

But does it work? Here is the progression of values used to store the various stops of light, as above. To generate these numbers, I took the 14-bit numbers (the first set, i.e.. 8000-16000, 4000-8000, and so on), and took the root of them. Then we can see how many values are used to cover the different ranges of the image:

 

top stop: 89-126

second stop: 63-89

third stop: 44-63

fourth stop: 31-44

fifth stop: 22-31

sixth stop: 15-22

seventh stop: 11-15

eighth stop: 8-11

ninth stop: 5-8

tenth stop: 4-5

11th stop: 2-4

 

We see that the brightest stops still have more subtlety in the stored values, but there is an overall shift of emphasis towards the shadows, and although we have only 128 values here, this format outperforms JPG for the fifth stop and on, exactly where we need the improvement. This format almost pulls even with a full 14 bits of uncompressed RAW in the shadows, but not quite. Maybe they should have taken the 3rd root Leica might also do something tricky, like multiply the 14-bit numbers by 4 before taking the square root, which extracts a little more accuracy. I could do another list at this point, but I need to shower and head out to the Martin-Gropius-Bau to meet a friend for a photo exhibition.

 

---

 

As I said, this all sounds very positive, but there is still less information there than with an uncompressed RAW, so I wish Leica had made compression an option, and I will test heavily to make sure that I don't see the difference in my work.

[carstenw]

There is another interesting point here, something which the article does not mention, and which I have mentioned before on this forum, but no one picked up on it. As background, it is necessary to understand

...

 

Whoops, I got side-tracked, and never got to the interesting part:

 

As background, we understand that we must expose to the right to maximise our use of the bits in an image. In other words, when taking a picture of a black cat in a coalmine, if we don't want all the subtle colour variations to be stored in just a few shades, we must expose it as a white cat in a snowmine (cough), and then reduce the exposure in software. This gives us thousands of shades of black, instead of a half-dozen.

 

But that was before. With the Leica M8, this may no longer be necessary, because the shift in the number of bits used for the various stops of exposure in the image is so subtle, and probably of no practical consideration.

 

In other words, Leica has found a way for us to be able to return to the old expose-correctly paradigm! Yay! Three cheers for Leica!

[cbretteville]

Carsten,

I might be reading this to quickly, but aren't your limoting the binary values to 7 bit? 0-127 (or 00h-7fh). Keping your coding and using the high bit (80h) as a flag you can increase the number range further. If you know it is a number in a certain range set the flag, suntract the base number and store only the 'interesting' part of the value Reversing it could also be implemented in a look up table and this be really fast. Still not full 14/16 bit, but more than plain 8. Alternatively just double the ranges in your table.

 

- Carl

[scott kirkpatrick]

Leica does in fact allow 16 bits to be used to capture the raw data off the pixels. The chip can hold up to 60,000 electrons per pixel, but the uncertainty (noise) is in the range of 10-20 electrons, depending on temperature, exposure speed and other factors. However, in restoring the 8 bits of square root information, the lookup table provided in every DNG file says to divide by 4 after squaring the dng pixel strength, so the maximum value resulting is 2^14. But the 8-bit processing uses values from 0 to 255.

 

There's a second advantage to the square root encoding. It is almost the gamma transformation that is made to RGB data before display on a monitor or transfer to a printer. So with the raw data in this form, one can do a very quick interpolation or even just a selection of values to display on the LCD at the back of the camera, even before a full JPG compression has been carried out. This could be a big factor in the noticeably quick image feedback from the M8.

 

scott

[ho_co]

Advertisement (gone after registration)

Carsten, Scott, Mark et al--

Thanks, all! You are bringing me slowly up to snuff on the matter of 8- vs 16-bit storage.

 

Sounds as if the math technique is common knowledge, and someone simply had to apply it to writing RAW.

 

Leica seems to possess the ability to surprise (and excel) at whatever they approach.

 

--HC

[carstenw]

Hi Carl,

 

yes, I suspect that they do something, but the net effect is rather minimal, compared to the decision to use th 8-bit encoding at all, so I didn't think it was worth discussing too much.

 

Scott, that is interesting. Where can I find such information? I would love to learn more about how all this works.

[carstenw]

Hello all!

 

No comments on the expose-correctly vs. expose-right discussion? This is what I find most interesting of all. Although I understand the logic behind expose-right, I always hated not simply exposing correctly.

[scott kirkpatrick]

Hello all!

 

No comments on the expose-correctly vs. expose-right discussion? This is what I find most interesting of all. Although I understand the logic behind expose-right, I always hated not simply exposing correctly.

 

Your discussion shows that there are still more levels available to describe each stop in the highlights than there are in the mid and shadow tones, so I don't think "expose to the right" has gone away. And the M8's CCD has more than eight stops of dynamic range, so there will always be some need to use post processing of the raw data file to get the most information out of an exposure, so I don't see that it can be argued that exposing correctly will leave you with a file ready to print or display. I guess that the difference between a file exposed to preserve all possible values and a file "exposed correctly" as you ultimately want to place those values my have just gotten a bit less, making for easier post processing. The M8 files that I have processed (from volunteered samples) have not needed great tone scale transformations.

 

scott

[j. borger]

Hello all!

No comments on the expose-correctly vs. expose-right discussion? This is what I find most interesting of all. Although I understand the logic behind expose-right, I always hated not simply exposing correctly.

I do not know how the M8 exposes, the discussion above is too technical for me ... but i dial in -1/3 or -2/3 constantly and have to dial in another -1/2 to -1 in raw processing or use a gamma-crush curve in PS to get to the image i like if using the internal meter of the M8. Histograms always look normal distributed ...... but images are too bright to my taste in 98% of the cases .............. but all the tonal information is there: no blown highlights and very open shadows! So i put exposure way down in post-processing..... and there is a lot of lattitude for local contrast enhancement!

THis is the opposite of what i do with the R-D1 or did with the Canons .... i have/ had to be carefull not blowing highlights .. and as a result very often had to boost the shadows in post!

Does this make sense:confused:

[scott kirkpatrick]

I do not know how the M8 exposes, the discussion above is too technical for me ... but i dial in -1/3 or -2/3 constantly and have to dial in another -1/2 to -1 in raw processing or use a gamma-crush curve in PS to get to the image i like if using the internal meter of the M8. Histograms always look normal distributed ...... but images are too bright to my taste in 98% of the cases .............. but all the tonal information is there: no blown highlights and very open shadows! So i put exposure way down in post-processing..... and there is a lot of lattitude for local contrast enhancement!

THis is the opposite of what i do with the R-D1 or did with the Canons .... i have/ had to be carefull not blowing highlights .. and as a result very often had to boost the shadows in post!

Does this make sense:confused:

 

Shooting with the Olympus E-1 (which uses a much earlier relative of the CCD used in the M8 and DMR), I also found that I could always pull additional detail from the highlights by reducing exposure during development, and at the same time use the C1 "film extra shadow) long tone scale to squeeze up information from the shadows. Kodak seems to make chips which hold highlight detail nicely.

 

scott

[carstenw]

Your discussion shows that there are still more levels available to describe each stop in the highlights than there are in the mid and shadow tones, so I don't think "expose to the right" has gone away. And the M8's CCD has more than eight stops of dynamic range, so there will always be some need to use post processing of the raw data file to get the most information out of an exposure, so I don't see that it can be argued that exposing correctly will leave you with a file ready to print or display. I guess that the difference between a file exposed to preserve all possible values and a file "exposed correctly" as you ultimately want to place those values my have just gotten a bit less, making for easier post processing. The M8 files that I have processed (from volunteered samples) have not needed great tone scale transformations.

 

scott

 

My point was that the advantages of "expose-right" are now minimal compared to before. We are talking something like 15 times the bits in the highlights, compared to 250, for the eighth stop, for example. And you now have to move the exposure a lot more to get the advantage than before. With such a small relative advantage, we might as well just forget about it.

[carstenw]

Shooting with the Olympus E-1 (which uses a much earlier relative of the CCD used in the M8 and DMR), I also found that I could always pull additional detail from the highlights by reducing exposure during development, and at the same time use the C1 "film extra shadow) long tone scale to squeeze up information from the shadows. Kodak seems to make chips which hold highlight detail nicely.

 

scott

 

I don't think that the Kodak chips handle the max levels any differently than the other chips, i.e. once we hit the maximum level, the remaining photons just increase the chance of spill-over into neighbouring pixels. I wonder what it is that is different? What software were you using, respectively?

[twf]

It is a new world for Leica and it has taken them a long time to come out with the M8. Nothing as new as this will be trouble free, but I do feel that because of the forum Leica has had to react to the problems. It will be interesting to see how well they follow through on all of these problems.

 

I am still waiting for my M8, and I think it is taking a little longer than I was told because of the problems. I would like to thank those that have discovered these problems and have been responsible for getting some action from Leica.

 

Tom

[lct]

...I would think there is a solution involving additional baffling if that is a real danger. But it is not a quick fix, it is an engineering path requiring further work...

Thank you Scott.

So all hope is not lost for those who don't want to put IR-cut filters on their lenses i guess.

Now a lot of people seem to stay unaware of the M8 problems so far.

In the last issue of 'Le Photographe', the chief editor of this usually good French mag states that the M8 is 'un peu sensible' (a bit sensitive) to IR rays but that this has 'pas d'incidence sur les images' (no influence on pictures).

How will react M8 users when they understand that IR-cut filters are virtually mandatory on their Leica lenses?

The same positive way as most this list hopefully...

[ennjott]

The output of (nearly) gamma-encoded 8 bits data is a pretty ugly crutch and will kill headroom for exposure corrections or significant changes to the white balance. And this headroom is already small enough at 12 bits compared to film.

Also, Nikon's compressed NEFs use the same method so it's nothing new. You can disable it on the higher end models, and it should certainly be possible to disable it on a Leica.

 

I've been taking part in this discussion in the German forum:

http://www.leica-camera-user.com/digitalforum/11644-eine-frage-der-kommunikation-4.html

 

On page 8, I posted some samples, linear and gamma/color-corrected versions of the same image, to demonstrate the large processing step that gets taken out of your hands by the M8's current firmware.

[jaapv]

German translated into English is generally bad news and comes across as all formal and ambiguous. Carsten is an exception - don't know if he is English or German but his English is completely fluent.

 

You should see the Dutch manual for Canon DSLR's. Sometimes utterly incomprehensible as it was from Japanese to some kind of English to Dutch by translators who had never seen a camera. "B setting" came out as "lamp setting."

{B=originally blower bulb (to operate the air-driven ancient shutters)=bulb=lightbulb=lamp. }

[sdai]

ennjott., this bit-depth topic has been beaten to death in the 8bit vs. 16-bit thread posted in the English forum almost two months ago. The LFI magazine was able to grab a prototype camera writing 16-bit DNGs albeit very slow ... I'm not sure if they've mentioned how slow it is because I haven't got the magazine yet but, it seems to prove my argument in the original thread that there's severe hardware limitation in the M8 and the 8-bit DNG is simply another tradeoff in favor of write speed.

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[ennjott]

You should see the Dutch manual for Canon DSLR's. Sometimes utterly incomprehensible as it was from Japanese to some kind of English to Dutch by translators who had never seen a camera.

Funny, the German manual is quite alright. But generally, the inability of most of my compatriots to speak comprehensible English or understand its native speakers (even phonetically) is ridiculous. Some of Leica's announcements made my toes curl. Not to mention the accents of most people here. I never understood why many are completely unable to mimic intonation different from the one of their own language.

 

this bit-depth topic has been beaten to death in the 8bit vs. 16-bit thread posted in the English forum almost two months ago.

Ok I wasn't aware of that. IMHO it's a tradeoff that isn't worthy of a Leica.