Erwin Puts: The Leica M9: part 5: M8/9 noise and dynamic range

[xrogers]

Advertisement (gone after registration)

I agree about the headache.

 

I just read a bit about how this is done on the Imatest website. Their measure of dynamic range is based on noise (which appears primarily in dark regions). They do state that "48-bit TIFF conversion has lower noise, hence higher dynamic range at any given quality level".

 

They also state that "Digital camera images with excessive noise reduction will have an unusually rapid falloff of the noise spectrum." Take a look at the Nikon "Noise spectrum", which starting at ISO 320 drops like a rock. Maybe this is what they're talking about?

 

From Erwin's writeup: "Here a small section of the total image has been selected on a scale of 100% without any post processing, except the proprietary JPG algorithms. For this type of analysis, there is no problem using the JPG processing."

 

After reading even a just a little, it seems like this is too much of a simplification. I don't see how even going to a RAW file helps, with their noise reduction. This comparison is complicated enough that I don't understand what this data is supposed to tell me. And it looks like something that could be "gamed" (where specific sorts of processing could result in a great Imatest "score", but terrible photographic results).

 

It's confusing enough that it's worthless to me. I'm just going to forget about it.

 

Later,

 

Clyde

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[pnoble]

Without fully understanding the methodology or the science, my surprise was simply that the M9 had a consistently lower DR and S/N than the M8 according Put's measurements. I assume that for the same sensor type and the same or very similar digital processing, and presumably using the same lens, both DR and S/N should be the same or very similar, no?

 

Instead, at all ISOs the M8.2 had 1-2 stops more DR and somewhat better S/N than the M9 and to quote Puts: "The M9 is remarkably somewhat less good than the M8.2 I used."

 

I brought this to the forum's attention not as an M8.2 owner happy to find justification for not upgrading, but as someone who purchased an M9 just this morning!

 

I really don't care if the M9 is any better than the M8 beyond the frame size and resolution but would hope that it was not for some reason a step back in performance.

 

Nothing I have seen or read from Sean Read and others doing careful visual tests suggests so, but Puts is maybe the first to do a more technical analysis and comparison of the two cameras.

[Guest roey]

From Erwin's writeup: "Here a small section of the total image has been selected on a scale of 100% without any post processing, except the proprietary JPG algorithms. For this type of analysis, there is no problem using the JPG processing."

 

I might be missing something, but doesn't that mean that he compares the quality of the in-camera JPG conversion rather than the dynamic range and noise characteristics of the sensors?

[haroldp]

Unless I am missing something, these tests are of 'per pixel' performance, The M9 having 80% more sensor area and 80% more pixels of the same pitch than the M8, should also result in 80% less magnification than M8 for any given print size, and 80% smaller pixels on print, if lenses are chosen for similar FOV.

 

This should lead to almost a 1 f-stop advantage for the M9, in perceived noise on print, as well as better optical performance etc., etc, etc.

 

As a user of both Nikon and Leica systems, I totally agree that IQ is not the main consideration in deciding which gear to take to a shoot, as all other characteristics of these systems are so different.

 

Regards ... Harold

[arthury]

I agree that reviews like this (and Puts is not the only one) are pretty pointless. If we're talking about IQ (other factors are important as well, of course), then what I'm interested in is how the prints look up to a certain size[...]

 

Interesting conclusion ... so, you are telling us that if image A has more resolution than image B at a certain ISO, then it is possible for the print of image B to be better than image A?

 

Whatever "magic" you are doing to image B, if you were to do the same thing to image A, wouldn't the print look better than B --- all the time?

[rolu]

What seems clear to me is that the M10 will have a CMOS - no improvement in CCDs so far. Did I get Stephan Daniel's hint to CMOS in the interview right ?

 

Regarding the M10 we should follow the performance and firmware of the X1-sensor.

 

 

... I instinctively keep my idea that Leica has still something to do on M9 firmware... ...

 

God bless your idea. I hope for a little firmware-help, too.

 

But what disadvantages would come with that ?

More vignetting, ... ?

[thrice]

Advertisement (gone after registration)

Unless I am missing something, these tests are of 'per pixel' performance, The M9 having 80% more sensor area and 80% more pixels of the same pitch than the M8, should also result in 80% less magnification than M8 for any given print size, and 80% smaller pixels on print, if lenses are chosen for similar FOV.

 

This should lead to almost a 1 f-stop advantage for the M9, in perceived noise on print, as well as better optical performance etc., etc, etc.

 

As a user of both Nikon and Leica systems, I totally agree that IQ is not the main consideration in deciding which gear to take to a shoot, as all other characteristics of these systems are so different.

 

Regards ... Harold

 

You are correct as is roey. Perhaps the DR will be superior vs M8 once scaled down I am not sure, but the noise certainly will be. For this to be a valid test Puts should have used RAW files, and zero'd then exported as tiffs if the testing software doesn't support RAW files.

[thompsonkirk]

Here are some reasons for my ‘Puts headache’:

 

1. Didn’t he say he compared JPGs & this makes no difference? Hasn’t he just bypassed the advantages of M9’s 14-bit uncompressed RAW files & entered the area where the firmware of different cameras processes JPEGs differently?

 

2. Did he miss Sean Reid’s review, which already made the point that M9 files require less enlargement or res-ing up, & will therefore reveal less noise in prints of the same size?

 

3. In comparing M8 & 9, didn’t he consider the possibility that M8’s JPG firmware works harder at noise reduction than M9? If I were writing the firmware for a larger sensor (with same pixel size), I’d program for a bit less noise reduction, to capitalize on the greater IQ of the larger sensor & the above factor: less enlargement / resing up.

[Guest joewehry]

Can someone simplify the practical application of the dynamic range of a sensor?

 

Does it mean this, for example?

 

1) A sensor has a dynamic range of five stops

2) A given photo meters at f/5.6 at 1/60

3) The sensor would effectively process highlights that metering f/5.6 at 1/250 down to darks metering f/5.6 at 1/15? And anything above or below that range would be clipped?

[xrogers]

I might be missing something, but doesn't that mean that he compares the quality of the in-camera JPG conversion rather than the dynamic range and noise characteristics of the sensors?

 

Imatest has two different DR tests, one where they say jpg is fine, and another where they suggest raw or tiff. It appears that they make some assumptions about jpg processing that are based on limited internal testing, not on fundamental properties of sensors or data processing---a weak foundation for such precise-seeming results.

 

It looks like a camera with a high resolution sensor and aggressive luminance noise reduction will get an artificially inflated score. It looks like a moderate or low quality jpg may provide an artificially reduced score. It looks like an unusual or poorly tuned jpg engine could further skew results in either direction. By the same token, noise reduction in raw files also will change Imatest results, again giving high scores to noise-free shadows, whether beautiful or muddy.

 

Basing DR evaluation on noise provides a simple, repeatable metric. Perhaps the metric provides a mathematically convenient definition of DR rather than a photographically useful definition.

 

Bafflegab. It seems more valuable (and much easier) to look at some good raw conversions from wide-range subjects.

 

Later,

 

Clyde

[Scott Root]

Clearly, the culprit for the degradation of M9 images is the IR absorbing filter that Leica has admitted had to be toyed with to strike a balance between dealing with the IR issue without wrecking sharpness. Either Leica will find another way to filter the IR, CCD technology will evolve to deal with IR, we can ask for an M9.2 without an IR absorbing filter and go back to using our IR lens filters, or the M10 will be CMOS and include focus confirmation or even better, auto focus with a new range of autofocus M-bayonet lenses.

 

What seems clear to me is that the M10 will have a CMOS - no improvement in CCDs so far. Did I get Stephan Daniel's hint to CMOS in the interview right ?

 

Regarding the M10 we should follow the performance and firmware of the X1-sensor.

 

 

 

 

God bless your idea. I hope for a little firmware-help, too.

 

But what disadvantages would come with that ?

More vignetting, ... ?

[diogenis]

Can someone simplify the practical application of the dynamic range of a sensor?

 

Does it mean this, for example?

 

1) A sensor has a dynamic range of five stops

2) A given photo meters at f/5.6 at 1/60

3) The sensor would effectively process highlights that metering f/5.6 at 1/250 down to darks metering f/5.6 at 1/15? And anything above or below that range would be clipped?

 

yes thats a 5 stops span from your correct setting of 1/60. It represents the sensor's ability to record very dark regions or very bright at the same time- very useful to avoid clippings and/or exposure compensate

[michael_b_elmer]

I once sold a Leica IIIb from 1938 with Elmar 3,5:5 to buy a Nikon SLR and lots of Nikon glass - I have never regretted anything more in my life.

 

So, being from the age of Kodachrome 50 and using fast Leica glass, I am happy that M9 even in a scientific test keeps up with Nikons best SLR up to ISO 320-400. In a practical test, I am sure that pictures at ISO 160 from the M9 with my fast Leica glass will beat or at least be on par with pictures from the Nikon.

[georg]

"No, I think isn't so surprising : Nikon has a longtime (in "digital" terms) experience in sensors, and lot of technology of its own into them: I wouldn't expect the Leica/Kodak agreement to give something better."

 

Well, Kodak was making sensors when Nikon still used film and they still make sensors while Nikon doesn't.

The M8/9 uses a full-frame CCD with a 6.8µm-pixel-pitch, while the Nikon uses a CMOS with 6µm-pixel-pitch - the photodiodes from Kodak must be much larger and therefore have a much higher saturation signal (important for handling of highlights).

I want to see real-world images which contain fine detail in highlights & shadows (to make noise reduction visbile) and would be really surprised to see any DSLR with more actual DR in comparison to current full-frame-CCD-cameras.

The problem is that Mr. Puts "tests" pretend scientific accuracy because of cryptic explanations, complex schematics and bizarre test objects but when he comes up with the conclusion that the M8 is as good as a Canon Rebel IQ-wise and the M8 is still better than the M9 regarding DR, it makes me cringe...

[mjh]

I might be missing something, but doesn't that mean that he compares the quality of the in-camera JPG conversion rather than the dynamic range and noise characteristics of the sensors?

That appears to be a succinct description of what he did, yes.

[okram]

Can you explain how and why you think they are wrong?

 

The man used in camera JPG for testing.

7 stop of dynamic range???????

 

Maybe the next part will be with RAW?

 

 

This kind of "technical testing" or reviewing is useful only for the reviewer- the best example is a DXO test where Canon G10 sensor scores overall better than P45....- it is a waste of time to read for somebody who actually uses the camera.

 

7 stop of dynamic range???????

[Stevez4]

Have any of you malcontents ever taken photos with the M9 and printed them without any fooling around with post processing? That is the beauty of Leicas to photographers IMHO.

My M9 is four days old and is a great improvement over the cropped M8. JAAp commented somewhere that a photo is judged more on aesthetics than a bit of noise. He is so right.

[nhabedi]

so, you are telling us that if image A has more resolution than image B at a certain ISO, then it is possible for the print of image B to be better than image A?

 

No, I didn't tell you that...

 

But what I would think (not having a M9 or D3x to test) is that image B can be as good as (not better than) image A for a given print size.

 

For example, the M9 has 5,212x3,472 pixels for a 36x24mm frame which results in approximately 3600 pixels per inch. If you're using a very good printer like, say, an Epson R2880, the recommended printing resolution is - depending on who you ask - 360 or 240 pixel per inch. (I've even read tests claiming that its effective resolution is "only" 200 ppi and my own experience is that I don't see advantages if printing at 360 instead of 240.) So, at 240ppi you could multiply the size of the sensor with 15 without the need for more resolution, i.e. you could print at 36x54 centimeters which is enough for A2 with a border around the photo. How are more pixels supposed to give you a better print in this case? That's not immediately clear to me.

 

There's lots of math like this that you can do, but that wasn't actually my point. If I were to evaluate the M9 against the D3x for my own, I'd shoot pictures of things I'm likely to shoot in real life and then I'd print them at sizes I'm usually printing. That would be interesting for me. And it'd also be interesting to see how high ISO shots look when printed in reasonable size. How their pixels look at 200% magnification I don't need to know.

 

I'm sure you can measure that the D3x is "better" than the M9 in terms of "IQ". The question is whether it matters for the final product which for me is the print.

[Advertisement]

Hallo,

Look here: Leica M

[Angelos Viskadourakis]

I have tested my M8 with IMATEST which by the way is a world standard -especially in the manufacturer world--and i have to inform the members in that forum that i had exactly the same results as ERWIN PUT.I tested my M8 against my CANON 5D in jpeg and in tiffs from raw convertion in CAPTURE ONE.Many people dont like what IMATEST shows for their cameras but is absolutely fact and the way to use it is to compare atleast two systems-i use to test a lot of systems.CANON 5D has over 2,5 stops better dynamic range compared to M8 but the most important evaluation part in the IMATEST is the rating in LOW.MEDIUM,HIGH level and someone has to look at that.M8 stops are all in the HIGH quality-i tested mine at 640 asa- and CANON 5D stops are little less in HIGH some in MEDIUM and 2 stops in LOW.

Personally i like the images from M8 in jpeg or raw for a certain style of photography and the images from CANON 5D for other style.I remember that we were much impressed with the much lower dynamic range-exposure lattitude in film language-of the positive-slide-film compared to the wide lattitude of the negative.Photography i believe is a give and take process there is no perfection but there is personal style.

M9 cannot be better than M8 technically for those they have deeper knowledge of the electronic image and that is why i strongly disagree with the M9 concept as is not a step forward is a marketing trick to make noise in the sales.LEICA should wait one more year and use the already existing technology that would make the new M really better.When using the same pixel pitch,same photosite,same circuit and you add some more circuit in the corners to make the sensor bigger you DONOT ALTERING THE IMAGE CHARACTERISTICS,they are the same,but you increase the power demand,you increase the data flow you strees the processor and you add vigneting because you use the same tech sensor-ok with a liitle more tilted microlenses-The whole effort was in software and this should be done first in the M8;s and the new full frame M should be based on a new tech sensor which is already out in the market and it was ignored..

PS:I DEMAND THE FUNCTIONALITY CORRECTIONS APPLIED TO M9 TO BECOME IMMEDIATELY AVAILABLE TO M8 CAMERAS,THAT WOULD BE REAL EXCITING MARKETING NEWS,i still dont know what to do with that 'PROTECT' button.

[mjh]

How are more pixels supposed to give you a better print in this case? That's not immediately clear to me.

Suppose the pixels you are comparing give you similar results on a per-pixel basis, i.e. signal-to-noise ratio and dynamic range are similar. Now when you’ve got more of those equally good pixels and the increased resolution is more than either your printer or your eyes (or both) can handle, then some of those pixels will get averaged (by the printer or your eyes), effectively reducing noise and thereby increasing dynamic range. So more pixels give you increased resolution, up to the point where still more resolution in the image would be wasted. From that point onwards, more pixels increase the signal-to-noise ratio and dynamic range.

 

In most cases these days we are dealing with more pixels crammed onto the same sensor size; in those cases, the signal-to-noise ratio and dynamic range drop, measured on a per-pixel basis, and the increased resolution cannot quite make up for that. But in the case of the M9 versus the M8 we have rather similar (although slightly improved) sensor pixels, only more of those, resulting in a net gain.