DXO Mark Leica M10 Score

[Chaemono]

Advertisement (gone after registration)

Chaemono, can you explain again exactly what you are trying to demonstrate? And how it relates to the DxO report on the M10 (in which lenses were not a factor)?

 

I feel like I am getting "carpet-bombed" with pictures that don't show me any significant differences at all. Which may be the point, but 3-4 examples would be enough to do that.

Edit Edited January 8, 2018 by Chaemono

[Chaemono]

 

I feel like I am getting "carpet-bombed" with pictures that don't show me any significant differences at all. Which may be the point, but 3-4 examples would be enough to do that.

This thread is about “It’s like a nice, little APS-C camera but it comes with a red dot.” This theme keeps recurring as the thread moves on. You may not have noticed. Now one day I’m likely to want to sell the M10, so I’m basically protecting my investment. “It’s more like the α6300 but with cooked ISO.” Right. Pictures will keep coming as long as these claims keep coming. Simple. Edited January 8, 2018 by Chaemono

[rodf]

This thread is about “It’s like a nice, little APS-C camera but it comes with a red dot.” This theme keeps recurring as the thread moves on. You may not have noticed. Now one day I’m likely to want to sell the M10, so I’m basically protecting my investment. “It’s more like the α6300 but with cooked ISO.” Right. Pictures will keep coming as long as these claims keep coming. Simple.

A very good point Chaemonno and your work is appreciated.  What it demonstrates to me is that Leica more than likely are correct in their opinion that 24 megapixels is the optimum for a full frame camera of this type. (note that Canon still used a 20 mps sensor ifin their recent upgrade of their flagship camera the 1DX).Your examples and my experience with my Canon 5ds,demonstrate to me that 40 Megapixels does not achieve a great deal. Ok 40 Mps might give us more technical sharpness which is not really visible unless it is on a huge print,in a very narrow depth of field and if a tripod was used. And as another observer noted there is less "pop" in a 40 MPS image. Brightness and colour saturation etc seem to suffer.

 

I have had no experience with a Sony Camera and before I bought my M10 I did consider one but after comparing the overall specs, viewfinders lenses etc and importantly the problems Sony users were having as detailed on various Sony forums I decided on the M10 and so glad I did.

[LocalHero1953]

TBH, I found Andy's single post extremely enlightening with regards to the original subject, and I could easily have missed it among all of Chaemono's sample images, which I lost track of pages ago, valuable though they may be to others.

 

Is there a way of separating detailed test image results into a separate thread from the discussion? Scientific papers normally hive off test data into an appendix just for convenience of readability.

 

I'm happy for Chaemono to protect his investment (though I doubt resale values of M10s will be affected by DXO scores), but not at the expense of missing other valuable posts on the same subject.

Edited January 8, 2018 by LocalHero1953

[rodf]

TBH, I found Andy's single post extremely enlightening with regards to the original subject, and I could easily have missed it among all of Chaemono's sample images, which I lost track of pages ago, valuable though they may be to others.

 

Is there a way of separating detailed test image results into a separate thread from the discussion? Scientific papers normally hive off test data into an appendix just for convenience of readability.

 

I'm happy for Chaemono to protect his investment (though I doubt resale values of M10s will be affected by DXO scores), but not at the expense of missing other valuable posts on the same subject.

In your comment that "I doubt resale value of M10s will be affected by DXO scores" implies that the DXO score is irrelevant and I think that is exactly what Chaemono is trying to prove by posting his images. If anything, in my opinion, the M10 images posted are superior to the Sony images despite the fact that DXO claims the Sony has a superior sensor. This may mean that the high MP sensor is only one factor amongst many that result in the final image. I tend to back the expertise of Leica who selected the sensor for the M10 based on their knowledge of what their customer base wanted from a camera over and above what a bunch of technologists at DXO might say about their sensor.

[LocalHero1953]

In your comment that "I doubt resale value of M10s will be affected by DXO scores" implies that the DXO score is irrelevant and I think that is exactly what Chaemono is trying to prove by posting his images. If anything, in my opinion, the M10 images posted are superior to the Sony images despite the fact that DXO claims the Sony has a superior sensor. This may mean that the high MP sensor is only one factor amongst many that result in the final image. I tend to back the expertise of Leica who selected the sensor for the M10 based on their knowledge of what their customer base wanted from a camera over and above what a bunch of technologists at DXO might say about their sensor.

To be absolutely clear, my comment was intended to imply that the DXO score is not a significant factor in the market value of Leicas i.e. only a minority of head banging propeller heads actually use DXO score in buying decisions. Most of those will always choose a camera with more pixels rather than less, with more buttons rather than fewer, with more customisation options......in fact with more of anything that can be given a easy quantitative score than something that requires personal judgement.

I suspect Leica has given up on marketing to this group, and continues to rely on photographers who base their buying decisions on the use of the camera in the hand and at the eye, and on the images that they can produce*.

 

 

* Plus those who buy it for the red dot, of course! They are also unlikely to be swayed by a DXO score (huh, what?).

[pgk]

Advertisement (gone after registration)

Inspired by pgk's wiki link in post 153, I decided to do a little ISO testing of my own.

 

 

Good job Adan! With a background in photoscience/sci-phot I should do this sort of thing myself, however I've done too much in the past so tend to rely more on my actual usage of a camera to determine whether it is fit for my purpose theses days, as opposed to being as per specifications. I'm no longer into comparisons either but I would comment that my M9 and Sony A7II have decidedly different characteristics and despite their obvious differences in terms of sensor age and so on, I find that the M9 files remain a little more flexible in terms of some shadow recovery. I'm not going to try to determine this because it doesn't enthuse me to do so, but I am going to say that it would be really great if we could start viewing cameras and lenses in terms of their 'fitness for purpose' as opposed to their marginal differences.

Edited January 8, 2018 by pgk

[Chaemono]

Inspired by pgk's wiki link in post 153, I decided to do a little ISO testing of my own. The setup is shown below - a Gretag/Macbeth ColorChecker (for its medium gray squares) and two Tiffen (Kodak-style) color-reference targets either side of the Gretag gray squares, since I am not sure that either of Gretag's squares conform to "18% reflectance."

 

M10testISO1.jpg

 

The setting was outdoors, direct low-winter sunlight at 5280 feet elevation, no clouds.

 

The scene was metered 5 different ways, for a "concensus" reading. M10 meter off gray patch - M10 meter, whole scene - Sekonic L-308S reflected reading off gray patch, Sekonic reflected reading whole scene - Sekonic incident reading. Set to ISO 200, the M10 and Sekonic reflected read the lighter ColorChecker gray patch at 1/500 @ f/13 - the overall scene readings and the incident reading gave me 1/500 @ f/11. I used the latter exposure for 200, and adjusted it accordingly at other ISOs (e.g. ISO 3200 was 1/4000th @ f/16). I shot .dng + jpg, using a Leica 75mm Summarit f/2.4. All M10 jpeg settings were set to "standard."

 

(a side note on metering - interesting how closely my M10 meter and my Sekonic "synced" in similar usage).

 

First test - a Standard Output Sensitivity test using jpeg images. The basis of the SOS test is that if an ISO is accurate, it will render an 18% gray card with a brightness of 118 on the standard 8-bit scale of 0-255, in a .jpg in the sRGB color space. The results are tabulated below as: ISO set on the M10 - measured value of Tiffen scale left, Gretag darker gray, Gretag lighter gray, and Tiffen scale right.

 

100     109    77    119    116

200     115    78    120    120

1600   148   101   149    143

3200   137   101   149    145

 

Put simply, the M10's ISOs are pretty much correct, or if anything, higher than the nominal ISO - IF one is aiming to get an 18% gray correct (i.e. according to this particular test). Only ISO 100 is darker than the nominal exposure should be.

 

Key points: 1) the darker Gretag patch is obviously an outlier - darker than 18% 2) the M10 .jpg at ISO 200 conforms very closely to the SOS standard, at worst 3/255s (1.2%) darker than nominal. 3) the higher ISOs are substantially brighter (higher effective ISO - perhaps 2000 and 4000) than nominal. 4) ISO 100 was the "darkest." 5) those results are almost the reverse of the DxO results, using a different test measuring Saturation-based speed. And 6) they apply directly only to jpg output, which most of us do not use.

 

Therefore, I moved on to my own "invention" - a quasi-Standard Output Sensitivity test. Opening the .dngs shot simultaneously with the test .jpgs used above, with Adobe Camera Raw, as an sRGB 8-bit file to match the jpgs. Call it the "qSOS_srgb" test. ACR exposure settings were all set to "0," Leica M10 profile was used (presumably the same one the camera uses for creating .jpgs in-camera). No recovery of highlights or shadows. The results:

 

100     109    77    119    116

200     115    78    120    120

1600   145   100   149    151

3200   139   100   149    147

 

Virtually identical to the results from the in-camera .jpgs

 

Just for grins, since I normally use the Adobe 1998 working space, I repeated that trial opening the same .dngs into that space. Call it the qSOS_A1998 test. Results:

 

100     108    78    118    115

200     118    80    121    123

1600   144   101   148    149

3200   137   101   147    146

 

Now, my quasi-SOS tests are not officially sanctioned by the ISO. The SOS is not to be used with raw files, because it can't be compared across cameras of different bit depths (118 in 8-bit may or may not be any particular value in 12/14/16-bit). Still, I found it interesting that the M10's jpegs do meet or surpass the SOS for ISO in jpegs, and the .dng files track the jpgs very closely. In my next attempt, I am going to try to at least approximate the idea of the Saturation-Based Speed test that DxOMark uses for measuring ISO. (More or less, finding the actual ISO that avoids clipping highlights, rather than reproducing am 18% gray).

 

For reference here is a closeup of my measuring areas for the data listed above, across the four different gray samples. I used Photoshop's eyedropper set to a radius of 11 to avoid too many spurious readings of speckles or noise.

 

M10testdetail.jpg

It's post #160. Always learn a lot from these posts. 

Edited January 8, 2018 by Chaemono

[M11 for me]

To be absolutely clear, my comment was intended to imply that the DXO score is not a significant factor in the market value of Leicas i.e. only a minority of head banging propeller heads actually use DXO score in buying decisions. Most of those will always choose a camera with more pixels rather than less, with more buttons rather than fewer, with more customisation options......in fact with more of anything that can be given a easy quantitative score than something that requires personal judgement.

I suspect Leica has given up on marketing to this group, and continues to rely on photographers who base their buying decisions on the use of the camera in the hand and at the eye, and on the images that they can produce*.

 

 

* Plus those who buy it for the red dot, of course! They are also unlikely to be swayed by a DXO score (huh, what?).

Hi Paul,

 

Imagine that you have your incredible 4,000th anniversary 2 post later and I am sure that you contributed an awful lot to this forum. What should we then do with your post # 3,998? Well I am awaiting post # 4,000.

[TrickyMrT]

Until ISO 400 Leica´s M10 Sensor is great. But over ISO 400 there is a noise. 

 

 

In the ISO 6400 examples, you can see it already. The images do not look this good anymore.

 

 

The "DXO score" is 100%, the sensor could come from 2007 or 2008. Of course, I was shocked to read that Leica was not able to create a better sensor.

 

 

 

I guess we need to accept the reality. Leica is a fantastic company when it is about the Camera Body design and the Lenses, but for all other parts like sensor, the software they are not this great.

 

 

 

For example, no WIFI connection for OSX or windows. The iPhone app does not allow to download all images ... You can´t setup minimum shutter timings ...

 

and many many more....

 

 

 

 

But the worst stays the sensor.... it's years behind the competitors.

 

 

Don´t get me wrong, i like my M10. because of the design. M = motivation to take pictures.

Sony does not feel the same 

[Steve McGarrett]

Yeah, great, it's like a 2007 sensor... 

Edited January 8, 2018 by Steve McGarrett

[dunos]

Yeah, great, it's like a 2007 sensor... 

 

Well... in 2007 Canon introduced their 1Ds II that had ISO sensitivity of 100 to 1600 which could be pushed to 3200. It was a beast of a camera looking at the Wikipedia page.

[pgk]

Well... in 2007 Canon introduced their 1Ds II that had ISO sensitivity of 100 to 1600 which could be pushed to 3200. It was a beast of a camera looking at the Wikipedia page.

 

I had a couple, beasts they were and their sensor was not up to the M9's even. I have many files from them and I don't find them as flexible as the M9s and the noise they produce can quickly become 'ugly'.

[pgh]

I had a couple, beasts they were and their sensor was not up to the M9's even. I have many files from them and I don't find them as flexible as the M9s and the noise they produce can quickly become 'ugly'.

The M9 sensor at the time was one of the better ones out there (when it didn't corrode). Since then, sensors from competitors have jumped ahead considerably, while the improvement with the Leica sensors has been only incremental. Canon files were never the most flexible, and still aren't. Those will come from Sony, Nikon and Fuji cameras these days - though the Canon has improved of course. 

Edited January 8, 2018 by pgh

[rodf]

While trying to find out who funds the DXO  Mark research I came across an article in Android Central titled "Why DXO Mark Scores are basically worthless"  it is worth a read  but here is the paragraph that is most relevant to this conversation:-

 

  "Like a wily student preparing for a standardized test, manufacturers who partner with DxO, and get access to its hardware and software, can tune their image processing to ace the firm's synthetic tests (within the limits of the hardware, of course). As a result, their review scores are higher when DxO eventually publishes them — because they've had access to the testing hardware all along. Manufacturers who don't partner with DxO are at an automatic disadvantage in terms of their score, even though real-world, outside-of-the-lab image quality might not be substantially worse. When that happens, as it is bound to, consumers who put faith in comparisons between scores from partners and non-partners are potentially misled."

 

Which may go some off the way in explaining why Sony cameras in general do so well in DXO Mark compared to some other manufacturers !

Edited January 8, 2018 by rodf

[Mark II]

Put simply, the M10's ISOs are pretty much correct, or if anything, higher than the nominal ISO - IF one is aiming to get an 18% gray correct (i.e. according to this particular test). Only ISO 100 is darker than the nominal exposure should be.

A very nice set of tests, that shows why the ISO labels are correct.

 

However, this does not tell you very much about the sensor hardware, because measurements of a single grey level are susceptible to changes in the curve that is applied to the sensor data (either when rendering the JPEG’s in camera or in Lightroom etc when rendering the RAWs). This is why DXO uses the highlight clipping and (arbitrary, but consistent) noise floor measurements. These provide an objective measure independent of aesthetic choices in the firmware/software that are independent of the sensor hardware.

 

As an aside, the 18% grey and DXO measurements show that the default curves used for processing JPEG’s etc are different between the M10 and M240 - something which is obvious when you look at contrast in the JPEG output.

 

Interesting thread, though more technologically and sociologically than photographically ;-)

[Chaemono]

Commercially very interesting and potentially very damaging to Leica. Leica chose the sensor in the M10 with their lenses in mind, obviously. And photographically we’ll continue to show this in this thread.

[pgk]

..... can tune their image processing to ace the firm's synthetic tests ....

 

VW cameras anyone?

[jaapv]

And as I understand it is the DESIGN of a sensor that represents the technical knowhow. The building of the sensor is then common technology. And does not Leica design its sensors or make at least part of the design? And as far as I know there are then several suppliers adding parts to the sensor.

No, Leica does not design its own sensors. They use specialist firms for that (like CMOSIS), but they will specify the parameters for design, and obviously participate in the design process.

[Advertisement]

Hallo,

Look here: Leica M

[jaapv]

I had a couple, beasts they were and their sensor was not up to the M9's even. I have many files from them and I don't find them as flexible as the M9s and the noise they produce can quickly become 'ugly'.

And the M10 files are miles better. So: 2007???

 

I've said it before:  It is ridiculous to judge a sensor on the way the camera noise reduction works - it is a camera software matter full of compromises and trade-offs. It is legitimate to have a preference - but not to have an absolute judgement.