Who Does DXO Rate the Leica APO-Summicron-SL 35mm f/2 ASPH so Poorly?

[lencap]

Advertisement (gone after registration)

I'm trying to make sense of the DXO lens ratings, and I can't.  Any help is appreciated.  I own the Leica Summicron-SL 50mm f/2.0 ASPH, and I'm very pleased with it.  After reading several reviews of the APO version of that lens I thought I'd check with DXO to get some measured results.  Needless to say the results surprised me.

The rankings raise several questions that I hope someone can help me clarify.  Apparently DXO determines their rankings includinging the camera body, but it isn't clear how this impacts the overall scoring criteria.  The tested camera, the Lumix DC-S1R, isn't the current state of the art, nor were any Leica SL bodies used in the ranking determination.  

Is it possible that Leica includes automatic lens correction that corrects some of the areas that appear rather "average" on the DXO Scale?  If so, does that suggest that using a non-Leica body with a L mount Leica branded lens can/does reduce the overall image quality of the lens?  Is the DXO methodology appropriate to evaluate the lens, rather than MTF charts?  I expected a closer correlation to the MTF published results and the DXO results, and can't understand the variance.  

It was also interesting that the highest DXOMARK rated lenses were often Nikon Z mounts, using the Nikon Z7 bodies, tested in 2019.  I owned the Nikon Z camera body and lenses, but didn't like the menu system and less useful controls.  It is surprising to me that given my positive experience with the Leica SL2-S/Summicron ASPH lens that I enjoy, the ranking difference compared to the Nikon Z system was so dramatic.  It also suggests that the same lenses mounted on the newer Z9 body may actually be rated even higher, given the updated sensor and other improvements.

Any thoughts are welcome.

The DXOMARK ratings are shown below:

Leica APO-Summicron-SL 35mm f/2 ASPH. On: Panasonic Lumix DC-S1R

$4595

Feb. 2019

DXOMARK Score: 39

SHARPNESS: 32

DISTORTION: - 0.1

VIGNETTING: -1.8

TRANSMISSION: 2.0

CHR. ABBERATION: 6

 

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[huwm]

No idea, I just know that the SL APO 35mm is the best lens I've ever used

[tomasis7]

i dont take DXO seriously. 

[Ivar B]

It appears that for some reason DXO really gets it wrong now and then. Maybe one reason which holds a little true, is that they use Panasonic camera bodies and also other testers have noted that performance is marginally below what you get with SL bodies. But DXO also rate a 95$ Chinese 35 mm lens above the Apo-Summicron-SL 2.0/35mm, and that I find just ridicoulus. I also recall a test in a French magazine not that long ago where APO-SL lenses were tested and I believe the magazine buys these tests from DXO. All lenses scored well but not more than that. I own the APO 35/50/90 and performance is amazing. 

[lencap]

I agree that Leica APO lenses are exceptional, which is why I can’t understand the DXO ratings.  What is it in their methodology that creates their ratings disparities,or is it possible that lens correction built into Leica cameras corrects distortions that are inherent in the lenses (I find that not very likely).

From the opposite side, is there something in the Leica MTF charts that makes the lenses appear better than other tests?

I’m seriously interested in WHY there is such a difference.  Noted reviewers, like Reid reviews, known for his detailed measurements and testing process, don’t find any significant issues with APO lenses, yet the poor ratings remain.  I’d really like to know how these differences can be explained from a technical perspective.

[dpitt]

16 hours ago, Ivar B said:

But DXO also rate a 95$ Chinese 35 mm lens above the Apo-Summicron-SL 2.0/35mm, and that I find just ridicoulus.

I am not an expert in optical testing and measurements, but this is very similar to what I encountered in the audio field. The main issue is that when your senses notice a difference, it is not straightforward to measure it and quantify it in a proper way.Most of the published measurements are very basic at best and in some cases they are totally besides the question.

So in this case it seems DXO has some homework to do, If their measurement classifies a mediocre lens above an excellent one like the APO Summicron -SL 2.0/35mm then you can see these possible conclusions:

1. The Lecica lens is a scam and thousends of pro and addvanced amateur users are clearly wrong in their enthousiasm for the IQ af this lens, compared to the ones from other brands that they rate higher.
2. DXO is biased and big money from advertisers dictates their results more than the actual measurements.
   It could also be a very bad case of sample variation, but IMO, DXO should try to eliminate this possible issue by requesting a second sample before publishing results that are this far of what most people seem to see in practical use.
3. The measurements are correct, but they are totally irrelevant to measure what we want to see in an image.

I think we can rule out case 1. We are not all blinded by the Leica logo and we are not all so rich and snobbish that we use Leica just because it is expensive. Most of us have other gear too and we spend hard earned money on Leica because it gives results that we like and we do not get  the same with other gear.

Case 2 is a possibility. I just hope this is not the case for DXO.

Case 3 seems far fetched, but IMO it is the most probable. "They must know what they are doing" is what most reader would assume. DXO's equipment is no doubt expensive and very accurate. Probably it is considered to be industry standard too. Frommy  experience I know it is sometimes very hard to find a good way to measure what we see or hear. Some would conclude that our ears and eyes deceive us, and that is maybe true to some extent. I think it is mostly the other way around. If some measurement says there is no difference between two samples where we clearly see or hear a difference, then I conclude that the measurement is wrong. I do not mean the results are inaccurate, they just are useless for this purpose.

It can be very hard to find measurements that really show  differences like we see them. The best firms probably have more know how than DXO does and it is probably one of the best kept industry secrets at Leica, Hasselblad, Zeiss... The last thing they would do is publish the way how they really measure their R&D or production.
I don't care as long as they make products that can please my eyes.

[lencap]

Advertisement (gone after registration)

Thanks DPitt for the comments.  As a long time audiophile I identify with the audio analogy.  I remember when the first issue of Sterophile came out decades ago.  They raised the same thoughts you did.  Measurements are helpful, but not the final arbiter of sound quality.  The debate raged for years, especially as digital CDs became the dominant music format.  "Perfect sound" was the tagline, and it did measure nearly ruler flat from 20-20KHz, giving the "measurements matter" crowd a lot to be happy about.  It was only years later that new measurements were introduced - Jitter.  That newly defined parameter revealed what until then could be heard, but not measured.  The jitter measurements indeed revealed that although subtle, there were significant distortions in the "perfect sound" CD format.  That in turn led to new designs, addressing the jitter issue, improving the sound for all.

That's what I hope will emerge in the new era of 100MP sensors, and MTF charts that suggest perfection, but may not capture the equivalent of "visual jitter" (if that term catches on I'll copyright it).  It's also possible that Leica uses software to optimize the final image.  If so, I don't particularly care - the final image is what matters.  Peter Karbe in an interview said that MTF charts aren't from measurements, but from calculations.  If so, maybe there is some other way to evaluate lenses.

Perhaps what DXO measures, in some way, is just that - a distortion that isn't easy to identify or measure, but nonetheless does have an impact on the final image.  I'm not suggesting that Leica lenses are inferior - far from it, but it is possible that MTF charts are more appropriate for lenses creating images on FILM, and the extra resolving power of DIGITAL SENSORS requires a new measurement tool that captures the impact of the sensor when measuring the lens quality?  Since DXO used a Panasonic body for its measurements, and since we know Leica optimizes sensors for Leica lenses (including the M/S/R mounts), perhaps both are correct.  The MTF charts measuring the glass, and the DXO including the señor impact, even if it may be unintentional.

Perhaps Reid Reviews, Mr. Slack (a wonderful reviewer), or some others can elaborate.  What say thee gentlemen?  Is it possible that the move to digital sensors, and/or the use of digital correction, has created a new type of potential distortion not captured in traditional MTF Charts, and if so, how might we identify and measure that impact?

Edited March 21, 2024 by lencap

[BernardC]

10 hours ago, lencap said:

I agree that Leica APO lenses are exceptional, which is why I can’t understand the DXO ratings.  What is it in their methodology that creates their ratings disparities,or is it possible that lens correction built into Leica cameras corrects distortions that are inherent in the lenses (I find that not very likely).

From the opposite side, is there something in the Leica MTF charts that makes the lenses appear better than other tests?

I’m seriously interested in WHY there is such a difference.

They've only tested one Leica SL lens, a long time ago, on a Panasonic body, using very early firmware. They've tested only five L-mount lenses in total; none since 2021.

You'd have to ask them why, but it seems like they don't bother with L-mount at all.

[jaapv]

They had equally silly outcomes in their sensor tests in the past , nowadays it is a bit better. My explanation: sheer sloppiness. 

[Ivar B]

1 hour ago, BernardC said:

They've only tested one Leica SL lens, a long time ago, on a Panasonic body, using very early firmware. They've tested only five L-mount lenses in total; none since 2021.

You'd have to ask them why, but it seems like they don't bother with L-mount at all.

I don`t think the first assertion is completely correct. A French Magazine tested all APO-Summicron lenses, and also as I recall the tests were done by DXO. All lenses came out OK, but not much more than that.

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[BernardC]

I had another look around their site. Frankly, it doesn't seem like they are actively maintaining the lens test section. Even Canon RF and Nikon Z only have a handful of entries, mostly dating back to 2019.

Their sensor tests are reasonably up-to-date, but most of the lens tests go back to 2017 or earlier. Their recent articles are all about smartphones, so it seems they've shifted away from interchangeable lenses.

[dpitt]

13 hours ago, lencap said:

Thanks DPitt for the comments.  As a long time audiophile I identify with the audio analogy.  I remember when the first issue of Sterophile came out decades ago.  They raised the same thoughts you did.  Measurements are helpful, but not the final arbiter of sound quality.  The debate raged for years, especially as digital CDs became the dominant music format.  "Perfect sound" was the tagline, and it did measure nearly ruler flat from 20-20KHz, giving the "measurements matter" crowd a lot to be happy about.  It was only years later that new measurements were introduced - Jitter.  That newly defined parameter revealed what until then could be heard, but not measured.  The jitter measurements indeed revealed that although subtle, there were significant distortions in the "perfect sound" CD format.  That in turn led to new designs, addressing the jitter issue, improving the sound for all.

That's what I hope will emerge in the new era of 100MP sensors, and MTF charts that suggest perfection, but may not capture the equivalent of "visual jitter" (if that term catches on I'll copyright it).  It's also possible that Leica uses software to optimize the final image.  If so, I don't particularly care - the final image is what matters.  Peter Karbe in an interview said that MTF charts aren't from measurements, but from calculations.  If so, maybe there is some other way to evaluate lenses.

Perhaps what DXO measures, in some way, is just that - a distortion that isn't easy to identify or measure, but nonetheless does have an impact on the final image.  I'm not suggesting that Leica lenses are inferior - far from it, but it is possible that MTF charts are more appropriate for lenses creating images on FILM, and the extra resolving power of DIGITAL SENSORS requires a new measurement tool that captures the impact of the sensor when measuring the lens quality?  Since DXO used a Panasonic body for its measurements, and since we know Leica optimizes sensors for Leica lenses (including the M/S/R mounts), perhaps both are correct.  The MTF charts measuring the glass, and the DXO including the señor impact, even if it may be unintentional.

Perhaps Reid Reviews, Mr. Slack (a wonderful reviewer), or some others can elaborate.  What say thee gentlemen?  Is it possible that the move to digital sensors, and/or the use of digital correction, has created a new type of potential distortion not captured in traditional MTF Charts, and if so, how might we identify and measure that impact?

Your analogy is correct. Jitter is one of the first issues with digital that was "discovered" by the measurements matter crowd. It is only one of the manay aspects of the digital CD format that was fundamentally flawed, and which could be heard by almost everyone that was self confident enough to trust their own ears. I almost felt like I was living the story of the "The Emporer's New Clothes". Suffice to say that the revival of vinyl was not a surprise for me and a lot of pro's in the field.

I agree with you that digital photography is probably not as mature as digital audio is now. So a lot of insights will become more clear in the future. But I draw different conclusions from what I see happening in the " measurement matter" crowd. I see the same hype and mistakes being made as in the 80s and early 90s with digital audio. I see lots of people that seem as blind as they were deaf for aspects which are, and should be clear for the majority of people. Sure, some people see better than others and some see different aspects that others can not see or do not pay attention too...

What I see when I test Leica lenses (vintage and new) to other brands, is that sharpness and contrast are not what they are all about. I really like what I see with most of the Leica lenses much better than with any other brand. And I see the qualities of these lenses on whatever body I use, certainly for 35mm and longer. In fact, I never used a body where the diferences where the other way around. I did see some where the differences were smaller though. But in that case all results were not as good as on bodies that showed a larger difference. I do not see these differences reflected in MTF charts (not all of it anyway), and certainly not in these DXO tests. My reaction would be to do extra research for a better measurement that will reflect what I see, rather than conclude that what I see is not relevant and that Leica actually makes mediocre lenses. I did some research in the audio domain. And after trying long and hard, I finally found a way to measure the differences that I heard all the time.

As it is now, I like the Leica results better than the results from so called perfect or better lenses (according to DXO)... This just means that the measurements by which they are more perfect are apparently not important for my eyes. And the measurement (if it exists or is available at all) by which Leica lenses are superior is not in these tests, otherwise they would be on top of the list. Clearly lots of people that trust their eyes like me are convinced of the same, and are prepared to pay a hefty premium to use Leica glass in stead of cheaper alternatives...

[chrismuc]

The SL 35f2 Asph. Apo has very strong (maybe 6-8%) distortion which is hidden by the automatic distortion correction. But the distortion correction of course reduces the resolution towards the image corners. 
That may reflect in the not-stellar test reviews. 

[Kreeshp]

4 hours ago, chrismuc said:

The SL 35f2 Asph. Apo has very strong (maybe 6-8%) distortion which is hidden by the automatic distortion correction. But the distortion correction of course reduces the resolution towards the image corners. 
That may reflect in the not-stellar test reviews. 

Nonsense.
 

“Even without applying the correction profile, the Leica APO-Summicron-SL 35mm F2.0 has virtually no distortion, measured at a maximum of just 0.1%.”

 

source: https://www.dxomark.com/leica-apo-summicron-sl-35mm-f2-asph-lens-review/#:~:text=Even without applying the correction,a maximum of just 0.1%.

 

 

[chrismuc]

I don’t believe DXO, I think they were not really able to deactivate the automatic distortion correction. 
0.1 % sounds completely unreasonable. 
Lloyd Chambers showed an image without distortion correction, it looks nearly like a fish eye lens. 

[chrismuc]

Can someone do an architecture photography, open the image in Iridient, deactivate the in-camera distortion correction and upload the image, then we see the real optical performance of the lens.

[chrismuc]

26 minutes ago, chrismuc said:

I don’t believe DXO, I think they were not really able to deactivate the automatic distortion correction. 
0.1 % sounds completely unreasonable. 
Lloyd Chambers showed an image without distortion correction, it looks nearly like a fish eye lens. 

[I mean … that’s the kind of Leica signature distortion: The 28mm lens in the Q1/2/3 is the same, the SL 24-90 @ 24mm too 😝.]

[jaapv]

On 3/26/2024 at 5:45 AM, chrismuc said:

Can someone do an architecture photography, open the image in Iridient, deactivate the in-camera distortion correction and upload the image, then we see the real optical performance of the lens.

Nowadays nearly all lenses are of hybrid design, which means that digital distortion correction is designed into the complete optical formula. Viewing the lens without the digital part of the design is equivalent to removing one lens element and then judging the lens. Not only with Leica but with virtually all lens makers, except for a few small ones that lack the resources to implement hybrid designs. 

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[chrismuc]

12 hours ago, jaapv said:

Nowadays nearly all lenses are of hybrid design, which means that digital distortion correction is designed into the complete optical formula. Viewing the lens without the digital part of the design is equivalent to removing one lens element and then judging the lens. Not only with Leica but with virtually all lens makers, except for a few small ones that lack the resources to implement hybrid designs. 

Yes, basically all lens manufacturers today allow stronger distortion in their lens design and compensate this with digital distortion correction, mainly to achieve more compact lens designs. The digital distortion correction in the best case removes the distortion completely. But: It always reduces the resolution / sharpness, so it is definitely a compromise to apply it. If a lens has let's say 6% distortion, the whole image is squeezed to remove the distortion and all image pixels are 'stretched' and interpolated which leads to a reduction of resolution of that amount distortion in certain areas plus interpolation losses, in total, in our example maybe 10%. So that is quite relevant! And the big question is: Is the MTF data provided for example by Leica and Sigma for their L-mount lenses with or without distortion correction?

Enclosed image edges from six Sigma L-mount lenses with distortion reduction turned off. The 20f2 and 24f2 are worst. I had both and kept the 24f2 which is generally sharper towards the image corners, so also after distortion correction.

The 28mm lens in the Q cameras and the L 24-90 at 24mm have even stronger distortion than the Sigma L 20f2 and 24f2, so the digital distortion correction will have quite an impact on the corner resolution because the pixels are stretched to such a high extend.

Btw. 1: 

In a lens with barrel distortion, the digital distortion correction will reduce the sharpness more in the image corners than in the center.

In a lens with pincushion distortion, the digital distortion correction will reduce the sharpness more towards the image center while not much modifying the image corners.

Btw. 2:

the-digital-picture.com shows well the distortion of lenses. Unfortunately no L mount lenses, but some Sigma which are available in E mount and L mount.

Sigma / Leica 14-24f2.8

https://www.the-digital-picture.com/Reviews/Lens-Distortion.aspx?Lens=1535

Sigma / Leica 24-70f2.8

https://www.the-digital-picture.com/Reviews/Lens-Distortion.aspx?FLI=0&FLIComp=0&LensComp=0&CameraComp=0&Lens=1518

 

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[jaapv]

Yes, the electronic corrections will cause some resolution loss towards the corners but that is compensated by the extra freedom allowed by being able to correct the optical part better; optical design is always about shifting aberrations.
I would like to see a comparison to a lens with an optical distortion correction which will lead to quality loss as well. I strongly suspect that the digital correction leads to fess negative impact.

Comparing a lens with uncorrected distortion to one with distortion correction makes little sense. One should compare a lens with digital correction to one with optical correction.

BTW I am a bit lost when you mention stretching pixels. How does one perform an analog operation like stretching on a digital entity like a pixel?