Why Does the Leica APO-Summicron-SL 35mm f/2 ASPH Rate Poorly at DXO?

[lencap]

Advertisement (gone after registration)

Greetings!

I don't typically rely upon DXO to evaluate lenses, but I have some avid photography friends that do.  One pointed out that DXO doesn't rate the 35mm f/2 APO lens very highly.  DXO also notes that the lens was tested on the Panasonic Lumix DC-S1R, not a Leica body.  In the DXO scoring methodology this suggests the APO lens has below average ratings for sharpness and vignetting; about average for T stop transmission and chromatic aberration, and a bit above average for distortion.

These results surprised me, especially sharpness which is an APO strong point.  What am I missing when looking at these rankings?  Would the results have been different if a Leica SL family body was used instead of the Panasonic?  Could it be that the APO lens has significant in camera correction using a Leica SL body that isn't included in the Panasonic body, which would offset the raw scores DXO obtained?  

This is more a curiosity to me, not a knock on Leica lenses or cameras, or DXO's methodology of using the Panasonic body instead of Leica.  I'm merely trying to understand how the measured ratings seem to be uncorrelated with user experiences.

The DXO link is here: https://www.dxomark.com/Lenses/Leica/LEICA-APO-SUMMICRON-SL-35mm-F2-ASPH-mounted-on-Panasonic-Lumix-DC-S1R__1294

DXOMARK Score
Best at f=35mm & f/2	39

Lens Metric Scores [?]
Sharpness [?]	32 P-Mpix
Transmission [?]	2 TStop
Distortion [?]	0.1 %
Vignetting [?]	-1.8 EV
Chr. aberration [?]	6 µm

 

Edited December 17, 2023 by lencap

[Stuart Richardson]

Whatever the reason, it is totally wrong, at least with respect to my copy. I have never seen a sharper or better corrected 35mm lens. Not by a long shot. It has also been excellent on my S1, but I have not checked in as much detail, as I use that almost exclusively for video. 

Edited December 17, 2023 by Stuart Richardson

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[Einst_Stein]

31 minutes ago, lencap said:

Greetings!

I don't typically rely upon DXO to evaluate lenses, but I have some avid photography friends that do.  One pointed out that DXO doesn't rate the 35mm f/2 APO lens very highly.  DXO also notes that the lens was tested on the Panasonic Lumix DC-S1R, not a Leica body.  In the DXO scoring methodology this suggests the APO lens has below average ratings for sharpness and vignetting; about average for T stop transmission and chromatic aberration, and a bit above average for distortion.

These results surprised me, especially sharpness which is an APO strong point.  What am I missing when looking at these rankings?  Would the results have been different if a Leica SL family body was used instead of the Panasonic?  Could it be that the APO lens has significant in camera correction using a Leica SL body that isn't included in the Panasonic body, which would offset the raw scores DXO obtained?  

This is more a curiosity to me, not a knock on Leica lenses or cameras, or DXO's methodology of using the Panasonic body instead of Leica.  I'm merely trying to understand how the measured ratings seem to be uncorrelated with user experiences.

The DXO link is here: https://www.dxomark.com/Lenses/Leica/LEICA-APO-SUMMICRON-SL-35mm-F2-ASPH-mounted-on-Panasonic-Lumix-DC-S1R__1294

DXOMARK Score
Best at f=35mm & f/2	39

Lens Metric Scores [?]
Sharpness [?]	32 P-Mpix
Transmission [?]	2 TStop
Distortion [?]	0.1 %
Vignetting [?]	-1.8 EV
Chr. aberration [?]	6 µm

 

Who you think is real expert? Would you trust Leica or DXO?

[lencap]

I appreciate the replies, and I agree that the APO lens is very special.  My question has more to do with why doesn't the DXO evaluation provide more impressive results?  What is it in their methodology that creates the gap between owner's experience and their testing?

As an aside, several of the Nikkor Z lenses are at the top of the ratings, along with Sigma ART and others.  The Nikkor lenses are well regarded by many, so the DXO ratings seem to agree with these ratings instead of questioning them.  It's the poor Leica rating that I can't explain.

DXO also doesn't seem to rate Leica lenses in general.  I could only find the APO Summicron SL on their list, and the ranking dates to 2019.

[LBJ2]

1 hour ago, lencap said:

Greetings!

I don't typically rely upon DXO to evaluate lenses, but I have some avid photography friends that do.  One pointed out that DXO doesn't rate the 35mm f/2 APO lens very highly.  DXO also notes that the lens was tested on the Panasonic Lumix DC-S1R, not a Leica body.  In the DXO scoring methodology this suggests the APO lens has below average ratings for sharpness and vignetting; about average for T stop transmission and chromatic aberration, and a bit above average for distortion.

These results surprised me, especially sharpness which is an APO strong point.  What am I missing when looking at these rankings?  Would the results have been different if a Leica SL family body was used instead of the Panasonic?  Could it be that the APO lens has significant in camera correction using a Leica SL body that isn't included in the Panasonic body, which would offset the raw scores DXO obtained?  

This is more a curiosity to me, not a knock on Leica lenses or cameras, or DXO's methodology of using the Panasonic body instead of Leica.  I'm merely trying to understand how the measured ratings seem to be uncorrelated with user experiences.

The DXO link is here: https://www.dxomark.com/Lenses/Leica/LEICA-APO-SUMMICRON-SL-35mm-F2-ASPH-mounted-on-Panasonic-Lumix-DC-S1R__1294

DXOMARK Score
Best at f=35mm & f/2	39

Lens Metric Scores [?]
Sharpness [?]	32 P-Mpix
Transmission [?]	2 TStop
Distortion [?]	0.1 %
Vignetting [?]	-1.8 EV
Chr. aberration [?]	6 µm

 

Strange DXOMark never updated these results from the initial SL 35/2 APO lens announcement period back in 2019. I've long since forgotten but thought for sure they would have updated results including the SL2 by now.

 Read LUF forum reaction from 2020. 

 

[Photoworks]

Leica said in the past that this is the best lens they ever made for SL system.

Does Leica use in-camera lens correction? Yes, they work on mechanical and software combinations to get the desired result. that is usually shared across L mount.

I think the problem is DxO here. the first mistake was using the Lumix camera in 2019. Leica was not giving out loaners for tests. The Lumix is a different sensor contraction of micro prisms, every lens will perform differently on the lumix. In fact M lenses are not that good on the S1R camera.

In my experience, the APO 35 is a great lens and balanced performance.

[Stuart Richardson]

Advertisement (gone after registration)

Sample variation is a real phenomenon, and it is possible that they (and Leica) could have been very unlucky to get a copy that was not a good performer. I had their No. 1 ranked Sony 35mm lens and it was so bad at the edges wide open that I returned it. Totally decentered to the extent that it was unusable. By contrast the 35mm APO SL is in a whole other universe of performance. At least in the comparison between my 35mm FE and 35mm APO SL. It is also way way better than the 35mm f2 Sigma DG DN that I have. I have posted my comparison before, but it is worth revisiting. I think the best thing to do is not rely on these third party testing services...or at least compare several of them to each other. This review is clearly a huge outlier, as most people who have used this lens have very few criticisms of its optical performance. The crop below is what both lenses look like at f2 on the SL2. It is not really a sharpness comparison (I have done others of those too, and the Leica is also way sharper and has much less vignetting), but it does show that one is truly APO, and the other is a mess of CA wide open...both lateral and longitudinal CA. 

 

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

Edited December 17, 2023 by Stuart Richardson

[lencap]

Thanks for the replies - I didn’t realize that this was discussed in a post earlier.  I need to brush up on my search skills.

Stuart - Thanks for the image comparison to the Sigma - the difference is remarkable!

[helged]

Among the lenses I have experience with, SL35-APO is the best of the best. Simply amazing. 

 

[FlashGordonPhotography]

I’ve used most of the SL APO’s on both the SL2 and S1R. The difference is generally nil, with the occasional shot showing *slightly* more micro contrast on the SL2 due to the thinner filter stack. Even then the difference is so minor as to be irrelevant in the real world.

DXO Mark and P2P should never be used in isolation. For starters DXO only tests one copy of a lens and it also tests a specific set of tests at a specific distance. Just like P2P equalises all files to 8MP for testing. They just don’t line up with real world shooting even when they’re accurate. Having owned a bunch of cameras and lenses tested by DXO I’ve just learned to ignore them as often they just don’t get it right compared to the real world.

Gordon

[Ivar B]

I don`t think there is any doubt that the APO SL lenses are extraordinarily good performers. I noted that a French magazine Chasseur d`imges gave kind of average scores for APO SL lenses, and also the German magazine Color Foto rated the APO SL 2/50 below the Panasonic S 1.4/50. However it seems that they both buy the test results from DXO? I really have no idea whant DXO misses, but it must be fair to argue that they got this completely wrong. I see that a lens I have never heard of (Yongnuo 2/35) which it is said carries a price tag of USD 278 receives a higher overall score than the APO SL 2/35. 

Edited December 18, 2023 by Ivar B

[Stuart Richardson]

Over the years I have learned to trust my own eyes and MTF charts over most reviewers. I know MTF can get a bad rap, but if you learn how to read it, it can be very helpful, especially when demonstrated with several different apertures and distances, like with Leica. It is not the only factor by any stretch, but when it comes to performance it is extremely helpful. Usually that with a combination of looking at sample pictures and maybe a few reviewers like Sean Reid (both for the insight and the photos that let you make your own conclusions) is usually enough. As Gordon said, sometimes these tests are just not correlated at all to the real world. Trying to compare at 8mp is ridiculous when you are using a camera that does 47mp native and up to 187mp in multishot. That's basically like measuring only for macro contrast, and not at all for resolution or microcontrast. Most modern lenses are going to look similar in resolution at 8mp. It is also a completely different thing if all you do is put stuff on the web or print 8x10s. If you are doing large prints for exhibitions and so on, then it is a totally different scenario.

[Ivar B]

10 minutes ago, Stuart Richardson said:

Over the years I have learned to trust my own eyes and MTF charts over most reviewers. I know MTF can get a bad rap, but if you learn how to read it, it can be very helpful, especially when demonstrated with several different apertures and distances, like with Leica. It is not the only factor by any stretch, but when it comes to performance it is extremely helpful. Usually that with a combination of looking at sample pictures and maybe a few reviewers like Sean Reid (both for the insight and the photos that let you make your own conclusions) is usually enough. As Gordon said, sometimes these tests are just not correlated at all to the real world. Trying to compare at 8mp is ridiculous when you are using a camera that does 47mp native and up to 187mp in multishot. That's basically like measuring only for macro contrast, and not at all for resolution or microcontrast. Most modern lenses are going to look similar in resolution at 8mp. It is also a completely different thing if all you do is put stuff on the web or print 8x10s. If you are doing large prints for exhibitions and so on, then it is a totally different scenario.

I also subscribe to Sean Reid and I value and trust his reports. You have probably read his review of SL 50mm lenses, where he unsurprisingly finds that nothing matches the APO SL 2/50. But interestingly, the Sigma Art 1.4/50 DG is not far behind an even surpasses the Leica lens in some measurements. Now the Sigma is replaced by the new and more compact version DG DN, and for what its worth, DXO reports that no other lens they have ever tested matches it (tested on a 60MP Sony). So even relatively low cost producers can deliver!

[SodaO]

I returned the 50mm APO Summicron-SL after one day.  The amount of aberrations (stained-glass window effect) between tree branches was almost laughable.  Based on the ridiculous premium for rebadged, average lenses, the weight and heft (and average performance) of real Leica SL glass, I've relegated my SL2-S to adapted M lenses.  I love my M10 and if it weren't for the occassional need for video, would be done with the L mount system.

[jaapv]

Unfortunately the better edge microcontrast a lens has, the stronger the purple fringing will be. It is mainly a sensor/interpolation artefact (despite Internet disinformation) which is -and that is the good thing-  easily resolved in postprocessing by using the basic slider provided. So I fear that you returned the lens for the wrong reason. 
It was simply better than your M lenses -and have you seen the amount of fringing from a Summilux-M asph? 

https://www.imatest.com/docs/sfr_chromatic/

[Photoworks]

7 hours ago, SodaO said:

I returned the 50mm APO Summicron-SL after one day.  The amount of aberrations (stained-glass window effect) between tree branches was almost laughable.  Based on the ridiculous premium for rebadged, average lenses, the weight and heft (and average performance) of real Leica SL glass, I've relegated my SL2-S to adapted M lenses.  I love my M10 and if it weren't for the occassional need for video, would be done with the L mount system.

the 50 SL APO is not "rebadged".

can it be your terminology is mixed up?

[Stuart Richardson]

2 hours ago, jaapv said:

Unfortunately the better edge microcontrast a lens has, the stronger the purple fringing will be. It is mainly a sensor/interpolation artefact (despite Internet disinformation) which is -and that is the good thing-  easily resolved in postprocessing by using the basic slider provided. So I fear that you returned the lens for the wrong reason. 
It was simply better than your M lenses -and have you seen the amount of fringing from a Summilux-M asph? 

https://www.imatest.com/docs/sfr_chromatic/

I don't think this is borne out by my experience. In my experience, the APO Summicron SL lenses do not really show any of this purple fringing, and they have extremely high microcontrast. Meanwhile, certain Sigma lenses and M lenses show a lot of fringing.

For example:

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

I realize this is not purple fringing per se (just straight up CA), but I do not recall seeing purple fringing on the SL, at least not with my lenses. It is my experience that both the 50 APO Summicron SL and 35mm APO Summicron SL perform extremely well with regards to chromatic aberration and purple fringing. Whether that is the result of the lens or the lens/sensor combo, I don't really know, and frankly it is irrelevant for me as the results speak for themselves.

If Soda8 was actually using a 50mm APO Summicron SL and found a lot of fringing, it sounds to me like that lens was damaged. There is nothing average about the performance of that lens, nor have I ever seen it show any fringing or CA, and I have used it for over 40% of my pictures on the SL2 over the last four years, including to do demanding work like artwork reproduction and exhibition work.

Edited March 21 by Stuart Richardson

[jaapv]

Well, I tend to believe the explanation of Imatest. After all, they are a leading lens test equipment company. 

[Stuart Richardson]

I am sure they are correct within the parameters of their explanation, but for whatever reason that does not line up with my experience of the SL APO Summicrons on the SL cameras. So either Leica corrects for it seamlessly or their particular lens/sensor combination does not exhibit this kind of false color around high contrast edges.

[Advertisement]

Hallo,

Look here: Leica L-Mount

[davidmknoble]

I think the answer is simple.  Leica designed the Leica M APO 35 to have effectively the same MTF results as the SL lens.  That M lens clearly sells for a high price.

It is amazing how much gear in so many brands gets released.  I cannot imagine how reviews are made for all that gear that don’t have mistakes…