Correct settings to compare SL2 vs other camera's noise performance

[LBJ2]

Advertisement (gone after registration)

For instance, SL2 vs M11 noise performance.

Due to differences between these cameras, simply using the same ISO setting might not be a true comparison. Others have suggested setting the same shutter speed and f-stop is the correct way to compare ISO performance between two different cameras models: 

E.g., M11 + M 50 Lux asph and SL2 + SL 50 APO Cron:

M11 1/25s @ F2.8 = Auto ISO 400 

SL2 1/25s @ F2.8 = Auto ISO 640

Or is there a better approach/settings for a more equal noise comparison between these two cameras SL2 and M11 ? 

[LocalHero1953]

I take an empirical approach. Take a shot of the same scene with two cameras, with preferred settings for each camera. Which one would I choose to give the best noise performance for the same purpose? Billboard? Framed on my living room wall at A2/A3/A4? WhatsApp/FB/Instagram? Photobook?

An alternative approach is purely scientific research. In this case, I would keep everything constant: shutter speed, aperture, ISO, lens, image viewing size. Then I am just comparing the difference between cameras.

Introducing other variables, such as scaling, adjusting resolution (up or down), etc, just opens a can of worms. Like trying to prove Ohm's Law by measuring across two different metals, or conductors of different length.

Edited November 3, 2023 by LocalHero1953

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[Geoff C. Bassett]

I've shot 12800 images that I've found acceptable on the SL2. Especially with the modern noise reduction algorithms in software I think all cameras are good enough in terms of noise. If your constantly shooting in the dark, the SL2s is much cleaner.

[goodbokeh]

2 hours ago, LBJ2 said:

For instance, SL2 vs M11 noise performance.

Due to differences between these cameras, simply using the same ISO setting might not be a true comparison. Others have suggested setting the same shutter speed and f-stop is the correct way to compare ISO performance between two different cameras models: 

E.g., M11 + M 50 Lux asph and SL2 + SL 50 APO Cron:

M11 1/25s @ F2.8 = Auto ISO 400 

SL2 1/25s @ F2.8 = Auto ISO 640

Or is there a better approach/settings for a more equal noise comparison between these two cameras SL2 and M11 ? 

LBJ2, I know it's apples to oranges but I find my SL2 generally exposes best for highlights @ -1/3 EV less than my Sony A7R5 with it's 60 MP sensor. BTW, the noise levels are pretty close, the Sony has a fraction of a stop advantage on the noise ( I limit to 1600...3200 max) but there's not much in it. Honestly, I can take or leave the A7R5's 60MP sensor, it's the 50MP stacked sensor of my Sony A1 that has the real value added in certain circumstances.

[dpitt]

Maybe I see it a little bit too simple, but I think using the same settings and preferably the same lens is a good start. So set the camera to RAW and ISO (fixed value). And why not choose the same F stop and shutter time? Exposure should be done for the highlights. And I expect that will result in the same aperture and shutter time when everything else is the same.

Now, to compensate for the difference in MP, I think that you should at least resize to the smallest size sensor. Although this would result in a disadvantage for the SL2, because down sampling the 60MP files will possibly reduce noise too. Why not resize to 24MP for both? It will allow for comparisons with other M and SL systems and it seems large enough for most purposes.

Then there is the question of WB... it should be set as close as possible for both.

Now, I think comparing the output JPEG files of both camera's shooting the same scene can provide some interesting info. I suppose repeating this test at different ISO values will reveal a lot about the noise OOC.

In the end, all files will get more processing before you will use them anyway, so a second set of results could be to process both files the best you can to minimize noise, and for the most pleasing result and then compare at 24MP again.

Just my 2 cents...

[jrp]

https://www.photonstophotos.net/Charts/RN_ADU.htm#Leica M11_14,Leica SL2_14,Leica SL2-S_14,Sony ILCE-7RM5_14

[FlashGordonPhotography]

Advertisement (gone after registration)

You can always look at sources like P2P or even the DP Review charts but neither really translates well to real world usage. That's like saying a lens is good or bad when only testing at one aperture at one focus distance. Part of the story but never the whole.

I have a lot of cameras and I do compare them often. I have three questions I want to answer. 1. How does camera A compare to camera B? 2. What are the optimal settings to get the best results from camera X? and 3. Why would I choose camera A over camera B and vice versa?

For question 1 I'll use same shutter speed and aperture and float the ISO plus I'll do same ISO comparisons. Boring as bat shit but it helps answer and leads to question 2.

For question 2, I'll do a set of exposures with EV compensation and then push and prod in post to see where they fall apart. Camera A, I might shoot to never clip a highlight and camera B I might see more highlight recovery. I also test IBIS and various lenses against each other on the camera. Do I need a tripod or monopod at night. What's my hit rate at 1/10th of a second? Do I need an add on grip, strap etc. How does the camera handles lights and strobes? What lenses have the best performance, balance, colours?

For Question 3, I'll compare the lens selections, size, weight, controls handling, etc. IQ is a given. M's don't go on safari and an S007 isn't getting into a street photography situation. Everything in between is a compromise of everything else.

Generally question 1 is just for laughs. All modern cameras are almost always more than good enough. Question 2 is the most important, always. If you know where your gear excels and struggles then it's much easier to get optimum IQ in the field. Mostly when it comes to reviewing my files it's almost always user error that spoils a shot, not how noisy it is. For me it's more important to know where an SL2 shows the least noise or retains the best highlight detail. rather than how it compares to an M11. I wouldn't use them in the same situations. Since I can't put a super tele zoom on an M11 it's irrelevant if the M11 performs fractionally better on P2P. 

Gordon

[jaapv]

The first thing to do is to make sure the images are properly exposed. An underexposed image is always noisy. So set the cameras to show identical histograms. The actual settings are irrelevant. 

[LBJ2]

On 11/3/2023 at 12:57 PM, goodbokeh said:

LBJ2, I know it's apples to oranges but I find my SL2 generally exposes best for highlights @ -1/3 EV less than my Sony A7R5 with it's 60 MP sensor. BTW, the noise levels are pretty close, the Sony has a fraction of a stop advantage on the noise ( I limit to 1600...3200 max) but there's not much in it. Honestly, I can take or leave the A7R5's 60MP sensor, it's the 50MP stacked sensor of my Sony A1 that has the real value added in certain circumstances.

Thank you for jogging my memory!

I saw this Sony ISO Invariance video about a year ago which describes how to see and compare various Sony camera ISO Invariance performance by recording black frames ( no lens) and using Lightroom to compare results. Not sure why I didn't think of this black frame approach when starting this thread. Regardless, I appreciate the feedback/suggestions from others on this topic too--always something to be learned from other's experience. 

I've already started the fairy easy to complete black frame comparison project and am in the process of comparing and evaluating SL2/M11 black frames in Lightroom. 

For reference incase anyone else is interested:

 

[goodbokeh]

Thanks LBJ2,  that's an impressive and instructive video. The dual gain in the extreme test examples did show a big noise advantage. I didn't see the dynamic range improvements but that could be the YouTube delivery or my 5K-P3 monitor I suppose.

[Photoworks]

I am not sure why you need to compare these two cameras.

In any case, M11 performs much better in lower iso up to 64-400. at 3400-6400, they are about the same.

The SL2 has the same noise at 400ISO and 800ISO, and I see a difference in 50-100ISO which is interesting.

In any case, the difference is not huge. when you do your tests make sure you turn off sharpening and noise reduction in your software to see the difference. The programs are ISO-aware and apply different amounts shot to shot.

 

[LBJ2]

On 11/8/2023 at 1:25 PM, Photoworks said:

I am not sure why you need to compare these two cameras.

In any case, M11 performs much better in lower iso up to 64-400. at 3400-6400, they are about the same.

The SL2 has the same noise at 400ISO and 800ISO, and I see a difference in 50-100ISO which is interesting.

In any case, the difference is not huge. when you do your tests make sure you turn off sharpening and noise reduction in your software to see the difference. The programs are ISO-aware and apply different amounts shot to shot.

 

"make sure you turn off sharpening and noise reduction in your software to see the difference" a good point and yes, I did remember.

[SrMi]

To compare cameras for noise performance, you must equalize the main contributor of noise: exposure (aperture, shutter speed, and scene light). You may need to adjust the brightness in the post, as ISO setting and/or metering is not standardized.

[LBJ2]

On 11/17/2023 at 2:28 PM, SrMi said:

To compare cameras for noise performance, you must equalize the main contributor of noise: exposure (aperture, shutter speed, and scene light). You may need to adjust the brightness in the post, as ISO setting and/or metering is not standardized.

Thank you. That's the path I was down. But after multiple applications/experiments I'm thinking recording a black frame is probably the best baseline as far as technical equalization between different camera models and offered the best approach to eliminating the light/exposure factor as much as possible. FWIW, I think the most consistent  approach to eliminate light "black" is to use the camera body cap and no lens attached to the camera to record the black frame. Lens+ lens caps can have some light leak.

I think I read somewhere black frame RAW/DNG recording is how Photons to Photos requests users create camera data files for submission, but I could be mistaken. Maybe those of you that have contributed to P2P can reply how you prepared those files. 

[SrMi]

1 hour ago, LBJ2 said:

Thank you. That's the path I was down. But after multiple applications/experiments I'm thinking recording a black frame is probably the best baseline as far as technical equalization between different camera models and offered the best approach to eliminating the light/exposure factor as much as possible. FWIW, I think the most consistent  approach to eliminate light "black" is to use the camera body cap and no lens attached to the camera to record the black frame. Lens+ lens caps can have some light leak.

I think I read somewhere black frame RAW/DNG recording is how Photons to Photos requests users create camera data files for submission, but I could be mistaken. Maybe those of you that have contributed to P2P can reply how you prepared those files. 

Here is the link describing P2P's collaboration data:

https://www.photonstophotos.net/Collaborations/Collaboration_Overview.htm

 

[LBJ2]

2 hours ago, SrMi said:

Here is the link describing P2P's collaboration data:

https://www.photonstophotos.net/Collaborations/Collaboration_Overview.htm

 

Thanks for posting. Very insightful to read this process( not to say I understand it all) to include the various additional separate instructions docs. Lot more to this then I had anticipated. I'm now definitely reconsidering what I previous thought might be erroneous SL2 data posted on photonstophotos.