Jump to content

Is C1 removing the DC offset in the DNG in Sean's test


t024484

Recommended Posts

Advertisement (gone after registration)

Looking at the ISO samples thar Sean Reid has made, two things are striking me.

 

1) In all cases the M9 has the lowest noise of the three Camera's in the Apple, the white and grey balance card from ISO 160 till ISO 2500, which corresponds very well to the fact that the M9 has the lowest possible sensor and A/D noise, lower than the 5D II.

2) In the black area, the two Leica's are displaying more detail al lower ISO, but also more Noise at higher ISO.

 

After having converted the images to B&W and after analysing them in LR, the black area above the grey card, has an almost constant black level for the 5D, between 0.4 to 0.8% for the different ISO values from 160 to 1250.

For the M8 the increase of the black level is from 1.0 % to 1.5%, the increase in line with expectation because of the higher noise values of the M8 compared to the 5D.

 

But the values of the M9 are not at all what is to be expected. At ISO 160 the blacks are at roughly 0.6%, climbing to over 2.5% at ISO 1250.

 

The decompressed M8 DNG starts with a black point at 0 and and a white point at 16.383.

The difference in the DNG of the M9 compared to the M8, is that a DC offset is added before digitizing, causing a black point at roughly 40/16.383 ( 0,25%) at ISO 160, going to 310/16.383 (1.9%) at ISO 1250.

Analyzing Sean's Crops, I get the feeling that C1 does not substract this black point from the DNG values, but starts at 0, just like for the M8.

This would explain why the Black part in the image raises from 0.6% to 2.5% in luminance.

If the black area would heve been kept at the intial level of 0.6%, the result would have been a much lesser amount of noise in the black area.

It simply does not correspond to the fact that the M9 is probably the camera with the lowest possible hardware noise on the market.

This is visable in the areas with the higher luminances, but not in the Black area, that's what causes my suspicion.

 

Hans

Link to post
Share on other sites

  • Replies 40
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Hans,

 

Couple of comments:

 

1. C1 does correctly handle black point - getting the black point wrong is instantly visible on high ISO images (That's what caused the previous version of CornerFix not to be compatible with C1, but C1 has been ok with all "native" M9 files I've run through it)

 

2. However, its not clear to me that the black point as set in the DNG is correctly matched to the DC offset. The DNG black point appears fixed, while in practice, DC offsets drift with temperature, etc.

 

3. Also, are you sure you're not seeing the compression glitch I described here http://www.l-camera-forum.com/leica-forum/leica-m9-forum/98212-lightoom-mangling-m9-images-there-compression.html ? That can increase apparent noise.

 

Sandy

Link to post
Share on other sites

Hans,

 

Couple of comments:

 

1. C1 does correctly handle black point - getting the black point wrong is instantly visible on high ISO images (That's what caused the previous version of CornerFix not to be compatible with C1, but C1 has been ok with all "native" M9 files I've run through it)

Sandy

That is exactly what I am trying to discuss. In Sean's Crops, the Blacks are raising in level much faster as is to be expected.

 

2. However, its not clear to me that the black point as set in the DNG is correctly matched to the DC offset. The DNG black point appears fixed, while in practice, DC offsets drift with temperature, etc.

 

There is a black point dependent on Temp drift and whatever, that is calculated in the integration time that takes place before the image is actually transferred to the DNG file of the camera. This black level is determined from the pixels that are not exposed. This correction is already in the DNG that comes from the camera.

 

What I found is that Leica has added an extra offset to the analog voltage in order to have an even better means of calculating the black point, just like some other high end DSLR cameras are doing.

 

When reading a DNG from a M9, that is exposed with the lenscap on, with 1/4000 second, one can see that the average level of the data in the DNG is ca. 40 /16,383 for ISO 160 with a Standard deviation of 1.55, with an offset going up to 625/16,383 for ISO 2500, with a standard deviation of 16.

All samples in that way will always produce a positive digital figure.

With the M8, you will find many zero samples in the DNG where the cumulated noise caused the signal to become zero or negative ( this is 50% of the time), and only noise samples that are positive will result in a digital value above 0.

It is this offset of 40 at ISO 160 going up to 625 at ISO 2500 that I doubt is being taken care of by C1 in the tests that Sean showed us, because I see the luminance of the black area in the picture rising with the same value of the Offset I have just described, which is not the case for the M8 and the 5D II.

 

3. Also, are you sure you're not seeing the compression glitch I described here http://www.l-camera-forum.com/leica-forum/leica-m9-forum/98212-lightoom-mangling-m9-images-there-compression.html ? That can increase apparent noise.

 

Sandy

Sean made his tests with uncompressed DNG's, so compression glitches are out of order.

 

I hope I expressed myself in an understandable way.

 

Hans

Link to post
Share on other sites

It is this offset of 40 at ISO 160 going up to 625 at ISO 2500 that I doubt is being taken care of by C1 in the tests that Sean showed us, because I see the luminance of the black area in the picture rising with the same value of the Offset I have just described, which is not the case for the M8 and the 5D II.

 

Umm, I guess I wasn't being clear. M9 DNGs have the DNG BlackLevel tags set. Those tags increase from 44 at ISO 160 to 624 at ISO 2500, at least on the samples I have. Those blacklevel settings are subtracted from the data in DNG prior to display, thus compensating for the DC voltage. C1 correctly does that. C1 does not, and cannot, measure the actual black level - it must rely on the BlackLevel tag.

 

What may be the case is that the M9 is not accurately matching what it is setting BlackLevel tag to, to the actual DC voltage offset. E.g, the DC you are measuring (I assume from the raw data) does not match the BlackLevel of 44. So I'm suggesting that what you're seeing is not C1 ignoring the tag of 40, but the impact of the difference between 40 and 44. Which is an M9/Leica issue, not a C1 issue.

 

Regards,

 

Sandy

Link to post
Share on other sites

Umm, I guess I wasn't being clear. M9 DNGs have the DNG BlackLevel tags set. Those tags increase from 44 at ISO 160 to 624 at ISO 2500, at least on the samples I have. Those blacklevel settings are subtracted from the data in DNG prior to display, thus compensating for the DC voltage. C1 correctly does that. C1 does not, and cannot, measure the actual black level - it must rely on the BlackLevel tag.

 

What may be the case is that the M9 is not accurately matching what it is setting BlackLevel tag to, to the actual DC voltage offset. E.g, the DC you are measuring (I assume from the raw data) does not match the BlackLevel of 44. So I'm suggesting that what you're seeing is not C1 ignoring the tag of 40, but the impact of the difference between 40 and 44. Which is an M9/Leica issue, not a C1 issue.

 

Regards,

 

Sandy

O.K., this brings me one step further, thank you for that.

As I understand, the DNG from the M9 has a Blacklevel tag,to be used as a value that should be substracted from the DNG data.

In what way could you see that C1 substracted these blacklevel tags from the DNG data, is C1 converting the file to another DNG where the Tag value is substracted ?

 

Hans

 

P.S.For simplicity, I rounded the figure to 40, but actually what I measured was close to 44.

Link to post
Share on other sites

In what way could you see that C1 substracted these blacklevel tags from the DNG data, is C1 converting the file to another DNG where the Tag value is substracted ?

 

Hans,

 

You can see, just visually - this is the same ISO1600 image (original posted in another thread here last week), first the original, then with blacklevel ignored.

 

Sandy

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

Link to post
Share on other sites

Advertisement (gone after registration)

This is an interesting thread but I have to admit it's way over my head. Could any of you explain the consequences of this number handling in terms a boob like me might grasp?

 

From the pictures posted I wonder if what you're discussing might have an effect on reducing the effect of IR contamination (the shirt looks rather magenta in one image).

 

What is the practical effect -- more or less high ISO noise as compared to the Canon?

 

Thanks!

Link to post
Share on other sites

From the pictures posted I wonder if what you're discussing might have an effect on reducing the effect of IR contamination (the shirt looks rather magenta in one image).

 

That's what I though when I first looked the the image. However, the color cast is really a optical illusion, brought on by the way C1 (which generated the JPEGs) works and the difference in RGB channel sensitivities.

 

Sandy

Link to post
Share on other sites

This is an interesting thread but I have to admit it's way over my head. Could any of you explain the consequences of this number handling in terms a boob like me might grasp?

For a start, in this context ‘DC’ stands for ‘dark current’ – I don’t think this had been explained. Dark current is a phenomenon leading to some extra electrons collected within each sensor pixel that bear no relationship to the number of photons hitting the pixel, so they are skewing the results. The number of extra electrons accumulated during exposure depends on the temperature (higher temperature –> more dark current) and its effective contribution to the image data also depends shutter speed (longer exposure time –> more electrons accumulated) and the ISO setting (a higher ISO setting requires a higher amplification, also amplifying the contribution of dark current). To compensate, the contribution of dark current is measured using reference pixels shielded against light, so all the electrons collected there during exposure should be due to dark current. A number representing the contribution of dark current is then stored in the DNG file, to be subtracted from the value of each sensor pixel in the raw conversion process. Since dark current isn’t completely uniform, i.e. its contribution varies somewhat from pixel to pixel, it also introduces some noise that cannot be similarly just subtracted away. Still, for common shutter speeds and temperatures the contribution of dark current to noise is negligible compared to other sources of noise.

 

This, by the way, is standard stuff; it is in no way specific to the M9.

 

From the pictures posted I wonder if what you're discussing might have an effect on reducing the effect of IR contamination (the shirt looks rather magenta in one image).

As Sandy explained, IR contamination is a totally different issue. There’s a reason for the two phenomena looking similar, though, that Sandy already hinted at. The color filters in front of the sensor pixels have quite different transmission curves blocking different percentages of the light hitting each pixel. Green filters generally have the highest transmission and blue filters often the lowest. When all of the sensor pixels were evenly lit, you would still get higher values from the green-sensitive pixels than from the red- or blue-sensitive ones as the filters for red and blue are absorbing more light. This has to be corrected for by effectively amplifying the signals from the red and blue pixels, as otherwise the image would have a greenish tint when it should be white or grey.

 

This is all well and good, but there are some phenomena applying uniformly to each sensor pixel, regardless of the filter color. Dark current is one, IR contamination (IR passes through the red, green, and blue filters with nearly equal ease) another, and also smearing (not an isue with the M9, but with compact digicams) affects all pixels uniformly. When the red and blue values get amplified, assuming they got hit by less photons due to their stronger filters, the roughly uniform contribution of dark current, IR, or smearing to all the pixels gets amplified as well, resulting in a magenta tint. So the effects of all those phenomena, their different causes notwithstanding, look rather similar.

Edited by mjh
Link to post
Share on other sites

As Sandy explained, IR contamination is a totally different issue. There’s a reason for the two phenomena looking similar, though, that Sandy already hinted at. The color filters in front of the sensor pixels have quite different transmission curves blocking different percentages of the light hitting each pixel. Green filters generally have the highest transmission and blue filters often the lowest. When all of the sensor pixels were evenly lit, you would still get higher values from the green-sensitive pixels than from the red- or blue-sensitive ones as the filters for red and blue are absorbing more light. This has to be corrected for by effectively amplifying the signals from the red and blue pixels, as otherwise the image would have a greenish tint when it should be white or grey.

 

This is all well and good, but there are some phenomena applying uniformly to each sensor pixel, regardless of the filter color. Dark current is one, IR contamination (IR passes through the red, green, and blue filters with nearly equal ease) another, and also smearing (not an isue with the M9, but with compact digicams) affects all pixels uniformly. When the red and blue values get amplified, assuming they got hit by less photons due to their stronger filters, the roughly uniform contribution of dark current, IR, or smearing to all the pixels gets amplified as well, resulting in a magenta tint. So the effects of all those phenomena, their different causes notwithstanding, look rather similar.

 

Nice explanation!!!

 

Sandy

Link to post
Share on other sites

Thank you! I think this gives me a much better (still not perfect) understanding of this thread. If I follow (and please tell me if I'm not) -- something, either C1 or the processing going on in the M9, is artificially raising shadow values and accentuating noise.

 

And if that's way off the mark it's my limited brain, not your explanations.

 

Thanks again

Link to post
Share on other sites

Joesph,

 

If I understand correctly, it's actually that C1 is respecting the DC offset that is supplied by the DNGs and consequently does the right thing: that's why you get a good-looking file from it.

 

The issue in the compressed DNGs, if I understand it correctly, is that the compression isn't quite right, and the M9 is apparently clipping more information than it should be in the compression scheme.

 

This may or may not have a visual effect; it depends on exposure and subject matter. But it's a matter of tweaking the compressed DNGs, not a horrible image flaw or anything of the kind.

 

Sandy--did I get that mostly right? :)

 

@ Michael--awesome explanation! Thanks!

Link to post
Share on other sites

Joesph,

 

If I understand correctly, it's actually that C1 is respecting the DC offset that is supplied by the DNGs and consequently does the right thing: that's why you get a good-looking file from it.

 

The issue in the compressed DNGs, if I understand it correctly, is that the compression isn't quite right, and the M9 is apparently clipping more information than it should be in the compression scheme.

 

This may or may not have a visual effect; it depends on exposure and subject matter. But it's a matter of tweaking the compressed DNGs, not a horrible image flaw or anything of the kind.

 

Sandy--did I get that mostly right? :)

 

@ Michael--awesome explanation! Thanks!

 

Jamie, yes, with a one subtlety, which is that the oddity that Hans has identified occurs in uncompressed files as well, but has only been identified in C1 processed images, and we're bouncing around trying to work out whether it's (a) real, (B) something to worry about, and © just something that relates to C1, or is broader. I don't think we've got a conclusion yet :confused:

 

The compression issue I think is indeed real, but not something to worry about as the workaround is easy.

 

Sandy

Link to post
Share on other sites

For a start, in this context ‘DC’ stands for ‘dark current’ – I don’t think this had been explained. Dark current is a phenomenon leading to some extra electrons collected within each sensor pixel that bear no relationship to the number of photons hitting the pixel, so they are skewing the results. The number of extra electrons accumulated during exposure depends on the temperature (higher temperature –> more dark current) and its effective contribution to the image data also depends shutter speed (longer exposure time –> more electrons accumulated) and the ISO setting (a higher ISO setting requires a higher amplification, also amplifying the contribution of dark current). To compensate, the contribution of dark current is measured using reference pixels shielded against light, so all the electrons collected there during exposure should be due to dark current. A number representing the contribution of dark current is then stored in the DNG file, to be subtracted from the value of each sensor pixel in the raw conversion process. Since dark current isn’t completely uniform, i.e. its contribution varies somewhat from pixel to pixel, it also introduces some noise that cannot be similarly just subtracted away. Still, for common shutter speeds and temperatures the contribution of dark current to noise is negligible compared to other sources of noise.

 

This, by the way, is standard stuff; it is in no way specific to the M9.

.

The Dark Current compensation process that you have described is what happens within the camera before the data is written to the DNG File.

What is extra to the M9, is that a fixed DC voltage has been added to the ISO amplifier before digitizing, increasing each sample in value with exactly this offset voltage.

This offset voltage is supposed to generate a digital value of 44 at ISO 160, going up to 624 at ISO 2500, because the DNG file coming from the M9 tells C1 or Lightroom to subtract this value of each individual pixel in the file.

 

It is very important that this offset voltage is extremely stable, but this can all be done with relative ease.

A point that is more obvious to me, (2500/160)*44@ISO160=687@2500 and not 624, so this will have an effect to the black level in a way to be explained, because a figure too low will be subtracted.

At ISO 320, 640 and 1250, the indicated black level in the DNG file is resp. 3, 11 and 23 too low when doing the same calculation.

 

If an offset 687 is added before digitizing, and later on a figure of 624 is subtracted, this will have an effect to the black level in the picture, because it will not be 0,0,0 for RGB but 63,63,63 at ISO 2500, and 23,23,23 for ISO 1250 etc. This could be an explanation for the fact that in Sean test the luminance of the black area is going up with ISO, what should not be the case. That is why I started this discussion in the first place.

 

It will also have an effect on the colours in the dark areas, because what should be 10,50,0 is now suddenly 73,113,63 being a complete different colour.

The higher the pixel value is, the lesser the effect. At a pixel value of 15.000, an error of 63 is only .4% and will be invisible.

 

The question is, why has Leica added this offset to the DNG that should be subtracted afterwards.

To my opinion the answer can only be that because of this offset, all digitized pixel samples are now represented by a positive figure, also the unexposed pixels that are covered, and that are used for calculating the dark current within the camera.

 

Without offset, we can only see noise on the dark current level that has a positive value, but now we can see the full noise, which enables the possibility to do something that is more accurate than subtracting just a figure that is in the DNG file.

 

There are in total 2 extra lines above and 2 extra lines below the picture that are unexposed, but digitized and in the DNG.

Also for all remaining 3468 lines, there are 2 extra pixels to the left and 2 extra pixels to the right that are not exposed. In total this gives 24,304 pixels that are not exposed.

When calculating the average value of these very pixels, will give an exact black level value that has to be subtracted from all exposed pixel values in the DNG file instead of using the fixed level in the DNG file, that is what I believe.

 

Maybe Sandy could try to do this in Cornerfix, and substitute the fixed value in the DNG for this calculated Black Level figure

 

Hans

Link to post
Share on other sites

Maybe Sandy could try to do this in Cornerfix, and substitute the fixed value in the DNG for this calculated Black Level figure

 

While possible in theory, there is a practical complication, which is that this would break compatibility with Capture One 4.8.3. So there would really need to be demonstrated image quality advantages to go that route.

 

Sandy

Link to post
Share on other sites

I may have got this wrong but I thought that even the uncompressed DNG discarded the bottom bit of the original 16 because the DC contaminated the signal to the point it was useless. A similar situation at the top end where the sensor is unable to provide significant information in brightness levels in ultra bright situations, so the top bit is also discarded, resulting in the 14 bit data of the uncompressed DNG. At the dark end, as the bottom bit is discarded, I assume either the analogue to digital converter in the camera or the processor resets the data from the next bit to provide a 0,0,0 pure black or does it? Presumably this is where C1 4.8.3 wins out using the additional camera specific file information provided with the DNG, against the other two main processors, Aperture and LR, which I understand currently do not.

 

Wilson

Link to post
Share on other sites

While possible in theory, there is a practical complication, which is that this would break compatibility with Capture One 4.8.3. So there would really need to be demonstrated image quality advantages to go that route.

 

Sandy

When the Black Level Tag is changed in the DNG, is there a reason why compatibility is broken? If you could provide me with an uncompressed DNG, if possible taken at ISO 2500, together with the adress information where the black Level tag is stored, I could do some investigation, and find out if this Tag can be changed without compatibilty problems.

 

I've been trying to follow this esoteric thread... My question - as the discrepancy increases with the ISO level, could it be that you are seeing part of the in-camera processing to reduce noise?

 

I personally do not think that this is playing a role, but further investigation will hopefully show.

 

I may have got this wrong but I thought that even the uncompressed DNG discarded the bottom bit of the original 16 because the DC contaminated the signal to the point it was useless. A similar situation at the top end where the sensor is unable to provide significant information in brightness levels in ultra bright situations, so the top bit is also discarded, resulting in the 14 bit data of the uncompressed DNG. At the dark end, as the bottom bit is discarded, I assume either the analogue to digital converter in the camera or the processor resets the data from the next bit to provide a 0,0,0 pure black or does it? Presumably this is where C1 4.8.3 wins out using the additional camera specific file information provided with the DNG, against the other two main processors, Aperture and LR, which I understand currently do not.

A 14 bit A/D converter is used, where all 14 bits are in use. No bits are discarded, but since they make use of a 16 bit word, 2 bits remain unused. There is also no reset to a certain bit. What the camera does, is to asses the value of the dark current, and substracts this from the sensor data before digitizing.

LR as well as C1 are forced to using the extra Black level Tag (which is zero for the M8) , to be substracted from the pixel values in the DNG file, after the M9 has added this extra level to the DNG values before. Leica adds something that should be substracted by LR or C1, what is the whole point of this?

 

Hans

Edited by t024484
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...