Is C1 removing the DC offset in the DNG in Sean's test

[sandymc]

Advertisement (gone after registration)

When the Black Level Tag is changed in the DNG, is there a reason why compatibility is broken? If you could provide me with an uncompressed DNG, if possible taken at ISO 2500, together with the adress information where the black Level tag is stored, I could do some investigation, and find out if this Tag can be changed without compatibilty problems.

 

Hans,

 

Its only for C1 that compatibility is broken - LR and Aperture will still read the file fine. The issue is that C1 has never had a "real" DNG reader, just a reader for specific formats that makes all sorts of assumptions about where tags are in the file. Any small change, it falls over.

 

To throw a bit more light on this, I have done some digging into data values, and below is a plot of the black levels for both green channels in a ISO 2500 image.

 

Average level for all channels is 610,0464491 611,7939859 614,3293666 611,4042226 (R G1 G2 , and standard deviation is 102,5165799 102,7696443 102,0994712 102,9970364

 

Post black level clipping to 625, the averages are: 31,97786308 32,82917466 33,43480486 32,93921945

 

The data samples are from the data cuts that CornerFix uses to analyze vignetting, btw. If you'd like the spreadsheet, PM me your e-mail address.

 

Looking at the data, I'd say two things:

 

1. The 624 black level appears to be deliberately chosen to be somewhat above the actual DC level, presumably to minimize noise in black areas of the image

 

2. Looking at the data, I see no signs of in-camera noise reduction - if its happening, it's pretty subtle.

 

Sandy

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[scott kirkpatrick]

This is a pretty cryptic thread because the main posters have been trying to be brief, when something as tricky as the process that gets us from raw image data to a Bayer-interpolated RGB image needs to be explained step by step for clarity, even if it takes more words than all but a few of us will be willing to read. I still don't think I have gotten what Hans and Sandy want to say is bothering them. Here's what I think they have said and a few questions:

 

1) on a side issue, Sandy is saying that C1 will only read an M8 or M9 DNG file if it is written exactly the way they have found Leica to be writing it, based on sample files and perhaps discussions with Leica. The Adobe DNG spec has more freedom than that, and Sandy in producing modified Leica DNG files from Cornerfix has seen examples in which C1 has misinterpreted what are probably valid DNG constructions. An extreme example is C1's handling of LR-compressed DNG's which started life as M9 files. In C1's defense, they probably feel a need to stay "bug for bug" compatible with whatever Leica does, whether the files are valid DNG or not.

 

2) The business of how a DC ("dark current" but apparently it is a voltage) offset is determined and whether it combines in a bad way with the compression scheme... It doesn't seem that either Hans or Sandy is prepared to say exactly what is happening, whether the combination of the square root compression and a black point offset is not performed correctly (for example, is the black point subtracted before taking the square root, or does the square root trick ignore the black point, which is subtracted after the compressed data is restored to its 16 bit form. Or do they suspect a more subtle error, such as setting the black point too high. The M8 and M9 presumably handle ISO shifting very differently, since the M8 only offered full stop shifts, which can be done basically by digital shifts while the M9 offers one-third stop ISO settings, which I would assume are most easily done in the analogue hardware. Whatever is involved, the M8 did not pass along a black point offset in its DNG files output.

 

Am I following this discussion so far?

 

Are lookup tables in the compressed M9 DNG file the same as those used in the M8 DNG files? If they are different, does either lookup table depend on ISO? (This could indicate that the DC offset might be entering twice.)

 

Or is the problem that the black point in the DNG is too high?

 

scott

[jaapv]

I think you are missing the point here, Scott. The Black Level Tag is written before the DNG. The M9 outputs both square root compressed and uncompressed DNGs, (users choice) so that aspect is irrelevant in this discussion. I am sure Sandy and Hans are using uncompressed DNG files.

[scott kirkpatrick]

I think you are missing the point here, Scott. The Black Level Tag is written before the DNG. The M9 outputs both square root compressed and uncompressed DNGs, (users choice) so that aspect is irrelevant in this discussion. I am sure Sandy and Hans are using uncompressed DNG files.

 

I don't think so, Jaap. Sandy says something is wrong with the compressed files, and nobody knows where the black point data in the DNG is coming from, since it doesn't show some of the variations that the actual measured voltage would exhibit. Let's see what Sandy and Hans, who are apparently studying actual files, have to say.

 

scott

[jaapv]

I based myself on this post by Hans, and Michael's explanation, but will happily stand corrected by the experts.

The Dark Current compensation process that you have described is what happens within the camera before the data is written to the DNG File.

What is extra to the M9, is that a fixed DC voltage has been added to the ISO amplifier before digitizing, increasing each sample in value with exactly this offset voltage.

This offset voltage is supposed to generate a digital value of 44 at ISO 160, going up to 624 at ISO 2500, because the DNG file coming from the M9 tells C1 or Lightroom to subtract this value of each individual pixel in the file.

[sandymc]

Apologies for the cryptic posts(!)

 

The (slightly) less technical and (slightly) less cryptic version of the thread - at least my version, anyway is the following:

 

1. When Hans analysed Sean's ISO tests, he found that the exposure of deep blacks as displayed by C1 showed as brighter at high ISO than low ISO. So in effect, what were deep blacks at low ISO we becoming very dark greys at higher ISO.

 

2. The subsequent discussion has been around why that is happing. Han's initial hypothesis was that C1 wasn't getting the black levels right, and we went through a discussion of how C1 would set its black levels, which is the whole black level tag discussion. The end result of that was (in my opinion anyway) that C1 is getting it right, at least in the sense of doing what the data in the DNG file told it to do.

 

3. The balance of the thread is around the hypothesis that the setting of the blacklevel tags, which is done by the M9, doesn't exactly correspond to the actual black level of the image. What my last post basically said was that the measured black level and the blacklevel tag are indeed slightly different. Whether they are different enough to account for what Hans sees, Hans will have to comment on, as I'm not sure how exactly Hans performed his measurement.

 

For what its worth, in my opinion, the difference between the actual black level and the black tags is too small to cause any practical problems for anyone. And it appears deliberate, to reduce the level of visible noise in high ISO images.

 

The DC story is a bit of a diversion from the main thread - the offset (black level) could either come from dark current (which is a natural feature of the sensor), or be deliberately introduced as an offset, or be a combination of both. For the purpose of this thread it doesn't really matter. The reason why you would deliberately introduce an offset is to ensure that neither the actual electronics or the software in the M9 ever saw a zero level - at or close to zero level, there are non-linearities, noise becomes more difficult to remove, etc. All-in-all you get better image quality more easily by staying away from zero as long as you can.

 

 

 

The compression story, btw is different, and largely unrelated to this thread, except that it also involves the (now rapidly becoming famous) blacklevel tag. The issue there is that while the compression curves in the M8 and M9 are the same, the end result of the compression process is different. Because the M9 compresses "below black" data, a portion of the 256 codes available get used by data that will never be seen. The impact is potentially substantial - at ISO 2500, 50 of the available 256 codes are being used for below black data. The M8 on the other hand preprocesses the image data to remove the black level, and hence always uses all 256 levels for "real" image data.

 

Sandy

[t024484]

Advertisement (gone after registration)

Here's what I think they have said and a few questions:

 

The business of how a DC ("dark current" but apparently it is a voltage) offset is determined and whether it combines in a bad way with the compression scheme... It doesn't seem that either Hans or Sandy is prepared to say exactly what is happening, whether the combination of the square root compression and a black point offset is not performed correctly (for example, is the black point subtracted before taking the square root, or does the square root trick ignore the black point, which is subtracted after the compressed data is restored to its 16 bit form. Or do they suspect a more subtle error, such as setting the black point too high. The M8 and M9 presumably handle ISO shifting very differently, since the M8 only offered full stop shifts, which can be done basically by digital shifts while the M9 offers one-third stop ISO settings, which I would assume are most easily done in the analogue hardware. Whatever is involved, the M8 did not pass along a black point offset in its DNG files output.

 

Am I following this discussion so far?

 

Are lookup tables in the compressed M9 DNG file the same as those used in the M8 DNG files? If they are different, does either lookup table depend on ISO? (This could indicate that the DC offset might be entering twice.)

 

Or is the problem that the black point in the DNG is too high?

 

scott

Scott, I haven't seen a compressed file from an M9, so I have no idea if the same Black Level issue plays a role here.

If an offset has been added for the black level, then indeed this will reduce the 255 levels that are used, and this will force another compression scheme than used fot the M8, and a different one for each ISO setting, a thing that I find hard to believe, but again I haven"t seen an compressed M9 File.

In general compressing to 8 bits, causes an extra source of noise that is equal to the shot noise, that is why the SQRT compression has been used.

Adding these two noise, increases the noise wit 3 dB, or 1/2 stop over the larger part of the exposure range. Decreasing the compression to less than 255 levels, will increase this amount of compression noise even more.

A lossless compression can be done from 10 bits, but you will need a lot of calculation to do this, so an uncompressed file with 14 bit values is just as convenient.

 

I have only concentrated myself on the non compressed file, and was amazed that Leica added a fixed black level tag to the file, whis is indicating the black level, but in fact the value of this black level tag has no relation to the DNG file itself, and only differs for the ISO that has been used.

Another thing that surprised me was that there was no linear relation as there should be.

If the black level at ISO 160 has to be 44, then the value at ISO 2500 has to be (2500/160)*44= 687, and not the 624 that Leica writes in the ISO2500 DNG File.

Doing so gives a black level that is either too high or too low.

 

That brought me to the point that with the offset that Leica has added to the DNG, it is now perfectly possible to calculate the EXACT black level, as opposed to the value of the Tag in the DNG.

 

Calculating the black level can be performed by calculating the average value of all 24.304 pixels that are not exposed, and substituting this value for the Tag value in the DNG.

 

Now according to Sandy, this could confuse C1. I took a file taken at ISO2500 with a black level tag of 624. I copied this file into a second and a third file with a tag value of resp. 0 and 687.

These black level values were indeed accepted by LR but also by C1 4.8.3, so this restricted experiment proves that it is possible to manipulate the black level without a problem.

This now opens he way to have a black level set to an accuracy that was never possible before, by taking the average of the unexposed pixels.

 

About switching the ISO sensitivity, I do not believe that the M8 did this in the digital domain, because doing so will also amplify the a/d noise.

The m9 is also doing it in the analog domain, but probably has a second amplifier that can switch for the intermediate values, added to the hardware that was already available from the M8.

 

Hans

[t024484]

1. When Hans analysed Sean's ISO tests, he found that the exposure of deep blacks as displayed by C1 showed as brighter at high ISO than low ISO. So in effect, what were deep blacks at low ISO we becoming very dark greys at higher ISO.

 

Exactly what puzzled me.

 

2. The subsequent discussion has been around why that is happing. Han's initial hypothesis was that C1 wasn't getting the black levels right, and we went through a discussion of how C1 would set its black levels, which is the whole black level tag discussion. The end result of that was (in my opinion anyway) that C1 is getting it right, at least in the sense of doing what the data in the DNG file told it to do.

[/Quote]

Yes, you convinced me that C1 is doing what it shoud do.

 

3. The balance of the thread is around the hypothesis that the setting of the blacklevel tags, which is done by the M9, doesn't exactly correspond to the actual black level of the image. What my last post basically said was that the measured black level and the blacklevel tag are indeed slightly different. Whether they are different enough to account for what Hans sees, Hans will have to comment on, as I'm not sure how exactly Hans performed his measurement.[/Quote]

My question to you is, which pixels did you use for your calculation? Did you use 7751 unexposed pixels from the first and second line ?

 

For what its worth, in my opinion, the difference between the actual black level and the black tags is too small to cause any practical problems for anyone. And it appears deliberate, to reduce the level of visible noise in high ISO images.

I would like to analyse a bit more images to come to a conclusion.

 

The DC story is a bit of a diversion from the main thread - the offset (black level) could either come from dark current (which is a natural feature of the sensor), or be deliberately introduced as an offset, or be a combination of both. For the purpose of this thread it doesn't really matter. The reason why you would deliberately introduce an offset is to ensure that neither the actual electronics or the software in the M9 ever saw a zero level - at or close to zero level, there are non-linearities, noise becomes more difficult to remove, etc. All-in-all you get better image quality more easily by staying away from zero as long as you can.

This does not explain why Leica did not substract this level before writing to the DNG. It must have a purpose to pass this information to the RAW processor, a sort of either/or.

The raw processor can use the value in the Black level tag, or calculate it from the unexposed pixels. That is still my best guess.

 

The compression story, btw is different, and largely unrelated to this thread, except that it also involves the (now rapidly becoming famous) blacklevel tag. The issue there is that while the compression curves in the M8 and M9 are the same, the end result of the compression process is different. Because the M9 compresses "below black" data, a portion of the 256 codes available get used by data that will never be seen. The impact is potentially substantial - at ISO 2500, 50 of the available 256 codes are being used for below black data. The M8 on the other hand preprocesses the image data to remove the black level, and hence always uses all 256 levels for "real" image data.

Sandy

 

I would love to have a few compressed DNG's from the M9 for evaluation. I mentioned to Scott that I found it hard to believe that compression is different as used for the M8. From what I understand here, is that you have noticed that it is indeed different, so I eat my words. This in effect would generate extra noise, because fewer levels are available. It is probably better to avoid compression at all.

 

Hans

[wlaidlaw]

Hans,

 

I wonder if it is worth sending Jakub at Phase One a link to this thread. I think they might find it very interesting. However maybe you have a direct contact with someone else at Phase. If there is a methodology of setting more accurate black levels as appears to be the case from your experiments, it would be to our advantage to have this at least as an option for processing M9 files. Could this be done as an alternative profile for M9 - say M9 (black adjusted) in addition to the M9 (standard)?

 

Wilson

[sandymc]

My question to you is, which pixels did you use for your calculation? Did you use 7751 unexposed pixels from the first and second line ?

 

No, I cheated - the analysis was done on the live pixels of a "black" image, looking at horizontal and vertical middle lines of pixels, and the two diagonals. That is what CornerFix analyses, so I could easily just get a dump of that data. Looking at the dark pixels would have been a lot more work(!)

 

Re C1, just to clarify, there is no problem (that I know of) with just changing the value in a tag if you leave the tag in the same position. CornerFix's problem is that because it changes the file significantly, tag positions change, which C1 doesn't seem to like. You could write a program that just changed the blacklevel tag which C1 would read, but what CornerFix does is a lot more than just that. Phase One are aware of the issue, and have verified for themselves that it does exist, but given that it doesn't affect unmodified M9 files, and the current version of CornerFix has a workaround that allows it to write compatible files, I doubt that fixing the issue is going to be on the priority list, as no user will ever see the issue.

 

Sandy

[t024484]

Hans,

 

I wonder if it is worth sending Jakub at Phase One a link to this thread. I think they might find it very interesting. However maybe you have a direct contact with someone else at Phase. If there is a methodology of setting more accurate black levels as appears to be the case from your experiments, it would be to our advantage to have this at least as an option for processing M9 files. Could this be done as an alternative profile for M9 - say M9 (black adjusted) in addition to the M9 (standard)?

 

Wilson

Wilson,

 

No I have no contact to C1 or anyone else in this industry. But you are right, a Raw converter could just ignore the Black Level Tag, and calculate the Black Level from the image itself.

This has nothing to do with profiles, though an incorrect black level has its effect on the colours.

 

Hans

[t024484]

Sandy,

 

 

The standard deviation of 102 that your calculation produced puzzled me a bit, since my attempts gave me a different value.

But your calculation was absolutely correct, I had an unfortunate error in my program.

 

This brings me to the point that the standard deviation at ISO 160 is 9.7 instead of the 1.5 that I had mentioned before.

Now all at the sudden the M9 is not at all as quiet as I thought. In fact the M9 is just as noisy as the M8 with respect to Sensor noise and A/D converter noise.

The fact that Sean’s images are showing the same amount of noise in the deep dark for the M8 and the M9 proves this point.

The M9 for that matter has three times more noise in the deep dark as the D3.

 

At low ISO and higher luminance’s though, the uncompressed M9 image’s are showing less noise as the compressed M8 images have.

This is the range where three types of noise are dominating: shot noise, PRNU (Pixel Related Non Uniformity) noise and noise made by compressing the Image to 8 bits for the M8

 

Shot noise is there because of physical reasons, and cannot be influenced other than changing the size of the pixel. But for the M8 and the M9 the shot noise will be the same.

Shot noise has a SQRT relation to the captured light. That must also be the reason why Leica is using a SQRT compression for the M8.

PRNU is caused by different sensitivities to light exposure for different pixels. For the M8, the nominal differences between pixels is 12%, so the more light the bigger the absolute differences.

At low ISO and at lower Luminances, shot noise is dominating because of the SQRT relation, and at the higher Luminance levels the PRNU becomes the dominating noise.

At higher ISO, since less light is captured by the sensor, PRNU is getting less important, being more and more dominated by shot noise.

Compressing the image to 8 bits according to a SQRT law, generates about as much noise as the shot noise.

 

Maybe Leica is correcting the captured M9 Image for PRNU errors, leading to a better image with lesser noise. But also having an uncompressed image from the M9 helps to reduce noise over the M8 with theoretically ½ stop . Both factors would explain why the images from the M9 are having less noise, together even a full stop as Leica claims.

 

At ISO 2500 PRNU noise plays no longer any role, but the absence of noise from non compressing the image is still in favour of the M9.

But maybe the Images are still further improved by a software noise reduction algorithm, which would explain the longer processing time at this speed.

 

So to conclude, the noise elements in a picture from low to high luminance are:

 

Lower ISO: (Sensor+A/D converter noise ) / Shot noise / PRNU noise

Higher ISO: (Sensor+A/D converter noise) / Shot noise

 

Shot noise and A/D noise are about the same between the M8 and the M9, and so is the shot noise.

PRNU might be corrected for in the M9, and for an uncompressed M9 Image, the compression noise from the M8 is missing, being of equal magnitude as the shot noise.

At ISO 2500 a noise reduction algorithm might be used, to be proven by means of a 2 dimensional FFT.

 

Hans

 

P.S. This noise discussion is just a different topic, and has no relation to the black level discussion above

[sandymc]

Hans,

 

Very interesting. So essentially, the raw sensor performance as regards noise is very similar to the M8. That's consistent with the Imatest test results I've seen - they show a small improvement, but nothing dramatic. However, I don't really trust Imatest's noise measurements, so its good to see your results.

 

Are you going to perform the 2-D FFT test? It would also be very interesting to know whether there is any noise reduction applied. Just visually looking at the data, I didn't see any sign of noise reduction, but a 2D FFT would be a far better test.

 

Regards,

 

Sandy

[Jamie Roberts]

{snipped}

Are you going to perform the 2-D FFT test? It would also be very interesting to know whether there is any noise reduction applied. Just visually looking at the data, I didn't see any sign of noise reduction, but a 2D FFT would be a far better test.

 

Regards,

 

Sandy

 

Yes, if Leica is doing NR at high ISOs, from what I've seen they're doing something really right

[rosuna]

A bit of NR in camera, before the DNG is written, is a good idea. Chromatic noise is ugly and it can be removed without affecting detail. Leica can find ways for removing some luminance noise preserving detail...

 

The sensor of the S2 camera incorporates 4 channels for data extraction (at a lower frequency), and the M8 sensor only two of them (and the M9 sensor is based on the M8 sensor, not the S2). Leica may update the sensor for future versions of the M9 camera.

 

How much improvements can be expected?

 

Is the S2 processor (Maestro) more powerful than the two DSP processors of the M9? Are there differences in programming? To sum-up, May the S2 do things the M9 can't regarding noise treatment?

[Jamie Roberts]

{snipped}

Is the S2 processor (Maestro) more powerful than the two DSP processors of the M9? Are there differences in programming? To sum-up, May the S2 do things the M9 can't regarding noise treatment?

 

...I hope so for the difference in price (sorry, couldn't resist!).

[rosuna]

...I hope so for the difference in price (sorry, couldn't resist!).

 

Do you think the difference in price is due to the processor and sensor's architecture? I doubt it.

[t024484]

To perform a 2D FFT analysis, could someone provide me with 3 DNG's taken at 640, 1250 and 2500 ISO of a white wall, not overexposed, but nice bright.

This will show if Leica is using a in camera noise reduction algorithm.

 

Hans

[Advertisement]

Hallo,

Look here: Leica M

[Jamie Roberts]

Do you think the difference in price is due to the processor and sensor's architecture? I doubt it.

 

I honestly think that's one of the differences, yes... as well as the actual size of the sensor.

 

I can't imagine it would be inexpensive in the slightest to have a company like Fujitsu custom-design the processors / pipeline. Maybe they do a lot of digital camera work, but if they do I've never heard about it.

 

I have the feeling somehow (though I don't know, of course) that Jenoptik is working with more or less off-the-shelf parts for the M8 and M9.

 

So yes, I do think there's going to be some significant differences, and they don't come cheaply.

 

Once the $$ has been sunk into the S2 and recovered, however, there's no theoretical reason why a future M camera couldn't exploit the work (although there may be tons of practical reasons).

[ho_co]

Jamie--

I think you may be right.

 

IIRC, when Leica bought the M8's Kodak sensors, they explained that they weren't restricting sales. That is, although Kodak and Leica had developed the sensor specifically for the M8, Kodak was now free to sell it to anyone.

 

In the Reichmann interview, I believe Stefan said that the M9 sensor was proprietary. If so, then Kodak is producing only for Leica, and Leica would need to pay for that privilege.

 

(Keeping this sensor unavailable to others sounds to me like a good business move.)