Speculating on why profiling works: impossible colors?

[blakley]

There have been assertions here and elsewhere that the magenta shift problem can't be fixed in firmware (or, presumably, by profiling in a Raw processor) because the sensor will "see" truly magenta fabric and black fabric exhibiting the magenta shift as the same color.

On the other hand, there have also been demonstrations here that profiling can significantly improve magenta shift problems, if not eliminate them.

The first position makes logical sense, but doesn't explain the observed facts of profile adjustment.

While I was pondering this it occurred to me that my mental model of how the sensor is actually working might not be right. If the sensor is literally just recording "RGB", and the "true magenta" and "magenta black" objects are really inducing the same set of RGB values in the sensor, then the "it's impossible" argument seems hard to refute. But that's not the only thing the sensor could be doing. Since I don't know how the sensor actually works, I can only fall back on other things I do know about. One thing I know about is Dan Margulis' notion of "impossible colors" in LAB. I'll quote Margulis (from his book "Photoshop LAB Color", page 162); impossible colors are:

"Colors that do not and cannot exist. Something as brilliantly green as exploding fireworks, but at the same time as dark as the night sky surrounding them - such a color is impossible, inconceivable, a preposterous offense against logic, a contradiction in terms. And yet it can be created, however theoretically, however fleetingly, in LAB."

I'll spare you another quote, but Margulis goes on to say that what happens when you try to convert a dark impossible LAB color to another space with smaller gamut is that the color gets lightened to bring it into gamut.

Which leads me to the point: if the M8's sensor gamut is wider than its DNG gamut, then maybe what's going on is that certain "dark impossible colors" (IR magentas which look black to the eye, for example) from the sensor color space are being lightened when converted to the DNG gamut, when what should really be happening is that they should be left dark.

If this is really the case, it would hold out some hope for a firmware solution. I'd love to hear from someone who really knows how sensors work about whether I'm just breathing fumes here.

[rvaubel]

Consider a piece of charcol. It is the closest common thing to "black" that we have in our everyday world. It is "black" because it reflects very little light of any frequency .

Consider a "black" dyed fabric. Often, such a dye will be very reflective in the infrared but not at all in the visible spectrum. We call it "black" because it looks black to us. But to an animal that sees in the infrared it may be white.

The M8's sensor is an "eye" that sees in both visible light our eyes see and the infrared light that we don't see. The colorimetric software that controls the display function simply interprets that infrared signal as the color purple (in this case) which is a false color, not actually the color of the infrared. Remember the infrared frquency has a color we can't see!

Thats why an object that has a carbon based pigment in it will stay "black" while an object with a dye based "black" will appear purple.
Ironically the M8 is actually doing a better job representing reality than a more heavily filtered camera is.

Both software filters and physical filters can theoritically strip the IR signal from the reality that the object is reflecting. There is a solution

Rex

[eronald]

Yes and no. A software solution depends on the characteristics of the Bayer filters - may or may not be possible, may or not work depending on the exact IR "colors" of the objects out there. My feeling is it will work for the near IR, not for the further stuff.

Originally Posted by rvaubel

Both software filters and physical filters can theoritically strip the IR signal from the reality that the object is reflecting. There is a solution

Rex

[Peter D Lux 2]

Originally Posted by eronald

Yes and no. A software solution depends on the characteristics of the Bayer filters - may or may not be possible, may or not work depending on the exact IR "colors" of the objects out there. My feeling is it will work for the near IR, not for the further stuff.

Interesting discussion guys,

Eronald: When you say near, do you mean:

- near in nanameters (to visible spectrum)
- fysically near the lens?

Thanks,

Peter

[blakley]

The more I think about this the more I think discussions to date have been too simplistic because they fail to differentiate between "object space", "input space", "output space", and "eyeball space".

When people say things like "As pointed out here by others, the problem isn't an overall purple cast, but that certain objects, especially some synthetic materials, are purple. The sensor sees purple. Software and firmware has no knowledge of the original subject's material and true color so it can't help out." (quoted from a Photo.net thread), I think they're just wrong. The object isn't "purple" - it reflects a whole bunch of wavelengths which my eye, in "eyeball space", interprets as purple. Other objects which reflect DIFFERENT bunches of wavelengths are also purple in "eyeball space".

But the sensor doesn't see in "eyeball space", it sees in "input space". The picture on Michael Reichmann's website (of the M8 sensor gamut) is a picture of "input space".

"input space" is mapped to various different "output spaces" by the camera's firmware; these "output spaces" include various RGB choices in DNG and JPEG. Each "output space" value lights up red, green, and blue pixels on a monitor, which results in emission of a bunch of wavelengths which I then see in "eyeball space".

Now, there are lots of possible combinations here, but let's just imagine two. Let's say I have a purple (eyeball space) object and a black (eyeball space) object which reflects a lot of IR.

It *might* be true that these two objects "look the same in input space" - i.e. that the sensor behind its Bayer array sees the same RGB values for both objects. But this might also NOT be true.

If it's true that the two objects look the same in "input space" then there is indeed nothing we can do to correct a magenta cast in the black (eyeball space) object without also affecting the color of the purple (eyeball space) object.

But if it's not true that the two objects look the same in "input space", we have a choice to make - and that choice is how to map the two different "input space" colors to "output space"; this is certainly true in at least some cases, because that's the definition of what we mean when we say that the sensor's gamut is wider than RBG.

The question is really how many black "eyeball space" objects reflect bundles of wavelengths which are the same as non-black "eyeball space" objects. I don't think it's easy to answer that question without doing a lot of testing of the sensor - but I'm willing to hold out hope that the number may turn out to be small, which would mean most color shift problems could be fixed with a profile adjustment. Time will tell.

[ChrisC]

Yet again I am at the cutting edge of my ignorance with this M8 roller coaster.

Are you suggesting [when RAW converting} that mapping only those colours existing in our visible spectrum into our archive colourspace would constitute an IR software filtration solution? If colourspace mapping can be that efficient would we need IR hardware filtration in front of the CCD?It would nopt help the Jpegs, but t's a fascinating thought, even if the next poster is about to shoot it down in flames.

Have you any idea how out of my depth I am?

If you do, keep it polite please.

Sincerely..................Chris

[markedavison]

The M8 is not a mysterious device. You can think of its color response as a machine which takes in a spectral power distribution (SPD) from a small portion of the scene (this is a function of wavelength that tells you the amount of power at each wavelength), and then produces device R, G, and B values. Each of these values is obtained by multiplying the incoming spectral power distribution by the spectral sensitivity of the corresponding channel, and then summing over all the frequencies for which the channel has a non-zero response. (Followed by scaling by a constant.) The resulting R, G, B values are the linear device R, G, B.

There is a central intuition here: cameras do not photograph colors, they photograph spectral power distributions. Two objects which appear to have the same color to the human eye may have different SPDs and yield different R, G, B outputs from the camera.

If two different spectral power distributions get mapped into the same R,G, and B values, there is no way to tell them apart by looking at the camera output. These two SPDs are said to be metameric pairs for the device.

A camera profile is a mathematical description of a function which takes a device R, G, B triple and maps it into a color space which describes human color response, such as CIE X,Y,Z space. (Note that CIE X,Y,Z space describes human color response by three particular spectral sensitivity functions, xbar, ybar and zbar.)

There are roughly two types of profiles.

The first type is a generic profile, intended for unrestricted use. You can show that it cannot be made to be accurate for all possible SPDs, unless each of the spectral sensitivity curves of the camera is an exact linear combination of the CIE spectral sensitivity functions. At best the results will be pleasing, but they will not be accurate for all input SPDs. (Technical note: if you examine a camera profile with a profile inspector you will discover that the look up table for the mapping contains output values for every possible combination of device R, G, and B. However, you can show mathematically that as the input SPD ranges over all possible spectra, the camera R, G, B triples lie inside a restricted gamut. Some of the LUT output values may indeed lie in the "impossible colors" of CIE X, Y, Z space, but this doesn't tell you if the range of values which would come from realizable device R, G, B triples ever hits an impossible color. Also, examining the range of the LUT output values doesn't tell you anything about the IR sensitivity of the camera--you need to see the spectral sensitivity curves of the camera.)

The second type of profile is intended for use on a restricted set of SPDs. For instance, if you are using the digital camera to copy color photographic prints of a given type, then the space of possible input SPDs is restricted to those which appear in the prints. Since there are only three dyes in the prints, this is going to be a much smaller space than the space of all possible SPDs seen in the world. If you profile a digital camera using an IT8 photographic target, this is the type of profile you are creating. A somewhat larger set of SPDs will be encountered if you use a color checker chart.

It is quite possible to correct IR casts by profiling if you are only using SPDs from a restricted space, there are no two SPDs in the profile test set which yield the same device R, G, B values, you only photograph scenes which are well represented by the set of test SPDs, and the spectral power distribution of the illuminating light is the same as the light used to create the profile.

Note: if you look at linear device R, G, B values it is straightforward to understand the effects of device sensitivity curves which are non-zero in the infra-red wavelengths. Suppose that we could find two filters, one which transmitted visible wavelengths perfectly, but blocked all IR, and another which did the reverse. Take a spectral power disrbution S. Then the camera response for S is simply the sum of the responses with the visual pass filter and the IR pass filter:

(R, G, B) = (Rvis, Gvis, Bvis) + (Rir, Gir, Bir).

I have been taking some pictures through the M8 with an infrared pass filer (a HOYA R72) and the results are quite interesting. You can see that under incandescent illumination, which is rich in infrared, a piece of black anodized aluminum (like a black Leica M lens) has a very sizable infrared response, which is high in red and blue. In the visible wavelengths, the piece is quite dark, and appears black, with no hue. Therefore the infrared contribution to the sum is big enough to push the hue to magenta.

The effect, however, is not limited to black synthetic fabrics and anodized aluminum. Under incandescent illumination even something as organic as an apple has a visible infrared response with the M8. It is small, but enough to change the hue just a little bit.

Rephotographing the apple with an IR cut filter (I used a Tiffen Standard Hot Mirror), produces a visible change in the hues of the apple, and gives them a much more natural appearance.

Skin tones of fair skinned people also show similar small but quite visible shifts.

The IR sensitivity of the M8 is not unique in the world of digital cameras. I have a Nikon D2h which displays similar IR responses.

There is a general misunderstanding among digital photographers that physical photographic filters are not necessary with digital because all filtration can be done after the fact. But that is simply not true, as the case of extended IR sensitivity proves.

The straightforward conclusion is: if you have an IR sensitive digital camera, and you want to create natural looking color photographs of the world without restricting the types of objects or the illumination, buy a good IR cut filter.

[fotografr]

"The straightforward conclusion is: if you have an IR sensitive digital camera, and you want to create natural looking color photographs of the world without restricting the types of objects or the illumination, buy a good IR cut filter."

Or, convince the manufacturer to place a good IR filter over the CCD.

[johnll]

Additionally (and this is not necessarily M8- or even Leica-related) I am planning to experiment with blue CC filters in tungsten light in digital capture, because this will give me a better balance (I think) between the RG and blue channels, so that I have relatively more information in the latter than I would have had without filtration. I'll do this some time in December because that's when I'll be able to collect the CC filters I've ordered.

[rosuna]

Originally Posted by fotografr

Or, convince the manufacturer to place a good IR filter over the CCD.

It is not a question of "good". It is a question of "thicker".

I don't want a thicker filter over the CCD because it has an impact on resolution and contrast.

It would be preferable to divide the task between a CCD filter (thiner than usual) and an external filter (softer than usual).

"No hay mal que por bien no venga".

Maybe the M8 with a IR filter over the lens is the best possible solution after all.

[guy_mancuso]

Here is my best guess at what Leica will do. Take the existing sensor that we have already than apply a coating that will block more IR than come up with a fix for cyan on wide angles with a firmware upgrade , but you will need coded lense to take advantage on the cyan corners and wide angles. that seems to me the only solutuion besides replacing the sensor. I'm a betting man
__

[blakley]

Take a spectral power distribution S. Then the camera response for S is simply the sum of the responses with the visual pass filter and the IR pass filter:

(R, G, B) = (Rvis, Gvis, Bvis) + (Rir, Gir, Bir).

Right. But the "(R, G, B)" on the left of the equation are scaled differently than the R, G, B in "output space" - in principle it's possible to have a range for each color of greater than 0,..,255 - because the sensor is capturing information from a wider gamut than the output device can display.

This is what would create the possibility that some correction can be done in the absence of an IR filter - because we can choose the mapping between "input space" (R, G, B) values and "output space" (R, G, B) values.

When people ask "what if there's an object in the picture which is the same color as the magenta-shifted black object?", they're asking a misleading question. Translating it into my terms above, what they're really asking is:

Imagine that you have two objects: B and P. B is black in "eyeball space". P is purple in "eyeball space".

What if B is the same color in "output space" as P is in "eyeball space"?

That's the wrong question, because the answer doesn't matter. What matters, from the point of view of ability to correct in software in the camera, is whether B is the same color as P in "input space" - which is the space in which the "(R, G, B)" on the left hand side of the formula above live in.

It's VERY hard for us input kibitzers to know whether there exist a lot of pairs of objects such that one looks black in "eyeball space" but the same color purple as the other in "input space", because we can't see "input space"!

We can see what color an object is in "object space" by pointing a spectrograph at it. We can see what color an object is in "output space" by using the Photoshop eydropper on an M8 image of the object on a monitor. We can see what color it is in "eyeball space" by pointing our eyes at it.

But we never see "input space" - we only see how "input space" is mapped onto "output space" by the M8's firmware; only the Leica and Kodak engineers can really see what objects look like in "input space", so only they can really tell how much of this problem can be fixed in firmware and how much will require other solutions (including IR filters).

I'm guessing we'll have to wait & see what they come up with.

[billh]

Originally Posted by blakley

[i]........

I'm guessing we'll have to wait & see what they come up with.

How long would you guess this might take them? I do hope they have done the same thought and actual experiments as those posted on this list.

Another question(s) I have is, can a coating be applied to the glass now covering the sensor which will be as effective as the (for example) B + W IR 486 cut filter? If so, will there be any image quality changes all or some of us would prefer not to have?

Was it Nikon that had this problem? How did they solve it, and what effects did the solution have on the images from this camera?

[Runkel]

Originally Posted by blakley

Take a spectral power distribution S. Then the camera response for S is simply the sum of the responses with the visual pass filter and the IR pass filter:

(R, G, B) = (Rvis, Gvis, Bvis) + (Rir, Gir, Bir).

Maybe it helps to plug in some numbers.

(20, 30, 40) = (15, 25, 35) + (5, 5, 5)

(20, 30, 40) = (10, 17, 15) + (10, 13, 25)

Etc.

It is easy to see that as long as there are non-zero inputs for Rir, Gir and/or Bir, any number of combinations will be mapped as the same (20, 30, 40) RGB output. This output then goes to the hypothetical "fixed" firmware. How is it possible for any non-magical firmware to tell whether a (20, 30, 40) input is coming from one or the other of the input combinations above or any number of others?

In order for the (20, 30, 40) RGB output to mean any one single thing, the equation has to be (20, 30, 40) = (20, 30, 40) + (0, 0, 0). The only way to get those zeros is by filtering out the infrared.

If the job could be done with firmware, then why couldn't firmware take the place of the visual pass filter too?

[sdai]

Originally Posted by billh

Was it Nikon that had this problem? How did they solve it, and what effects did the solution have on the images from this camera?

Hi, Bill ... the problem with the original D2H was not profound at all and to the best of my knowledge, Nikon has never bothered to deal with it. Personally, I've never found it a problem under my normal shooting conditions and have never used a hot mirror filter on mine so I can't comment on its effect.

[Jamie Roberts]

This is an interesting thread. I'm also completely out of my depth here.

But I'm inclined to believe that the IR sensitivity of this particular class of sensors, especially under tungsten light, is very well known to Kodak and Phase.

I suspect that Leica rushed the camera to market on the heels of good reviews, but Phase didn't quite understand that they were essentially dealing with a mini P30 back (or maybe they did understand exactly that--there's a conspiratorial thought for you--JUST KIDDING).

Anyway, the fact that a profile effectively fixes the problem (or at least puts the camera into "better than Canon skintones without the magenta blacks" range) seems to suggest that the original poster was completely correct:
the input color of the IR reflectivity is "imaginary" or outside any normal output.

Anyway, it's not "visible magenta", exactly--that's for sure; I've shot lots now with magenta and black in same shot and it works just fine.

I'd love to see a shot where it doesn't, but I haven't found it yet.

Edmund--have you found one?

Guy--I'm not a betting man. I can't see the cyan thing on non-coded lenses being acceptable either. We'll see what happens.

Some day there will be a supply of IR cut filters again and Ill be able to compare the SW with HW solution

[jrc]

I think it's possible that Leica will offer a range of solutions -- at-cost UV/IR cut filters, OR a recall with more IR filtering on the sensor, etc. Some people might actually prefer to keep the IR sensitivity (especially Leica photographers.)

Here's a question for the mathematicians among us. Would it be possible to examine the auto white balance and then, judging from the amount of red, blue and green light coming through the sensor, to make a *mathematical* best-guess solution to how much of the magenta response should be eliminated? I've been told that Canon does some pre-adjustment of the RAW signal (else how would they get that smooth watercolor patina?) so why couldn't some nice sophisticated equations be applied to cut an excess of magenta? Not all the magenta, so that magentas would stay magenta, but would limit *flares* in the red channel (or red/blue channels) that suggest IR interference? Would there be a characteristic signature of this kind of IR overflow?

JC

[blakley]

Matthew wrote

(20, 30, 40) = (15, 25, 35) + (5, 5, 5)

(20, 30, 40) = (10, 17, 15) + (10, 13, 25)

Etc.

It is easy to see that as long as there are non-zero inputs for Rir, Gir and/or Bir, any number of combinations will be mapped as the same (20, 30, 40) RGB output.

Indeed. Now let's try some other numbers.

"input space" (400, 30, 40) = (10, 30, 40) + (390, 0, 0)

Since the Rv can't be higher than 255, something weird is clearly going on.

How do we map "input space" (400, 30, 40) to "output space"? Well, we could decide that we have REALLY BRIGHT visible red light and a little IR, in which case we'll choose

"output space" (255, 30, 40)

Or we could decide that we have a dark subject of some "normal" color and a LOT of IR, in which case we'll choose something like

"output space" (35, 30, 40) - not perfectly neutral, but close.

This is the point I'm trying to make. You & I don't KNOW what the "input space" values are, and we don't know (yet) whether there's a mapping between "input space" and "output space" which produces the right colors most of the time.

[blakley]

By the way, Mark, that was an exceptionally useful post - thank you. I agree with you that for ideal reproduction of colors better IR filtration is the best solution. But in the absence of better IR filtration, your explanation seems to suggest that a modified curve which deemphasizes the red and blue channels in dark objects under tungsten lighting might go some distance toward more visually accurate color reproduction. True?

[markedavison]

Everyone,

I think our intuitions about infrared contamination may be based on different experience.
At
Infrared filtration examples

I have posted some example photographs to illustrate the effects of IR filtration on IR sensitive cameras. The scene was shot with incandescent illumination from ordinary lightbulbs. The camera white balances were set to 2800 K where possible. The Epson R-D1 was set to incandescent.

The first example is the D200, which is very insensitive to IR. The colors in the first D200 photograph are a very accurate rendition of the way the scene appears to my eye. Take special note of the maroon and green pile blankets, the black Leica M lens, and the black pile jacket at the bottom of the photograph. The second photograph shows the D200 with IR cut filtration (via a Tiffen standard hot mirror filter). There is hardly any visible change in the colors. The third photograph is with the D200 and the IR pass filter (a Hoya R72), taken at the same exposure as the first two photographs. There is no visble IR at all at this exposure.

The photographs continue in sequence for 3 more cameras: the Leica M8, the Epson R-D1 and the Nikon D2h. For each camera I show an image with no filtration, with IR cut, and IR pass, all at the same exposure. Note how much IR is recorded by the M8--it is the most IR sensitive of all the cameras. Note also how the IR contamination has completely bleached the green out of the green pile blanket, how the maroon blanket has shifted color, how there is a purple sheen on the barrel of the Leica lens, and how the black pile jacket has turned dark purple. The shot with IR cut filtration knocks down the purple sheen on the lens barrel, improves color saturation and contrast overall, but doesn't quite return the green pile blanket to the correct color. Note also that there was a glowing IR reflection from the "black" pile jacket on the bottom of the apple which is taken out by the IR filtration.

Similar comments apply to the Nikon D2h, but the infrared sensitivity is weaker and the corrections with the IR cut filter look better to my eye.

My point is that IR contamination is not something that only affects synthetic black objects and dark anodized aluminum--it contaminates practially all synthetic pile fabrics that I can find in my house. So you can't just hunt down dark purple things and change their color. (By the way, if you shoot social events and students in classrooms in Seattle in the winter, you are going to encounter a lot of pile jackets and incandescent light, so this is not some obscure rare combination, at least for my use.)

If someone can send me a Capture One profile that supposedly takes out the IR contamination I would be happy to try it on the original RAW files associated with these photographs.

Mark Davison