M8-why 10MB-vs-DMR 20MB

[miami91]

Advertisement (gone after registration)

What takes it so hard for Leica to add one more line into their spec. sheet or whatever ... just to say the damn camera has a color depth of 16-bit instead of letting folks scratching their head out there?

 

Umm, their spec sheet does say this, quite explicitly. See further up the thread (post #66), or d/l it yourself from Leica Camera AG - Home.

 

I think that's what makes this so disconcerting, is that everything we've seen/read up to this point would lead one to conclude 16-bit color depth, except for this (misleading?) EXIF data.

 

Jeff.

[sdai]

Umm, their spec sheet does say this, quite explicitly. See further up the thread (post #66), or d/l it yourself from Leica Camera AG - Home.

 

I think that's what makes this so disconcerting, is that everything we've seen/read up to this point would lead one to conclude 16-bit color depth, except for this (misleading?) EXIF data.

 

Jeff.

 

I actually have read that, but thanks for the pointer again, Jeff. It isn't that clear and seems to me very confusing too ... FWIW, it says "16-bit" color resolution ... correct me if I'm wrong but I always think that resolution is measured in the number of dots in one-inch line, also referred to as dots-per-inch (dpi). While in the DMR spec sheet, they did explicitly say that the color depth is 16-bit.

 

And a friendly reminder to those who don't want to read my posts ... please add me into your ignore list - it is a mouse click that easy.

[Jamie Roberts]

Umm, their spec sheet does say this, quite explicitly. See further up the thread (post #66), or d/l it yourself from Leica Camera AG - Home.

 

I think that's what makes this so disconcerting, is that everything we've seen/read up to this point would lead one to conclude 16-bit color depth, except for this (misleading?) EXIF data.

 

Jeff.

 

I think you're right Jeff. This is just confusing. So you have the spec. sheet implying one thing, and people with actual (though pre-release) files saying something else.

 

But just saying "oh these can't be 8 bit files" isn't an answer. As someone also buying M lenses and with actual money down for an M8 for my business, I really, really want to know what the RAW files will deliver.

 

So Simon (and I) are asking legitimate questions, and we're far from trolls. At least, I suppose I could be a troll with a R9/DMR and boatload of R glass

 

As for 8 bpp files, hey--I've seen some wonderful JPEGs--it takes a lot to "show" they're only 8 bit (since that's mainly what your monitor shows anyway!) with normal subject matter.

 

But I still want a true 16bpp RAW file, for a number of reasons. so I hope that's what they deliver.

 

I'll guess I'll just breathe a lot better once Guy gets word back from Leica or once Sean distributes a DNG or three

[ho_co]

Jamie--

Trolls are defined by behaviour, not by queries, I think.

 

I understand the question and your interest in accuracy. I've gone round and round with someone at Leica over how to interpret part of a list of lenses that can or can't be used with the M8. I see ambiguity, he doesn't.

 

So I'll just repeat that Leica's DMR is the only camera on the market with true 16-bit RAW, and they are not about to try to get away with less on the M8. I understand that you and Simon can continue to say you want proof.

 

As Sean said above, it's a red herring. And I think we'll soon have proof of that surmise when the cameras become available.

 

--HC

[plasticman]

The thread seems to be getting overheated - but i think some of the irritation stems from uncertainty: Leica should really clear-up some of the question marks asap imho.

 

Anyways, i still maintain that there is NO POSSIBLE WAY on earth that the M8 will produce only 8-bit RAW files. The whole thing strikes me as a possible misreading of a string of data. I have no experience with the applications that've been used here to extract that data - but the line is at best ambiguous:

 

"TIFF 320x240 320x240+0+0 DirectClass 8-bit"

 

Those dimensions "320x240+0+0" refer to thumbnail files, as i understand it - the two extra numbers being available to the programmer to modify for screen display. So it's entirely possible that the 8bit info refers also to the thumbnail produced with the RAW file.

[Kaj Jensen]

Take a look at the detailed information provided in the Leica Talk Forum:

 

-----http://forums.dpreview.com/forums/readflat.asp?forum=1038&thread=20479105

 

8bit RAW data may be a preliminary emergency solution. Leica may have severe problems in getting 16bit up and running on production models.

[scott kirkpatrick]

Advertisement (gone after registration)

An overheated thread is a natural result when none of us really knows that much about how a RAW file can be constructed under the .dng ground rules (I don't think they are definite enough to be called a "standard"). So instead we each argue that some position other than our own guess cannot possibly be true for some philosophical reason. I'm in the same boat -- I have an old copy of dcraw.c and have played with it, but haven't looked at the latest version, which might have Dave Coffin's comments in it, if he has broken the code already. But I do know something about how coding and compression work.

 

Let me summarize an issue raised about the M8's dng files and the facts that bear on them:

 

The DMR produced a RAW file of a constant 20 MB from a Kodak chip with 10 MegaPixels (MP), by the simple method of writing each pixel's value into a 16bit field and adding some metadata. The Olympus E-1 does something similar, writing each of its 5M pixels into a 16bit field to avoid having to devote any signal processing power to packing them denser. Although there was discussion that the Kodak chip used in the E-1 -- a predecessor of the DMR and M8 chips -- could support a 14bit AtoD, Olympus has never claimed that more than 12 bits per pixel were valid data. Olympus .orf RAW files are always the same size, regardless of the image content.

 

Canon performs what they describe as lossless compression of their RAW file data. I just looked at two files from a 20D, taken a minute apart so that the firmware is the same. For an 8.25 MP image, one is 6991 KB and the other 7395 KB. The second image had more details, because it was zoomed out a bit. Canon's algorithm is no mystery, because DCRAW (used in IrfanView) can open it, and here results in about 0.85 B per pixel.

 

Most lossless encoders analyze the raw data and build a decoder table on the fly which assigns the shortest symbols to the most frequently encountered data. Another trick is to replace repeated or missing information with a short summary code. I don't have a .CR2 file showing a black cat in a coalbin at midnight, but if I did, it would be very small indeed.

 

The M8 specs promise that the RAW files for its 10.3 MP will be 10.2 MB each. If each M8 .dng file is exactly the same size, regardless of its content, that doesn't look like a lossless encoding, but like a transformation, applied to each pixel, which maps the data into one Byte. (The difference between 10.3 MP and 10.2MB is because a MB is really 1024x1024 = 1,048,576 Bytes.) So are they? There are lossless encoding schemes which control their output to achieve constant total size, but these take a lot of computation and could produce a file smaller than 10.2 MB.

 

So it's reasonable to ask if all M8 .DNG files are the same size, one B per pixel, and if so, how are they encoded?

 

scott

[Jamie Roberts]

{snipped} If each M8 .dng file is exactly the same size, regardless of its content, that doesn't look like a lossless encoding, but like a transformation, applied to each pixel, which maps the data into one Byte. {snipped}...

So it's reasonable to ask if all M8 .DNG files are the same size, one B per pixel, and if so, how are they encoded?

 

scott

 

Scott, another "exactly" from me

 

Once we see the production RAW file sizes that will be a clue, unless Leica is doing something very strange. If they're exactly the same size each time, then I would suspect lossy compression.

 

The Canon "losslessly" compressed files also get larger with ISO increases, presumably because noise adds more data in the file (?)...

[reprobit]

no computer is able to write 12bit data, these guys need either 8 or 16 bit words.

no camera is able to send 16 bit signals, they deliver from 10 to 14 bit

 

these camera data are written into raw files with full information and stored.

if a camera has advanced processors, they are able to do a lossless compression.

 

this is the reason why some cameras have smaller raw files than others

 

in both cases no information is lost

 

 

bernd

[sdai]

Enough energy has been wasted and so far none of the ones who really knows and understands was talking ... let's hope it's just an error in the EXIF tag and will change in the product firmware - it's interesting to note that the cameras (well, at least one of them) shown at the Kina was already running firmware version 1.04 so people may get the real ones at version 1.10 or maybe 1.20?

 

For the record, here's the complete EXIF of the "Photokina Lady" DNG ... some may be interested in coming back to the discussion:

 

File Name : L9994925.dng

File Size : 10 MB

File Modification Date/Time : 2006:10:19 00:28:50

File Type : DNG

MIME Type : image/x-raw

Software : 1.04

Artist :

Subfile Type : Full-resolution Image

Image Width : 3920

Image Height : 2638

Bits Per Sample : 8

Compression : Uncompressed

Photometric Interpretation : Color Filter Array

Strip Offsets : (Binary data 1315 bytes, use -b option to extract)

Samples Per Pixel : 1

Rows Per Strip : 16

Strip Byte Counts : (Binary data 989 bytes, use -b option to extract)

X Resolution : 300

Y Resolution : 300

Planar Configuration : Chunky

Resolution Unit : inches

CFA Repeat Pattern Dim : 2 2

CFA Pattern 2 : 0 1 1 2

Linearization Table : (Binary data 1244 bytes, use -b option to extract)

White Level : 16383

Default Crop Origin : 2 2

Default Crop Size : 3916 2634

Bayer Green Split : 0

Anti Alias Strength : 1

Make : Leica Camera AG

Camera Model Name : M8 Digital Camera

Orientation : Rotate 270 CW

Date/Time Digitized : 2006:09:27 16:49:31-04:00

Flash Fired : False

Flash Return : No return detection

Flash Mode : Off

Flash Function : False

Flash Red Eye Mode : False

Modify Date : 2006:09:27 16:49:31-04:00

Creator Tool : 1.04

Rating : 0

Serial Number : 3100333

Version : 3.5

Raw File Name : L9994925.DNG

White Balance : As Shot

Temperature : 5100

Tint : -16

Auto Exposure : True

Exposure : -0.85

Auto Shadows : True

Shadows : 4

Auto Brightness : True

Brightness : 64

Auto Contrast : True

Contrast : 0

Saturation : 0

Sharpness : 25

Luminance Smoothing : 0

Color Noise Reduction : 25

Chromatic Aberration R : 0

Chromatic Aberration B : 0

Vignette Amount : 0

Shadow Tint : 0

Red Hue : 0

Red Saturation : 0

Green Hue : 0

Green Saturation : 0

Blue Hue : 0

Blue Saturation : 0

Tone Curve Name : Medium Contrast

Tone Curve : 0, 0, 32, 22, 64, 56, 128, 128, 192, 196, 255, 255

Camera Profile : Embedded

Has Settings : True

Has Crop : False

Copyright :

Exposure Time : 1/500

Exposure Program : Manual

ISO : 320

Exif Version : 0220

Create Date : 2006:09:27 16:49:31

Shutter Speed Value : 1/512

Exposure Compensation : 0

Max Aperture Value : 1.0

Metering Mode : Center-weighted average

Light Source : Unknown (0)

Flash : Off

Focal Length : 0.0mm

Warning : [minor] Possibly incorrect maker notes offsets (fix by 2314?)

File Source : Digital Camera

Scene Type : Directly photographed

Digital Zoom Ratio : 0

Focal Length In 35mm Format : 0

Scene Capture Type : Standard

Image Unique ID : 000000000000000000000000000001d5

Self Timer Mode : 0

Date/Time Original : 2006:09:27 16:49:31

Focal Plane X Resolution : 3729

Focal Plane Y Resolution : 3764

Focal Plane Resolution Unit : inches

TIFF-EP Standard ID : 0 0 0 1

DNG Version : 1 0 0 0

Unique Camera Model : M8 Digital Camera

Color Matrix 1 : 0.6863 -0.1407 -0.0775 -0.3086 1.139 0.1921 -0.0971 0.2791 0.6609

Color Matrix 2 : 0.6863 -0.1407 -0.0775 -0.3086 1.139 0.1921 -0.0971 0.2791 0.6609

Camera Calibration 1 : 1 0 0 0 1 0 0 0 1

Camera Calibration 2 : 1 0 0 0 1 0 0 0 1

As Shot Neutral : 0.4750638 1 0.7966159

Baseline Noise : 1

Baseline Sharpness : 1

Camera Serial Number : 3100333

Calibration Illuminant 1 : Unknown (36068)

Calibration Illuminant 2 : Unknown (38768)

Raw Data Unique ID : 3148EC60B7B5219BF48EA94E0E87DC9C

CFA Pattern : [Red,Green][Green,Blue]

Image Size : 3920x2638

Scale Factor To 35mm Equivalent : 1.3

Shutter Speed : 1/500

Circle Of Confusion : 0.022 mm

Focal Length : 0.0mm (35mm equivalent: 0.0mm)

[Jamie Roberts]

no computer is able to write 12bit data, these guys need either 8 or 16 bit words.

no camera is able to send 16 bit signals, they deliver from 10 to 14 bit

 

these camera data are written into raw files with full information and stored.

if a camera has advanced processors, they are able to do a lossless compression.

 

this is the reason why some cameras have smaller raw files than others

 

in both cases no information is lost

 

 

bernd

 

Bernd, this is maybe just a language thing, and I think I know (and agree) with what you're saying.

 

But there's a difference between 12bit per channel colour data (which the Canons, for example, absolutely do capture) and the way that's represented on a standard system (padded to 16bpp) *and* the way a program (8 / 16 / 64 bit on Windows) works with that data.

 

All we're questioning is if the colour depth / sampling is 8 or 16 bit. There are certainly 16bit per pixel images out there. How a computer handles them is something else again.

 

What we seem to be hearing here is that inside the DNG file the actual RAW data has been mapped to 8 bpp. This could be an EXIF mistake, it could be a funky provisional thing Leica did for testing, or it could be somethign else entirely.

 

That has nothging to do with how computers handle the data.

 

And yes, if the compression is lossless, then no data is lost

[Andrej Kolev]

Hi All. John Bean gave me a link to this forum in dpreview thread. I see a big discussion here, so there is my comments.

 

Someone has posted thoughts that there is some hardware limitations, I dont think so. Maybe it's a simply test Leica config, but why bother with mapping of data instead of simply writing of 16 bit data.

 

It can not be a tag error because I've successfully loadded and interpolated D8 image according to that scheme.

 

Processing pipeline of M8 from my point of view may be a next:

 

CCD ->Analog to Digital Converter (ADC) 14 bit -> Camera processor applying nonlinear law using so called Lookup Table (LUT - very fast operation) -> writing to DNG. After back-decoding by RAW software (using another LUT attached to DNG file with back transformation) output looks like usual 12-16 bit output.

 

Output image from many RAW converters may be look 16 bit because most of them are scaling any bit-depth to 16 bit for uniform processing.

 

You can check M8 DNG if you want by yourself using some HEX viewer. The M8 DNG with a girl from photokina that I've used was posted in Leica thread at DPreview. Camera firmware is 1.04 - this is useful for comparison with more recent camera versions.

 

Based on PDFs with EXIF and DNG specification (you can download it from adobe site if you want):

 

Dont confuse next values with tags relating to thumbnail data:

 

Exif Tag Bit per Pixel: 0x0102 - file has II signature so you need to reverse bytes - 02 01 next 03 (data type Short) next bytes 01 - on number next 4 bytes of data 08 - 8 bits per pixel

 

Exif Compression Scheme tag 0x0103 = 0301 (II) , data type 03 (Short), 01 (One number) , data = 01 - ( Look to DNG Spec - possible values:

 

Value = 1: Uncompressed data.

 

Value = 7: JPEG compressed data, either baseline DCT JPEG, or lossless JPEG compression.)

 

Also is present Lianerization Table tag 50712 in decimal notation = 0x 18 C6 (II) which contain offset to a lianerization table, consisting of 256 elements (as a number of different luminance levels encodend by one byte=8bit of data, the maximum output value from with table is equivalent to 2^14, so after lianerization maximum value will be 2^14).

 

Also DNG spec has a description of lianerization table:

 

"LinearizationTable describes a lookup table that maps stored values into linear values. This tag is typically used to increase compression ratios by storing the raw data in a non-linear, more visually uniform space with fewer total encoding levels.

 

If SamplesPerPixel is not equal to one, this single table applies to all the samples for each pixel."

 

It is clear that some loss of information is present, because you have only a 256 levels of data at range of 2^14 instead of 2^14. So there is the only one question - is that loss of inormation significant or not. Usually it is considered that in case of linear coding to smootly shade from black to white you need about 9,900 codes, or about 14 bits per component, in case of nonlinear coding (according to Rec. 709) you need 460 levels (9 bits). But it doesnt take in account that photographer will apply gamma/curve corrections to linear image to emphasize some tonal range, so there is a need in more bits. And even Rec.709 demand 460 levels and M8 gives only 256.

 

The only reason why I've started the discussion on DPReview is that maybe it is interesting for a people at forum and for possible users of that camera to know some properties of M8 RAW data. But I'm afraid my post may confuse some people, who may mix up concepts of linear/nonlinear coding and bit capacity in that cases. And it is not a final version of camera (but why firmware has "finally looking" 1.04 ver. number?) and it is unknown will be a loss of information significant or not.

 

Anyway, I think possible users of this camera (which is not a cheap instrument) need to know what they are

getting. The best way for Leica, if they dont want to use lossless compression, I suppose, is to make a selection option (just like somebody said is realized in Nikon cameras) - to write 16 bits of data or write 8 bit nonlinear data. Will wait for final camera version.

 

Regards, Andrej Kolev , kolevraw.com

[scott kirkpatrick]

Andrej, if your observations --

the data in the M8 DNG file makes sense when interpreted 8 bits at a time

there is a lookup table which maps these 8 bits into 14 bits of valid signal per pixel, which is the linearized data.

 

remain correct when production firmware files are accessible,then the M8 is indeed compressing each pixel into 8 bits and the files will have to be treated a bit carefully when trying to remap their tone curves for whatever purpose.

 

The problem that occurs when you do this is a histogram that looks like a comb -- banding. Dithering the pixel data (adding some random low order bits) when it is expanded into the 16 bit space for editing in C1 or ACR or whatever might be a reasonable cure for this, and it is possible that this is already quietly being done.

 

Is the lookup table different in different files, or always the same? I know you can't answer this with one premature sample, but soon...

 

scott

[scott kirkpatrick]

The Canon "losslessly" compressed files also get larger with ISO increases, presumably because noise adds more data in the file (?)...

 

That's true in general. Try this experiment -- make several versions of JPEG output from the same raw file in C1, increasing the sharpening or decreasing the noise filtering. Then look at the size of the resulting files. Noise and sharpening lead to significantly bigger files. Another experiment with a Canon is to shoot the same scene at a range of ISOs and plot the RAW file sizes. I bet they increase steadily, but the increase tapers off at high ISO because of internal noise filtering before the RAW is written. But I don't have the camera to try this with.

 

scott

[sdai]

I bet they increase steadily, but the increase tapers off at high ISO because of internal noise filtering before the RAW is written.

 

That is correct, Scott. Canon has indicated at many occasions that their NR circuitry is pre-built onboard the CMOS sensor. So noise filtering is done before RAW is written.

[Andrej Kolev]

If someone is interested, this is how a curve look. I suppose it is the same in all files, you can see it from the plot:

http://www.kolevraw.com/img/m8lut.png

[gravastar]

If someone is interested, this is how a curve look. I suppose it is the same in all files, you can see it from the plot:

http://www.kolevraw.com/img/m8lut.png

 

So it's heavily biased towards more gradation steps in shadow areas compared to highlights. I guess what's important is does this give rise to visible banding in highlights if you carry out reasonable manipulation of the image and is there a noticable change in overall image quality compared to an unmapped image.

 

Bob.

[fotografr]

Personally, I'm not one bit (pardon the pun) worried about this issue of 16 bit vs 8 bit. For 99% of digital photo work, the 8 bit depth is absolutely fine. Unless you are doing fine art reproductions, working in the Pro Photo RGB color space and printing on high end printers, the only thing working with 16 bit files will do is slow down your workflow and your computer. Otherwise, you will see no difference whatsoever. Just my 2 cents worth.

[plasticman]

For 99% of digital photo work, the 8 bit depth is absolutely fine. Unless you are doing fine art reproductions, working in the Pro Photo RGB color space and printing on high end printers, the only thing working with 16 bit files will do is slow down your workflow and your computer. Otherwise, you will see no difference whatsoever.

 

I'd have to respectfully disagree with this statement. If the camera has any pretensions to being a pro-level tool, then its output has to be capable of withstanding the processing that's often required to make a magazine spread or even an online ad look as great as possible.

8 bits just isn't enough as a starting-point for that type of post-processing.

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[scott kirkpatrick]

If someone is interested, this is how a curve look. I suppose it is the same in all files, you can see it from the plot:

http://www.kolevraw.com/img/m8lut.png

 

The expression in the plot must be the same for all files, because it is so simple. This looks like an optimization for speed (doubling the number of pictures that fit into the RAW buffer) and hardware simplicity.

 

scott