reading M10-D RAW/DNG files with linux/python/gimp

[Jul]

Advertisement (gone after registration)

Just found this thread and thought I could chime in.

I use just one computer and OS for work and anything else,

to make my life easier at work, it runs OpenBSD.

Still with the M9 but most I write here should apply to the M10 anyway.

 

Didn't do anything special with the SD card, it might already have been in FAT32,

just shot with it and stuck it into the SD card slot of the laptop.

hotplugd(8) cares for the import of data to a memory based file system (maybe not relevant anymore

with the speed of today SSDs but it made a difference with spinning disks)

dcraw or its unmaintained floating point fork cares for demosaicing the DNG into a white balanced

16 bit linear gamma tiff with a so called "Raw color" (unique to each camera model) colorspace

as asked and will do nothing more. 

graphicsmagick will assign an ICC profile and further processing such as lightness, saturation, gamma,

contrast, etc… as needed or wished.

The dcraw/graphicsmagick part is just a one-liner thanks to unix pipe and can be recalled

with a few keystroke thanks to the shell history. Compression and resizing to fit forum requirement

is part of one of these one-liners. The only button I click during the whole process is on the web browser

when I want to upload the picture.

Who said lazy? 

 

 

[fenykepesz]

...not to forget to mention that Slackware is pretty BSD by its very nature, and unix that way.

thanks for the demosaicing links.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[budjames]

Why not set up you PC for dual boot - Linux and Windows? That way you can run mainstream photo apps on Windows.

Regards,
Bud James

Please check out my fine art and travel photography at www.budjames.photography or on Instagram at www.instagram.com/budjamesphoto.

[fenykepesz]

for the same reasons as Jul : "I use just one computer and OS for work and anything else, to make my life easier at work, it runs..."

[indergaard]

As much as I love Linux, BSD (OpenBSD particularly), Unix and Python in general (it’s what I base most of my income generating work on), it’s sad that you guys choose to convert the raw files to tiff before applying any adjustments to shadows or highlights, especially considering that the M10 files can recover tremendous amounts of shadow detail. 

Converting to tiff before adjusting shadows or highlights will result in a huge loss of available dynamic range and especially shadow detail (data).

Photography is a visual medium and the command line might not be the best place to process a visual medium. There is a reason why really old film scanners still provided visual previews of scans. The guys at the development lab did in fact make adjustments prior to scanning. The same was also done in the darkroom. Very few processed with blindfolds on in the darkroom. Which the command line is the equivalent of doing. 

The best part of Emacs is definitely this easter egg: emacs -batch -l dunnet

Its the only thing I use emacs for. Otherwise I prefer nano these days

Edited July 20, 2019 by indergaard

[fenykepesz]

this whole field of RAW/DNG conversion is new-land to me.  i am  therefore entirely open to suggestions.  as you, indergaard, seem to live in a similar computing environment i am curious and keen to hear your advice regarding linux compatible solutions.  i just thought that those 14bit Leica DNG files are fully represented in the 16bit TIFFs and therefore could the subsequent adjustments be done in the TIFF space.  i am actually already experiencing quite some difficulty (in Gimp) correcting my flat and dark looking files, and and was wondering what color correction scheme i could apply to and perhaps extract from the DNGs.  if possible i would prefer to do the data-prep step in python and thereafter switch to the next phase using Gimp for data processing.  i vaguely remember reading years ago doing raw conversion in Gimp.  i guess i have to go behind the books = rtfm !

regarding scanning film (and darkroom), i actually scanned over 5 years all my negatives, mainly b&w which is obviously simpler, but then also color negatives and diapositives.  so, at times i adjusted my settings prior to scanning where it deemed worth the time.  in that sense i fully agree with your comment.

 

[fenykepesz]

Advertisement (gone after registration)

ok, i am an ox - there are many manuals & instructions for opening RAW/DNG files with Gimp & Co.  the market changed since i last looked into this years ago, but now with v2.10 things improved a lot.  so, i  have to adjust a bit my 'workflow'.  more about this another time...

[Jul]

2 hours ago, indergaard said:

As much as I love Linux, BSD (OpenBSD particularly), Unix and Python in general (it’s what I base most of my income generating work on), it’s sad that you guys choose to convert the raw files to tiff before applying any adjustments to shadows or highlights, especially considering that the M10 files can recover tremendous amounts of shadow detail. 

Converting to tiff before adjusting shadows or highlights will result in a huge loss of available dynamic range and especially shadow detail (data).

Photography is a visual medium and the command line might not be the best place to process a visual medium. There is a reason why really old film scanners still provided visual previews of scans. The guys at the development lab did in fact make adjustments prior to scanning. The same was also done in the darkroom. Very few processed with blindfolds on in the darkroom. Which the command line is the equivalent of doing. 

I won't comment on the "huge loss" as I'm still amazed at how much one can, even from the M9 which is not famous for high dynamic range, recover from the shadows even with the method I wrote in an above post. I stick with base iso, that's why the raw converter picks up a low black point which might help in this regard. If your method works even better I'm happy for you. I won't talk either about adjusting highlights after the tiff conversion as I never do it and tend to avoid blowing them at exposure as much as I can.

Regarding the command line part, you are aware that it is just an input method as is a GUI? Maybe are you confusing the command line in a graphical session with the text only console session? Using command line input does not prevent one from seeing the resulting picture at the same time in an image viewing utility, I'm not sure where the blindfolds are coming from.
 

[phib]

I can recommend rawtherapee as a "all in one" solution. Works under all operating systems (there even is an OpenBSD port). Another open alternative would be darktable.

Edited July 20, 2019 by phib

[indergaard]

The dng file is pure raw data from the sensor. Any conversion to tiff, regardless of settings, will result in a noticeable loss in data. Linear conversions can probably keep most of the data from the M9, but the M10 has a whole lot more shadow detail that can be recovered. But a linear approach is probably the best way to convert to tiff.

To get optimum dng > tiff conversions, manual labor is required. Copy the files of the card, open them in Darktable or Rawtherapee, adjust and recover the data you would want to preserve, then export to tiff, and then proceed with the rest of the routine. The darktable or rawtherapee stage cannot be automated, because each image is unique, and the data recovery needed will differ from image to image, depending on the scene, light, exposure, etc.

A dng > tiff file conversion will only result in the tiff file keeping what is "visible", but not what is -5 stops in the shadow areas of an image. In my opinion, a tiff file is the final export/output archival medium AFTER adjustments has been made, and the image has been finalized. Then base future prints, jpeg exports of various sizes, etc. on the tiff itself, but never alter the archival file. Or just keep the raw / dng file as the archival file, and apply non-destructive processing in darktable or rawtherapee.

[pico]

1 hour ago, indergaard said:

The dng file is pure raw data from the sensor. Any conversion to tiff, regardless of settings, will result in a noticeable loss in data.

Yea, fer sure. If ya can't visualize pure raw binary yer a loser.

(I call BS, indergaard)

 

 

[indergaard]

2 minutes ago, pico said:

Yea, fer sure. If ya can't visualize pure raw binary yer a loser.

(I call BS, indergaard)

 

 

You can call BS as much as you want, Pico. But that doesn’t change the fact that you are wrong. A raw-to-tiff conversion is a destructive process. Just like raw to jpeg is. The tiff format might offer lossless compression, but that is a totally different matter, that doesn’t apply for this scenario. 

[Jul]

11 hours ago, indergaard said:

The dng file is pure raw data from the sensor. Any conversion to tiff, regardless of settings, will result in a noticeable loss in data. Linear conversions can probably keep most of the data from the M9, but the M10 has a whole lot more shadow detail that can be recovered. But a linear approach is probably the best way to convert to tiff.

To get optimum dng > tiff conversions, manual labor is required. Copy the files of the card, open them in Darktable or Rawtherapee, adjust and recover the data you would want to preserve, then export to tiff, and then proceed with the rest of the routine. The darktable or rawtherapee stage cannot be automated, because each image is unique, and the data recovery needed will differ from image to image, depending on the scene, light, exposure, etc.

Well, I think we have to agree on the fact that we disagree, the possible quantity of detail to be recovered does not change the method. Just for fun I tried mine with a M10 sample DNG shot against the light from the Leica website and I got perfectly detailed highlights and shadows. On the "manual labor is required" assertion, as long as lightning condition and exposure are consistent on a session, the only uniqueness on a picture is the content and much of the processing routine can be automated, maybe that is the reason why raw converters developers implement a batch mode functionality in their product?

On a WYSIWYG workflow, you are working on interpolated data as soon as you can see your photo on the screen. For you to see the photo correctly, at least some gamma correction has already been applied and most raw converters default to some other transformation as well. Do you believe that these interpolated and gamma corrected data are still "pure raw data from the sensor"? Even if the software did the whole processing one more time on the unaltered raw data before exporting to tiff, it doesn't change the fact that you were visually working on already altered data. My point is that no matter what, we are all working on data that may not be exactly what the sensor recorded but are still more than adequate to yield fine results and hair splitting won't change that.

[fenykepesz]

many thanks for all your inputs and at times lively discussions - i learned quite a bit in this forum.

i think i got this RAW/DNG workflow business worked out for my Linux environment where i through my Python based photo manager program (fm.py) simply call RawTherapee and Gimp as needed.  for me, the main learning curve - coming from film, the M10-D is in fact the first digital camera i ever bought - was in understanding that a so called photo raw data converter does a different job as compared to a picture editor.  having worked in image processing for decades, it was for me hard to imagine the difference between a DNG and a TIFF file for example : data is data, 16(14)bit RGB is still 16bit RGB, right ?  i understand that re-scaling of a vector can degrade information content.

i still assume that converting RAW to a TIFF-A file with rawpy without any data alteration, then using RawTherapee (or LR) for the raw-adjustments in a TIFF-B file, and lastly doing the picture editing with Gimp (or PS) in a TIFF-C file can lead to the same result than doing it the usual way raw->tiff->tiff with LR & PS.  there is no magic about these raw converters, right ?  to me they just look like specialized picture editors.  but please correct me if i am wrong here.

i still don't know how much of the lens distortion, aberration and vignetting stuff gets in the M10 camera itself corrected at time of image capture... ?

[01maciel]

Having used Linux since an early pre-alpha 1.0 kernel version I have no doubt the rough and inexpedient time of Linux is over. Once the desktop is installed and configured its pretty similar to Windows or Apple OS. If you like the concept of UNIX you will probably make use of the fabulous OS underneath the Desktop-GUI. 

The ideal workflow to develop DNG files depends on your needs and preferences, sure. I do the entire post processing in RT before I post a photo in the LUF. That's all I do as I don't like to spend too much time editing photo to my taste. In my opinion there is no need to convert a DNG into TIFF first as DNG is the raw file. RT comes with a decent GUI but it's not that easy. You probably  have to RTFM. If you like the command line interface - here you go: there is comman-line app called rawtherapee-cli with loads of parameter to play with. The combination with imagemagick opens up a whole new world. 

$ rawtherapee-cli -help is your friend:

Options:
 rawtherapee-cli[-o <output>|-O <output>] [-q] [-a] [-s|-S] [-p <one.pp3> [-p <two.pp3> ...] ] [-d
] [ -j[1-100] -js<1-3> | -t[z] -b<8|16|16f|32> | -n -b<8|16> ] [-Y] [-f] -c <input>

 -c <files>       Specify one or more input files or folders.
                  When specifying folders, Rawtherapee will look for image file types which comply
                  with the selected extensions (see also '-a').
                  -c must be the last option.
 -o <file>|<dir>  Set output file or folder.
                  Saves output file alongside input file if -o is not specified.
 -O <file>|<dir>  Set output file or folder and copy pp3 file into it.
                  Saves output file alongside input file if -O is not specified.
 -q               Quick-start mode. Does not load cached files to speedup start time.
 -a               Process all supported image file types when specifying a folder, even those
                  not currently selected in Preferences > File Browser > Parsed Extensions.
 -s               Use the existing sidecar file to build the processing parameters,
                  e.g. for photo.raw there should be a photo.raw.pp3 file in the same folder.
                  If the sidecar file does not exist, neutral values will be used.
 -S               Like -s but skip if the sidecar file does not exist.
 -p <file.pp3>    Specify processing profile to be used for all conversions.
                  You can specify as many sets of "-p <file.pp3>" options as you like,
                  each will be built on top of the previous one, as explained below.
 -d               Use the default raw or non-raw processing profile as set in
                  Preferences > Image Processing > Default Processing Profile
 -j[1-100]        Specify output to be JPEG (default, if -t and -n are not set).
                  Optionally, specify compression 1-100 (default value: 92).
 -js<1-3>         Specify the JPEG chroma subsampling parameter, where:
                  1 = Best compression:   2x2, 1x1, 1x1 (4:2:0)
                      Chroma halved vertically and horizontally.
                  2 = Balanced (default): 2x1, 1x1, 1x1 (4:2:2)
                      Chroma halved horizontally.
                  3 = Best quality:       1x1, 1x1, 1x1 (4:4:4)
                      No chroma subsampling.
 -b<8|16|16f|32>  Specify bit depth per channel.
                  8   = 8-bit integer.  Applies to JPEG, PNG and TIFF. Default for JPEG and PNG.
                  16  = 16-bit integer. Applies to TIFF and PNG. Default for TIFF.
                  16f = 16-bit float.   Applies to TIFF.
                  32  = 32-bit float.   Applies to TIFF.
 -t[z]            Specify output to be TIFF.
                  Uncompressed by default, or deflate compression with 'z'.
 -n               Specify output to be compressed PNG.
                  Compression is hard-coded to PNG_FILTER_PAETH, Z_RLE.
 -Y               Overwrite output if present.
 -f               Use the custom fast-export processing pipeline.
 

btw: RT uses a patched version of dcraw code to read and parse raw formats. Lensfun will do the lens correction. Unfortunately, the support for M-Lenses could be much better.

 

[fenykepesz]

thanks, 01maciel, for your input.

in this case i like to ask you if you can explain why there is a substantial difference in image outcome when using RT (without any adjustments) as compared to RT-cli ?  my M10-D DNG files look always much more pleasant and more 'correct' to my eye when processed with RT.  i speak hear about TIFF (or JPEG) files converted from DNG with RT or with RT-cli.

btw, regarding pre-alpha 1.0, are you a slacker as well ?  i started using Slackware in the fall 1993 with version 1.00 and dozens of floppies...

[01maciel]

vor 34 Minuten schrieb fenykepesz:

there is a substantial difference in image outcome

no idea. there shoudn't be any differences as the cli and GUI use the same code base. The GUI  uses mainly slider,s curves, the cli values. You probably compare apples with oranges.

In the beginning I was a SuSE Linux user. Tried different distributions. Debian, Ubuntu, etc. In the meantime Fedora does a very good job.

[Jul]

3 hours ago, 01maciel said:

there shoudn't be any differences as the cli and GUI use the same code base.

Using the same code base does not mean using the same default settings.

It would show a lack of consistency if the default settings were not the same though.

[frank14542]

For me something like this works fine on Linux (Ubuntu/LinuxMint):

    def convertimage(self, dngfile):
        raw = rawpy.imread(dngfile)
        rgb = raw.postprocess()
        imageio.imsave('{}/{}.tiff'.format(self.outputverz, dngfile.split('.')[0]), rgb)
        return

and

    def exifwrite(self, dngfile):
        os.system('exiftool -TagsFromFile {}/{} --Orientation {}/{}'.format(self.inputverz, dngfile, self.outputverz, dngfile.split('.')[0] + '.tiff' ))

and

    def openmirage(self):
        os.system('mirage {}'.format(self.outputverz))
        return

It is very similar to what Camera RAW, and Bridge do. I use exiftool instead of exifread, because it seems to be more powerful, but this is more a question of taste. Rawfile-adjustments with Darktable or RawTherapee are available, although I prefer making adjustements on  tiff-files. Unfortunately image-processing on GIMP is not yet compatible with Photoshop, but as with GIMP 3.2 adjustment-layers will be integrated. ...

Regards

Frank

 

[Advertisement]

Hallo,

Look here: Leica M

[fenykepesz]

dear Frank, thank you for your informative reply.  first i also used rawpy but then i switched to RawTherapee & Co :

 

subprocess.call([ me.RawEditorCli, '-q', '-o', PathImgJpeg, '-c',  PathImg])
im = Image.open( PathImgJpeg)
if im.mode != 'RGB':
    im = im.convert( 'RGB')
f       = open( PathImg, 'rb')
tags    = exifread.process_file( f)

 

i actually don't understand why this raw2tiff/jpeg conversion cannot be done directly in Gimp/Photoshop or similar image editing software.  i mean, a 16bit DNG matrix is not different from a 16bit TIFF matrix.  what's the big fuss about this RAW conversion business ?  there is no magic involved here.  i understand that there are these profile curves, very helpful indeed - but why does this preprocessing step need to be separate ?  can't it just be incorporated in the regular processing flow inside our favoured image editor ?

anyway, i am more than happy with my M10D and my current image processing workflow using lnx/python3/RT/Gimp.  only the b&w conversion could be better, the current c2g algorithm has its flaws, and i hope the next version of Gegl/Gimp will have these flaw ironed out.