Does LR alter your referenced raw files

[glenerrolrd]

Advertisement (gone after registration)

Having asked this question of adobe and on three forums... seemed reasonable to discuss with other M8/M9 users.

 

I have always operated with the understanding that ..if you referenced your raw files without moving ..they would be unaltered in anyway. I shoot raw and create Leica generated .DNG files. My import to LR2 references the files without moving.

 

In working with Lr3 I needed to create some test files...so I found a half dozen selects and proceeded to export the 6 files from LR2 . I specified "original" . This is a workflow that some us to create a folder to rework in say Capture One. Sort of an exception process used for a limited number of worthy files.

 

When I imported the files (again referencing them without moving) into Lr3 ..I was surprised to see that all my develop settings (from Lr2) were retained and that the process version came across as Process 1(the Lr2 convertor). This implies that the DNG I assumed was unaltered ,in fact included the Lr2 processing). (I do understand how this could be important in migrating form Lr2 to Lr3.....but I wanted unaltered files).

 

Adobe responded by indicating that my original DNG s were infact converted by Lr2 and you could not go back to the original raw file. I can work around Lr2 s export outputting a converted file but if LR changed my original dng ..thats a big problem.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[sandymc]

Having asked this question of adobe and on three forums... seemed reasonable to discuss with other M8/M9 users.

 

I have always operated with the understanding that ..if you referenced your raw files without moving ..they would be unaltered in anyway. I shoot raw and create Leica generated .DNG files. My import to LR2 references the files without moving.

 

In working with Lr3 I needed to create some test files...so I found a half dozen selects and proceeded to export the 6 files from LR2 . I specified "original" . This is a workflow that some us to create a folder to rework in say Capture One. Sort of an exception process used for a limited number of worthy files.

 

When I imported the files (again referencing them without moving) into Lr3 ..I was surprised to see that all my develop settings (from Lr2) were retained and that the process version came across as Process 1(the Lr2 convertor). This implies that the DNG I assumed was unaltered ,in fact included the Lr2 processing). (I do understand how this could be important in migrating form Lr2 to Lr3.....but I wanted unaltered files).

 

Adobe responded by indicating that my original DNG s were infact converted by Lr2 and you could not go back to the original raw file. I can work around Lr2 s export outputting a converted file but if LR changed my original dng ..thats a big problem.

 

Yes, by default LR adds metadata to your DNGs. For "normal" raw files, e.g., Nikon or Canon, LR stores the development setting in a separate "sidecar" file. However for DNGs Adobe stores the same information within the DNG file itself.

 

Sandy

[ho_co]

So if I understand--

1) If I present LR with a DNG and tweak it in LR,

there's no "reset" switch, no way to go back to the camera original (from M8, say) except from an archived copy?

 

But

2) If I present LR with an RWL or NEF file and tweak it in LR,

my changes are stored only in a sidecar file and the camera original is unchanged?

 

 

At first blush, that sounds potentially nasty. Why would I want this two-sided file handling? Assuming my workflow required that I re-access unchanged DNGs, is there an easy way of handling this?

 

I know that lightroom will on request produce compressed DNGs. Is there a way to separate my LR-tweaked DNGs from my M8-original DNGs?

 

Or is it that once I know what I'm doing, I won't care?

 

(Obviously, I'm just beginning to start playing with Lightroom, so my apology for any signs of paranoia here. )

[adan]

(It should be noted that, unless you are a software engineer, you never really see a RAW image just as it was from the camera. All RAW developing programs have to apply SOME kind of default settings just to make the pixels visible as a full color image for you to work on. Default WB, default sharpening (even if it is zero), default exposure and brightness and all the rest. An unprocessed RAW file is basically a B&W TIFF with a visible checkerboard revealing the effects of the Bayer filters.)

 

I think the deal is this (coming from Adobe Camera Raw, which is basically the same processing engine in a different user interface (except that ACR won't catch up to LR3 until its next iteration):

 

The actual per-pixel data is not changed, but parameters to be applied to those pixels (such as WB, sharpening, noise reduction, exposure, saturation, calibration, lens corrections) are changed, and saved either as metadata in DNG files, or as the sidecar files (".xmp") for non-dng RAW formats. Saving the parameters alongside the pixels means you don't have to start from scratch each time your reopen one of your dngs, so it is not a bad thing.

 

With a non-.dng file, you could just find and trash the .xmp file and LR/ACR would have to start over with the data as straight from the camera - but still applying the program's default settings.

 

In .dngs, resetting all the parameters on every tab to either "zero" (sharpening, lens corrections, saturation, vibrance, lens corrections, etc.) or the Adobe factory defaults (brightness, contrast) is about as close as you can get to "rebooting" the image as it came from the camera.

 

Again, it is not the pixel data that is changed - it is the instructions or notes that the program makes to itself on how to display and output the pixels as a full RGB image that get changed and saved.

 

There may be one exception to that - I notice some raw developing programs now offer things like dust removal, and while I guess that could also be stored as side data ("when opening this file each time, replace pixels 2,010,246 through 2,042,759 with surrounding colors"), it seems like they may instead be altering actual pixel data and overwriting it. I just don't know.

[glenerrolrd]

Two issues that bother me about what appears to be the process:

 

1. I have always operated as if original raw file would be unaltered in any way by LR. That the original capture and all backups/copies would be identical.

 

2. I have conflicting information from adobe about the raw conversion itself. One post indicated that LR2 would convert the original DNG and others indicated that only the addition of the sidecar would occur.

 

Back to Adobe.

[adan]

Well, there is one way to test it in a rough way. Shoot a .dng picture, copy the file to your computer, open it in whatever RAW program you want, and hit "open" or "save" depending on workflow.

 

Then change the file number of that .dng original to something that won't overwrite the file still on your SD card or reset your camera's counter (L10000001.dng would be good unless you JUST GOT your M9), copy the renumbered file back to the SD card in the right folder, put the card in the camera, and hit "play". You should be able to see the original untouched-by-Adobe picture you shot next to the resaved version and see if there are any differences in WB or other things.

 

But you're right - there is not a cut-and-dried answer - yet.

[glenerrolrd]

Advertisement (gone after registration)

Yes, by default LR adds metadata to your DNGs. For "normal" raw files, e.g., Nikon or Canon, LR stores the development setting in a separate "sidecar" file. However for DNGs Adobe stores the same information within the DNG file itself.

 

Sandy

 

sandy

 

The meta data and develop settings don t seem to be much of a problem. If I take the image into C1 they appear to be ignored. Into LR3 I can reset to the original.

 

I am much more concerned about the raw conversion itself. Has LR altered the core DNG ( I mean aside from the sidecar information).

 

The assumption being that raw processors will continue to get better and I may want to go back to the original raw file.

[stunsworth]

So if I understand--

1) If I present LR with a DNG and tweak it in LR,

there's no "reset" switch, no way to go back to the camera original (from M8, say) except from an archived copy?

 

That's not the case. You can revert to the original of an M8 file in the develop module by either pressing the 'Reset' button, or going to the history panel and clicking on the 'Import' history state.

[Shootist]

Not sure if this is the same thing but if I open a Leica DNG in ACR and make adjustments to it then try to open that same image in PO C1 4 the DNG file is no longer seens as a Leica M8 file. If I then try to apply any of the C1 M8 profiles to that DNG file the outcome is hideous.

 

I would guess this happen because ACR, or LR, adds the adjustments to the original DNG file as apposed to writing a sidecar file as it does to other RAW file formats.

 

Just did a test where I made some adjustments to a DNG in ACR. Copied that file to another folder and in bridge removed all the adjustments I had made. In C1 both the original and the copy with the adjustments made and then removed still should in C1 with a ICC profile of DNG file Neutral and if I apply any of the C1 M8 profiles the images look bad.

So ACR/LR is changing the original DNG file in someway that C1 does not see it as a Leica M8 (or M9) file.

[sandymc]

Has LR altered the core DNG ( I mean aside from the sidecar information).

 

No, LR never alters the actual image data (other than losslesly compressing it, if you enable that). The only thing its does is to add "instructions" on how to process the data.

 

However, as Shootist correctly points out, you do need to be aware that even though the data isn't changed, some programs may not be able to read a DNG file that has been modified from its "native" form, even if the data is the same. Capture One is a particular offender here. The symptoms that Shootist describes are what happens if C1 reads a file that had been losslessly compressed by LR.

 

So I would strongly (very strongly!!) recommend that, unless you only ever intend opening your DNGs with Adobe products, you keep an unmodified backup of the camera generated version.

 

Sandy

[glenerrolrd]

This totally SUCKS. Thats a technical term .

 

I think its very important to differentiate between files that have been referenced without moving and those that have truly been imported into LR.

 

LR was supposed to be nondestructive to the original raw file . After asking this question seven ways to sunday...I now understand that LR will add the sidecar develop settings and meta data to the original DNG.

 

I still do not know if LR alters the DNG in any other way..e.g. does it do the basic raw conversion using the profile selected. e.g has the file been converted like it would be if I specified output as a TIFF. The only thing I am interested in learning here is the raw file no longer available for conversion?

 

I am not having any problem with C1 processing my DNG s and using the M9 profile . Keep in mind that I never really import the files into LR..I merely reference them without moving.

[sandymc]

LR was supposed to be nondestructive to the original raw file

 

LR is always non-destructive of the original data. However, what LR will do is to reformat that data into alternate representations. Those representations are entirely valid in terms of the DNG specification, and will be correctly read by any fully compliant DNG reader.

 

The problem that Adobe, and various folks the evangelize the DNG format, don't mention is that only Adobe's products, or products based on Adobe's SDK (of which CornerFix is one, btw) are fully DNG spec compliant. The rest are usually designed and tested only to read specific camera generated DNGs - aka, a subset of the full DNG specification.

 

Bottom line - treat anything except an Adobe product as able to read only very specific subsets of DNG.

 

Sandy

[glenerrolrd]

Sandy

 

This is helpful but quite disturbing to learn after processing 50K images. Reformating is in itself destructive if it keeps other programs from reading the file as an original. Leica users should be warned about this especially if they are considering loading their originals into LR .

[Shootist]

Sandy

 

This is helpful but quite disturbing to learn after processing 50K images. Reformating is in itself destructive if it keeps other programs from reading the file as an original. Leica users should be warned about this especially if they are considering loading their originals into LR .

 

Just did another little test with Bridge/ACR. I openned a M8 DNG in ACR from Bridge and then selected "Done" which will apply my basic settings, Sharpening, Contrast and profile and that file is still seen by C1 as a M8 file.

 

So it seems only when you make certian edit or many edits to a M8 DNG that it changes the DNG format so other programs, or at least C1, don't see the file as coming from a M8. I have no idea where or what the cutoff point is.

 

In LR you can import images without applying any settings. Then you make duplicates or variants (or whatever it's called in LR) of the ones you want to edit so the original does not get changed in any way. I think that is why Adobe incorporated the duplicate feature/function.

[Guest]

AS posted in another forum, one needs to set catalog settings such that:

 

"Automatically write changes into XMP

 

Warning: Changes made in LR will not automatically be visible in other applications."

 

Stay unchecked!

 

One should also not use "Update DNG Preview & Metadata" under metadata.

 

I assume that this will keep the original DNG in its native form and you can make duplicates to edit...

 

Probably best workflow will be to import DNGs to computer...archive a copy and use another to import into LR...suggested by Peter Krogh.

 

Bob

[roguewave]

Roger, here's my workflow to prevent what happened to you & others. I download the RAW files to a directory called RAW, where I have individual subdirectories named for the date and a secondary code if more than 1 card was used. When importing for LR, I import by "converting to DNG" into a new subdirectory with the exact same name (date) under the top level directory name of the year, i.e., 2009. I NEVER, EVER touch my RAW files. NEVER !!!! With my Drobo, I have at least 2 copies of ALL MY RAW FILES.

 

As fast as the technology is moving, I'm sure that Adobe & others will bring magic wands to our ability to get the most from the RAW data from now to well into the future. NEVER, EVER touch those archived RAW files!!!!

[glenerrolrd]

I put more time into understanding this than I should. Before I decided to spend all my time working through digital photography challenges, I ran large scale system integration projects. Thats consulting speak for projects that allow business information software to work seamlessly as a single business process. I was designing backup,recovery and restart processes before Adobe existed.

 

When I established my workflow, I created a folder system that goes back to the original card . My imports from the cards are loaded to a drive and backed up. The native ,original raws are safe and with some effort (thats the recovery part) I can get back to my originals , Knock on wood ...but I have never lost a single image since I went to a digital archive about 4 years ago.

 

One of the primary value propositions of LR is the ability to work non destructively with your raw files (or so I thought). Your live version of your raw files can be compromised by LR in several ways and there is no way to have LR export a true original unaltered file. This was news to me and I asked this question a bunch when I started my current workflow.

 

The problem seems to come with the fact that LR treats your Leica generated native DNG files as a standard DNG. An advantage of a DNG is that it can store a lot of relevant data.....like metadata,your develop settings ,the camera profile you used etc. The problem comes because you didn t know that LR would also update the DNG to a universal standard(think of this like the current version of a word document).

 

Now if the settings are all positioned to not update the raw file ..then you are ok for actions that just reference the raw file. But if you make and error ..you just updated your raw file. And guess what some off these updates make you raw file unreadable in software like C1. Now when I talk to the LR forum they uniformly think its a problem with C1 and get tied up in their shorts over DNG specifications. How dare C1 not maintain a current standard? (they may very well be doing that but importing files processed in LR has been a documented problem).

 

If you think about this in the context of a workflow ..its a big deal. You just can not trust LR to not change your raw images.....you might make a mistake, the program may have a bug or you might not understand what its really doing. This in my terminology means that your use of LR might destroy your primary copy of the raw file.

 

This can all be put together into a bullet proof workflow .....but it would be easier if you could just set the system to never update your original. (You don t have this problem with the proprietary versions of the raws files because they export a sidecar .xmp) .

 

I am always interested in learning so if someone has better knowledge set me straight.

[roguewave]

Roger, again, this is the Achilles heel, "One of the primary value propositions of LR is the ability to work non destructively with your raw files (or so I thought). "

 

I was instructed by a friend at Adobe to NEVER import the RAW file into LR. NEVER, NEVER, NEVER. As you pointed out, the Leica RAW file is designed by LEICA, etal. YOU MUST COPY & THEN COVERT the COPIED RAW file into a DNG. Then import ONLY the DNG. PERIOD. STOP. NO OTHER CHOICE.!!!!!!

 

SECOND. MAKE AN ADDITIONAL (READ 2nd) COPY OF ALL RAW FILES ON A DIFFERENT MEDIUM/DEVICE.

 

Never allow the raw files to exist on a platform where it is easy to import the original RAW files any where. NEVER. These files are GOLDEN. In the future, the images that we will be able to create with them are going to be outrageous. DON"T SPOIL THEM!

 

People don't take this seriously. Archive the original RAW files and NEVER touch them. You should always make a copy of any RAW file you want to convert to a DNG & then import that into LR or any other software. NEVER, EVER, trust any software company with your ORIGNAL SOURCEWORK!!!!!!

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[cocker]

That's not the case. You can revert to the original of an M8 file in the develop module by either pressing the 'Reset' button, or going to the history panel and clicking on the 'Import' history state.

 

Steve - where is the "Reset" button?

[roguewave]

That's not the case. You can revert to the original of an M8 file in the develop module by either pressing the 'Reset' button, or going to the history panel and clicking on the 'Import' history state.

 

 

This is MISLEADING. The problem is that you want to have the ORIGINAL RAW FILE, not the DNG. The DNG is not equivalent to the RAW file. DNG's are altered. The files start as RAW, then are CONVERTED into DNG's. PERIOD. Let's get this straight, A DNG IS NOT A RAW FILE!!