PNCD Posted September 30, 2014 Share #1 Posted September 30, 2014 Advertisement (gone after registration) I had a M9 and then moved to the M240 when it came out. I keep the pictures in Leica RAW format and then use current version of Adobe Lightroom to edit and, if needed, save in JPG. I returned from a trip to Greece and was astonished to see files that had been perfectly normal in the camera get loaded into Lightroom and before my eyes become corrupted. I quickly copied the RAW file to another NAS drive and to a USB drive. I also picked clear images from the different directories and converted them into JPG. I also used Capture Media One to inspect the files to see if it was a corruption of the sidecar files in Lightroom. The image below Shows two sets of identical DNGs in different directories. On the left are the same files that I converted from JPG, using clean looking images in Lightroom. Has anyone ever seen anything like this? Welcome, dear visitor! As registered member you'd see an image here… Simply register for free here – We are always happy to welcome new members! Quote Link to post Share on other sites Simply register for free here – We are always happy to welcome new members! ' data-webShareUrl='https://www.l-camera-forum.com/topic/234799-bizarre-changes-in-leica-dng/?do=findComment&comment=2680978'>More sharing options...
Advertisement Posted September 30, 2014 Posted September 30, 2014 Hi PNCD, Take a look here Bizarre Changes in Leica DNG. I'm sure you'll find what you were looking for!
Exodies Posted September 30, 2014 Share #2 Posted September 30, 2014 Do the thumbnails sort themselves out if you get Lightroom to refresh the previews? Quote Link to post Share on other sites More sharing options...
PNCD Posted September 30, 2014 Author Share #3 Posted September 30, 2014 I went to Library-Previews and Discarded existing previews and Built new previews. The result is the same. Of course, the other directories have different files that have been changed - all from the same source! Quote Link to post Share on other sites More sharing options...
hoppyman Posted September 30, 2014 Share #4 Posted September 30, 2014 (edited) I don't know anything about Capture Media One and which DNG standard that they currently implement. I can provide a couple of facts that may assist a little. You can't convert to DNG files from JPEGs but you can of course create new versions from your DNG originals. There are no separate sidecar files for DNG. The preview you see on the camera LCD and initially in Lightroom is a (particularly) low resolution JPEG that is embedded in the DNG. Depending on your LR settings and selections, it will build new previews. A DNG that has been edited in one program may not have those instructions visible in another program (and that is dependant on which parts of the DNG standards are implemented too) In Lightroom, the higher quality previews are stored with the catalog file. They don't replace the embedded ones made in the camera. The develop instructions are written into the DNG itself although they don't change the original data. If you can see the previews in another application it suggests that the original files are not corrupted. However new previews and edits in one application are not going to be visible in another. Under 'Metadata" in LR there is an option to 'update DNG preview and metadata' Under 'Library' there is an option to 'validate DNG files' You can choose to have the edit instructions written into XMP however its not clear to me how that works in the case of DNGs when there are no separate sidecar files anyway. You could try creating a new catalog (without deleting the original) and importing new copies of some DNG originals It is also possible to delete the entire previews file to force Lightroom to recreate all of them but be aware that this can be a large operation depending on how many images you have. You will lose all of the develop info from every image if you start again though. you may like to do some searching on-line for LR instructional sites. For example, Julianne Kost and Laura Shoe offer a lot of free tutorials FAQs etc. Edited September 30, 2014 by hoppyman Quote Link to post Share on other sites More sharing options...
CheshireCat Posted September 30, 2014 Share #5 Posted September 30, 2014 It smells like bit-errors in lossless compressed DNG. The images you see in-camera is the JPG preview, unless you zoom-in. The JPG preview has a low chance of being corrupted in case of media error, as it is much smaller than the RAW sensor data. Next steps: 1) If you still have the files in your DS, then put it back in your camera and try to zooming-in to 1:1. Let us know if you see the artifacts. 2) Send us a "broken" file to check (DropBox). 3) Report SD card brand and model. 4) Do not use that SD card until we understand what the problem is. Quote Link to post Share on other sites More sharing options...
FlashGordonPhotography Posted October 1, 2014 Share #6 Posted October 1, 2014 Normally, Lightroom loads the embedded preview before it creates its own preview based on the actual data. The change to the "corruption" as you describe it, is this process in action. Lightfoot never ever modifies a raw file. Any changes you see are only metadata until a file is exported. One possibility, since you have upgraded your camera is that you are using an older version of LR. is your version up to date? It's also possible the copy from the card caused the corruption. you could try copying the images from the card again with a different card reader. The files could have been corrupted when the camera was writing the file to the card. unfortunately this would be unrecoverable. I would try to copy the files from the card again. Gordon Quote Link to post Share on other sites More sharing options...
PNCD Posted October 1, 2014 Author Share #7 Posted October 1, 2014 Advertisement (gone after registration) HoppyMan. Thank you for the information on the DNG file format. I was not trying to create DNG from JPG, though I did convert DNG to JPG for visibly 'clean' DNG files in the fear that something on my computer was corrupting DNG files. That is not the case - every previous photo series from the M240 is unaffected. I did update the DNG previews as you instructed. The validation process is running now. CheshireCat I deleted the files from SD card. It is a current spec SanDisk and it too had worked fine in the M240 on previous photo series. It is an ExtremePro 32gb. I will upload a file, though I am not sure where to drop it on DropBox. FlashGordon I am using the latest LR and the latest Capture MediaPro. I cannot recopy from the SD card since I moved the files off the SD. Here is the strange thing: I copied the moved files to two different NAS and to an external USB 3.0 drive. The corruption shows in different files though all were copied from the same files moved off the SD card. At this point I am relieved that I am not looking at some media-consuming virus. The only material change that I see is an M240 firmware upgrade before I left on this trip. However I have seen no comments of others have such problems after the update and it is not a subtle thing! I really thank you for your help. Quote Link to post Share on other sites More sharing options...
PNCD Posted October 1, 2014 Author Share #8 Posted October 1, 2014 Look at the file in the middle of the second raw in the screenshot above: Santorini-005. I converted the 'damaged' photo in LR (Santorini-005LR) and in MediaPro (Santorini-005MP). The LR preview showed it was damaged while the MP preview showed that it was not damaged. Now look at the two results for the same file on the same directory! MediaPro does show some files as damaged but far fewer that LR. It is definitely bizarre and I think the only cure is going to be more malt whisky.... Welcome, dear visitor! As registered member you'd see an image here… Simply register for free here – We are always happy to welcome new members! Quote Link to post Share on other sites Simply register for free here – We are always happy to welcome new members! ' data-webShareUrl='https://www.l-camera-forum.com/topic/234799-bizarre-changes-in-leica-dng/?do=findComment&comment=2681771'>More sharing options...
CheshireCat Posted October 1, 2014 Share #9 Posted October 1, 2014 I cannot see any corruption in the second picture. I will definitely need to download the DNG from a public DropBox or Google Drive link. You should gnore the previews. Some applications get them from the embedded JPG, while others recompute the preview using the raw data (Lightroom does this in a background thread, so you'll see the embedded JPG previews at first, and after a while they are updated with the recomputed ones). In Lightroom, if you select "Update DNG Preview and Metadata" function, then the embedded preview JPG in your DNG is replaced by a fresh one recomputed using raw data and current processing setting. It is interesting to note that while doing this, an uncompressed DNG will be magically lossless compressed. In fact this was the trick I used to lossless compress images from my M9. Quote Link to post Share on other sites More sharing options...
Hookeye Posted October 1, 2014 Share #10 Posted October 1, 2014 It is definitely bizarre and I think the only cure is going to be more malt whisky.... I think you may be on to something there.. Cheers, HP Quote Link to post Share on other sites More sharing options...
wlaidlaw Posted October 1, 2014 Share #11 Posted October 1, 2014 I too have been having problems with corrupted previews in Capture One (V. 8.0 Pro). I thought it was just me. I will try switching over to uncompressed to see if that cures the problem. It seems very variable. Some corrupt but some don’t seemingly at random. Sometimes you can cure it by navigating away from the image and going back again but sometimes you can’t. I will try the malt whisky cure, which at least has its upside. Wilson Quote Link to post Share on other sites More sharing options...
colonel Posted October 2, 2014 Share #12 Posted October 2, 2014 there may be some corruption in the DNG. i.e. the embedded jpg is fine and most of the DNG is fine, but some key control bits for a certain section are corrupted. Mediapro may simply be doing some interpolation based on some auto error correction I would chuck the SD card (or have it exchanged if guaranteed) and get a new one If you bought it on ebay be aware that a high percentage of sandisk cards on ebay are fake The fake cards are often 16gb cards posing as 32gb, 8 as 16, etc. Which means they work up to a point and then after that point eat into the area of the card that was marked as unwritable (typically alot for SD - hidden from user) and have all sorts of odd effects Quote Link to post Share on other sites More sharing options...
jaapv Posted October 2, 2014 Share #13 Posted October 2, 2014 I had this a couple of years ago on M9 files. It turned out to be the card that got unstable at higher temperatures. I I threw the card inn the bin. 1 Quote Link to post Share on other sites More sharing options...
CheshireCat Posted October 2, 2014 Share #14 Posted October 2, 2014 Mediapro may simply be doing some interpolation based on some auto error correction Indeed it seems that Mediapro has a more robust decompression algorithm. As DNG lossless compression is differential, it may just be enough to clip pixel data to avoid overflows. I don't know if DNG allows ECC, but it would be very helpful in these cases. In any case, an alert on CRC failure would warn the photographer about the fact he/she is using a faulty flash card to replace immediately. It may save the day in case of a critical assignment. Quote Link to post Share on other sites More sharing options...
01af Posted October 2, 2014 Share #15 Posted October 2, 2014 Have you checked your hard-disk drives for bad sectors lately? Quote Link to post Share on other sites More sharing options...
jdlaing Posted October 2, 2014 Share #16 Posted October 2, 2014 Looks like a bad memory card. Quote Link to post Share on other sites More sharing options...
pico Posted October 2, 2014 Share #17 Posted October 2, 2014 Indeed it seems that Mediapro has a more robust decompression algorithm. As DNG lossless compression is differential, it may just be enough to clip pixel data to avoid overflows. DNG is the data, not the image. DNG data is not supposed to be compressed using a differential method, is it? The preview or JPEG output may/many not be compressed using differential methods, however. Overflow is not likely an issue, IMHO. I am certainly willing to be corrected. Differential is amazingly good, but its hash is not saved. Once compressed, we have no method of recapitulating the data. ...it is early. I need coffee. . Quote Link to post Share on other sites More sharing options...
wlaidlaw Posted October 2, 2014 Share #18 Posted October 2, 2014 On my images which looked corrupted on C1, the actual DNG was fine, it was just the preview that was corrupted. However I am not going to take a chance for my trip to Taiwan. Am I correct in thinking that the Transcend 64GB Ultimate SDHC cards (http://www.amazon.co.uk/Transcend-Ultimate-Class-UHS-I-Memory/dp/B00AFTV3FC/ref=sr_1_4?ie=UTF8&qid=1412267625&sr=8-4&keywords=Transcend+SDHC+64GB) are the current flavour of the month for the M240? Wilson Quote Link to post Share on other sites More sharing options...
CheshireCat Posted October 2, 2014 Share #19 Posted October 2, 2014 DNG is the data, not the image. DNG data is not supposed to be compressed using a differential method, is it? The preview or JPEG output may/many not be compressed using differential methods, however. Overflow is not likely an issue, IMHO. It is DPCM compression, where "D" stands for Differential. Lossless JPEG - Wikipedia, the free encyclopedia Differential is amazingly good, but its hash is not saved. Once compressed, we have no method of recapitulating the data. The purpose of a compression scheme is getting rid of redundancy. In that, differential compression is not different than other compression algorithms. Redundancy is usually added at a different layer, whether it is a CRC (identify a broken file) or ECC (identify and correct up to a certain amount of bit errors). A DNG 1.2 file can contain an optional "RawImageDigest" tag which is a MD5 hash, but unfortunately, it doesn't seem to be used. With that, one could immediately tell whether a file is corrupt (and immediately know that the flash memory is broken). But the ideal thing would be having flash cards with proper ECC. I wouldn't mind paying twice the price for a good ECC flash. 2 Quote Link to post Share on other sites More sharing options...
Gary Clennan Posted October 2, 2014 Share #20 Posted October 2, 2014 I have experienced the same issue and am starting to hear of a few other cases. Here is my thread which may help a bit: http://www.l-camera-forum.com/leica-forum/leica-m-type-240/345130-strange-image-output.html There was a lot of good suggestions by forum members. I have not solved my problem but it doesn't happen very often so I try not to worry. One thing I was able to find out was that if you exit out of LR and open the file in Photoshop (not sure about other software), the file will actually render properly. Another issue I recently noticed was that after import of images into LR, there would be some oddball picture in the middle of them all which was taken months prior. If you then open that picture in Photoshop, it will render as a completely different (and proper) image. Pretty strange behavior and I really don't think it is related to they SD card. Best of luck and let us know how it works out. Quote Link to post Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.