M11's new bottom, corrupted DNG's and a proposed fix

[Edax]

Advertisement (gone after registration)

“Talking about an issue is one thing, but it is often much more effective just to have a thorough look at it.”

 When a card is inserted in my M10, the backside red LED briefly lights up. It looks like the camera is being notified of the insertion. It got me thinking, why isn’t there a LED lit up on the M11 during card change? Well, the answer is extremely simple: on the M11 the card can only be changed when the battery is removed, in other words the camera is powerless during card change meaning it can never be notified of it. So, every time the M11 is switched on, it must check the card to get all the parameters needed for writing and reading. 

As it seems, most probably to save power up time, the M11 with FW 2.0.2 performs some kind of smart parameter caching. At power down, it will record “somewhere” the parameters together with the card id, and at power up it will retrieve those parameters from “somewhere” for the card id found inside. If the card id is yet unknown to the camera, the parameters will be retrieved by fully scanning the card. One of those parameters is the id of the first free sector where the M11 will be writing next.

 This smart card check works under the assumption that nothing changes to the card when the camera is off. But as soon as the card is inserted into a PC or a Mac without write protection, things will change on the card. If not already there, Windows will create a hidden “System Volume Information” folder with 2 files, and MacOS will create, if not already there, a hidden “.fseventsd” folder that normally contains a file called “fseventsd-uuid”. This “fseventsd-uuid” file will be updated every time the card is inserted. Now where are these hidden folders created? Exactly at the same free position as where the camera intents to do its next writing on that card.

 Put the card back into the camera and leave everything on it. The next write of the camera, will just overwrite the hidden files, without changing their directory entries as “deleted” (I checked this with a Windows program called FileSystemAnalyser). This is the start of card corruption (“damage to the data structure”) which will eventually lead to corrupted folders and/or corrupted DNG’s and is IMHO the reason behind that message in the FW2.0.2 release notes:

 “In rare cases, the import of DNGs from the SD card to image processing programs like Adobe Photoshop and Adobe Photoshop Lightroom would result in damage to the data structure, rendering the DNGs unreadable. This error will be remedied shortly”.

 Is there a fix to this problem? Yes, just write protect the card when inserting it in the PC or Mac. Use the tab or use a card reader with a write protection mode. Now what if you don’t have a reader with write protection and use Sony Tough cards without write protect tab? Then there is one more thing.

One more thing: I think I found a proper fix for this issue, a fix that prevents these corrupted DNG’s (when cards with pictures are brought back into the camera):

1)        Insert the SD card into the M11 and format the card IN CAMERA

2)       Take one picture of “nothing” and delete it immediately IN CAMERA.

The card is now ready for shooting.

 How does it work? The deletion of that single picture creates a kind of “safe room” for the hidden files written by Windows and/or MacOS because the camera, as far as I can tell (again verified with FileSystemAnalyser), will never write at this deleted location again.

 

 

 

 

 

 

[Rand]

I think you are something.

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[Stevejack]

15 hours ago, Edax said:

 

 

One more thing: I think I found a proper fix for this issue, a fix that prevents these corrupted DNG’s (when cards with pictures are brought back into the camera):

1)        Insert the SD card into the M11 and format the card IN CAMERA

  

  

Unfortunately this is my workflow every time the card gets put back into the camera, and I still get the corrupt DNG issue. IF there were any files on my card which were put there by OSX, they aren't there by the time I take my first picture because the card is formatted in-camera every time. 

Edited March 25 by Stevejack

[Edax]

6 minutes ago, Stevejack said:

Unfortunately this is my workflow every time the card gets put back into the camera, and I still get the corrupt DNG issue. IF there were any files on my card which were put there by OSX, they aren't there by the time I take my first picture because the card is formatted in-camera every time. 

If you always format the card when it is brought into the M11, and get corrupted DNG's, then the problem is indeed not caused by OSX littering those files. I suggest a complete reset of the camera with afterwards all settings done manual from scratch and see what happens. I know, doing a complete reset is a hassle. But it solved severe problems for me a few times.

[Photoworks]

I think the M11 has a new DB system in comparison to the M10. 
It is probably to deal with the presence of internal storage.

The M11 does not read any cards coming from other cameras, not even Leica SL or M

I had great success just formatting the SD card on the Mac and using it on the M11

 

PS. Sony Toght cards are a problem on many cameras, they must use a controller that buffers and stops working. I stopped using them.

[Edax]

42 minutes ago, Photoworks said:

I think the M11 has a new DB system in comparison to the M10. 
It is probably to deal with the presence of internal storage.

They are just putting files on the SD card according to the DCF (design rule for camera file system) standard. 

42 minutes ago, Photoworks said:

The M11 does not read any cards coming from other cameras, not even Leica SL or M

I can format a card in my M10, put it for a moment in my Mac, and then use it in the M11. In this way it is even Mac "litter-resistant".

42 minutes ago, Photoworks said:

I had great success just formatting the SD card on the Mac and using it on the M11

If the card has never been in the M11 yet, yes it works and the card even becomes "litter-resistant". But if the card has already been used in the M11, depending on conditions, it is a recipe for damaging the SD card data structure, eventually leading to disaster (corrupted files and/or camera freezes). E.g. format a card in the M11, then format it again on the Mac, take a few pictures and the card data structure is already damaged!

42 minutes ago, Photoworks said:

PS. Sony Toght cards are a problem on many cameras, they must use a controller that buffers and stops working. I stopped using them.

My Sony Tough 64GB G card does not work in my M10, it does work in my M11-p.

Edited March 26 by Edax

[Jamie Roberts]

Advertisement (gone after registration)

52 minutes ago, Edax said:

I can format a card in my M10, put it for a moment in my Mac, and then use it in the M11. In this way it is even Mac "litter-resistant".

If the card has never been in the M11 yet, yes it works and the card even becomes "litter-resistant". But if the card has already been used in the M11, depending on conditions, it is a recipe for damaging the SD card data structure, eventually leading to disaster (corrupted files and/or camera freezes). E.g. format a card in the M11, then format it again on the Mac, take a few pictures and the card data structure is already damaged!

I have some logical questions...

1) Why would anyone format a card in an M10 then put it in an M11 (even without a trip through MacOS)? Are you saying the formatting in the M10 is somehow protective for the M11 against bad writes?  
2) I understand folks think that MacOS is "littering" SD cards with hidden files. So that means you always need to re-format the card. But how on earth could the Mac writing files "damage the SD card structure?" And If I format my cards every time they go into the M11 (which I always do), the M11 has erased the "littering" put there by a Mac, no? (FWIW I use both Mac and Windows).
3) If I format my card every time it goes into an M11, then get an error, then the M11 in-camera formatting (or a card incompatibility--which seems to be a Leica thing) is at the root of the issue.  

4) Doing a full reset seems wise to me, though how that would affect camera writes is a little hard to tell... the only "new" thing that would affect the write, you would think, would be the security / originality confirmation signature (which I have turned off, right now :)) 

For me: 3K photos so far on latest firmware: no bad DNGs or lockups yet (except when I forgot to remove the cap and had to wait for noise reduction. But that's on me :))  

[Edax]

12 hours ago, Jamie Roberts said:

I have some logical questions...

1) Why would anyone format a card in an M10 then put it in an M11 (even without a trip through MacOS)? Are you saying the formatting in the M10 is somehow protective for the M11 against bad writes? 

Yes it may sound very weird, but it is true. The M11 treads cards formatted by another M as "unknown yet" until the moment it has written something on it. It doesn't do this however with cards that are formatted on Mac/PC and were written on in the M11 before.

12 hours ago, Jamie Roberts said:

2) I understand folks think that MacOS is "littering" SD cards with hidden files. So that means you always need to re-format the card. But how on earth could the Mac writing files "damage the SD card structure?" And If I format my cards every time they go into the M11 (which I always do), the M11 has erased the "littering" put there by a Mac, no? (FWIW I use both Mac and Windows).

Yes the in-camera format deletes the files, and shooting will be without any problems. It is a safe workflow.

12 hours ago, Jamie Roberts said:

3) If I format my card every time it goes into an M11, then get an error, then the M11 in-camera formatting (or a card incompatibility--which seems to be a Leica thing) is at the root of the issue.  

In-camera format before shooting should never lead to problems on compatible cards. Card failure or incompatibility is then root of issue. 

12 hours ago, Jamie Roberts said:

4) Doing a full reset seems wise to me, though how that would affect camera writes is a little hard to tell... the only "new" thing that would affect the write, you would think, would be the security / originality confirmation signature (which I have turned off, right now :)) 

The full M11 reset will wipe an internal M11 database which contains records for "already written to" cards on "where to bluntly write next without consulting the cards's data structures". 

12 hours ago, Jamie Roberts said:

For me: 3K photos so far on latest firmware: no bad DNGs or lockups yet (except when I forgot to remove the cap and had to wait for noise reduction. But that's on me :))  

Yes, with your "always format in camera before shooting" this is what is expected!!

Personally I want to keep pictures on cards; ingest them in the field on my Macbook, and ingest the same pictures from the same cards again at home on my Mac Studio. So I have to be careful. Write protecting cards is of course the best way to prevent problems, but with the all knowledge I gathered by now I can also safely perform my workflow without that.

[Jamie Roberts]

8 hours ago, Edax said:

{snipped}
Personally I want to keep pictures on cards; ingest them in the field on my Macbook, and ingest the same pictures from the same cards again at home on my Mac Studio. So I have to be careful. Write protecting cards is of course the best way to prevent problems, but with the all knowledge I gathered by now I can also safely perform my workflow without that.

Thanks for clarifying--that all makes sense!

As for workflow, I understand wanting to back up in the field. But why ingest the card twice (it's slow, too)? Why not, say, copy files to your Mac in the field, then copy them again to a fast external SSD? That way you could format the card in-camera, have a backup, and still ingest raws (from the SSD) when you're home. 

I understand it seems to be an unnecessary step, but when I was shooting a lot of weddings (mainly with an M9!), that's basically how I'd do it (no SSDs back then, but external hard drives).

The "always format in-camera before use" mantra came from issues from many manufacturers 15 years ago--not just Leica--where the camera would get confused re-reading re-used, previously formatted cards (and heaven help you if you put a card from one camera into a different brand without re-formatting!). 

PS--love your photography (from Instagram)--it's too bad you're having such trouble with the M11. 
 

[Tailwagger]

Very promising theory which seems to fit most of the facts. Thanks!  It may well be that those who are never experiencing the problem are xfering via Fotos or cables. 

I have always experienced the corruption on the first shot after power up, but more often that not that shot was not in the first series taken after the card was inserted.  It's been many years since I did any serious file system work, but I think what you are suggesting is that this initial miswrite screws up the TOC's notion of what's on the card and eventually you lose a file due to bad pointers.  I can definitely buy into that, but then why is it that the lost file is never in the middle of a series of shots, only after a power on?  The other potentially interesting datapoint, one I have not personally experienced but has been claimed by others is that they have experienced file corruption on the internal card as well.

Again, Im not up on SD protocols, so your supposition may indeed be spot on, but OTOH it might not be the only bug in 2.0.2 that can result in corrupted files.  

[Edax]

On 3/27/2024 at 10:48 PM, Tailwagger said:

Very promising theory which seems to fit most of the facts. Thanks!  It may well be that those who are never experiencing the problem are xfering via Fotos or cables. 

Exactly. Also, those that format a card in camera after each Mac/PC picture ingest, will not experience the problem. And those who use the M11' internal copy IN => SD, a process that requires an empty card, will not experience the problem even if they did formatted the card in the M11.

On 3/27/2024 at 10:48 PM, Tailwagger said:

I have always experienced the corruption on the first shot after power up, but more often that not that shot was not in the first series taken after the card was inserted.  It's been many years since I did any serious file system work, but I think what you are suggesting is that this initial miswrite screws up the TOC's notion of what's on the card and eventually you lose a file due to bad pointers.  I can definitely buy into that, but then why is it that the lost file is never in the middle of a series of shots, only after a power on? 

That might indeed have another cause. My M11-p is used in luddite mode: so no Foto's, no cable transfer, no Visoflex, and until now I only used two 6-bit coded lenses. So might never be hit by that problem...

Did you ever check if the data structures on your affected cards are still OK? (You can do that on the Mac with Disk Utility, and apply First aid on the LEICA M volume). I would be interested to know that.

On 3/27/2024 at 10:48 PM, Tailwagger said:

Again, Im not up on SD protocols, so your supposition may indeed be spot on, but OTOH it might not be the only bug in 2.0.2 that can result in corrupted files.  

Yes, there can/will be other bugs too on 2.0.2 that cause corrupted files. 

I believe that most, if not all of the freezes that I encountered were eventually caused by this problem. The freezes sometimes left the M11-p in a bad, unstable state; e.g. I experienced rest file counters and after one freeze the camera started produce *unneeded* 101LEICA, 102LEICA folders in the DCIM folder. I had to fully reset the camera to get it stable again. I wouldn't rule out that a M11 in such an unstable state can produce corrupted files too.

This morning I observed something weird: a picture on the card that could not be displayed by the M11. After a slight panic I found the cause of this issue: a little bit earlier today (by accident) I openend a picture (named L003300.DNG) directly from the SD card with PTGUI. Nothing strange to see in Finder, and First Aid reported a healthy card. So I took the card to the PC with that mighty FileSystemAnalyzer.exe utility. It found a ._L003300.DNG file in the 100LEICA directory (toc) that was physically written in the space that I "created" by deleting that first file after format (as described in my OP). The file is obviously metadata for the file I accidentally opened, and apparently the M11 can see it and thinks that it is a picture file too... Good to see that the fix dealt properly with this unexpected write to the card!

[Tailwagger]

3 hours ago, Edax said:

That might indeed have another cause. My M11-p is used in luddite mode: so no Foto's, no cable transfer, no Visoflex, and until now I only used two 6-bit coded lenses. So might never be hit by that problem...

Did you ever check if the data structures on your affected cards are still OK?

No fotos, no cable xfer here as well. All 6bit coded of late.  But Visoflex 24/7. 

I haven't checked... hadn't considered the possibility to be honest.  Being an OS guy back in the day, my sense was that freezes had the classic feel of multi-level flag race conditions. I saw freezes pre-1.6 (and all previous Ms as well), but none subsequent.  The file corruption issue did strike me as being some sort of corrupt NVRAM situation... given it has always occurred on startup, but it hadn't occurred to me that the problem had an external stimulus.  Really should have, as this isnt the first issue to surface in this space. I seem to recall a few years back with the M10 that if you xfered files onto the Mac through the car reader rather than LR, the card wound up with some MacOS cruft files and things got wonky... though I can recall the symptoms. Wasnt  a corrupted file, might have been that the camera refused to write the card or something. 

Anyhoo, I just checked my card (San disk Extreme Pro 300MB/s) via First Aid and it was pronounced clean. If I encounter another bad file (and I expect I will... had about a dozen so far), I'll check the card then and report back. 

[Edax]

A card became corrupted yesterday. I took pictures on it, removed it and confirmed its healthy status with Disk Utility / First Aid. Inserted another card, and took pictures on that one. Removed the second card, and put back the first one and took again some pictures on that. Immediately after the next M11 power up with that card still in the camera came that miserable “SD Card Error” message.

Time for more tests with help of FileSystemAnalyser.exe, and time to revise my theory. It is even much simpler than initially thought.

After the M11 takes a picture, it records it at a certain cluster position on the card, and it calculates the cluster number where to write the next picture.

Now don’t take that next picture, but replace that card A by another card B. Don’t format it. Take a picture on card B. It will be just recorded at that calculated next cluster number as if card A was still present in the camera. On the second card this B position might already contain pictures, so a true clusterf**k!

 

[LocalHero1953]

@Edax, I hope you are reporting your findings to Leica?

[Edax]

2 minutes ago, LocalHero1953 said:

@Edax, I hope you are reporting your findings to Leica?

Yes I plan to do that. But never heard back from my previous report...

[charlesphoto99]

I've probably shot 100,000 plus photos with my M10 and now M10-R. All downloaded via SD card into a Mac, and then the cards reformatted always inside the camera. Various Sandisks, and others. Even cards that had been used previously in a Nikon. Never once had an issue with the card, or lockups or lost files from said cards. Never locked the cards or used SD formatter. Same with many other cameras. It's an M11 issue. 

[Jamie Roberts]

37 minutes ago, charlesphoto99 said:

I've probably shot 100,000 plus photos with my M10 and now M10-R. All downloaded via SD card into a Mac, and then the cards reformatted always inside the camera. Various Sandisks, and others. Even cards that had been used previously in a Nikon. Never once had an issue with the card, or lockups or lost files from said cards. Never locked the cards or used SD formatter. Same with many other cameras. It's an M11 issue. 

But some people are saying that because you format the card in-camera you won't see a problem with the M11. 

I'm 4K photos in on a new M11P and so far zero issues. But I always format the various (fast) cards I use in-camera whenever they are used, and I did that in my previous Ms, Canons, Panasonics and Nikons. 

If I *didn't* do that all the time there was always going to be a non-zero chance of things screwing up at some point under heavy use (especially with video).

Even early on, with fast equipment, I remember my Canon 1ds2 going south during a shoot--card issues--and a colleague had the same trouble with 5ds, and others. My R5 is also card and format picky. I had less trouble with Nikon, but by then I was always formatting in-camera, so who knows?

FWIW, I've also had a couple of bad (probably forged) Sandisk cards, so I don't use them anymore in any camera, just as a precaution.  

[Smogg]

I always format the card when I put it in the camera before shooting again. Unfortunately, this does not help in my case with both M11 cameras. It feels like shooting on internal memory (also with pre-formatting) causes freezes a little less often. At the same time, my previous cameras M10, M10p, M10r froze so rarely that this could be neglected.

[JRoberts]

2 hours ago, Smogg said:

I always format the card when I put it in the camera before shooting again. Unfortunately, this does not help in my case with both M11 cameras. It feels like shooting on internal memory (also with pre-formatting) causes freezes a little less often. At the same time, my previous cameras M10, M10p, M10r froze so rarely that this could be neglected.

Interesting … I thought you said in another thread that you backup the cards into a Mac then put them back into the camera—without a format? 
 

Sorry in advance if I misunderstood your workflow! 

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[Smogg]

32 minutes ago, JRoberts said:

Interesting … I thought you said in another thread that you backup the cards into a Mac then put them back into the camera—without a format? 
 

Sorry in advance if I misunderstood your workflow! 

No, apparently I was misunderstood. I do use a Mac to transfer files into Lightroom, but after inserting a memory card into the camera I always format it