Rethinking my digital photo storage - how do others do it?

[jhluxton]

Advertisement (gone after registration)

On 8/16/2021 at 10:07 AM, leica dream said:

OK, I accept fair comment but it never hurts to float the unthinkable. A millennial thing, well perhaps at 80 I should be proud! I do support the view that after a lifetime of taking pictures, keeping them is futile for those who follow but will have no interest in the subjects. Times, tastes and attitudes move on.

Interesting perspective - but it might depend what you photograph. There is someone I follow on flickr who "rescues" discarded photos which he I think he picks up from skips, house clearances etc. There are some real gems in there. Historical interest, and it is a shame that they have been discarded. Yes quite a few are not very good - but he posts the good, bad and ugly and even in some of the less that perfect there can be interest. I love playing spot the location as most are not captioned.

Continuing the point of why keep images when those who come after you may not be interested, or perhaps in my case where there isn't anyone coming after me, I still don't want all my images to end up in a skip when I pass on. Here in the UK there is an organisation known as "The Online Transport Archive" or OTA that takes collections of transport / street scene material digital / negative and slide and makes it available for historians, writers, publishers etc. They also publish the odd book in their own right. 

When I get round to rewriting my will which is a must do job in the near future the OTA is where I want my images to go as I have slides going back to my teenage years in the 1970s and digital images from around 2000/2001. 

I think today we are in great danger of losing images, much more so at any time in history. How many Victorian / Edwardian photos, plates, negatives were just stashed away and were discovered much more recently. Analog photograph lends itself to leaving in the attic, back of a wardrobe - but digital - unless one keeps it on current media which is readable can so easily be lost. 

My interest in photography has always been an adjunct to my interest in history / transport etc and I just value and enjoy images from the past as record of a period of time and subsequent change so perhaps I look at things differently from those who just undertake photography from a creative / artistic perspective. 

I think we all need to think about ensuring our photographic legacy is there for future generations - even if it is not necessarily within our own families. 

Anyway that is my perspective!

John

[jhluxton]

On 8/17/2021 at 4:24 PM, ianman said:

That is why mine is backed up to Amazon Glacier. It is astonishingly cheap. The point with this system is that it really is a last resort. You only really pay when you download. Storage is frankly peanuts. I pay something like €2,50 per month for storage. IIRC if you do need to download you can do so at very slow speeds to reduce cost.

That sounds interesting - I have been an Amazon customer probably since they started and I didn't even realise this existed. Though I imagine uploading might take a VERY long time! 

[Advertisement]

The Leica Q3 is available for pre-order at B&H Photo Video and Adorama

[ianman]

2 hours ago, jhluxton said:

That sounds interesting - I have been an Amazon customer probably since they started and I didn't even realise this existed. Though I imagine uploading might take a VERY long time! 

Glacier is part of the Amazon Web Services (AWS). They have had different hosting services available for a very long time now but it's true that they are more targeted to the IT industry so not at all advertised in the Amazon store front end. They have so many different services it can get quite confusing!!

Yes, uploading is slow and I think you can set limits yourself. Glacier is deliberatly slow as it's what they call deep storage. I use it as a last resort. To be honest I've never needed to retrieve anything and hope I never need to, but the storage is so cheap I use it anyway. As you rightly wrote in a previous post, we can have as many backups as we want, but if they are all in the same place, they could all be destroyed or stolen together. A friend of mine stores backups in a bank vault. I can be bothered TBH, Glacier is probably cheaper and easier.

If you need faster (but more expensive) storage Amazon S3 is a possibility.

Storage prices are here: https://aws.amazon.com/s3/glacier/pricing/  For example if you store in their "London" region, it's $0.0045 per GB / Month. Uploading is also very cheap ($0.0318 per 1,000 requests)

Retrieval is more expensive and cost depends on the speed at which you want to download and how many files. Here the trick is to have some sort of backup stratergy so that if you need to retrieve, you download as little as possible. So maybe make a folder every month, or every x months depending on how much you shoot. I don't think you can retrieve single files, but I'm not sure about this.

Anyway, I don't want to come across as an Amazon salesperson... just sharing what I do

 

[Stuart Richardson]

Honestly, this is all kind of moot. As said above, 2TB is tiny these days. You can buy a 2TB drive brand new for around 50-60 dollars at B&H. 4TB is around 75 dollars. Buy a couple drives and set up some basic back up procedures. You could buy a good, fast 4-8TB drive to put your new work on as your main drive, and have another drive as a time machine backup (if you are with Apple...I am sure Windows has an equivalent. Then using a cloning software (Carbon Copy Cloner is a good one for Mac) to make mirrors every week or so and keep it off site if you are especially worried. Or pay for a monthly service online. This works just fine until you get into video or medium format digital...

 

[jhluxton]

2 hours ago, ianman said:Anyway, I don't want to come across as an Amazon salesperson... just sharing what I do

 

Very useful info - no you don't come across as a sales person. Something which I will put in the consideration pot when deciding a final strategy.

Thank you

[jrp]

And if you’re an Amazon Prime subscriber you get unlimited photo storage.

[jrp]

Advertisement (gone after registration)

But 2.5” spinning rust drives go to 5tb. Sandisk and Crucial do 4tb ssd drives. 

[jhluxton]

On 8/21/2021 at 1:57 PM, jrp said:

But 2.5” spinning rust drives go to 5tb. Sandisk and Crucial do 4tb ssd drives. 

I have tended to disregard SSD as in the past when I have looked at them they appeared very pricey for the capacity offered,

However, just having a look at the Crucial 4TB externals I see these are now available in the UK for just over £400 each.

The price/storage capacity has certainly improved in recent times and seems to be heading in the right direction with prices falling and capacity increasing very much as happened with hard drives.

If this trend continues one presumes shortly they will become the default method of storage. Anyone know of any disadvantages of SSD over HD?  

John

Edited August 24, 2021 by jhluxton

[piran]

9 hours ago, jhluxton said:

If this trend continues one presumes shortly they will become the default method of storage. Anyone know of any disadvantages of SSD over HD?  

Yes, unless something better comes along 😎 Disadvantages? More costly, more limited sizes, need TRIM events (can be automated), the portable ones can 'walk' more easily (office or public environment) as they are quite small. Be aware SSD flavours can be different - some are physically the same size as for a 2.5" flying rust HDD replacement - some are portable jobbies (usually smaller) - some are NVme installable 'cards' into computer extension bus. I have one of the latter in my PC running my scanned stuff. Also have a brilliantly portable one for quickie nightly events ...it's about the size of a skinny Swan Vesta oblong matchbox. Really small. REALLY quick. Also helpful w.r.t. ransomware as it spends 99% of the time unplugged/offline. My intranet-located NAS boxes doing the daily runs actually have bunches of big flying rust HDDs but the Synology boxes could've been fitted out with SSDs. YMMV

[marchyman]

On 8/24/2021 at 12:25 PM, piran said:

the portable ones can 'walk' more easily (office or public environment) as they are quite small.

Quite true.

The small size can also be a good thing.  In these days of fire, floods, earthquakes, whatever if I have to leave my house in a hurry I can fit both my system backup drive and my photos drive in the same shirt pocket.  They are about 50x95x9 mm  in size.

[piran]

2 hours ago, marchyman said:

In these days of fire, floods, earthquakes, whatever if I have to leave my house in a hurry I can fit both my system backup drive and my photos drive in the same shirt pocket.  They are about 50x95x9 mm  in size.

Agreed and I thoroughly commend your planning, not enough people think seriously about that, but I cite 'horses for courses'. Yes, my quickie nightly portable 2TB SSD is a bit smaller but it's only got my current/changed stuff.

My own 'planned emergency departure' would need ...a bigger pocket. More of a tucked-under-the-arm lug with a shoebox sized 40TB NAS (weighing maybe 10Kg), another smaller NAS under the other arm and, of course, Leicas over my shoulder. That would still leave behind at least three active systems each installed with 10TB drives. A small cupboard of ancient/legacy HDDs would just have to be lost to being drowned, burnt or buried.

Though San Andreas Fault earthquakes are not common here (solid granite albeit peppered with tin mining shaft holes and tunnels) I am considering a meagre 4TB 'Swan Vesta' SSD for just such a pocketable 'hurried leaving' that you've described.

Long term I'm awaiting the costs of 40-50TB replacement SSDs to come down some... Might give me a better chance to escape 😎 or I could just 'go down with the ship' which might render what I do with all the pixels afterwards moot ...as John's original thread starting post also discussed.

[Herr Barnack]

Here's how I do it:  I use a Macbook Pro, external hard drives and Lightroom.  Once my DNG files have been downloaded via the library module, about half of them mysteriously vanish off into the sweet by and by, never to be f*****g seen or heard from again.  Of course, the useless preview images are always available at my fingertips.

I would highly recommend this method of image storage and retrieval to one and all. 

[ianman]

Pocketable primary drives and back ups in the same pocket are pretty much useless if you are wading in five feet of water. There is IMHO nothing better than off site backups which are themselves distributed in multiple locations worldwide.

[John Robinson]

Lots of hard disks.  Every two years or so they double in size for the same price.  Given they need replacement anyway after about three years, just double your storage each time you update.  I have a PC with two internal disks (8Tb) that mirror each other using robocopy.  On a Mac it is more difficult, probably relies on external USB3 or Firewire disks. On a Mac you can use rsync (with appropriate command-line switches) to do mirroring similar to robocopy on a PC.

Every week or so mirror the master disk(s) onto external disks and put them into a datasafe, preferably in a different part of the house or some other safe location.

Migrating from one set of disks to another is a matter of mirroring from the old to the new, then upgrade the new disk as the master disk(s).  In my case, two 4Tb disks in a small 2-disk array became an external 8Tb disk to back up the new internal 8Tb disk(s).  Another neat trick is to find an old PC (or maybe a Mac Mini) and use it with something like Linux or Macos to create an SMB volume.   Such a volume can be mounted on both a PC and Mac, providing a means of copying files between them.

[piran]

@ianman Yes... and no. Depends on the risks, scenarios and logistics.

Given a few minutes I can load a Pelican tough case (waterproof) big enough to take both NASs, Leicas, phones & cards etc. It'll also float with enough buoyancy for itself ...and me for that matter👀 Another miniature Pelican can take a half a dozen portable SDD 'Swan Vestas' and in which my quickie nightly thing lives. It's more easily grabbed on-the-run. That's why I'm thinking of another 'meagre' 4TB portable SSD to bolster the offline or disaster option.

I'm never going to get 5ft of water here (it's on a windy ridge) but I accept your point.

From my POV I might compare how many stories abound that cite 5ft wades carrying IT storage media ...to the number of stories abounding from data leaks/outages afflicting 'other people's computers' aka 'clouds' around the world...

Just balancing risks. I believe I can protect my stuff here better than somebody else can be paid to do something allegedly similar on my behalf on their computers. The transfer logistics for remoting multiple tens of TBs are daunting. 

I used to keep a spare NAS in a Pelican up at my parents' house (as geographical diversity a whole county away) until they both passed away. 

@John Robinson Agreed... LOTS of disks. I normally go more for 4x than 2x at update time. I used to lose drives decades ago but stuff nowadays seems more robust or maybe I'm just more careful and buy vibration resilient stuff. My Linux box is massively over-engineered and I also occasionally put its own RAID spare capacity to good use.

A 'toaster slot' bench/engineering rig gets used to fire up those ancient cupboard bare HDDs from time to time. It keeps their bearings from rusting up, also to rewrite or update the storage as old HDDs can lose their magnetism unless the writes are renewed. Push comes to shove if their capacity is just too small or slow the rig gets used to securely wipe the flying rust.

Nice talking to you both, hope @jhluxton gets some ideas, still don't know what to do with all my pixels when I'm finally gone 😎

[jhluxton]

On 8/24/2021 at 8:25 PM, piran said:

Yes, unless something better comes along 😎 Disadvantages? More costly, more limited sizes, need TRIM events (can be automated), the portable ones can 'walk' more easily (office or public environment) as they are quite small. Be aware SSD flavours can be different - some are physically the same size as for a 2.5" flying rust HDD replacement - some are portable jobbies (usually smaller) - some are NVme installable 'cards' into computer extension bus. I have one of the latter in my PC running my scanned stuff. Also have a brilliantly portable one for quickie nightly events ...it's about the size of a skinny Swan Vesta oblong matchbox. Really small. REALLY quick. Also helpful w.r.t. ransomware as it spends 99% of the time unplugged/offline. My intranet-located NAS boxes doing the daily runs actually have bunches of big flying rust HDDs but the Synology boxes could've been fitted out with SSDs. YMMV

Yes I have spotted the cards when checking them out but I would want externals. Not much concern about them walking as I would only be using them at home and perhaps taking one away with me as a security backup. I generally take one back up of all my images whenever I go out. 

[piran]

@jhluxton Have had a more realistic rethink about my 'rushing out - almost - in panic' scenario. Have come to the conclusion that my long term planning is more of the 30min or an hour's 'warning to evacuate', say, while a UXB is investigated/detonated ...much like the recent one in Exeter (within view and earshot of my parents' old house that I used when they were alive). So, now on the lookout for a supplier of two 4TB SanDisks Extreme Pro 'Swan Vestas' - A & B - that'll nicely fit into that miniature Pelican. Much simpler less stressful set of instructions for my wife and also for me to carry out (literally). Have to downsize or condense what I want to keep into 4TB 😎 

One thing I glossed over in my earlier response to your question about possible negative points of SSDs. You probably already know it but I'll add it here JIC for anyone else. SSDs have a specific and finite 'lifetime' w.r.t. the overall number of writes supported. Complicated (look it up if you care or are curious)... but glibly they 'wear out' eventually in use. There are provisioning measures to even things out to slightly extend their maximal operational life (aka TRIM events). Variously capacity headroom is built in (depending on brand, costs etc) but wear out they will eventually (in use). However, for our purposes ie archiving, this really does not matter. 

[jhluxton]

On 8/29/2021 at 4:58 PM, piran said:

rious)... but glibly they 'wear out' eventually in use. There are provisioning measures to even things out to slightly extend their maximal operational life (aka TRIM events). Variously capacity headroom is built in (depending on brand, costs etc) but wear out they will eventually (in use). However, for our purposes ie archiving, this really does not matter. 

I thought having no moving parts they would be much less likely to wear out. Does anyone know there mean time to failure rate? Have they been tested? Or does that only apply if constantly updated?

[marchyman]

This report is 5 years old now, but in a paper published in 2016 

Quote

 This paper provides a large-scale field study covering many millions of drive days, ten different drive models, different flash technologies (MLC, eMLC, SLC) over 6 years of production use in Google’s data centers. We study a wide range of reliability characteristics and come to a number of unexpected conclusions. For example, raw bit error rates (RBER) grow at a much slower rate with wear-out than the exponential rate commonly assumed and, more importantly, they are not predictive of uncorrectable errors or other error modes. The widely used metric UBER (uncorrectable bit error rate) is not a meaningful metric, since we see no correlation between the number of reads and the number of uncorrectable errors. We see no evidence that higher-end SLC drives are more reliable than MLC drives within typical drive lifetimes. Comparing with traditional hard disk drives, flash drives have a significantly lower replacement rate in the field, however, they have a higher rate of uncorrectable errors.

 

Full paper at https://www.usenix.org/system/files/conference/fast16/fast16-papers-schroeder.pdf

[Advertisement]

Hallo,

Look here: Interesting Leica gear

[Harvard Kiwi]

I only shoot DNG and have three grades of photos:

1. Those I really think have potential

2. Some that may have some potential with significant work in post

3. Irrecoverable rubbish (including pictures taken with a lens cap on and those of my ex-wife 😝)

Those in #3 get purged on a regular basis as no amount of post will fix them and all they do is take up space. Those in #2 will usually exist on 1 or 2 drives, whereas those in #1 are stored eventually stored in 3 drives at 3 sites (my house, my mum's house, and my office filing cabinet) along with completed edits as JPEG and TIFF files.

I also carry my iPad Pro and Samsung T7 Touch 2TB drive with me wherever I go so that adds a couple more locations. My T7 and iPad both have some of my favourite images for personal viewing as well as more recent backups from my SD card. I do use a 512GB Sandisk Extreme Pro as my sole M10 card so I should really get a backup for that.

I would like to add a cloud based service to the mix but am unsure as to which one I should use. I would like something more accessible than Glacier, although there is a chance that I could add that as well seeing how the prices are so cheap. Any recommendations on the best accessible cloud storage?