Jump to content

The Leica Forum uses cookies. Read the privacy statement for more info. To remove this message, please click the button to the right:    OK, understood.

Photo
- - - - -

Image storage: bigger and bigger storage drives?


  • Please log in to reply
31 replies to this topic

#1 LocalHero1953

LocalHero1953

    Erfahrener Benutzer

  • Premium Member
  • 3,892 posts
  • LocationCambridge
  • City / Ort:Cambridge

Posted 26 May 2017 - 20:39

Advertisement (Gone after free registration)

I'm facing the same problem everyone does from time to time: I'm running out of drive storage on my PC.

My PC has two 1Tb solid state internal drives. One is used as an image store, and it's where I keep my main Lightroom catalogue, and it's used by Adobe as a swap file store etc. The second drive is used for everything else: operating system, programs, documents etc.

I see I'm not far from filling the first drive with images. I don't want to overflow onto the other drive (but perhaps I should).

My images are stored in folders by year at the top level.

 

What's the answer? Just buy a 2Tb or 3Tb replacement disk and just keep going in the expectation that by the time I've filled it, the next size up will be available? Do you archive older images to external drives? Do you use a NAS, and if so, how big is it? How big is big enough for image storage? How do you manage access to your catalogue if it is spread across several drives?

 

These are not questions about backup and security. I have that sorted with a Crashplan subscription. This is about how to manage efficiently an ever growing collection of images (and growth is accelerating as sensor sizes grow).

 

Let me know how you do it!

 

(My first experience with hard disk storage was a 10k (or was it Mb?) drive that had to be removed from the Prime minicomputer cabinet every evening and locked in a desk drawer).


Edited by LocalHero1953, 26 May 2017 - 20:40.

Paul
=============================
Paul Ashley Photography
Paul Ashley on Instagram


#2 zeitz

zeitz

    Erfahrener Benutzer

  • Members
  • 626 posts

Posted 26 May 2017 - 21:12

You may be ready for a RAID.  I love my Drobo 5D with 5 x 4 TB drives, Thunderbolt connection and a solid state buffer drive.  Fast and big.  Others have had trouble with Drobos.


  • LocalHero1953 said thank you to this

#3 LocalHero1953

LocalHero1953

    Erfahrener Benutzer

  • Premium Member
  • 3,892 posts
  • LocationCambridge
  • City / Ort:Cambridge

Posted 26 May 2017 - 21:33

Aside from security/redundancy (which I agree is valuable), what's the advantage of a RAID?


Paul
=============================
Paul Ashley Photography
Paul Ashley on Instagram


#4 mikemgb

mikemgb

    Erfahrener Benutzer

  • Premium Member
  • 2,145 posts
  • City / Ort:St. Louis

Posted 26 May 2017 - 22:02

Aside from security/redundancy (which I agree is valuable), what's the advantage of a RAID?


RAID also allows you to configure multiple drives as a single drive. If you had a four drive system with 4 x 4TB drives installed you could configure it as a single 16TB drive if you wanted, or as two 8TB drives, one being the RAID duplicate of the other.

Edited by mikemgb, 26 May 2017 - 22:06.

  • LocalHero1953 said thank you to this

"Your life is a story, make it a good one"


#5 zeitz

zeitz

    Erfahrener Benutzer

  • Members
  • 626 posts

Posted 26 May 2017 - 23:37

The advantage of a Drobo RAID is that you can start with two drives and expand as your needs increase.  You can also swap smaller drives with larger drives when the smaller drives fill.  And the drives in the box don't have to match.  You can also mix HDDs and SSDs.  None of these are true for conventional RAIDs where all the drives have to match.  The downside is the cost of the Drobo device.

 

I can't imagine life without a RAID of some sort to get past the limits of the size of single HDDs.  I haven't looked at the price of conventional RAIDs recently, but I think they are very reasonably priced.

 

For those who don't know the RAID acronym, it stands for Redundant Array of Independent Drives.


  • LocalHero1953 said thank you to this

#6 Ko.Fe.

Ko.Fe.

    Erfahrener Benutzer

  • Members
  • 588 posts
  • LocationCanada, ON.

Posted 27 May 2017 - 00:08

I would get NAS (box from the store with software or DIY with 1 RU old file server and free OS).

 

https://www.newegg.c...2-207-_-Product

 

With some of the RAID configurations you could get to your files faster. Or you could get it more redundant. http://www.raid-calc...-reference.aspx


  • LocalHero1953 said thank you to this

#7 pico

pico

    Sponsoring Canine

  • Premium Member
  • 15,627 posts
  • LocationMinnesota
  • City / Ort:Winona

Posted 27 May 2017 - 00:33

I would get NAS (box from the store with software or DIY with 1 RU old file server and free OS).

 

https://www.newegg.c...2-207-_-Product

 

With some of the RAID configurations you could get to your files faster. Or you could get it more redundant. http://www.raid-calc...-reference.aspx

 

So what happens when the NAS box's controller smokes?

.



do_not_touch.png


#8 mikemgb

mikemgb

    Erfahrener Benutzer

  • Premium Member
  • 2,145 posts
  • City / Ort:St. Louis

Posted 27 May 2017 - 00:53

So what happens when the NAS box's controller smokes?

.

 

Mine did that a few months ago, I bought a new box and threw the old drives in it, everything is right back as it was.


"Your life is a story, make it a good one"


#9 zeitz

zeitz

    Erfahrener Benutzer

  • Members
  • 626 posts

Posted 27 May 2017 - 01:12

You still need to back everything up on a separate device, with off-site storage.


  • Berlinman said thank you to this

#10 pico

pico

    Sponsoring Canine

  • Premium Member
  • 15,627 posts
  • LocationMinnesota
  • City / Ort:Winona

Posted 27 May 2017 - 02:33

You still need to back everything up on a separate device, with off-site storage.

 

I roll my backups to the dog house out back.



do_not_touch.png


#11 LocalHero1953

LocalHero1953

    Erfahrener Benutzer

  • Premium Member
  • 3,892 posts
  • LocationCambridge
  • City / Ort:Cambridge

Posted 27 May 2017 - 09:07

You still need to back everything up on a separate device, with off-site storage.

As I posted earlier, I have a Crashplan subscription, which gives offsite backup, version control etc. I'm concerned here with efficient access to a large catalogue of images.

 

Backup is relevant though: I saw a note on the Drobo website saying that Crashplan will not back up a Drobo NAS, though it will work with a directly connected Drobo RAID. Is anyone else using Crashplan or Backblaze with a NAS or RAID?


Paul
=============================
Paul Ashley Photography
Paul Ashley on Instagram


#12 Exodies

Exodies

    Picturer

  • Premium Member
  • 3,461 posts
  • LocationLuxembourg

Posted 27 May 2017 - 09:36

A NAS served file is a file shared across a network; quite different from a locally connected disc. It would be very inefficient to do a crashplan type backup of a NAS. The NAS will be running an operating system, maybe there are backup clients available for that OS.

Just buy a 5TB USB disc and add it to your crashplan.
  • LocalHero1953 said thank you to this

#13 Exodies

Exodies

    Picturer

  • Premium Member
  • 3,461 posts
  • LocationLuxembourg

Posted 27 May 2017 - 09:45

Another thought, is your Lightroom catalogue over 1TB. Prolly not, so keep the catalogue on the SSD as it uses the speed best, and put the image and side-car files on a large USB disc. The image files are accessed one by one (e.g. When you enter development module) so the lower speed is not such a pain.

#14 LocalHero1953

LocalHero1953

    Erfahrener Benutzer

  • Premium Member
  • 3,892 posts
  • LocationCambridge
  • City / Ort:Cambridge

Posted 27 May 2017 - 11:27

Another thought, is your Lightroom catalogue over 1TB. Prolly not, so keep the catalogue on the SSD as it uses the speed best, and put the image and side-car files on a large USB disc. The image files are accessed one by one (e.g. When you enter development module) so the lower speed is not such a pain.

Hmm. I'll have to think about this. The raws and sidecar files, and any jpg/tif/psd files I have created from them, are kept together to make it easier for third party apps to read them (for the apocalyptic scenario of Adobe going out of business). The catalogue file (.lrcat) is 'only' 2.5Gb.


Paul
=============================
Paul Ashley Photography
Paul Ashley on Instagram


#15 Exodies

Exodies

    Picturer

  • Premium Member
  • 3,461 posts
  • LocationLuxembourg

Posted 27 May 2017 - 14:04

Yes, keep all images, dng, etc together but on a USB disc. Leave the catalogue where it is. Any other DAM program can then easily take over. If you decide to do this, move the images by dragging in Lightroom.
Oh, and keep a local backup as well as crashplan - crashplan is your disaster recovery, extra copy kept at home (time machine on Apple) is for casual recovery.

Edited by Exodies, 27 May 2017 - 14:10.

  • LocalHero1953 said thank you to this

#16 Steve Ash

Steve Ash

    Erfahrener Benutzer

  • Members
  • 2,394 posts

Posted 27 May 2017 - 14:19

I prefer a high-speed solution over a NAS-system: https://www.wdc.com/...DTB0120JSL-EESN


Gesendet von iPhone mit Tapatalk Pro
  • LocalHero1953 said thank you to this

#17 Luke_Miller

Luke_Miller

    Erfahrener Benutzer

  • Members
  • 813 posts
  • City / Ort:Rural Virginia

Posted 27 May 2017 - 16:30

My solution is an ever larger internal drive (currently 4 TB) for primary storage and a 5 disk 15 TB RAID for backup and network sharing.  I also have an external USB hard drive dock.  I use it to backup to bare hard drives.  Currently the ones I've outgrown.  Since my images are grouped by years I can put all but the last couple of years on one of the larger drives and then the remaining years on the smaller ones.  I store the drives in hard drive cases (similar to VHS tape cases) with labels.


  • LocalHero1953 said thank you to this

#18 Ko.Fe.

Ko.Fe.

    Erfahrener Benutzer

  • Members
  • 588 posts
  • LocationCanada, ON.

Posted 27 May 2017 - 18:39

So what happens when the NAS box's controller smokes?
.


You would have order and install another. If user is paranoid where are NAS with rediundant controllers. But I can't recall any of our company customers to have controller go out. I recommend, as usual, Google particular box manufacturer for reviews before buying it.

#19 LocalHero1953

LocalHero1953

    Erfahrener Benutzer

  • Premium Member
  • 3,892 posts
  • LocationCambridge
  • City / Ort:Cambridge

Posted 28 May 2017 - 07:36

Thanks for the advice, everyone. It looks like I shall go down the route of a RAID directly connected to my PC.

Drobo sounds attractive, with its mix and match drives. Has anyone had any problems with Drobos? I've seen Scott Kelby's blog, but the problems he had were several years old, and his recent change of heart seems more based on Drobo giving him personal attention, with no explanation of whether the system has technically changed.

 

And are there alternatives to Drobo, with similar flexibility?

As a PC user, Thunderbolt is not an option. Is USB 3.0 fast enough, or is there a faster connection?


Paul
=============================
Paul Ashley Photography
Paul Ashley on Instagram


#20 Luke_Miller

Luke_Miller

    Erfahrener Benutzer

  • Members
  • 813 posts
  • City / Ort:Rural Virginia

Posted 28 May 2017 - 14:00

And are there alternatives to Drobo, with similar flexibility?

As a PC user, Thunderbolt is not an option. Is USB 3.0 fast enough, or is there a faster connection?

I'm using a Synology 1515+.  It is network connection only, but on a GB Lan it is fast enough to use as primary storage with Lightroom and C1 and 10GB Ethernet is now an option.

 

My new Gigabyte X99 PC has onboard Thunderbolt 3 so I have that as a growth option.


Edited by Luke_Miller, 28 May 2017 - 14:01.

  • LocalHero1953 said thank you to this


0 user(s) are reading this topic