In my last post, I discussed the pitfalls of cloud storage for photo libraries and how the best, most reliable method for organizing your library for future reference is trusting in tried-and-true files and folders. But if you’re not depending on a cloud service and/or app like Apple Photos for your organization and storage, what is the best method for reliably storing photography and video captures, alongside a high performance workflow?
As a medium format photographer or even “only” 35mm, you might face storage challenges similar to mine. The size of my image library keeps growing. Single-shot RAW file captures with my 100MP Fujifilm GFX100 II are up to 130MB each with a single focus stacked image requiring from 3 to 30 frames. A single composition might require 0.5 to 4GB of captures, plus the finished stacked image. Or large volume for other reasons, which lands you in the same place.
Two key considerations stand out: (1) fast storage for performance-sensitive tasks, and (2) high-capacity overall storage. Sometimes these two can be the same thing, but often cost reasons preclude that due to the sheer volume of storage required—SSDs are still far more expensive that hard drives on a per-terabyte basis.
I covered my recommended backup method in a previous post, but in a nutshell: (1) backup every day at least once to at least one always-attached backup. (2) maintain at least three additional “offsite” backups stored safely away from the computer, never having those backups all together with the computer at once, (3) consider cloud-based backup as an adjunct if internet bandwidth makes it viable.
The discussion here is based on what works: for 15 years or so I have consulted for nearly a thousand clients about choosing and configuring a system for photography. You do not need technical knowledge to set up or use an outstanding system. What I cover here is the right solution for almost everyone.
Primary storage vs. backup, ideal system
“Primary storage” as used here means the original master copies of all your stuff. As compared to backups, which are copies of that primary storage. See Backup Best Practices: How to Make Fast Clone Backups on a Mac.
In my case, I have not graduated to a single large SSD, so I use two 8TB and one 16TB SSD. I would prefer a single 32TB SSD of course! But things evolve over time, and also cost money.
The ideal primary storage solution is a very large SSD e.g. a 16TB or 32TB OWC Thunderblade X8 (see pricing). But since that can be cost prohibitive, the next-best thing is a two-tier SSD + hard drive system.
Don’t Starve the GPU/GPU
A “balanced” system needs appropriate CPU and GPU processing power and memory, but without a fast SSD to deliver data, much of that potential is idled while the processing units wait for data. See The Ultimate Mac Buyer’s Guide Parts 1/2/3/4/5/6.
A prime example where SSD speed matters is the Adobe Lightroom catalog. Any operation such as accessing previews or using brushes or controls, setting metadata on many files, etc., tends to become one where response time matters, particularly as the catalog grows. Those interactive operations directly impact productivity whereas batch operations might or might not matter to you, since you can walk away for a break.
Drive speed starvation vs. ample
While the Lightroom catalog itself demands a fast SSD, RAW files stored on a reasonably fast hard drive have until recently had minimal impact on performance: for most operations, the time required to process the RAW file (eg. to generate a preview, export as JPG, etc.) had been much longer than the time needed to read it off disk. This is no longer true.
CPU and GPU speeds have greatly improved, particularly in chips like the 24-core Apple M2 Ultra in my 2023 Mac Pro and Mac Studio. The M3 Max chip is also blazingly fast. On my system, reading and converting and saving a 100-megapixel RAW file as JPG takes only 0.65 seconds.
If it takes only 2/3 second to process a 100MP RAW file, but the hard drive takes 1/2 second just to read the file (let alone write the JPG), the workflow is seriously degraded. And it is not just one CPU in use; there may be 10 or 14 or 24 CPU cores that all need data simultaneously e.g. bandwidth requirements approaching 3000 MB/sec — vs. 250MB/sec or so best case for a hard drive. A hard drive system becomes a severe bottleneck. Even a 4-drive RAID-0 stripe delivering 900MB/sec is a bottleneck.
A workflow involving hard drive storage is a recipe for stuttering performance.
A few years ago, it might have taken 5 to 10 seconds to render a large RAW file. Now it might only take a second or so. That changes the drive speed equation, depending on the workflow tasks. Hard drives have made only marginal speed gains over the past decade, making them an increasingly poor choice for high-performance computers when speed of workflow matters. Even a non-performance oriented SSD is now 4X the speed of the fastest hard drive.
Big SSD vs. 2-tier storage with migration
Before SSDs became so fast and so capacious, I recommended a hard drive RAID-0 or RAID-5 system for performance. Here in 2024, that’s a losing proposition; you are far better off with a fast SSD. And even a compact portable SSD like the OWC Envoy Pro Elektron (now available in up to 4TB) can easily outperform a 4-drive RAID-0 stripe of hard drives.
One large SSD (or two) are the ideal choice for primary storage. For those with very large capacity requirements, that might not fly for cost reasons. Thankfully, a 2-tier approach can get the job done just as well. Here’s how:
- Store current and recent projects on SSD (add SSD storage for all data if feasible).
- Migrate older projects off SSD to hard drive storage.
There are multiple benefits to this 2-tier approach: hard drives turned off mean no noise, something not just bothersome but also relevant for video editing. Power consumption is reduced. Hard drive life is extended by a far lower duty cycle. And if disconnected, the drives are immune to power surges or corruption while working.
For all these reasons, my primary storage has no hard drives involved at all. Hard drives are used for backups, and only for backups (I also have some SSD backup drives).
Recommended Solutions
Recommended products here are chosen for their performance and capacity and/or hitting a sweet spot. Many other options exist of lower capacity or performance and can be just as good for many users. See my SSD wishlist and storage wishlists.
A Big, Fast SSD
When spec’ing out your Mac, go with a 4TB or 8TB internal SSD, but remember to make a separate volume for your data.
When more space is needed, add either of these SSDs for king-of-the-hill performance. These are my two top picks.
Ultra-high performance USB4: 4TB or 8TB OWC Express 1M2. The Express 1M2 is a a USB4 external SSD that can be bought as a DIY, bring your own M.2 kit, or with the SSDs pre-installed. It’s a very flexible drive with speeds up to 3151 MB/s. You can read more about it here.
Ultra-high performance up to 32TB: OWC Thunderblade X8. It simply doesn’t get any better for an external SSD: Silent. Up to 32TB of capacity. Hyper-fast reading and writing. Fault tolerance. Professional build quality. Daisy-chainable. Compatible with Mac and PC. Read my review here. Pricing...
Hard Drive storage
*See also my external storage wishlist.
Many users are well served by single hard drive solutions for both main storage and backup.
No, you do not have to assemble it yourself, but this image shows what the unit looks like, should you for some reason want to install your own drive eg a 22TB drive until OWC offers it directly.
OWC Mercury Elite Pro: up to 20TB fanless single hard drive. Pricing…
Multi-drive enclosures
I like multi-drive enclosures of 4 or 8 drives for their flexibility. The OWC ThunderBay units can be run as single drives and/or combined with various flavors of RAID. You do not need to configure all drives as one large volume; while that is one option many users are well served using single drives and/or some RAID or non-RAID. How to do that is outside the scope of this article. The best flexibility comes with 8 drives, enough for single-drives and/or RAID-0 for performance or RAID-5 for fault tolerance and performance.
OWC Thunderbay 4: 4 hard drives each up to 20TB. Pricing…
OWC Thunderbay 8: 8 hard drives each up to 20TB. Pricing...
Conclusions
Workflow speed for today’s high-end Macs can easily be throttled by hard drive speed. One large and fast SSD is ideal, but a 2-tier SSD + hard drive system can get the same high-performance job done.
…
Need help working through your requirements? Consult with Lloyd before you potentially buy too much or too little, and see my Mac wishlist of current models, first pondering whether refurbished or used Macs might do the job for you.
View all posts by Lloyd Chambers… Lloyd’s photo web site is diglloyd.com, computers is MacPerformanceGuide.com, cycling and health are found at WindInMyFace.com, software tools including disk testing and data integrity validation at diglloydTools.com. Patreon page.
Hello,
Having recently migrated from PC to Mac after some 30 years Win 3.0 to Win 10, I am seeking some advice on the best way to go regarding external storage, my main reason for migrating was partly due to this problem.
I was using an External StarTech 2 Bay RAID housing with 2 x 3Tb Toshiba HHD, then about a year ago I started having issues with Windows, which could not be repaired, short of a full reinstallation, as I have a few friends and family who are Mac Users I decided to migrate with a 2019 19,1 iMac 500Gb 16Gb Ram, I have increased the RAM to 64Gb as I do a lot of photographic work. One thing that happened prior to migration was the Raid became corrupted, fortunately at some cost I was able to recover more than 95% of my files.
So I am wondering the best way to go rebuilding around the iMac. Over the years I have acquired a large number of various drives for storage, as follows,
HDDs 3 x 3Tb Toshiba & 1 x 1Tb Toshiba 3.5” .
2 x 1Tb Toshiba Canvio drives.
SSDs 2 x 2Tb Samsung in individual housings, a 1Tb and 500Gb drives.
My thoughts are to get a 4 Bay housing for the 4 Toshiba HDDs, set 2 of the 3Tb drives in Mirror Mode and see if the spare 3Tb & 1Tb can be set as individual or as one drive as a JBOD. The 2 Samsung SSDs then install them into the StarTech housing. The 1TB fix to the stand as working space second drive of iMac. Alternatively, I could use the 4Bay housing for 2 of the 3 Tb drives and the 2 Samsung SSDs setting up as 2 Mirrored drives, and the remaining Toshiba drives in the StarTech as a simple JBOD.
I sincerely hope you can provide some advice as the best path to take. I write this after reading your article for OWC April 30th this year.
Best regards,
Geoff Howard
Hi
Your inquiry suggests a number of solutions that stack up.
The starting point is to figure out how much storage you require for all your files + a bit of headroom.
Because you are dealing with images and videos, it is paramount to have a set-up that is indestructible as possible and recoverable as possible if something goes wrong. On top of that, speed may be important if doing large file video production.
You mention external enclosures. You should aim for a primary/working physical exclosure and a physical backup enclosure (plus an online backup service). The physical enclosures can be either 2 bay, 4 bay or more. The rule of thumb is 4 bay is better because they can be set to RAID 5 which is the more recoverable RAID configuration.
Now you could use TimeMachine as backup software or use SuperDuper or Carbon Clone to do automatic backups.
HDDs or SSDs can be put into those enclosures and there’s pros and cons for either. SSDs are fast and small, no longer overheat, but they are expensive and if they crash then the data is irrecoverable (unlike HDDs). But this is not a real problem if in a say 4 bay set up. The major online backup companies use HDDs in their server farms, but are gradually shifting to SDDs as their price goes down and reliability issues are being overcome. HDDs are still good and cheaper than ever, but not as reliable as some time ago. You just have to accept that SSDs and HDDs fail and simply have to be replaced.
If you are going to use external enclosures, then best to have the HDDs or SDDs the same size and specifications in the one enclosure.
If you go the enclosure/RAID approach with HDDs or SDDs, then to configure them, you can use Apple’s Disk Utility or software like SoftRAID. These do partitions, etc, as well. I use both approaches depending on the use of the enclosure. I’ve been using SoftRAID for a while because it seemed a good way of configuring HDDs and SDDs, but SoftRAID did give headaches. The SoftRAID issues are now largely fixed, but SoftRAID is now a subscription service and it is overpriced for what is on offer.
Hopefully others will respond to give you a range of options. Best of luck. There’s nothing better when everything is in place and all working nicely for a reasonable time.
IMO, hard drives should be left for backups, with SSDs used for current work.
Old and small drives 1/2/4/8TB should be eliminated, kept only as backups. If you still need hard drives, get ones that are at least 14TB.
Avoid RAID unless there is a GOOD reason for it such as fault tolerance. And remember that unless zero downtime is a requirement, more backups is always superior to the more complex and more expensive RAID setup.
If an SSD is the primary storage, then frequent backups of the SSD is required because a SSD can crash without giving warning signals unlike HDDs.
I am interested in your view in having an SSD RAID set up for the primary storage. An SSD RAID would give better performance and data security than a single SSD.
“an SSD can crash without giving warning signals” — Maybe you are thinking of cheap knockoff SSDs that claim a capacity but don’t actually have the chips, and then die suddenly because the flash isn’t actually there? That’s a scam, not an SSD behavior. Barring something like that, I disagree.
Anything can fail of course, but it’s all about the odds—and they are excellent odds for SSDs. And HDDs are far more likely to have problems, being mechanical in nature. However, we enjoy very high reliability with todays’ devices.
As far as a RAID-1 mirror (the only thing that makes sense in context of your comment), if you had 2 SSDs on hand, it is clearly inferior to a primary and a backup setup. A mirror offers zero protection against all sorts of user and software “oops” scenarios. Backups can be configured to run at high frequency if desired and/or made at the finishing of any important work on demand. Furthermore, the internal SSD is not viable for a RAID-1 mirror at all. Yet the internal SSD is for most users by far the best choice.
IMO today’s quality SSDs offer extremely high reliability. I would never even consider a RAID-1 mirror of SSDs—completely pointless, probabilistically speaking.
If you want a form of fault tolerance, you should go straight to the OWC ThunderBlade X8 and run it in RAID-4/5 mode. Otherwise, make and keep several “always attached” backups as discussed in recent articles.
https://eshop.macsales.com/blog/89210-owc-thunderblade-x8-review-performance-figures
“An SSD RAID would give better performance and data security than a single SSD.”
The “better performance” is not true. Meaning that if you require data security, that requires a RAID-1 mirror.
Unless those SSDs are on two separate busses you would be sharing write bandwidth, cutting performance in half (data has to be sent to both drives for writes).
On two separate busses, there is still overhead of doing two drives, plus the write cannot return until both drives have responded. This is guaranteed to be SLOWER than a single drive. For reads, you can in theory read from both for higher response (assuming separate bussses) eg a RAID-0 striping effect, but in practice I have not seen this work out; it’s pretty worthless in practice.
There’s plenty who disagree with your take. Bus speed is relatively a small part in the speed equation. With the exception of high level and large scale video production, bus speeds are neither here nor there for most image post production.
Data security is important if using SSDs given their crash without warning characteristic and their subsequent nil recovery consequence. Both RAID and backup regimes are important safeguards for the amateur and professional photographer.
No further comment.