Getting the Most Performance out of Spinning Hard Drives, Part 5: Validating Hard Drive Performance, Especially for RAID

The Previous article in this series discussed how to use 4 or 8 hard drives in a multi-drive enclosure, including RAID setups.

When working with 4/6/8 hard drives at a time, there is a good chance that one of those drives will be a dud whose performance may on average be similar but with oddball spikes of increased or decreased speed. Even by itself this is undesirable as it indicates a likely section of bad blacks that had to be mapped-out. High quality drives, particularly those for enterprise use, tend to have fewer such issues, but these can come at a stiff price premium.

Matching performance in a RAID

Using different brands or even different drive models in a RAID is not a good plan due to differing performance behaviors: transfer rates, seek times, latency, etc. The rest of the discussion assumes matched drives.

When used in a RAID, a variation of more than 4-5% in drive performance can significantly degrade RAID performance. With 3 or 4 or 7 of 8 drives having done their job, one laggard drive hangs up the entire transaction until it finishes—transactions are gated by the slowest drive. Any drive that deviates too much should be replaced with one that performs consistently with all the others.

There is no guarantee that any particular RAID vendor will detect this situation, since it requires a “burn in” period lasting 24 to 48 hours. In other words, it is costly to do so. But you can do it yourself.

Example: Well-matched hard drives

Below, this graph shows shows four 22TB hard drives tested simultaneously. In this case, even though they are four different models and two different brands, they all perform similarly.

However, the Seagate Iron Wolf Pro shows the highest read and write speeds by a little. Were I building a RAID, that’s what I would pick for the best performance, though it would not matter all that much. Note also that the WD Pro Gold has noticeably slower speed (the yellow and blue lines).

What you do not want to do is mix and match these drives, not just because of differeing transfer speeds, but because seek time and latency behavior might also differ, which could cause strange and unpredictable downward spikes in performance in a RAID.

**Four 22TB hard drives tested over entire capacity**

Example: One bad drive

The 6th drive crapped-out and could not complete the test, exhibiting erratic performance towards the end (orange line) utimately resulting in an I/O error, which the SoftRAID driver also reported as a failure.

DiskTester found the error:

Task FileReaderTask exited with error: disk error (I/O error) [-36]
Read 523 files totaling 1.89TB in 13258.3 seconds @ 150MB/sec
No verification failures

…and SoftRAID (highly recommended) confirmed it:

Testing the drives

As a rule, hard drives are not surface tested before being shipped. Bad drives can and do get shipped.

I am not aware of any tool that can emit data like that shown above other than my own diglloydTools DiskTester. Its fill-volume command writes 1000 files to fill the capacity of the drive. Benefits:

Graphable data of 1000 data points for write and 1000 for reach spanning the drive capacity, thus affording and at-a-glance assessment of the performance over the entire drive capacity for both reads and writes.
Oddball spikes or dropouts in performance stick out like a sore thumb.
Detects bad drives by forcing all sectors to be written.
Occasionally, a drive cannot complete the test; it’s defective out of the box, with an I/O error at some point during the test.
Detecting reliability problems with the drive connectivity eg cable problems, enclosure flakiness, etc.

Assuming the test completes for each drive, and assuming they are all within ±=3% of each other and there are no spikes or dropouts in the performance, you can rest assured that the set of drives is a good set for a RAID setup. If not, return/replace the outlier drive and test the replacement.

Simultaneous testing

Test each drive/volume in a separate Terminal window, so that they can run concurrently:

disktester fill-volume volume1 # in the 1st Terminal window
disktester fill-volume volume2 # in the 2nd Terminal window
…
disktester fill-volume volume8 # in the 8th Terminal window

The resulting data is graph ready using the supplied Excel template. Make the graph and scan it for anomolies.

High-End Solutions

While most user needs are well served by the OWC Thunderbay 4 and OWC Thunderbay 8 (discussed in previous article), high end professional video editing in a workgroup scenario leads to outstanding products like the OWC Jellyfish Tower and OWC R24 solutions. These solutions offer extreme performance (up to 100GBe), with capacities from 80 to 432TB.

For example, the OWC Jellyfish Tower has up to six (6) 10GBe connections with up to 240TB working space.

While preconfigured and pretested, the same performance testing discussed here can be used on such devices. It could for example be used to validate performance for 1/2/3/4/5/6 simultaneous clients. Hard drives can have contention with simultaneous users (head seeking), so it might be worthwhile to verify just how many clients can ‘hit’ the same storage simultaneously.

Conclusions

Most hard drives perform within ±= a few percent of each other. But occasionally a particular sample exhibits erratic and inconsistent performance, presumably from mapping out bad blocks.

View my external storage wishlist.

…

Need help working through your computing or photography system design? Consult with Lloyd before you potentially buy too much or too little, and see my Mac wishlist of current models, first pondering whether refurbished or used Macs might do the job for you.