RAID Levels
<< Disks and Interfaces | RAID Levels |
Local Disk Configurations >>
RAID is an acronym for Redundant Array of Independent (or Inexpensive) Disks,
and the term describes a group of multiple small hard disk drives organized to appear as one large drive.
RAID configurations improve system performance by increasing the speed at which you can read and write
your data. RAID arrays also add storage capacity and some RAID levels enhance your system reliability .
Typically, a RAID configuration consists of a number of hard disks, a specialized
controller, possibly one or more parity drives, and
hot
spare drives.
In the event of a drive failure, the parity drives in the array provide error detection and the
capability to recover the original information stored on the failed drive, making the drive
failure transparent to the end-user. Most RAID systems then use the hot spare drive(s) associated
with the RAID array as a replacement for the failed disk, rebuilding the lost data to the new
drive and returning the array to full performance more quickly.
- Hot spares always enhance RAID
reliability.
RAID technology is an integral building block for todays' high performance
storage systems, so it's important to understand some of the basics, and trade offs, associated
with the different RAID levels. Each RAID level can be characterized as having a greater or
less advantage over another RAID level in the following areas; performance, capacity, reliability,
cost, and computational cost (overhead).
The successful selection, configuration, and deployment of your storage
solution requires some knowledge of RAID technology. The following definitions should help
you understand the different levels of RAID and their inherent advantages and dis-advantages.
We've marked the commonly used RAID levels with a * in the title
for your convenience.
RAID Level:
*0
|
*1
|
0+1
|
3
|
4
|
*5
|
*6
|
*10
|
*50
|
*60
Click to expand all sections,
to expand a single
section, and
to open the reference as a pop up.
*RAID Level 0 (Data Striping)
Minimum Number of Disks: 2
A RAID 0 array splits your data evenly across two or more disks by using striping.
A stripe is simply a specified chunk of disk space of a certain size, such as
4,8,16, or 32 KBytes, that is allocated sequentially across the number of drives
you have, one stripe per disk. The first data block is allocated on the first disk,
the second data block on the second disk, and so on, until all disk have allocated
identical blocks of data. Your data is written to or read from these blocks in parallel,
increasing your data read/write performance by using all the disk drives in the RAID.
| Characteristic |
+/- |
Description |
| Performance |
++ |
Raid 0 can be quite fast. Data access speed can be at or near a linear progression of the
single disk members speed. If one disk allows you to write data at 65MB/s, two could
theoretically write at 130MB/s. Under optimal conditions, this progression can continue
until you reach the performance limits of the RAID controller, system bus, or software
RAID implementation.
|
| Capacity |
++ |
Very little capacity is lost to the RAID configuration, so data size is roughly equivalent
to the total number of disks times the smallest disk used.
|
| Reliability |
--- |
RAID level 0 has no additional reliability advantages over a single disk drive. In fact,
the more disks placed into a RAID 0 array, the lower the RAIDs reliability becomes.
Using RAID 0, the file system is distributed across all the disks, so any drive failure in
the array will cause the entire file system to be inaccessible, and all data to be lost.
Reliability is normally measured as mean time between failures, or MTBF, for a disk.
MTBF is measured by a running a large number of like components for a long time and seeing
how many fail during that time, deriving the statistical probability of failure for any
single unit within that time frame. So, the more disks that are placed into a RAID 0 array,
the more likely it becomes that one of the disks will fail, fatally effecting the entire
RAID 0 array. This characteristic makes a RAID 0 array totally unsuitable for long term
data storage.
|
| Overhead |
+++ |
RAID level 0 requires very little computational overhead, making it suitable for either
a hardware RAID controller solution, or a software RAID implementation.
|
| Usage:
RAID level 0 is only suitable to use for short term data storage or scratch space for transient
or easily reproduced data. It should never be used as long term data storage in any
HPC solution. The most common usage for RAID Level 0 in HPC is to deploy multiple disks
on compute nodes in a RAID 0 configuration for cheap, fast scratch space.
|
*RAID Level 1 (Data Mirroring)
Minimum Number of Disks: 2
A RAID 1 array duplicates your data across two or more disks, increasing data reliability geometrically over a single disk solution.
All data is written twice, normally on two identical disk drives. When one disk drive fails, all data is still
available on the other drive.
RAID 1 represents an easy and highly efficient solution for data security and system availability, but is
inherently expensive, as the capacity available is only half of the installed disk drives.
RAID 1 can also be implemented with a hot spare drive in many cases, further increasing data security and
system reliability.
| Characteristic |
+/- |
Description |
| Performance |
~ |
Raid 1 data read speed can be at or near a linear progression of the
single disk members speed if the implementation supports read duplexing.
Write performance is no better than the write speed of a single drive in the array, and sometimes a
bit worse.
|
| Capacity |
-- |
RAID 1 arrays lose at a minimum one half of the total disk space allocated for the array to the
mirroring of the data. If you had two 500 GB drives in a RAID 1 array (total 1000GB disk space), only
half the disk space installed, 500GB, would be available to format and mount as a file system.
|
| Reliability |
++ |
RAID level 1 is quite reliable. Should a member disk fail, your read/write access will continue unaffected.
You may lose a bit of read performance, depending on your implementation, but your data is safe. Adding
hot spares to a RAID 1 increases reliability.
|
| Overhead |
- |
RAID level 1 requires that a minimum of two writes be committed for every single write performed by the
operating system. However, no parity calculation is performed, so the operation can be performed with a
simple data copy, making RAID 1 suitable for a software RAID implementation.
|
| Usage:
In HPC, RAID level 1 is often used to mirror operating system disks on critical nodes. For instance, Aspen recommends
that all head nodes be configured with a RAID 1 for their operating system disk. This RAID can easily be implemented in
software or by using a hardware RAID controller.
|
RAID Level 0+1 (a mirror of stripes)
Minimum Number of Disks: 4
RAID level 0+1 combines the speed of RAID Level 0 (multiple spindle/disk striping) with the reliability
of RAID 1, but not in the most effective way possible. While quite similar to RAID 1+0 (RAID 10), it's not as reliable due to
the way it is implemented, so we recommend using RAID 0+1 only in very special circumstances.
RAID 0+1 is the first hybrid (combination of other RAID levels), or nested, RAID array type that we will discuss.
RAID 0+1 starts with two sets of striped disks, then mirrors the data across those striped sets. For example, lets say that
you have 10 disks. The first 5 disks are placed into a RAID 0 array, where data is striped across all 5 drives. We'll
call that array "array1". The second five disks are configured identically, another RAID 0 stripe across five disks. We'll
call the second array "array2". "array1" is then mirrored to "array2" to provide a RAID 0+1 array.
Both RAID 0+1 and RAID 10 protect you from a single disk failure, and provide very good performance. However, the single
disk failure that occurs in a RAID 0+1 takes out an entire RAID0 array, either "array1", or "array2", removing 5
drives from our example configuration, not one. Should another disk failure occur, and that disk fails in the operational
array, not the failed array, then your RAID become inoperative. RAID10 is more effective at protecting from multiple
disk failures, and we'll explain how in the RAID 10 section.
Although a RAID 0+1 array exhibits high performance, it is usually better to utilize a RAID 10 array instead.
RAID Level 3
Minimum Number of Disks: 3
RAID 3 stripes across all the data drives in the array byte by byte, then stores the parity information on a single drive.
This RAID level has high data transfer rates, but generally doesn't so well with simultaneous requests, making it
somewhat unsuitable for general purpose use.
RAID 3 is fast because data is sent to and from the disc drives in parallel, one I/O request at a time,
with parity data on the single dedicated parity drive. The disc spindles are synchronized to transfer data to all drives
simultaneously, giving a high degree of parallelism and making data transfers very fast.
If a single drive fails, data is still available by using the data on the working drives and the parity drive to reconstruct
the lost data. RAID-3 is typically used on image-manipulation processors and other applications where very high data-transfer
rates are needed. It is most efficient for large block transfers and is inefficient for short transactions with high I/O
request rates as is common in many HPC applications. For a given capacity, fewer drives are needed than for RAID-1, as only
a single drive for redundancy must be added to the data drives. However, the controller may be more complex and expensive.
RAID-3 is best for situations that require very fast data transfer rates or long data blocks.
Given those characteristics, RAID 3 is best used only in special circumstances for large block transfer at the highest
possible transfer rates, say, video editing or large image work. Even then other RAID levels can be tuned to give a better
balanced performance mix, so you won't see many RAID Level 3 arrays in common use.
RAID Level 4
Minimum Number of Disks: 3
RAID 4 works very much like RAID 0, utilizing block level striping. The data is striped across all
but one disk drive in the array. The controller calculates parity information, which is then stored on a separate
disk drive. Even when one disk drive fails, all data is still fully available. The missing data is recalculated from
the data still available and the parity information.
However, in situations with many small data block transfers, the parity disk drive becomes a throughput bottleneck.
With large data blocks, RAID 4 can show significantly improved performance, but its disadvantages usually outweigh
any advantages to the configuration, so RAID Level 4 isn't implemented very often.
*RAID Level 5 (Striped Data / Striped Parity)
Minimum Number of Disks: 3
In RAID Level 5, the parity data is striped across all disk drives in the array. RAID Level 5 delivers balanced
throughput even with small data transfers, so it is quite commonly used in many configurations, including HPC.
Response time of a RAID Level 5 array can be very good, and reliability is good as well. When one disk drive fails,
all data is still fully available, as the missing data is recalculated from the data still available and the parity
information. RAID 5 is particularly suitable for systems with medium to large capacity requirements, as it has a
good ratio of installed to available capacity.
| Characteristic |
+/- |
Description |
| Performance |
+ |
Raid Level 5 data write speed can be quite good if your block size isn't too large for your data mix. However, when
the average write size is smaller than the block size of the array, the parity update cycle can slow performance.
Using write-back cache can alleviate this issue. Highly concurrent random write performance is usually poor on a
RAID Level 5 array, so RAID 5 is not normally suitable for large multi-user databases where this write pattern is
common. Read performance on RAID Level 5 is quite good, however, almost as good as a RAID level 0 array, just a bit
worse because the read must skip over the parity data.
|
| Capacity |
++ |
RAID 5 arrays lose the capacity of one drive in the array to parity. If you have 6 2 TB drives configured as a RAID 5, 2 TB of space
will be used for parity calculation across all the drives, so your total usable space prior to file system creation would be
10 TB. This compares quite favorably to a RAID1, as half the space would be lost, giving you only 6 TB usable space before
file system creation. Given the enhanced read performance and the efficient use of disk space exhibited by RAID 5, this RAID level
is used quite often.
|
| Reliability |
+ |
RAID level 5 protects you from only a single disk failure. Should a member disk fail, your read/write access speed will
suffer slightly. Aspen highly recommends that you implement hot spares on all your RAID Level 5 arrays. We go into
quite a bit of detail about why we're so concerned about this in our
hot spare discussion in our
Detailed Buyers Reference.
|
| Overhead |
~ |
RAID level 5 should be tuned correctly for proper performance. The stripe size is critical. You don't want a configuration with many small
writes that are smaller than the capacity of a single stripe, as the RAID write performance will suffer greatly due to the parity update
overhead. You can use write back cache to combat this problem. As stated before, random write performance can be poor at high concurrency
levels because of the parity update overhead.
|
| Usage:
RAID level 5 is often used in HPC to service the large data partitions that model input and output data is stored on.
These partitions could be mounted as user home files systems, or as data partitions named according
to your organizational or administrative preference. Again, Aspen always recommends the use of hot spare disks when deploying
RAID 5 solutions.
|
*RAID Level 6
Minimum Number of Disks: 4
RAID 6 is similar to RAID 5, but it writes twice the parity information. So, it uses block level striping just as RAID 5
does, but it writes two parity blocks across the disks, instead of just one. RAID 6 has more overhead when writing than RAID 5, as
it has to write those 2 parity blocks for each transaction, and it utilizes twice the amount of space for parity storage.
The same example we used in a RAID 5 configuration, 6 2 TB drives, would yield only 8 TBs of usable space for file system creation instead
of the 10 TB RAID 5 yields. However, as disk storage capacities and the size of your RAID partitions go up, the opportunity for data loss
increases as well. RAID 6 provides more data protection during some pretty critical times, such as a RAID rebuild, protecting you against a
second disk error, perhaps the result of a bad block read on one of the operational drives, or even because an administrator pulled the wrong disk
when switching in the new drive (regrettably, not that uncommon of an occurrence).
Aspen recommends RAID Level 6 implementations with hot spares for all critical data storage.
| Characteristic |
+/- |
Description |
| Performance |
- |
Raid Level 6 data write speed can be slightly lower than an equivalent RAID Level 5 array due to the need to write two separate parity
blocks. Read performance is equivalent to RAID 5, which is very good.
|
| Capacity |
- |
RAID 6 arrays dedicate the capacity of two drives in the array to parity. If you have 6 2 TB drives configured as a RAID 6, 4 TB of space
(2 x 2 TB drives) will be used for parity calculation across all the drives, so your total usable space prior to file system creation would be
8 TB. This compares favorably to a RAID1, which loses half the total disk space, giving you only 6 TB usable space before
file system creation.
|
| Reliability |
++ |
RAID level 6 protects you from two disk failures. Should a member disk fail, your read/write access speed will
suffer slightly. Aspen recommends that you implement hot spares on all your RAID Level 6 arrays if your data is critical.
|
| Overhead |
- |
RAID level 6 writes each parity block twice, so write overhead is slightly higher. Newer controllers can compensate for this, sometimes
giving equivalent performance on RAID 5 or RAID 6. Read speed is equivalent
to RAID 5.
|
| Usage:
RAID level 6 is recommended for use on large data partitions which contain critical data. Even with RAID 6, Aspen recommends that you use dedicated hot spares.
|
*RAID Level 10 (a stripe of mirrors)
Minimum Number of Disks: 4
RAID 10, or RAID 1+0 as its sometimes called, is based on the combination of RAID 1 (Data Security) and RAID 0
(performance). Unlike RAID 4 and RAID 5, there is no need to calculate parity information. RAID 10 disk arrays offer superior
performance and data security. Like RAID 0, optimum performance is achieved in highly sequential load situations. However, like
RAID 1, 50% of the installed capacity is lost for redundancy.
| Characteristic |
+/- |
Description |
| Performance |
+++ |
Raid Level 10 is a hybrid RAID level, and consists of several RAID 1 level
arrays striped together. No parity information is needed, so read performance
can be increased easily by increasing the number of RAID 1 arrays involved
in the RAID 10 hybrid array. For example, if a RAID 10 array was composed
of 10 500 GB drives, the RAID 1 arrays could be 2 drive sets, allowing the
data be be striped across 5 RAID 1 sets. As no parity is calculated, read
performance will be as good as a 5 drive RAID 0 array, and writes are spread
across 5 RAID 1 sets as well, yielding superior performance for both
reads and writes. RAID 10, configured correctly, is the highest
performance RAID level.
|
| Capacity |
-- |
By definition, RAID 10 arrays lose half the installed disk space to the
mirrored set, so disk usage efficiency is as low as a RAID 1.
|
| Reliability |
+++ |
RAID Level 10 is very reliable, as it is made up of stripes of
RAID 1 mirrored sets. A RAID 1 can lose more than one disk, as long as
the failing disks are not in the same RAID 1 set. In some cases, vendor
implementations of RAID 10 accommodate this failure mode as well.
|
| Overhead |
+ |
RAID level 10 does not use parity calculations, so overhead is low. When writing
a stripe, two writes are performed to the RAID 1 set instead of only one, but
this can be done quite efficiently.
|
| Usage:
RAID level 10 is used for high performance RAID sets which contain critical data. RAID 10 is often
the highest performing RAID set which gives sufficient data retention reliability and protects against
multiple disk failures in the set. It is, however, expensive, as half of your total space is used to
mirror data.
|
*RAID Level 50 (a stripe of RAID 5 Arrays)
Minimum Number of Disks: 6
RAID Level 50 is a RAID 0 stripe across standard RAID 5 arrays. This RAID Level manages a good balance of
reliability and performance. Performance is increased by striping across the RAID 5 arrays, which improves
both read and write speed, while reliability is that of a RAID 5 array. One drive from each of the RAID 5
sets can fail without loss of data. Of course, that failed drive makes the entire RAID susceptible to a
catastrophic failure if another drive fails, so Aspen always recommends the use of
hot spare drives in all
RAID 50 arrays.
*RAID Level 60 (a stripe of RAID 6 Arrays)
Minimum Number of Disks: 8
RAID Level 60 is a RAID 0 stripe across RAID 6 arrays. Performance is increased by striping across the RAID 6 arrays
improving both read and write speed, while reliability is that of a RAID 6 array. two drives from each of the RAID 6
sets can fail without loss of data. As with RAID 50, Aspen always recommends the use of
hot spare drives in all
RAID 60 arrays.
Next we'll discuss some of the local disk options used in HPC solutions.
<< Disks and Interfaces | RAID Levels | Local Disk Configurations >>
drives.
In the event of a drive failure, the parity drives in the array provide error detection and the
capability to recover the original information stored on the failed drive, making the drive
failure transparent to the end-user. Most RAID systems then use the hot spare drive(s) associated
with the RAID array as a replacement for the failed disk, rebuilding the lost data to the new
drive and returning the array to full performance more quickly.
to expand all sections,
to expand a single
section, and 