A Comparison of Software RAID-5 With and Without Encryption
Linux Software RAID Performance Comparisons (2012)
The use of encrypted RAID-5 has historicaly halved throughput when
compared with unencrypted IO. Here we look at the performance
degradation imposed by cryptography on an otherwise unloaded system.
- LSI SAS9211-8i (SAS2008)
- 8 6Gbph ports
- PCIe 2.0
- Chipset: Fusion-MPT
- Linux driver: mpt2sas
- Cost: about $230 from amazon.com
- Configuration: JBOD
The Test System
- Motherboard: Supermicro MBD-H8DCL-IF-O
- Processors: Two 3.3GHz Opteron 4238 (Socket C32)
- RAM: 64GB 1600MHz DDR3 (PC3-12800)
- Slots: PCIe x8 (4000MB/s)
- Drives: Seagate Barracuda 7200 3000Gbytes ST3000DM001
- Drive cage: Supermicro CSE-M35T-1B 5-Bay Enclosure (fits in
three 5-inch chassis bays; sells for about $100-$120 from
- Debian Wheezy, Linux 3.2.0-3-amd64
The Test Matrix
- Read Percentage: 100% (pure read), 0% (pure write)
- Random Percentage: 100% (random)
- Thread counts: 1
- Small block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k,
1m, 2m, 4m
- Large block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k,
1m, 2m, 4m, 8m, 16m, 32m, 64m, 128m, 256m, 512m, 1024m
- Targets: All 5 driver were tested simultaneously, as well as
each drive individually.
- All small block I/Os are issued using the O_DIRECT flag.
- All large block I/Os use a sequence of 8KB blocks, followed
by an fsync, followed by a seek to the next "large block". This
simulates random I/O at small block sizes and sequential I/O at
large block sizes. By not using O_DIRECT and calling fsync
instead, the Linux block system is tested, which simulated
real-world NFS performance.
- All tests last 30 seconds.
The degradation in performance for encryption is about 5% for
small block random reads, rising to 20% for large sequential
writes. This is improved over earlier systems, which degraded by as
much as 50%, probably because of improved memory bandwith.
RAID-5 means IO to the raw /dev/md0 device created with
'mdadm -C /dev/md0 /dev/sd[bcdef]2 -n 5 -l 5'
Encrypted RAID-5 means IO to the cryptographic device /dev/mapper/r0,
created using 'cryptsetup -c aes-cbc-essiv:sha256 -s 256 -h
Ext4fs over Encrypted RAID-5 means IO to a 1TB file
created with 'dd if=/dev/zero of=/mnt/file bs=1024k count=1400000'
on an ext4fs file system created with 'mke2fs -t ext4 -E
stride=128,stripe-width=512 /dev/mapper/r0' that was mounted with
'mount /dev/mapper/r0 /mnt -o noatime,nodiratime'
Small Block Tests
Large Block Tests