A Comparison of Software RAID-5 With and Without Encryption

Linux Software RAID Performance Comparisons (2012)

The Problem

The use of encrypted RAID-5 has historicaly halved throughput when compared with unencrypted IO. Here we look at the performance degradation imposed by cryptography on an otherwise unloaded system.

The Controller

LSI SAS9211-8i (SAS2008)
- 8 6Gbph ports
- PCIe 2.0
- Chipset: Fusion-MPT
- Linux driver: mpt2sas
- Cost: about $230 from amazon.com
- Configuration: JBOD

The Test System

Motherboard: Supermicro MBD-H8DCL-IF-O
Processors: Two 3.3GHz Opteron 4238 (Socket C32)
RAM: 64GB 1600MHz DDR3 (PC3-12800)
Slots: PCIe x8 (4000MB/s)
Drives: Seagate Barracuda 7200 3000Gbytes ST3000DM001
Drive cage: Supermicro CSE-M35T-1B 5-Bay Enclosure (fits in three 5-inch chassis bays; sells for about $100-$120 from newegg.com)
Debian Wheezy, Linux 3.2.0-3-amd64

The Test Matrix

Read Percentage: 100% (pure read), 0% (pure write)
Random Percentage: 100% (random)
Thread counts: 1
Small block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1m, 2m, 4m
Large block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1m, 2m, 4m, 8m, 16m, 32m, 64m, 128m, 256m, 512m, 1024m
Targets: All 5 driver were tested simultaneously, as well as each drive individually.
All small block I/Os are issued using the O_DIRECT flag.
All large block I/Os use a sequence of 8KB blocks, followed by an fsync, followed by a seek to the next "large block". This simulates random I/O at small block sizes and sequential I/O at large block sizes. By not using O_DIRECT and calling fsync instead, the Linux block system is tested, which simulated real-world NFS performance.
All tests last 30 seconds.

Conclusion

The degradation in performance for encryption is about 5% for small block random reads, rising to 20% for large sequential writes. This is improved over earlier systems, which degraded by as much as 50%, probably because of improved memory bandwith.

Legend

RAID-5 means IO to the raw /dev/md0 device created with 'mdadm -C /dev/md0 /dev/sd[bcdef]2 -n 5 -l 5'

Encrypted RAID-5 means IO to the cryptographic device /dev/mapper/r0, created using 'cryptsetup -c aes-cbc-essiv:sha256 -s 256 -h sha256'

Ext4fs over Encrypted RAID-5 means IO to a 1TB file created with 'dd if=/dev/zero of=/mnt/file bs=1024k count=1400000' on an ext4fs file system created with 'mke2fs -t ext4 -E stride=128,stripe-width=512 /dev/mapper/r0' that was mounted with 'mount /dev/mapper/r0 /mnt -o noatime,nodiratime'