A Comparison of Various Ext4fs Options

The Problem

The Controller

• LSI SAS9211-8i (SAS2008)
  • 8 6Gbph ports
  • PCIe 2.0
  • Chipset: Fusion-MPT
  • Linux driver: mpt2sas
  • Cost: about $230 from amazon.com
  • Configuration: JBOD

The Test System

• Motherboard: Supermicro MBD-H8DCL-IF-O
• Processors: Two 3.3GHz Opteron 4238 (Socket C32)
• RAM: 64GB 1600MHz DDR3 (PC3-12800)
• Slots: PCIe x8 (4000MB/s)
• Drives: Seagate Barracuda 7200 3000Gbytes ST3000DM001
• Drive cage: Supermicro CSE-M35T-1B 5-Bay Enclosure (fits in three 5-inch chassis bays; sells for about $100-$120 from newegg.com)
• Debian Wheezy, Linux 3.2.0-3-amd64

The Test Matrix

• Read Percentage: 100% (pure read), 0% (pure write)
• Random Percentage: 100% (random)
• Thread counts: 1
• Small block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1m, 2m, 4m
• Large block sizes: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1m, 2m, 4m, 8m, 16m, 32m, 64m, 128m, 256m, 512m, 1024m
• Targets: All 5 driver were tested simultaneously, as well as each drive individually.
• All small block I/Os are issued using the O_DIRECT flag.
• All large block I/Os use a sequence of 8KB blocks, followed by an fsync, followed by a seek to the next "large block". This simulates random I/O at small block sizes and sequential I/O at large block sizes. By not using O_DIRECT and calling fsync instead, the Linux block system is tested, which simulated real-world NFS performance.
• All tests last 30 seconds.

Conclusion

First, we notice that reads are uneffected by these options, which are only expected to have an impact on writes.

For small block wirtes, only the lack of a barrier helps performance, and then raises it above the underlying encrypted RAID-5 performance documented elsewhere in this study. This suggests caching and the possibility of severe data loss in the face of a power failure. This option is not recommended.

For large block writes, placing the journal on an SSD partition, with or without data=writeback, provide the same level of performance as the elimination of barriers. This is an impressive result, since data=order and barrier=1 provide the best data integrity in the face of a power failure.

Recommendation: Place the journal on an SSD partition but leave other defaults in place (i.e., data=ordered,barrier=1). This has some security/privacy risk since the SSD is not encrypted, so up to 400MB of unencrypted metadata would be in the journal at any time.

Legend

Ext4fs over Encrypted RAID-5 means IO to a 1TB file created with 'dd if=/dev/zero of=/mnt/file bs=1024k count=1400000' on an ext4fs file system created with 'mke2fs -t ext4 -E stride=128,stripe-width=512,lazy_itable_init=0 /dev/mapper/r0' that was mounted with 'mount /dev/mapper/r0 /mnt -o noatime,nodiratime'

Ext4fs over Encrypted RAID-5 with Writeback was mounted with 'mount /dev/mapper/r0 /mnt -o noatime,nodiratime,data=writeback,nobh' (nobh was deprecated and ignored on the test system's kernel).

Ext4fs over Encrypted RAID-5 with Writeback and No Barrier was mounted with 'mount /dev/mapper/r0 /mnt -o noatime,nodiratime,data=writeback,nobh,barrier=0' (nobh was deprecated and ignored on the test system's kernel).

Ext4fs over Encrypted RAID-5 with Journal Device used an SSD partition for a journal, created with 'mke2fs -O journal_dev /dev/sda3' and 'mke2fs -t ext4 -J device=/dev/sda3 -E stride=128,stripe-width=512,lazy_itable_init=0 /dev/mapper/r0' and mounted with 'mount /dev/mapper/r0 /mnt -o noatime,nodiratime'.

Ext4fs over Encrypted RAID-5 with Journal Device and Writeback used the same journal device and writeback mount options documented above.

The lazy_itable_init=0 option is requried to prevent the background creation fo the itable from impacting test results.

Small Block Tests

Large Block Tests