I've set up ZFS on OpenSUSE Tumbleweed, on my T430 server using 8x SAS ST6000NM0034 6TB 7.2K RPM drives. The ZFS pool is setup as RAIDZ-2 and the dataset has encryption.
I'm getting very slow writes to the pool, only about 33MB/s. Reads however are much faster at 376MB/s (though still slower than I would have expected).
No significant CPU usage during writes to the pool, or excessive memory usage. The system has 26 physical cores and 192GB ram, so CPU and ram should not be the bottleneck.
ZFS properties:
workstation:/media_storage/photos # zfs get all media_storage/photosNAME PROPERTY VALUE SOURCEmedia_storage/photos type filesystem -media_storage/photos creation Sat Feb 15 16:41 2025 -media_storage/photos used 27.6G -media_storage/photos available 30.9T -media_storage/photos referenced 27.6G -media_storage/photos compressratio 1.01x -media_storage/photos mounted yes -media_storage/photos quota none defaultmedia_storage/photos reservation none defaultmedia_storage/photos recordsize 128K defaultmedia_storage/photos mountpoint /media_storage/photos defaultmedia_storage/photos sharenfs off defaultmedia_storage/photos checksum on defaultmedia_storage/photos compression lz4 inherited from media_storagemedia_storage/photos atime on defaultmedia_storage/photos devices on defaultmedia_storage/photos exec on defaultmedia_storage/photos setuid on defaultmedia_storage/photos readonly off defaultmedia_storage/photos zoned off defaultmedia_storage/photos snapdir hidden defaultmedia_storage/photos aclmode discard defaultmedia_storage/photos aclinherit restricted defaultmedia_storage/photos createtxg 220 -media_storage/photos canmount on defaultmedia_storage/photos xattr on defaultmedia_storage/photos copies 1 defaultmedia_storage/photos version 5 -media_storage/photos utf8only off -media_storage/photos normalization none -media_storage/photos casesensitivity sensitive -media_storage/photos vscan off defaultmedia_storage/photos nbmand off defaultmedia_storage/photos sharesmb off defaultmedia_storage/photos refquota none defaultmedia_storage/photos refreservation none defaultmedia_storage/photos guid 7117054581706915696 -media_storage/photos primarycache all defaultmedia_storage/photos secondarycache all defaultmedia_storage/photos usedbysnapshots 0B -media_storage/photos usedbydataset 27.6G -media_storage/photos usedbychildren 0B -media_storage/photos usedbyrefreservation 0B -media_storage/photos logbias latency defaultmedia_storage/photos objsetid 259 -media_storage/photos dedup off defaultmedia_storage/photos mlslabel none defaultmedia_storage/photos sync disabled inherited from media_storagemedia_storage/photos dnodesize legacy defaultmedia_storage/photos refcompressratio 1.01x -media_storage/photos written 27.6G -media_storage/photos logicalused 27.9G -media_storage/photos logicalreferenced 27.9G -media_storage/photos volmode default defaultmedia_storage/photos filesystem_limit none defaultmedia_storage/photos snapshot_limit none defaultmedia_storage/photos filesystem_count none defaultmedia_storage/photos snapshot_count none defaultmedia_storage/photos snapdev hidden defaultmedia_storage/photos acltype off defaultmedia_storage/photos context none defaultmedia_storage/photos fscontext none defaultmedia_storage/photos defcontext none defaultmedia_storage/photos rootcontext none defaultmedia_storage/photos relatime on defaultmedia_storage/photos redundant_metadata all defaultmedia_storage/photos overlay on defaultmedia_storage/photos encryption aes-256-gcm -media_storage/photos keylocation prompt localmedia_storage/photos keyformat passphrase -media_storage/photos pbkdf2iters 350000 -media_storage/photos encryptionroot media_storage/photos -media_storage/photos keystatus available -media_storage/photos special_small_blocks 0 defaultmedia_storage/photos prefetch all defaultworkstation:/media_storage/photos # While writing from /dev/random to a 4GB file:
workstation:/home/josh # zpool iostat -vly 30 1 capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim rebuildpool alloc free read write read write read write read write read write read write wait wait wait-------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----media_storage 25.9G 43.6T 0 471 0 33.7M - 87ms - 75ms - 768ns - 12ms - - - raidz2-0 25.9G 43.6T 0 471 0 33.7M - 87ms - 75ms - 768ns - 12ms - - - wwn-0x5000c5008e4e6d6b - - 0 60 0 4.23M - 86ms - 74ms - 960ns - 11ms - - - wwn-0x5000c5008e6057fb - - 0 58 0 4.23M - 85ms - 73ms - 768ns - 12ms - - - wwn-0x5000c5008e605d47 - - 0 61 0 4.21M - 84ms - 71ms - 672ns - 12ms - - - wwn-0x5000c5008e6114f7 - - 0 55 0 4.20M - 101ms - 87ms - 768ns - 13ms - - - wwn-0x5000c5008e64f5d3 - - 0 57 0 4.23M - 95ms - 83ms - 768ns - 12ms - - - wwn-0x5000c5008e65014b - - 0 59 0 4.18M - 85ms - 74ms - 672ns - 11ms - - - wwn-0x5000c5008e69dea7 - - 0 59 0 4.20M - 83ms - 72ms - 768ns - 11ms - - - wwn-0x5000c5008e69e17f - - 0 58 0 4.20M - 82ms - 71ms - 768ns - 11ms - - --------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----workstation:/home/josh #While reading from the same file (cache flushed first):
workstation:/home/josh # echo 0 > /sys/module/zfs/parameters/zfs_arc_shrinker_limitworkstation:/home/josh # echo 3 > /proc/sys/vm/drop_cachesworkstation:/home/josh # zpool iostat -vly 5 1 capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim rebuildpool alloc free read write read write read write read write read write read write wait wait wait-------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----media_storage 25.1G 43.6T 14.9K 0 376M 0 1ms - 596us - 201ms - 593us - - - - raidz2-0 25.1G 43.6T 14.9K 0 376M 0 1ms - 596us - 201ms - 593us - - - - wwn-0x5000c5008e4e6d6b - - 1.87K 0 46.8M 0 1ms - 615us - 201ms - 582us - - - - wwn-0x5000c5008e6057fb - - 1.97K 0 45.9M 0 747us - 412us - - - 324us - - - - wwn-0x5000c5008e605d47 - - 1.82K 0 47.5M 0 1ms - 623us - - - 491us - - - - wwn-0x5000c5008e6114f7 - - 1.79K 0 47.9M 0 1ms - 709us - - - 831us - - - - wwn-0x5000c5008e64f5d3 - - 1.95K 0 46.3M 0 922us - 491us - - - 444us - - - - wwn-0x5000c5008e65014b - - 1.81K 0 47.7M 0 1ms - 686us - - - 953us - - - - wwn-0x5000c5008e69dea7 - - 1.83K 0 47.0M 0 1ms - 603us - 201ms - 527us - - - - wwn-0x5000c5008e69e17f - - 1.86K 0 47.2M 0 1ms - 650us - - - 632us - - - --------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----workstation:/home/josh #Any ideas of what might be causing the bottleneck in speed?