Storage Upgrades: Special VDEV, Compression Changes, and Rebalancing
I’ve been making more improvements to my ZFS storage setup, focusing on performance and efficiency. The key updates include:
- Added a special vdev using 960GB Micron Pro 5300 SSDs
- Switched compression from ZSTD-9 to LZ4 for better performance
- Rebalancing the dataset to ensure even distribution across the pool
These changes should improve latency, space efficiency, and data distribution, making my storage more responsive for workloads.
Adding a Special VDEV for Metadata
I finally added a special vdev to my main ZFS pool using 960GB Micron Pro 5300 SSDs. The goal is to offload metadata and small files from my HDD-based vdevs, which improves performance in several ways:
✅ Faster directory lookups
✅ Lower latency for small files
✅ Improved ZFS snapshot performance
✅ Better overall IOPS on the pool
To add the special vdev, I ran:
zpool add my-zpool special mirror /dev/sdX /dev/sdY
Now, metadata-heavy operations like listing directories, accessing snapshots, and working with small files feel much snappier.
Switching Compression to LZ4
Previously, my pool was using ZSTD-9 for compression. While ZSTD-9 offers better compression ratios, it also comes with higher CPU overhead. Since my primary concern is real-time performance, I switched to LZ4, which provides:
✅ Lower CPU usage
✅ Faster read/write speeds
✅ Good compression without major performance trade-offs
To make the switch, I ran:
zfs set compression=lz4 my-zpool
LZ4 still provides decent compression but is much lighter on system resources, making it a better fit for my current workload.
Rebalancing the Dataset
Since I’ve been expanding the pool and adjusting the setup, I needed to rebalance the dataset to ensure even data distribution. Without rebalancing, older vdevs stay heavily utilized, while newer vdevs remain underused.
To fix this, I used ZFS In-Place Rebalancing from Markus Ressel:
Running the Rebalance
./zfs_inplace_rebalancing.py /mnt/my-zpool
This ensures data is evenly spread across all vdevs, optimizing performance and maximizing storage efficiency.
Formatted HDD Sizes
| Raw HDD Size (TB) | NTFS (TB) | ext4 (TB) | XFS (TB) | ZFS (TB) |
|---|---|---|---|---|
| 1 | 0.93 | 0.92 | 0.93 | 0.88 |
| 2 | 1.86 | 1.84 | 1.86 | 1.76 |
| 3 | 2.79 | 2.76 | 2.79 | 2.64 |
| 4 | 3.72 | 3.68 | 3.72 | 3.52 |
| 6 | 5.58 | 5.52 | 5.58 | 5.28 |
| 8 | 7.44 | 7.36 | 7.44 | 7.04 |
| 10 | 9.30 | 9.20 | 9.30 | 8.80 |
| 12 | 11.16 | 11.04 | 11.16 | 10.56 |
| 14 | 13.02 | 12.88 | 13.02 | 12.32 |
| 16 | 14.88 | 14.72 | 14.88 | 14.08 |
| 18 | 16.74 | 16.56 | 16.74 | 15.84 |
| 20 | 18.60 | 18.40 | 18.60 | 17.60 |
| 22 | 20.46 | 20.24 | 20.46 | 19.36 |
| 24 | 22.32 | 22.08 | 22.32 | 21.12 |
| 26 | 24.18 | 23.92 | 24.18 | 22.88 |
| 28 | 26.04 | 25.76 | 26.04 | 24.64 |
| 30 | 27.90 | 27.60 | 27.90 | 26.40 |
The Results So Far
After these updates, here’s what I’ve noticed:
✅ Snappier Metadata Performance – The special vdev makes a clear difference.
✅ Lower CPU Load – Switching to LZ4 compression reduces overhead.
✅ More Even Storage Utilization – The rebalancing process is improving data distribution.
These optimizations help future-proof my storage setup while keeping it efficient and responsive. I’ll continue monitoring performance and making adjustments as needed.
More updates coming soon!