Skip to content

Swapping to the SuperMicro JBOD, Adding a Special VDEV, and Rebalancing ZFS

Over the past few weeks, I’ve been making significant changes to my ZFS storage setup. I replaced my EMC VNX 25-bay JBOD with a SuperMicro CSE-216BE1-741JBOD, added a special vdev using 960GB Micron Pro 5300 SSDs, and now I’m rebalancing my ZFS pool to ensure an even data distribution.

Each of these upgrades has brought noticeable improvements in performance, flexibility, and noise reduction, and I’m excited to share the results.

Moving to the SuperMicro JBOD

Previously, I was using an EMC VNX JBOD, but it had several downsides:

  • Excessive Noise – The fans ran at full speed constantly, making it unsuitable for my homelab.
  • Limited Drive Compatibility – Some disks wouldn’t report their full capacity or even show up properly.
  • Noisy Power Supplies – Even with adjustments, the noise level was unbearable.

I decided to replace it with a SuperMicro CSE-216BE1-741JBOD, which provides:

  • Full 12Gb/s SAS Support – Works perfectly with my SAS3 SSDs.
  • Improved Drive Compatibility – Recognizes all SATA and SAS drives without issue.
  • IPMI-Based Fan Control – Allows me to fine-tune cooling and noise levels.

Adjusting Fan Speeds with IPMI

By default, the SuperMicro JBOD fans ran much louder than necessary. Fortunately, I was able to lower the fan speed to 10% using IPMI:

ipmitool -I lanplus -H 10.1.10.27 -U ADMIN -P 'Password' raw 0x30 0x70 0x66 0x01 0x00 0x0A

After applying this setting, the noise level dropped significantly, and temperatures remain stable.

Adding a Special VDEV for Metadata

With the SuperMicro JBOD in place, I decided to add a special vdev to my main ZFS pool. This helps improve metadata performance, especially for:

  • Directory lookups
  • Snapshots & clones
  • Small file access
  • ZFS deduplication (if enabled in the future)

For this, I’m using 960GB Micron Pro 5300 SSDs, which provide low latency and high endurance. Adding them as a special vdev moves metadata and small files off my spinning HDDs, significantly improving IOPS and latency.

To add the special vdev, I ran:

zpool add my-zpool special mirror /dev/sdX /dev/sdY

Now, all metadata and small files are automatically stored on the SSDs, keeping the pool responsive even under heavy loads.

Rebalancing My ZFS Pool

Since I’ve expanded my pool with new drives, I also need to rebalance data across all vdevs. By default, ZFS doesn’t automatically redistribute data when you add new drives, meaning older vdevs remain full while new ones stay underutilized.

Instead of manually copying data, I’m using ZFS In-Place Rebalancing by Markus Ressel, which redistributes data without destroying the pool.

Installing zfs-inplace-rebalancing

To install the script:

curl -O https://raw.githubusercontent.com/markusressel/zfs-inplace-rebalancing/master/zfs-inplace-rebalancing.sh
chmod +x zfs_inplace_rebalancing.sh

Running the Rebalancing Process

Once installed, I ran:

./zfs_inplace_rebalancing.sh /mnt/my-zpool

This process gradually redistributes data across all vdevs, ensuring even usage of old and new drives.

The Results So Far

After implementing these changes, here’s what I’ve noticed:

✅ Much Quieter JBOD – Fan noise is no longer an issue after IPMI tuning.
✅ Faster Metadata Operations – The special vdev makes a huge difference in responsiveness.
✅ More Even Storage Utilization – Rebalancing ensures all drives are used efficiently.

These upgrades have made a huge impact on my homelab storage performance. Now, I’ll be monitoring performance over the next few weeks and fine-tuning settings as needed.

More updates coming soon! I should be getting the drives in a few days.