Hi, I have an old Dell server I want to repurpose as a storage server and I decided to use zfs. I have read that gradual increases are best done with mirrored vdevs and only if I want to max out, I can choose Z2. What are your experiences in practice?
I like to go the gradual increase route because I cannot afford to maximize from the beginning. I like to start with ~10TB usable space. The server has at least 8 free drive bays. Should I go with a mirror as my first vdev or go 6x3TB with Z2? Ideally I want to be able to add more vdevs to the same pool in the future.
I am also curious whether it is a bad idea to have an uneven pool with mixed vdev types, the first being mirror and the second Z2. Will the vdevs eventually even out over time because zfs is a CoW filesystem?
One 5 drive z2 and add another when you decide to max it out? 10/2 is even. 10/5 is too. 10/3 is not.
@poinck I wouldn’t do a mirror and a Z2 in the same pool for fault tolerance reasons. Ideally, all vdevs in a pool should be able to tolerate the same number of faulted devices.
A pool of mirrors has three major advantages:
- Each disk can read independently, so you can get really high random read performance. In contrast, all disks in a raidz have to do the same thing at the same time. This can be a big deal for a pool of SSDs.
- Write performance of a pool correlates to the number of vdevs (with the exception of draid). A pool of mirrors typically gives you more vdevs, so it gets higher random write performance.
- If the pool has only ever contained mirrors, you can *remove* a mirror from the pool as long as you have enough capacity. The data will be evacuated to other vdevs.
ARC should prevent the disks from needing to be read often, minimizing advantage 1 at the cost of RAM. With transaction grouping, ZFS should only see random write load with sync writes, and a log vdev turns those into sequential writes on your pool devices, minimizing advantage 2.
With recent versions of ZFS, you can expand a raidz vdev a drive at a time. You could start a raidz2 with three drives (one drive of capacity), then add one drive at a time. There are some limitations with this, so research it further before relying on it.
Thx for your answer. I think, capacity is currently more important than speed. The idea of starting with a 3-drive-z2 sounds interesting but will cost more than a mirror with the same capacity. Maybe, I start with a mirror and wait how my users will fill it up to decide whether the next increase will be a z2-vdev with 3 drives, a second mirror or even a z2 with 6 drives.
But I see the point of just using one z2-vdev and adding new drives as needed. I would have more capacity in the end, but all drives need to have the same capacity. Or, if I start with a 10 TB mirrored vdev and add a second 20 TB mirror after 1 year, and 40 TB mirror after 2 more years or replace the first mirror with larger drives after many more years.
I fear, what likely will happen is that I will start with a single disk vdev, because my users don’t want to pay for redundancy. And only after some years, they see the value of it.
@poinck Once you add a raidz vdev to a pool, you can’t remove the mirror. You’d be stuck with the mismatch until the whole pool is destroyed and rebuilt. This isn’t a technical problem, but it makes maintenance more painful (which two drive bays are the mirror, again?). If you want raidz, you should really start with it to save yourself future pain.
You also can’t increase the fault tolerance of a raidz vdev. That is, you can’t take a raidz1 and add a drive to make it a raidz2.
I personally don’t trust single-fault-tolerant vdevs. Resilvering takes long enough I want double fault tolerance for all spinning disks bigger than maybe 4 TB. That means raidz2 or three-member mirrors.
