You’ve got three nodes, empty storage, and a decision to make
Longhorn wins for a straightforward 3-node k3s cluster where backups matter, and OpenEBS wins the moment you want to mix local, replicated, and legacy storage engines per workload.
So you’ve finally got k3s running on that home lab cluster. It’s humming along, pods are deploying, and life is good. Then you realize: stateful workloads need persistent storage, and k3s doesn’t ship with one out of the box. Time to pick between Longhorn and OpenEBS.
This is not a “one-size-fits-all” story. Both are solid choices, but they solve the problem from different angles. Longhorn is the Swiss Army knife that does 80% of what you need with minimal fuss. OpenEBS is the toolbox where you can pick exactly which tool you want for each job. The deciding question is narrow: do you want one engine that makes every decision for you, or per-workload control over replication that you configure and maintain yourself?
Longhorn: Simple, opinionated, just works
Longhorn is Rancher’s answer to “what if we made distributed block storage boring?” It’s a single operator that manages everything: replicas, snapshots, backups, failover. You install it, point it at your nodes, and it handles the rest.
Longhorn is the car bought with every option pre-tuned. You get a good experience out of the box because someone already made all the decisions for you.
How Longhorn works
Longhorn runs a manager pod on each node, plus one instance-manager pod per node. When you claim a PersistentVolume, Longhorn starts an engine process for that volume inside the instance manager on the node where the pod runs, and replica processes inside the instance managers on the nodes holding the copies. The replicas are written synchronously; if a node dies, Longhorn restarts the engine and keeps serving from the surviving replicas.
The thing to get straight, because a lot of older write-ups have it wrong: engines and replicas are processes inside a shared per-node pod, not one pod each. For a 3-node cluster with five 10GB volumes at Longhorn’s default of 3 replicas, you get five engine processes and fifteen replica processes spread across three instance-manager pods. kubectl get pods -n longhorn-system will not show you a pod per volume.
Installing Longhorn on k3s
Before the Helm command: Longhorn drives its block devices through the host’s iSCSI stack, so every node needs open-iscsi installed with iscsid running. Miss this and your first PVC hangs in Pending with an attach error that does not mention iSCSI anywhere. On Debian and Ubuntu nodes:
sudo apt install -y open-iscsi nfs-commonsudo systemctl enable --now iscsidnfs-common is only needed if you want ReadWriteMany volumes, which Longhorn serves over NFSv4.
# Add the Longhorn Helm repohelm repo add longhorn https://charts.longhorn.iohelm repo update
# Install with k3s-friendly defaultshelm install longhorn longhorn/longhorn \ --namespace longhorn-system \ --create-namespace \ --set defaultSettings.defaultReplicaCount=2 \ --set defaultSettings.storageMinimalAvailablePercentage=10 \ --set defaultSettings.storageOverProvisioningPercentage=200The key settings:
defaultReplicaCount=2: For a 3-node cluster, 2 replicas is sane. Longhorn defaults to 3, which leaves you only a third of your raw capacity. Two leaves you half.storageMinimalAvailablePercentage=10: Longhorn won’t accept writes if free space drops below this.storageOverProvisioningPercentage=200: Longhorn lets you claim 2x your actual disk space. Reasonable for home lab; adjust down if you’re paranoid.
StorageClass example
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: longhornprovisioner: driver.longhorn.ioallowVolumeExpansion: trueparameters: numberOfReplicas: "2" staleReplicaTimeout: "2880" # 2 days fromBackup: ""volumeBindingMode: WaitForFirstConsumerSet WaitForFirstConsumer so replicas land on nodes where the pod actually runs. No orphaned replicas in corners of your cluster.
What Longhorn does well
Snapshots & backups: Built-in. Take a snapshot with a kubectl command, back it up to S3 or NFS, restore it anywhere. Your morning self-hosted Nextcloud backup story is trivial.
Failover: Automatic. Node dies, Longhorn’s controller switches to a healthy replica. Your stateful pod reschedules, finds new storage, keeps going.
UI: Longhorn has a dashboard on port 80 of the longhorn-frontend service. You can see your volumes, snapshots, node resources, replica placement. Not fancy, but it works.
Simplicity: One operator, and in daily use one CRD you actually touch: Volume. Longhorn does register about 22 CRDs behind it (engines, replicas, backuptargets, recurringjobs and friends), but the operator drives those for you. No plugins to choose, no modes to understand.
The catch
Longhorn’s replica model costs disk space. A 3-replica volume eats 3x the data across your cluster. For a 3-node lab with 500GB per node, that 1.5TB of raw disk gives you only ~500GB of usable storage at 3 replicas. Not fun.
It also reserves CPU up front. The guaranteedInstanceManagerCPU setting defaults to 12, meaning 12% of each node’s allocatable CPU is requested by the instance-manager pod whether or not any volume is attached. On a 4-core mini PC that is roughly half a core gone before you schedule anything. You can lower it, but at zero Longhorn drops the CPU request entirely and your storage starts competing with everything else on the node.
One more: rebuilding replicas after a node comes back online can saturate your network. A node that was hosting a replica of ten volumes rejoins and Longhorn starts resyncing all ten at once. Your cluster feels slow for a while. concurrentReplicaRebuildPerNodeLimit is the knob that caps it.
OpenEBS: Modular, flexible, more levers to pull
OpenEBS is a storage framework. It doesn’t come with one opinion; it comes with several, and you pick which engine fits each volume. Want a volume pinned to one node’s disk? Use LocalPV. Want replication across nodes? Use Mayastor. Want LVM or ZFS underneath instead of a plain directory? Those are separate local engines too.
This is like buying a car and then choosing your own suspension, transmission, and engine separately. Powerful if you know what you’re doing; overwhelming if you just want to drive.
The OpenEBS engines
LocalPV StatefulSet / LocalPV HostPath: Volumes on the local node’s disk. No replication, no distribution. Fast as hell, but if the node dies, so does your data. Good for temporary storage or dev databases where you’re cool with that trade-off.
cStor (legacy, archived): A distributed block storage engine. Data is replicated across pools on different nodes. Similar to Longhorn’s approach, but with more knobs. As of OpenEBS 4.0 it’s been moved to legacy status, it still works and lives on in the openebs-archive org, but it’s no longer part of the main OpenEBS Helm chart. The project pushed everyone toward Mayastor instead.
Mayastor (Replicated PV): OpenEBS’s flagship replicated engine these days. A Rust engine built on SPDK and designed for fast NVMe storage. Lower latency than cStor, and by the current 4.6 release (August 2026) it is the default, actively maintained path for replicated storage rather than the experimental new kid it used to be.
LocalPV LVM / LocalPV ZFS: Same “stays on one node” model, but the volume is an LVM logical volume or a ZFS dataset instead of a directory, which buys you real snapshots and thin provisioning from the layer below. Both ship in the 4.x chart and both are enabled by default.
For a 3-node k3s cluster, LocalPV HostPath (dev/temp) or Mayastor / Replicated PV (stateful workloads) are the typical picks today. cStor is still an option if you’re maintaining an older deployment, but for anything new you’d reach for Mayastor.
Installing OpenEBS on k3s
If you want Mayastor, do the node prep first. The io-engine DaemonSet carries a nodeSelector of openebs.io/engine: mayastor, and it wants 2GiB of 2MiB hugepages plus the nvme_tcp kernel module on every node that runs it:
# Hugepages: 1024 x 2MiB = 2GiB, which is what the io-engine requestsecho 'vm.nr_hugepages = 1024' | sudo tee /etc/sysctl.d/99-mayastor.confsudo sysctl -p /etc/sysctl.d/99-mayastor.conf
# nvme_tcp is how the CSI node mounts a volume over the networksudo modprobe nvme_tcpecho nvme_tcp | sudo tee /etc/modules-load.d/mayastor.conf
# Tell OpenEBS this node is allowed to run the io-enginekubectl label node <node-name> openebs.io/engine=mayastorSkip the label and you get a healthy-looking install with zero io-engine pods, because the DaemonSet has nowhere to schedule. Skip nvme_tcp and the csi-node init container loops forever printing nvme_tcp module not loaded.... Neither failure says “you forgot a prerequisite” in any obvious place.
# Add the OpenEBS Helm repo. Note the URL.helm repo add openebs https://openebs.github.io/openebshelm repo update
# Install OpenEBS (local engines + replicated Mayastor)helm install openebs openebs/openebs \ --namespace openebs \ --create-namespaceThe repo URL matters. The old https://openebs.github.io/charts repo is still up and still serves a chart called openebs, but it stops at 3.10.0. Point Helm at it and you will install the 3.x layout, where the values keys below do not exist and cStor is still in the box. The 4.x chart lives at https://openebs.github.io/openebs.
Watch the value keys too. There is no engines.local.enabled: the chart splits local storage into engines.local.hostpath, engines.local.lvm, engines.local.zfs and engines.local.rawfile, and Helm will silently accept a --set for a key no template reads. As of 4.6 hostpath, LVM, ZFS and Mayastor are all enabled by default, so a bare helm install gives you everything. If you only want LocalPV, turn Mayastor off explicitly with --set engines.replicated.mayastor.enabled=false, which drops the io-engine, the etcd cluster and the CSI components. That is a lot less to run and a lot less to break.
The 4.x chart ships LocalPV and Mayastor only. cStor was archived and is no longer installed by this chart, so the cStor example below assumes a legacy install from openebs-archive.
LocalPV HostPath StorageClass
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: local-hostpath annotations: openebs.io/cas-type: local cas.openebs.io/config: | - name: StorageType value: "hostpath" - name: BasePath value: "/var/openebs/local"provisioner: openebs.io/localallowVolumeExpansion: false # hostpath doesn't support expansionvolumeBindingMode: WaitForFirstConsumerLocalPV HostPath is the odd one out: it is not a CSI driver, so the provisioner string is the old-style openebs.io/local and the engine choice lives in the cas.openebs.io/config annotation rather than in parameters. The CSI-style names (local.csi.openebs.io, zfs.csi.openebs.io) belong to the LVM and ZFS LocalPV engines, not to hostpath. Get this wrong and the PVC sits in Pending with no provisioner claiming it.
Data lives on whatever node the pod lands on. Expansion not supported (you’d have to manually resize the underlying filesystem). Good for caches, temp data, or apps that can tolerate loss.
cStor StorageClass
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: cstor-single-replicaprovisioner: cstor.csi.openebs.ioallowVolumeExpansion: trueparameters: cstorPoolCluster: cstor-pool replicaCount: "1"volumeBindingMode: ImmediateFor cStor, you first need to provision a CStorPoolCluster. The field that trips everyone up is blockDeviceName: it takes the name of a BlockDevice custom resource, not a device path like /dev/sdb. The node-disk-manager discovers your disks and names them for you, so list them first:
kubectl get blockdevice -n openebs -o wideThen reference those names:
apiVersion: cstor.openebs.io/v1kind: CStorPoolClustermetadata: name: cstor-pool namespace: openebsspec: pools: - nodeSelector: kubernetes.io/hostname: node1 dataRaidGroups: - blockDevices: - blockDeviceName: blockdevice-176cda34921fdae209bdd489fe72475d poolConfig: dataRaidGroupType: stripe - nodeSelector: kubernetes.io/hostname: node2 dataRaidGroups: - blockDevices: - blockDeviceName: blockdevice-2c8f1e0b47a6d3915ee7c2b8f0341a9c poolConfig: dataRaidGroupType: stripe - nodeSelector: kubernetes.io/hostname: node3 dataRaidGroups: - blockDevices: - blockDeviceName: blockdevice-9b4e7d215fc80a63bb1de49072c5f8e1 poolConfig: dataRaidGroupType: stripepoolConfig.dataRaidGroupType is required too; the admission webhook rejects the CSPC without it. Valid values are stripe, mirror, raidz and raidz2.
You’re explicitly telling OpenEBS which disk on each node to use. This is where the flexibility comes in, and also where the config complexity lives. You need to know your disk layout, device names, and be comfortable with manual pool provisioning.
What OpenEBS does well
Flexibility: Want different storage engines for different workloads? LocalPV HostPath for scratch space, LocalPV ZFS for a single-node database that wants snapshots, Mayastor for the volumes you cannot lose. All three run side by side, each with its own StorageClass.
No replica tax: The local engines have zero replication overhead. Mayastor lets you set the replica count per StorageClass, so a build cache and a Postgres volume don’t have to make the same trade.
Mayastor performance: Mayastor’s SPDK path is built to cut the kernel overhead that Longhorn’s iSCSI-based v1 engine pays. On NVMe with a fast network, that gap is worth having for database-heavy setups.
Expert-friendly: If you already know ZFS or LVM terminology, the pool model reads naturally, and the LocalPV ZFS engine lets you keep using the tools you know underneath.
The catch
OpenEBS is more complex. Mayastor wants labeled nodes, hugepages and a kernel module before it will start, and cStor on a legacy cluster wants you to manage BlockDevices, CStorPools and StorageClasses by hand. Either way you’re doing storage admin work that Longhorn abstracts away.
Documentation is spottier, and a lot of what you find by searching is written against the 3.x layout that the 4.x chart replaced. cStor has been archived in favor of Mayastor, so you’re swimming against the current if you start there for a new cluster. Mayastor is the maintained path now, but it carries more moving parts than Longhorn’s single operator: an io-engine DaemonSet, its own etcd cluster, and the CSI controller and node plugins.
LocalPV is simple but risky. You lose a node, you lose data. For a 3-node cluster where one node is your old laptop, maybe don’t use LocalPV for anything that matters.
No built-in disaster recovery like Longhorn. cStor has snapshots, but backups require integration with external tools (Velero, custom scripts). Longhorn’s backup story is nicer out of the box.
The real difference: Architecture
Longhorn is centralized-ish. Each volume has exactly one active controller, the engine, running on the node where the workload is attached; that controller orchestrates the replicas and writes to all of them synchronously. There is no second controller standing by, so the HA story is replica redundancy plus fast engine restart, not an active-active pair. If a node goes down, Longhorn’s management components (living elsewhere in the cluster) notice and rebuild the engine against a healthy replica.
OpenEBS (Mayastor) splits the job in two. Pools are declared per node and live independently of any volume, and a volume is a target plus N replicas placed onto those pools. The control plane keeps its state in its own etcd cluster rather than in your disks. If a node goes down, its pool is offline until the node returns or you remove it, and the volume runs degraded on the surviving replicas. You get more control over placement, and you own the pool lifecycle.
OpenEBS (LocalPV) is the simplest: just use the node’s disk. Pod dies, reschedule on another node, your old data is stranded on the original node until you clean it up.
Performance: what actually moves the needle
I am not going to hand you a latency table, because any number I quote would be a number from my disks, my switch and my NICs, and yours are different. What holds across setups is the ordering and the reason for it:
- LocalPV HostPath is the fastest, by a wide margin. It is the node’s own filesystem with a provisioner bolted on. No replication, no network hop, no storage protocol. Nothing else here can match it, and nothing else here loses your data when the node dies.
- Replicated engines pay for every write twice. With 2 replicas, a write is not acknowledged until it lands on a second node, so your write latency floor is set by your network round trip, not your disk. This is the single largest effect, and it is why gigabit Ethernet is the usual home lab bottleneck rather than the SSD.
- Mayastor is the low-latency option among the replicated engines. It is built on SPDK with a userspace NVMe path, which is designed to skip kernel overhead that Longhorn’s iSCSI-based V1 engine pays. That advantage is real on NVMe and mostly theoretical on the SATA SSDs and spinning rust in most home labs.
- Longhorn has its own SPDK engine now, and it costs a core. Longhorn’s V2 Data Engine is also SPDK-based, so the architectural gap is closing. The price is steep for a small cluster: with V2 enabled,
spdk_tgtinside each instance-manager pod busy-polls a dedicated CPU core at 100%, per node. Longhorn’s own docs recommend two cores. On a three-node lab of mini PCs, that is a real chunk of your compute budget for a latency win you probably cannot measure. - Reserved CPU shows up before any volume does. Longhorn’s instance manager requests 12% of each node’s allocatable CPU by default. Mayastor’s io-engine wants 2GiB of hugepages per node. Both are paid whether the volume is busy or idle.
If you want real numbers, run fio inside a pod on each StorageClass you’re considering, on your own hardware, before you commit. For typical home lab workloads (databases, wikis, document storage) you are bottlenecked by disk I/O and the network between nodes, not by the storage layer’s own overhead.
So which one?
Pick Longhorn if:
- You want to set it and forget it. Install, create a StorageClass, move on.
- Backups matter (databases, media libraries, configs). Longhorn’s backup story is excellent.
- Your cluster is remote or managed by someone who doesn’t want to deal with pool configuration.
- You’ve got spare disk space. Replicas are cheap if you’ve got headroom.
- You’re running 3 to 5 nodes. Longhorn’s replica model scales fine up to moderate cluster sizes.
Pick OpenEBS if:
- You want to optimize storage differently for different workloads. Mix LocalPV HostPath (scratch space, single-node workloads) with Mayastor (replicated stateful volumes).
- You’re comfortable with storage admin work (pools, block devices, raid groups).
- You’re exploring Mayastor for low-latency database workloads on NVMe.
- Your disks are precious and you want explicit control over replication.
- You want to learn how replicated storage actually works. Mayastor makes you configure the pools and replicas yourself; Longhorn hides those details.
For a 3-node k3s cluster specifically: Longhorn wins unless you have a specific reason to prefer OpenEBS. Three nodes is where Longhorn works best, big enough for meaningful replication, small enough that one operator handles everything. You set replicaCount=2, point it at your disks, and you’re done. Your 2 AM self will appreciate not having to debug pool membership or missing BlockDevices.
If you’re running a NAS-style setup (media library, file sharing), neither is ideal, both are designed for block storage. Use NFS on a real NAS. OpenEBS does have a dynamic NFS provisioner that layers ReadWriteMany volumes on top of a block StorageClass, but its last release was 0.11.0 in December 2023 and the repo has been quiet since, so do not build a media library on it.
The pragmatist’s checklist
Before you choose, ask yourself:
- What are you storing? Databases love distributed storage. Media libraries are fine with local + snapshots. Kubernetes configs? Ephemeral LocalPV is enough.
- How much disk space do you have? Replicas cost space. Three nodes with 500GB each is 1.5TB raw, which is ~750GB usable at 2 replicas and ~500GB at 3. Every replica you add is another full copy.
- Can you afford to lose a node? If yes, consider LocalPV or replicaCount=1. If no, 2+ replicas mandatory.
- Do you take backups? Longhorn makes this trivial. OpenEBS requires more plumbing.
- How much time do you want to spend on storage ops? Longhorn: 1 hour setup, 5 minutes maintenance. OpenEBS: 3 hours setup, 15 minutes maintenance.
You’re not picking the objectively best storage engine. You’re picking the one that fits your cluster size, risk tolerance, and patience level. For most 3-node home labs, that’s Longhorn. For a lab built to learn how replicated storage works, that’s OpenEBS Mayastor, which is the maintained engine and puts the pool and replica configuration in your hands. Either way, you’re winning. Both beat “just hope your pod doesn’t crash.”
Common Questions
Can I run Longhorn on a 2-node k3s cluster?
Yes, with defaultReplicaCount=2, but you lose the ability to survive a node failure cleanly. With two nodes there is no third place to rebuild a replica, so a node loss leaves every volume degraded until that node returns. Three nodes is the practical floor for replicated storage.
Do I need to install anything on my nodes before Longhorn?
Yes. Every node needs open-iscsi installed with the iscsid service running, because Longhorn’s V1 engine attaches volumes over iSCSI. Add nfs-common if you want ReadWriteMany volumes. Without iSCSI, your first PersistentVolumeClaim hangs in Pending with an attach error that never mentions iSCSI.
Is OpenEBS cStor still supported in 2026?
No. cStor moved to the openebs-archive GitHub organization and is not installed by the OpenEBS 4.x Helm chart. Existing cStor deployments keep working, but new clusters should use Mayastor for replicated volumes or one of the LocalPV engines for node-local volumes.
Can I run Longhorn and OpenEBS in the same cluster?
Yes. Each provisioner has its own StorageClass, its own namespace and its own CSI driver, so they coexist without conflict. The cost is duplicated overhead: two sets of DaemonSets, two control planes, and two reserved CPU and memory budgets on every node. Pick one unless you have a specific reason.
How much usable space do I get from 3 nodes with 500GB each?
Roughly 750GB at 2 replicas and 500GB at 3, out of 1.5TB raw. Every replica is a full copy of the volume, not a parity stripe, so usable capacity is raw capacity divided by the replica count. There is no erasure coding option in either Longhorn or Mayastor.
