Go back

External Secrets Operator + Vault on k3s

By SumGuy 15 min read
External Secrets Operator + Vault on k3s
Contents

Sealed Secrets Was Fine Until It Wasn’t

You’ve been shipping Sealed Secrets in your k3s cluster for a year. It works. Developers re-encrypt secrets when they rotate. Your backups are safe because the encryption key lives in the cluster. Life is… okay.

Then your ops team asks: “What if we need to share a secret with our staging cluster without deploying new sealed-secrets keys?” Or: “Can we audit who accessed what secret and when?” Or worse: “Our security audit says we need centralized secret rotation.”

Sealed Secrets will look at you and shrug. It’s a simple lock-the-secret-in-git solution, nothing more. For a home lab, that’s enough. For anything touching production, or anything where compliance people show up with checklists, you need something that talks to a proper secret backend.

External Secrets Operator (ESO) plus Vault is that something. It’s the car you take to work instead of the forklift you’ve been driving.

ESO doesn’t care where secrets live. You can use Vault, AWS Secrets Manager, HashiCorp Cloud Platform, Azure Key Vault, or even a plain HTTP API. Vault is the obvious choice for a self-hosted setup because you can run it on the same cluster, or on a separate rig if you’re paranoid about keeping secrets away from app pods.

Here’s the k3s setup: running Vault, installing ESO, and pulling secrets from Vault into your cluster without ever checking them into git.

What We’re Building

If you’ve got a home lab k3s cluster running and you’re tired of managing secrets manually, this is your move.

Install Vault (Standalone, In-Cluster)

First, we’ll run Vault as a single stateful pod with file storage. That skips HA, but for a home lab it’s fine. If you want HA later, move to Vault’s integrated Raft storage rather than Consul: Raft is what HashiCorp ships the official Helm chart with, and it needs no second cluster to babysit.

Two things before the YAML, because they are the traps that eat an afternoon:

  1. The hashicorp/vault image defaults to CMD ["server", "-dev"]. If you do not override the args, you get dev mode: in-memory storage, auto-unseal, a KV v2 engine already mounted at secret/, and every secret gone on pod restart. Your PVC and your config file are both silently ignored. Plenty of “why did my Vault empty itself” threads start here.
  2. Vault tries to mlock its memory. In a container without the IPC_LOCK capability it fails to start. Set disable_mlock = true or grant the capability. Below does the former, which is what the official Helm chart does.

Versions used here: Vault 2.1 and External Secrets Operator 2.x. If HashiCorp’s BUSL licence is a problem for you, OpenBao is the MPL-licensed fork of Vault 1.14 and speaks the same API, so everything below works against it with the image swapped.

Vault Namespace & Storage

Create a namespace and persistent volume for Vault’s data:

apiVersion: v1
kind: Namespace
metadata:
  name: vault


---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vault-data
  namespace: vault
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-path

Apply that. k3s gives you the local-path storage class by default.

Vault Deployment

Note the serviceAccountName and the system:auth-delegator binding. Vault validates ServiceAccount tokens through the Kubernetes TokenReview API, and it cannot call that API without that ClusterRole. Leave it out and Kubernetes auth fails later with a permission error that points nowhere useful.

vault.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
  namespace: vault


---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  strategy:
    type: Recreate          # one RWO volume, so never two pods at once
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      serviceAccountName: vault
      containers:
      - name: vault
        image: hashicorp/vault:2.1
        args: ["server", "-config=/vault/config/vault.hcl"]   # without this you get dev mode
        ports:
        - containerPort: 8200
          name: http
        env:
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        - name: vault-config
          mountPath: /vault/config
      volumes:
      - name: vault-data
        persistentVolumeClaim:
          claimName: vault-data
      - name: vault-config
        configMap:
          name: vault-config


---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: vault
data:
  vault.hcl: |
    ui            = true
    disable_mlock = true
    api_addr      = "http://vault.vault.svc.cluster.local:8200"
    listener "tcp" {
      address     = "0.0.0.0:8200"
      tls_disable = 1
    }
    storage "file" {
      path = "/vault/data"
    }


---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  ports:
  - port: 8200
    targetPort: 8200
  selector:
    app: vault

tls_disable = 1 is acceptable for cluster-internal traffic on a home lab and nowhere else. If Vault is reachable off-cluster, terminate TLS properly and point the ClusterSecretStore at https://.

Deploy it:

Terminal window
kubectl apply -f vault.yaml
kubectl -n vault logs -f deployment/vault

Wait for the pod to be ready. You’ll see something like “Vault server started!” in the logs.

Initialise and Unseal Vault

A real Vault server starts uninitialised and sealed. Dev mode auto-unseals and hands you a root token, which is exactly why we did not use dev mode.

Port-forward to Vault:

Terminal window
kubectl -n vault port-forward service/vault 8200:8200

In another shell, initialise it. One key share with a threshold of one is a home lab shortcut. Use three-of-five if anyone else depends on this cluster:

Terminal window
export VAULT_ADDR=http://localhost:8200
vault operator init -key-shares=1 -key-threshold=1

That prints an unseal key and an initial root token exactly once. Put both somewhere off the cluster right now: a password manager, a printed sheet in a drawer, anywhere that survives the cluster dying. There is no recovery path if you lose them.

Terminal window
vault operator unseal <unseal-key-from-above>
export VAULT_TOKEN=<initial-root-token>
vault status

You should see Sealed false and Version 2.1.x.

Vault re-seals on every restart. With file storage and no auto-unseal, a node reboot leaves Vault sealed and every ExternalSecret failing to refresh until you unseal by hand. That is the biggest running cost of self-hosted Vault. Live with it, or configure auto-unseal against a cloud KMS or a Transit engine on a second Vault.

Enable the KV Secret Engine

A freshly initialised Vault has no KV engine mounted. Enable one at secret/:

Terminal window
vault secrets enable -version=2 -path=secret kv

If that returns path is already in use at secret/, you are in dev mode. Go back and check the args: on the Deployment.

Create a test secret:

Terminal window
vault kv put secret/my-app/db-password username=appuser password="super-secret-db-pass"
vault kv get secret/my-app/db-password

Good. Vault is running and storing secrets on disk that survive a restart.

Install External Secrets Operator

ESO is a Kubernetes operator that watches for ExternalSecret resources and syncs them to Kubernetes Secrets.

Add the Helm Repository

Terminal window
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Install ESO

Terminal window
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

Verify it’s running:

Terminal window
kubectl -n external-secrets-system get pods

You should see external-secrets-webhook-* and external-secrets-* pods.

Create a Vault Auth Token for ESO

ESO needs credentials to talk to Vault. We’ll create a Kubernetes ServiceAccount and enable Vault’s Kubernetes auth method so that the ServiceAccount can authenticate to Vault.

Enable Kubernetes Auth in Vault

Run this inside the Vault pod. The config needs the pod’s own ServiceAccount token and CA certificate, and the @ syntax reads them as files, so the paths have to exist where the command runs. Running it from your laptop against a port-forward fails, because those paths are not on your laptop and $KUBERNETES_SERVICE_HOST is not set there either.

Terminal window
kubectl -n vault exec -it deployment/vault -- sh -c '
  export VAULT_ADDR=http://127.0.0.1:8200
  export VAULT_TOKEN=<your-root-token>
  vault auth enable kubernetes
  vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
'

token_reviewer_jwt is deliberately absent. When Vault runs in the cluster, it uses its own ServiceAccount token to call TokenReview, which is why the system:auth-delegator binding earlier matters. Passing a static JWT instead means pinning a token that Kubernetes will eventually rotate out from under you.

Create a Vault Policy for ESO

KV v2 splits reads and lists across two paths: data lives under secret/data/, and listing happens under secret/metadata/. A policy that grants list on secret/data/* grants nothing useful.

Terminal window
vault policy write eso-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read"]
}
path "secret/metadata/*" {
  capabilities = ["read", "list"]
}
EOF

Create a Kubernetes Auth Role

The bound namespace is where the ESO ServiceAccount lives, not where your apps live. ESO authenticates once on behalf of every ExternalSecret in the cluster, so external-secrets-system is the only namespace that belongs here.

Terminal window
vault write auth/kubernetes/role/external-secrets-role \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets-system \
  audience=vault \
  policies=eso-policy \
  ttl=1h

Recent Vault warns on roles with no audience, and the ESO documentation reports that Vault 1.21 and later can reject them outright. Setting it on both sides costs nothing and removes the question. The matching audiences: field goes in the ClusterSecretStore below, and the two must be identical.

Create a ClusterSecretStore

A ClusterSecretStore tells ESO how to authenticate to Vault. It’s a cluster-wide resource that external secrets reference.

cluster-secret-store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"        # match the KV engine you enabled
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets-role"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets-system   # required on a ClusterSecretStore
            audiences:
              - vault      # must match audience= on the Vault role

The serviceAccountRef points to the ESO ServiceAccount created by the Helm chart. ESO mints a short-lived token for that account through the TokenRequest API and presents it to Vault, which validates it via TokenReview. On a ClusterSecretStore the namespace field is mandatory: without it ESO has no idea which namespace to look in and the store never becomes ready.

Apply it:

Terminal window
kubectl apply -f cluster-secret-store.yaml

Create an ExternalSecret

Now the fun part. Write an ExternalSecret resource that pulls a secret from Vault and creates a Kubernetes Secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: app-db-secret
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        username: "{{ .username }}"
        password: "{{ .password }}"
        database_url: "postgresql://{{ .username }}:{{ .password }}@postgres.default.svc.cluster.local:5432/mydb"
  data:
  - secretKey: username
    remoteRef:
      key: my-app/db-password
      property: username
  - secretKey: password
    remoteRef:
      key: my-app/db-password
      property: password

This says:

Apply it:

Terminal window
kubectl apply -f external-secret.yaml

Check that the Kubernetes Secret was created:

Terminal window
kubectl get secret app-db-secret -o yaml

You should see:

data:
  username: YXBwdXNlcg==  # base64-encoded "appuser"
  password: c3VwZXItc2VjcmV0LWRiLXBhc3M=  # base64-encoded password
  database_url: cG9zdGdyZXNxbDovL2FwcHVzZXI6c3VwZXItc2VjcmV0LWRiLXBhc3NAcG9zdGdyZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo1NDMyL215ZGI=

If the Secret doesn’t exist, check the ExternalSecret status:

Terminal window
kubectl describe externalsecret app-db-secret

Look for errors like “failed to authenticate” or “key not found”. Common issues:

Using the Secret in a Pod

In your Deployment or Pod spec, mount the Secret as an environment variable or volume:

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: password
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: database_url

Your app reads these environment variables. No secrets in the Pod spec, no secrets in git.

Rotate Secrets Without Redeploying

Update the secret in Vault:

Terminal window
vault kv put secret/my-app/db-password username=appuser password="new-shiny-password"

ESO checks for updates every hour (controlled by refreshInterval: 1h in the ExternalSecret). When it detects a change, it rewrites the Kubernetes Secret.

Rewriting the Secret does not restart your pods, and this is where people get burned. What happens next depends on how the app consumes it:

For the env-var case, use Stakater Reloader: annotate the Deployment with reloader.stakater.com/auto: "true" and it performs a rolling restart whenever a referenced Secret or ConfigMap changes. Prometheus does not do this, whatever a search result told you. It scrapes metrics.

Vault Outside the Cluster (Optional)

If you want Vault on a separate machine or in a different cluster, just change the server: URL in your ClusterSecretStore:

vault:
  server: "https://vault.mycompany.com:8200"

And use a different auth method if Kubernetes auth doesn’t make sense. Vault supports AppRole (agent credentials), JWT (OIDC), or plain token auth.

When to Use This vs Sealed Secrets

Use Sealed Secrets if:

Use ESO + Vault if:

For a home lab with one cluster, Sealed Secrets is probably still fine. But once you’re managing anything more complex, or once you want to stop checking secrets into git even in encrypted form, ESO plus Vault is the obvious upgrade. The cost is honest and worth stating: one more service to run, and a Vault that seals itself every time it restarts.

Troubleshooting Checklist

ExternalSecret stuck in “pending”:

Vault sealed unexpectedly:

“Authentication failed” in ExternalSecret status:

Secret not updating when you change it in Vault:

Secrets out of git, auditable at every access, rotatable without redeployment. That’s the promise, and ESO plus Vault delivers it. Just budget for the unseal.

Common Questions

Does the hashicorp/vault container run in dev mode by default?

Yes. The image’s default command is server -dev, so a Deployment that sets no args: gets dev mode: in-memory storage, auto-unseal, a KV v2 engine already mounted at secret/, and every secret lost on restart. Pass args: ["server", "-config=/vault/config/vault.hcl"] to run a real server.

Why does Vault Kubernetes auth fail with a permission error?

Vault validates ServiceAccount tokens by calling the Kubernetes TokenReview API, and it needs the system:auth-delegator ClusterRole to do that. Bind that ClusterRole to Vault’s own ServiceAccount. This is the most common cause of “authentication failed” on an otherwise correct External Secrets Operator setup.

Do my pods pick up a rotated secret automatically?

Only if the Secret is mounted as a volume. The kubelet refreshes mounted Secret files in place within a minute or two. Secrets injected as env: variables are set once at container start and never change until the pod restarts. Use Stakater Reloader to restart deployments when a Secret changes.

Do I have to unseal Vault by hand after every restart?

Yes, unless you configure auto-unseal. A Vault using file or Raft storage seals itself on every restart, and every ExternalSecret stops refreshing until you unseal it. Auto-unseal against a cloud KMS or a Transit engine on a second Vault removes the manual step.

Can I use OpenBao instead of Vault?

Yes. OpenBao is the MPL-licensed fork of Vault 1.14 and keeps the same API, CLI and Kubernetes auth method, so every command and manifest here works with the image swapped. Pick OpenBao if HashiCorp’s BUSL licence is a problem, and Vault if you want the newer enterprise-adjacent features.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email

Send a Webmention

Written about this post on your own site? Send a webmention and it'll show up above once verified.

Next Post
Kustomize Without Helm: When Overlays Win

Discussion

Powered by Garrul . Sign in with GitHub or Google, or post anonymously.

Related Posts