Go back

Falco + Trivy in k3s: Runtime Security on Small Clusters

By SumGuy 15 min read
Falco + Trivy in k3s: Runtime Security on Small Clusters
Contents

You Can’t Secure What You Don’t See

Container security has two halves, and they hate each other.

One half lives at image build time, scanning for known vulnerabilities before anything touches your cluster. Trivy does this. It’s the bouncer at the door, checking IDs, finding the one guy with a fake credential.

The other half lives at runtime, watching what’s actually happening inside your cluster right now. Is a container spawning a reverse shell? Did nginx just try to write to /etc for some reason? Falco catches this. It’s the security guard on the floor, walking the beat, noticing when someone’s carrying a ladder in the wrong direction.

The interesting part is stitching them together in k3s. Not a managed platform with three teams staffing security. Actual k3s. The one on your home lab server, or a small VPS, or a trio of Raspberry Pis in a rack that your neighbors definitely don’t know about.

This isn’t about building Fort Knox. It’s about knowing when someone’s opened a door they shouldn’t have, and knowing it happened instead of discovering it three weeks later in your logs.

The Setup

Here’s what we’re building:

  1. Trivy Operator: scans images as they land in your cluster, flags CVEs
  2. Falco: watches syscalls, detects behavioral anomalies
  3. Alert routing: both feed into a webhook (Slack, Discord, whatever)
  4. Optional: Grafana: one dashboard, all the signals

You need:

Part 1: Trivy Operator (Image Scanning)

Trivy Operator is the easy half. It runs a Kubernetes controller that automatically scans images in your cluster and creates VulnerabilityReport custom resources. In practice, every time a pod spins up, Trivy runs a scan, stashes the results, and goes quiet again. If something gets marked critical, you react. Otherwise, life goes on.

Install Trivy Operator

Terminal window
helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm upgrade --install trivy-operator aqua/trivy-operator \
  --namespace trivy-system \
  --create-namespace \
  --set serviceAccount.create=true

Leave the Trivy image alone and let the chart pull its own pinned default. That way you get whatever version ships with the operator release instead of pinning a stale tag that goes out of date the moment you publish it.

That’s it. Trivy Operator is now watching your cluster. Every pod that gets created gets scanned. The scan takes 30 seconds to a few minutes (depends on image size and how fast your network is), and results show up as a VulnerabilityReport object in the same namespace as the pod.

Check it:

Terminal window
kubectl get vulnerabilityreport -A
kubectl describe vulnerabilityreport <name> -n <namespace>

You’ll see something like:

Vulnerabilities:
  - Fixed Version: 8.4.0
    Installed Version: 8.2.1
    Resource: curl
    Severity: CRITICAL
    Title: curl: heap based buffer overflow in the SOCKS5 proxy handshake
    Vulnerability ID: CVE-2023-38545

Trivy only scans artifacts you actually run, not your entire registry. You run nginx:latest with a known CVE in OpenSSL? It tells you. You run your own custom Python app with a pip dependency that’s three years old? It tells you that too.

Configure Trivy alerts

You’ll read blog posts telling you to build a CronJob that polls VulnerabilityReport objects with kubectl and jq and POSTs them somewhere. Don’t. Trivy Operator has shipped webhook broadcasting for a while, and it’s one Helm flag:

Terminal window
helm upgrade --install trivy-operator aqua/trivy-operator \
  --namespace trivy-system \
  --create-namespace \
  --set operator.webhookBroadcastURL="http://alert-aggregator.security.svc.cluster.local/trivy" \
  --set operator.webhookBroadcastTimeout=30s

That maps to the OPERATOR_WEBHOOK_BROADCAST_URL environment variable on the operator. Every report the operator produces gets POSTed to that URL as it’s written: vulnerabilityreport, exposedsecretreport, configauditreport, sbomreport, and the cluster-scoped variants. Two options worth knowing:

The payload shape matters, because most examples get it wrong. Trivy Operator sends an envelope, not a bare report:

{
  "verb": "created",
  "operatorObject": {
    "apiVersion": "aquasecurity.github.io/v1alpha1",
    "kind": "VulnerabilityReport",
    "metadata": { "name": "replicaset-nginx-6d4cf56db6-nginx", "namespace": "default" },
    "report": {
      "artifact": { "repository": "library/nginx", "tag": "1.29" },
      "summary": { "criticalCount": 1, "highCount": 4, "mediumCount": 11 },
      "vulnerabilities": []
    }
  }
}

So your receiver reads operatorObject.report.summary, not a top-level vulnerabilities key. If you’d rather not write a receiver at all, Aqua’s own Postee takes this webhook directly and fans it out to Slack, Teams, Jira, and friends.

Part 2: Falco (Runtime Detection)

Now we swap gears. Trivy told you what’s in your images. Falco tells you what your containers are doing. If a pod that’s supposed to run nginx suddenly starts trying to read /etc/shadow, Falco sees it. If something spawns a shell when it shouldn’t, Falco knows.

Falco taps the kernel for syscalls, either through a loadable kernel module or the modern eBPF probe embedded in the Falco binary. It matches those syscalls against rules in real time. It’s like a video camera that only records when something suspicious happens.

Install Falco

Terminal window
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm upgrade --install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falco.json_output=true

driver.kind=modern_ebpf uses the eBPF probe baked into the Falco binary instead of a kernel module, so nothing gets compiled or downloaded on your nodes. It works on most k3s setups, VPS included, as long as the kernel clears the 5.8 bar from the requirements above. The chart’s default is auto, which picks modern eBPF when the kernel supports it and falls back to the kernel module when it doesn’t. Setting it explicitly means you find out at install time rather than discovering a kernel module got built on your Pi.

falco.json_output=true matters more than it looks. Falco’s default output is plain text, and every HTTP receiver in this article (and falcosidekick, and most third-party tooling) expects JSON. Turn it on now or spend an hour wondering why your webhook handler throws on request.json.

Don’t copy the gRPC flags from older tutorials. Falco 0.43 deprecated the legacy eBPF probe, the gVisor engine, and gRPC output. Falco 0.44 removed all three outright, and the chart’s current app version is 0.44.x. --set falco.grpc.enabled=true and --set falco.grpcOutput.enabled=true do nothing useful now. HTTP output is the supported path.

Verify:

Terminal window
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50

You’ll see Falco initializing, detecting syscalls, and any rule violations. By default, Falco has rules for:

The default ruleset is good, but you’ll probably want to tune it.

Configure Falco Rules

Falco loads its rules from /etc/falco/falco_rules.yaml inside the container. Don’t hand-roll a ConfigMap and try to mount it over the DaemonSet. The chart has a customRules map for exactly this, and it wires up the volume and the rules_files entry for you:

falco-custom-rules.yaml
customRules:
  custom-rules.yaml: |-
    - rule: Unexpected shell in container
      desc: A shell spawned inside a container
      condition: >
        spawned_process and
        container and
        shell_procs
      output: >
        Shell detected
        (user=%user.name command=%proc.cmdline parent=%proc.pname
        container=%container.name image=%container.image.repository:%container.image.tag)
      priority: WARNING
      tags: [shell, container]


    - rule: Sensitive file read in container
      desc: Reading sensitive files from a container
      condition: >
        open_read and
        container and
        fd.name in (/etc/shadow, /root/.ssh/id_rsa)
      output: >
        Sensitive file read attempt
        (user=%user.name file=%fd.name container=%container.name)
      priority: WARNING
      tags: [privilege, file]

Apply it with helm upgrade falco falcosecurity/falco -n falco -f falco-custom-rules.yaml.

Three things in there will bite you if you copy a rule from a random gist:

Validate before you ship: falco -V /etc/falco/rules.d/custom-rules.yaml inside the pod tells you whether the file parses. The default ruleset is solid for most home labs, so customize only when you’re chasing a false positive or a workload nobody upstream models.

Part 3: Alert Routing (The Glue)

Both Trivy and Falco produce alerts, and by default they go to two different places. One lands in the operator’s webhook, the other in a pod log nobody reads. Put them in one place.

Falco to Webhook

Falco posts alerts to an HTTP endpoint through its http_output channel. The chart passes everything under falco: straight into falco.yaml, so the keys are snake_case, exactly as they appear in the config file:

Terminal window
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set falco.json_output=true \
  --set falco.http_output.enabled=true \
  --set falco.http_output.url="http://alert-aggregator.security.svc.cluster.local/falco"

Camel-case variants like falco.httpOutput.enabled are not chart keys. Helm accepts them without complaint, writes them into falco.yaml, and Falco ignores them. You get a healthy pod and zero webhooks, which is the worst failure mode a security tool has.

Keep json_output=true in there. Without it Falco POSTs a plain text line instead of a JSON body, and your receiver has to regex it. With it you get:

{
  "hostname": "k3s-node-01",
  "output": "16:45:12.123456789: Warning Shell detected (user=root command=/bin/sh parent=containerd-shim container=nginx image=nginx)",
  "output_fields": {
    "container.image.repository": "nginx",
    "container.name": "nginx",
    "evt.time": 1788280512123456789,
    "proc.cmdline": "sh",
    "user.name": "root"
  },
  "priority": "Warning",
  "rule": "Unexpected shell in container",
  "source": "syscall",
  "tags": ["container", "shell"],
  "time": "2026-09-04T16:45:12.123456789Z"
}

Parse output_fields, not output. The output string is a human-readable render of your rule’s template and it changes the moment you edit the rule. output_fields is a stable map keyed by the field names Falco actually resolved.

Simple Alert Aggregator

You need a tiny webhook receiver. Here’s a Python one:

from flask import Flask, request
import requests
import os


app = Flask(__name__)


SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK", "")


@app.route("/falco", methods=["POST"])
def falco_alert():
    data = request.json
    priority = data.get("priority", "Unknown")
    rule = data.get("rule", "Unknown")
    fields = data.get("output_fields", {})
    tags = ", ".join(data.get("tags", []))


    emoji = {"Critical": "🔴", "Error": "🟠", "Warning": "🟡"}.get(priority, "⚪")


    message = (
        f"{emoji} *Falco: {rule}* ({priority})\n"
        f"host={data.get('hostname', '?')} "
        f"container={fields.get('container.name', '?')} "
        f"image={fields.get('container.image.repository', '?')}\n"
        f"user={fields.get('user.name', '?')} "
        f"cmd={fields.get('proc.cmdline', '?')}\n"
        f"tags: {tags}"
    )


    if SLACK_WEBHOOK:
        requests.post(SLACK_WEBHOOK, json={"text": message})


    return {"status": "ok"}, 200


@app.route("/trivy", methods=["POST"])
def trivy_alert():
    msg = request.json
    if msg.get("verb") == "deleted":
        return {"status": "ignored"}, 200


    obj = msg.get("operatorObject", {})
    if obj.get("kind") != "VulnerabilityReport":
        return {"status": "ignored"}, 200


    meta = obj.get("metadata", {})
    report = obj.get("report", {})
    summary = report.get("summary", {})
    critical = summary.get("criticalCount", 0)
    high = summary.get("highCount", 0)


    if critical or high:
        artifact = report.get("artifact", {})
        image = f"{artifact.get('repository', 'unknown')}:{artifact.get('tag', 'unknown')}"
        message = (
            f"🚨 *Trivy: {image}*\n"
            f"critical={critical} high={high}\n"
            f"namespace={meta.get('namespace', '?')} report={meta.get('name', '?')}"
        )
        if SLACK_WEBHOOK:
            requests.post(SLACK_WEBHOOK, json={"text": message})


    return {"status": "ok"}, 200


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Deploy it as a service in your cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alert-aggregator
  namespace: security
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alert-aggregator
  template:
    metadata:
      labels:
        app: alert-aggregator
    spec:
      containers:
      - name: aggregator
        image: python:3.14-slim
        command:
        - sh
        - -c
        - |
          pip install flask requests && python /app/app.py
        env:
        - name: SLACK_WEBHOOK
          valueFrom:
            secretKeyRef:
              name: slack-webhook
              key: url
        volumeMounts:
        - name: app
          mountPath: /app
        ports:
        - containerPort: 5000
      volumes:
      - name: app
        configMap:
          name: alert-aggregator-code
---
apiVersion: v1
kind: Service
metadata:
  name: alert-aggregator
  namespace: security
spec:
  selector:
    app: alert-aggregator
  ports:
  - port: 80
    targetPort: 5000
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alert-aggregator-code
  namespace: security
data:
  app.py: |
    # The Flask receiver from the previous code block goes here, verbatim.
    # kubectl create configmap alert-aggregator-code -n security --from-file=app.py

Now both Falco and Trivy point at http://alert-aggregator.security.svc.cluster.local/falco and /trivy. Alerts show up in Slack in real time.

Part 4: Optional Grafana Dashboard

If you’re already running Prometheus + Grafana (and who isn’t these days?), you can expose Falco and Trivy metrics there.

Falco exposes Prometheus metrics from its own webserver. metrics is a top-level block in the chart, not a key under falco:, so falco.metricsEnabled does nothing:

Terminal window
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set metrics.enabled=true \
  --set metrics.convertMemoryToMB=true

That flips on falco.metrics.enabled and falco.webserver.prometheus_metrics_enabled together, and metrics land at /metrics on the webserver port, 8765, not the pod’s default port.

If you run the Prometheus Operator, skip the hand-written scrape config and let the chart create the ServiceMonitor:

Terminal window
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set metrics.enabled=true \
  --set serviceMonitor.create=true

Doing it by hand instead? The port matters, and this is the part most copy-paste configs get wrong:

prometheus-falco-job.yaml
- job_name: 'falco'
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - falco
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
    action: keep
    regex: falco
  - source_labels: [__address__]
    action: replace
    regex: '([^:]+)(?::\d+)?'
    replacement: '$1:8765'
    target_label: __address__

Without that last relabel, Prometheus scrapes the pod IP on port 80 and every target sits red.

For Trivy, you can query the VulnerabilityReport CRD directly with a simple Kubernetes exporter, or use a sidecar that converts reports to Prometheus metrics.

The Grafana dashboard is nice-to-have, not must-have. Slack/Discord notifications are usually enough.

The Reality Check

Look, security isn’t a feature you ship. It’s infrastructure you live in. Falco + Trivy in k3s gives you:

The k3s version is lean, but be clear about what “lean” means. Falco is a privileged DaemonSet: one pod per node, reading syscalls through an eBPF probe. Trivy Operator is a single Deployment that spawns short-lived scan jobs. What you’re skipping is the SaaS control plane, the per-node license, and the “security team” org chart. You are still running an agent on every node, and it will show up in your node resource graphs. It scales down to three nodes on your home lab and up to a small production cluster without re-architecting anything.

Start with Trivy. Get image scanning working. Watch for 48 hours. Then add Falco. Let it baseline on your actual workloads (it’ll have false positives on day one, everything does). Then tune rules based on what you care about.

The goal is to catch things that matter, and know about them before 3 AM.

Quick Checklist

Common Questions

Does Falco need a privileged container on every node?

Yes. Falco runs as a DaemonSet with elevated privileges because it reads kernel syscall events. The modern eBPF driver narrows this to specific capabilities rather than full privileged: true, which the chart handles for you. There is no unprivileged mode that still sees syscalls. Budget one pod per node.

Can Trivy Operator scan images in a private registry?

Yes. Trivy Operator reuses the imagePullSecrets already attached to the workload’s ServiceAccount or pod spec, so images your cluster can pull are images it can scan. For registries not covered that way, set operator.privateRegistryScanSecretsNames to map a namespace to a pull secret.

Which Falco driver should I pick on k3s?

Modern eBPF, if your kernel is 5.8 or newer and exposes BTF. Set driver.kind=modern_ebpf. It needs no compilation and no kernel headers. Leave driver.kind=auto and the chart probes for you, falling back to the kernel module on older kernels. The legacy eBPF probe was removed in Falco 0.44.

How much overhead does Falco add per node?

Expect a few hundred MB of RAM and low single-digit CPU percentage per node at rest on a home lab workload. Syscall-heavy pods push that up. Set resources.limits in the chart before you deploy, then watch actual usage for a week and tighten from there.

Do I still need this if I already scan images in CI?

Yes, for the runtime half. CI scanning tells you what CVEs an image shipped with on build day. It cannot tell you a container spawned a shell at 3 AM, read /etc/shadow, or started talking to a new IP. Trivy Operator also re-scans running images as new CVEs get published.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email

Send a Webmention

Written about this post on your own site? Send a webmention and it'll show up above once verified.

Next Post
Renovate for k8s GitOps

Discussion

Powered by Garrul . Sign in with GitHub or Google, or post anonymously.

Related Posts