Your Database Password Is Sitting in a Text File You Forgot About
You wrote Environment=DB_PASSWORD=hunter2 in a unit file at 11 PM, it worked, and you went to bed. Congratulations, that password now lives in a file under /etc/systemd/system, in the unit’s loaded properties, and in the environment of every process the service spawns.
The fix is built into systemd and has been for years: credentials. Put the secret in an encrypted file, point LoadCredentialEncrypted= at it, and the service finds a read-only file in $CREDENTIALS_DIRECTORY. No env var, no plaintext at rest. This post compares the two approaches side by side and shows the commands that are short enough that you’ll actually make the switch.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/linux/systemd-credentials
What’s Wrong With Environment=
Environment variables are the default way to hand config to a service, and they’re fine for LOG_LEVEL=debug. For secrets, the systemd authors themselves say no. The systemd.exec man page states that environment variables are “not suitable for passing secrets”, for three reasons:
- Variables set on a unit are exposed to unprivileged clients over D-Bus.
- They are generally not understood as data that needs protection.
- They propagate down the process tree, including across security boundaries like setuid executables.
You can watch reason one happen. This is a transient unit with a secret in Environment=:
systemd-run --user --unit=demo-bad \ -p Environment=DB_PASSWORD=hunter2 sleep 60
systemctl --user show demo-bad.service -p Environment# Environment=DB_PASSWORD=hunter2That’s the user manager. System units leak the same way: as a regular UID 1000 account, systemctl show <unit> -p Environment printed the Environment= line of system services running as root, no sudo needed. Any local account on the box can do the same.
Then there’s the process tree. Every child your service forks inherits the variable. Your app shells out to convert or git, and now those carry your database password too. Crash dumps and debug output that print the environment will print it as well. And EnvironmentFile= doesn’t help: the file keeps the secret in plaintext at rest and the variables still land in the environment.
A limit worth stating plainly: /proc/PID/environ is not world-readable. Reading another user’s process gave me “Permission denied”. The same user and root can read it. So a random local account can’t just grab the secret that way. The leaks are the quieter ones above, and those are harder to audit because there is no single file to point at.
What systemd Credentials Do Instead
A credential is a small named blob (the man page caps the total at 1 MB per unit) that the service manager copies into a private directory just for that service. The service finds the directory in $CREDENTIALS_DIRECTORY. From the man page:
- The location is read-only and, if possible and permitted, backed by non-swappable memory.
- Only the user the unit runs as (via
User=orDynamicUser=) and the superuser can read it. - The service manager reads the source file itself, so the file doesn’t need to be readable by the service’s user.
On my box (systemd 262.1) a user-service credential shows up like this:
dr-x------ 2 you you 60 Nov 4 15:02 /run/user/1000/credentials/run-p3007160-i11400625.service-r-------- 1 you you 7 Nov 4 15:02 /run/user/1000/credentials/run-p3007160-i11400625.service/dbpassDirectory mode 0500, file mode 0400, owned by the service user. Inside a service, systemd-creds list reports each credential as “secure” (backed by unswappable ramfs), “weak” (other memory), or “insecure” (any mode other than 0400). It’s a quick sanity check.
Picture Environment= as a forklift that drops the pallet in the hallway. A credential is the same forklift delivering to a locked room with one key. Same delivery, very different 2 AM.
The Four Directives You Need
The man page lists several ways to feed credentials in. This is the working set.
| Directive | Source | Safe for secrets? |
|---|---|---|
LoadCredential=ID:PATH | A plaintext file (or an AF_UNIX socket) | Yes, but the file sits in plaintext at rest |
LoadCredentialEncrypted=ID:PATH | An encrypted credential file | Yes, the recommended one |
SetCredentialEncrypted=ID:VALUE | Encrypted blob pasted into the unit | Yes |
SetCredential=ID:VALUE | A literal value in the unit | No |
That last row matters. The man page says of SetCredential=: “Do not use this option for data that is supposed to be secret, as it is accessible to unprivileged processes via IPC.” It’s for user IDs, public keys, and defaults. When I ran systemctl show on a unit with SetCredential= it did not print the value, but the man page warning outranks my one test, so treat it as non-secret.
There’s also ImportCredential=GLOB, which passes through credentials the service manager already has, searching /etc/credstore/, /run/credstore/, /usr/lib/credstore/ and their .encrypted counterparts. Only a single trailing * is allowed as a wildcard. It’s handy for vendor units that want “everything named myapp.*”.
For LoadCredential= and LoadCredentialEncrypted=, if you omit the path, systemd looks in the credstore directories by name. So LoadCredentialEncrypted=dbpass finds /etc/credstore.encrypted/dbpass. A missing credential in that short form isn’t fatal, so check your service starts with the value you expect.
Encrypting a Secret
systemd-creds encrypt takes a plaintext input and writes an encrypted, Base64-encoded output. The man page says it uses AES256-GCM, so you get integrity as well as confidentiality.
printf '%s' 'hunter2' | sudo systemd-creds encrypt --name=dbpass - /etc/myapp/dbpass.credsudo systemd-creds decrypt --name=dbpass /etc/myapp/dbpass.cred -# hunter2Two things to notice. First, printf '%s' avoids a trailing newline that would become part of your password. Second, the credential name (dbpass) is embedded in the ciphertext. Per the man page, that stops someone renaming an encrypted file and reusing it for a different purpose: decryption fails if the name doesn’t match. Use --name= whenever the output path doesn’t carry the right name.
Which key encrypts it
The --with-key= flag takes host, tpm2, host+tpm2, null, auto, or auto-initrd. The default is auto:
- If a TPM2 is found and you’re not in a container, the TPM2 key is used.
- If
/var/lib/systemd/is on persistent media, the host key is used. That’s the file/var/lib/systemd/credential.secret, readable only by root. - On a typical machine with both, the credential is bound to the TPM2 and the OS install. You need both to decrypt.
Check whether you have a TPM2 first:
systemd-analyze has-tpm2(The older systemd-creds has-tpm2 now just tells you it was replaced by that command.)
No TPM2? Fine. The host key still protects the file against casual theft of the file itself: someone who copies your .cred file off the box, or finds it in a backup, can’t read it without credential.secret. Be clear about what that does not cover: root on the running machine can decrypt anything, because the key is on the same disk. The credential protects secrets at rest (backups, copied files), not against a compromised root account.
Avoid --with-key=null. The man page says it provides no confidentiality and no authenticity.
Wiring It Into a Service
Here’s a system unit:
[Unit]Description=Demo app using an encrypted credential
[Service]ExecStart=/opt/myapp/app.shDynamicUser=yesLoadCredentialEncrypted=dbpass:/etc/myapp/dbpass.credDynamicUser=yes works nicely here. The service manager reads the credential file as root and drops a copy only the dynamic user can see, so you don’t have to open up file permissions for a UID that doesn’t exist until the service starts.
If you’d rather not keep a separate file, systemd-creds encrypt with -p (long form --pretty) prints a ready-to-paste SetCredentialEncrypted= line. It needs --name= and - as the output:
printf '%s' 'hunter2' | sudo systemd-creds encrypt --name=dbpass -p - -The man page’s own recipe puts that output in a drop-in like /etc/systemd/system/xyz.service.d/50-password.conf, prefixed with [Service], then runs systemctl daemon-reload. A drop-in keeps the ciphertext out of the main unit file you might paste into a forum thread.
Reading the Secret in Your App
The app reads a file. That’s it. The path is $CREDENTIALS_DIRECTORY/<name>.
#!/bin/shsecret=$(cat "$CREDENTIALS_DIRECTORY/dbpass")import osfrom pathlib import Path
db_password = (Path(os.environ["CREDENTIALS_DIRECTORY"]) / "dbpass").read_text()Yes, $CREDENTIALS_DIRECTORY is itself an environment variable, but it holds a path, not a secret. Leaking it tells an attacker nothing they can use without already being the right user.
Apps that only know env vars or config files
Plenty of software wants the secret as a variable or in a config file. You have options, in order of preference:
- A
*_FILEvariable. Many apps (the official Postgres image’sPOSTGRES_PASSWORD_FILE, for example) accept a path instead of a value. The%dspecifier expands to the credentials directory, soEnvironment=POSTGRES_PASSWORD_FILE=%d/dbpassworks. The man page uses the same trick withEnvironment=MYCREDPATH=%d/mycred. ExecStart=interpolation.${CREDENTIALS_DIRECTORY}/mycredcan appear directly in the command line, as in the man page’sExecStart=cat ${CREDENTIALS_DIRECTORY}/mycred.- A fixed path. For system services only, the man page says you can also use
/run/credentials/UNITNAMEwhere the software can’t interpolate anything. It adds that$CREDENTIALS_DIRECTORYis the primary interface because it also works in user services.
The wrapper-script approach (read the file, export it, exec the app) works too, but it puts you right back in the environment, so reach for it last.
User Services Work Too
You don’t need root to try any of this. With systemd-creds encrypt --user, a per-user service manager can do the same thing. The man page notes that credentials for the user manager must be encrypted with systemd-creds encrypt --user, and those for the system manager without it. A user-encrypted credential can’t be decrypted by the system manager or another user’s manager (unless the attacker can gain those privileges), because the UID, username, and machine-id go into the key.
The kit ships demo-user.sh, which runs the whole comparison under systemd-run --user. The core of it:
systemd-creds --user encrypt --name=dbpass db.pass db.credsystemd-run --user --wait --pipe \ -p LoadCredentialEncrypted=dbpass:$PWD/db.cred \ sh -c 'ls -l "$CREDENTIALS_DIRECTORY"; cat "$CREDENTIALS_DIRECTORY/dbpass"'On systemd 262.1 the good run shows the 0500 directory and 0400 file from earlier, and env | grep for the password finds nothing. The bad run shows Environment=DB_PASSWORD=hunter2 right in systemctl show, plus the same line from /proc/PID/environ when read as the same user.
Gotchas
- Machine-bound by design. Copy
dbpass.credto another server and decryption fails, because the key lives in that machine’scredential.secretand/or TPM2. Moving a service means re-encrypting from the plaintext on the new host. Keep the plaintext in your password manager, not in a git repo. - systemd 262 changed the TPM2 format. The
--with-key=section of the man page says v262 and later pin the encrypted credential to the TPM2’s SRK, and that makes the result incompatible with earlier systemd versions. If you manage a mixed fleet, encrypt on the oldest version you run, or test the decrypt on each. - PCR binding bites on updates.
--tpm2-pcrs=binds to specific PCR values, and the default is no PCRs. Binding to PCRs that change on a kernel or firmware update locks you out. The man page points to--tpm2-public-key=for setups where updates must not break access. - Mind the trailing newline.
echo hunter2 > fileadds one. Many apps don’t strip it. Useprintf '%s'. - Rotation is a restart. Credentials are loaded when the service starts. Change the secret, re-encrypt,
systemctl restart. - Plaintext
LoadCredential=is still plaintext. It beatsEnvironment=on exposure, but the source file sits unencrypted on disk. Lock it to root with mode 0600 or use the encrypted variant.
The SumGuy Take
If a value would hurt to see in a screenshot, it does not belong in Environment= or EnvironmentFile=. Use LoadCredentialEncrypted= for anything you control, *_FILE plus %d for apps that want a path, and keep Environment= for log levels and feature flags.
The setup cost is two commands and one unit line. The payoff is that your next systemctl show, crash dump, or “can you paste your unit file?” request doesn’t turn into a password rotation. Your 2 AM self will thank you.
Common Questions
Do systemd credentials work in user services?
Yes. The per-user service manager supports the same directives. Encrypt with systemd-creds encrypt --user, which the man page marks as added in version 256, and the credential decrypts only in that user’s manager. In a user service, $CREDENTIALS_DIRECTORY points under /run/user/<uid>/credentials/, and no root access is needed at any step.
Can root read systemd credentials?
Yes. The man page says credential data is accessible only to the unit’s user and the superuser. Root can read the runtime directory, and on the same machine root can also decrypt the file using the host key. Credentials protect secrets at rest and from other users, not from a compromised root.
What happens if I copy an encrypted credential to another machine?
Decryption fails. The credential is encrypted with a key derived from the TPM2 chip, the host secret in /var/lib/systemd/credential.secret, or both, and the new machine has neither. Re-encrypt the plaintext on the target host. TPM2-bound credentials made on systemd 262 or later also cannot be decrypted by older versions.
Is SetCredential= safe for passwords?
No. The systemd.exec man page says SetCredential= values are accessible to unprivileged processes via IPC and should only hold non-secret data such as user IDs or public keys. For a secret, use LoadCredentialEncrypted= or paste an encrypted blob with SetCredentialEncrypted=.
