Most Linux log guides open with a list of files in /var/log. Here is that list run against a current Arch box:
$ ls /var/log/{syslog,messages,auth.log,kern.log,dmesg}ls: cannot access '/var/log/syslog': No such file or directoryls: cannot access '/var/log/messages': No such file or directoryls: cannot access '/var/log/auth.log': No such file or directoryls: cannot access '/var/log/kern.log': No such file or directoryls: cannot access '/var/log/dmesg': No such file or directoryFive for five. Nothing on a default systemd install writes those files any more. rsyslog is not installed by default on Arch, Fedora, or a minimal Debian 12, and without it the events go to the journal and nowhere else. If you inherited a grep /var/log/syslog habit from a 2015 tutorial, it has been silently finding nothing for a while.
So: journalctl first. Flat files second, for the servers that still have them and for application logs, which have never been in the journal. And a section at the end for the popular one-liners that do not work, because several of the most-copied ones return empty and look like a quiet system.
journalctl, the Twelve Flags That Matter
Everything below was run against systemd 261.
journalctl -p err -b # errors only, this bootjournalctl -u sshd --since "1 hour ago"journalctl -k # kernel ring buffer, replaces dmesgjournalctl -f -u nginx # follow one unitjournalctl -g 'timed out' -S today # regex over MESSAGEjournalctl -b -1 -p warning # previous boot, warnings and worsejournalctl --disk-usage-p takes the syslog priority names, and it is inclusive of everything more severe. -p err gives you err, crit, alert and emerg. -p warning widens it. This one flag replaces most of the grep -i "error\|warn" pipelines people write.
-g is a regex search over the message field only, which is the difference between finding your error and finding every line that happens to contain the hostname.
-S and -U take human dates. -S "2026-08-29 09:00", -S yesterday, -U "10 min ago". Time-boxing is the single highest-leverage habit in log work, and journalctl makes it a flag instead of a regex.
Watch your disk:
$ journalctl --disk-usageArchived and active journals take up 47.1M in the file system.Cap it in /etc/systemd/journald.conf with SystemMaxUse=500M before it caps itself at 10% of the filesystem.
The Part That Makes journald Worth It
The journal is structured. Every entry carries indexed fields, so you can filter on them directly rather than guessing at column positions:
journalctl _UID=1000 -S todayjournalctl _SYSTEMD_UNIT=docker.service -p errjournalctl _COMM=sudo -S "7 days ago"And you can get real data out instead of text:
journalctl -p err -b -o json | jq -r '[._SYSTEMD_UNIT, .MESSAGE] | @tsv'That last one is the answer to almost every “count errors per service” question, and it does not care how many spaces are in the message.
Flat Files: grep and awk Still Earn Their Keep
Application logs are still files. Nginx, Apache, Postgres, and every container that writes to stdout and gets captured somewhere. Two tools cover most of it.
grep for finding lines. -i case-insensitive, -c count, -A3 -B3 for context around a hit, -E for extended regex.
awk for pulling fields out of lines that have a consistent shape. The thing to internalise is that awk splits on runs of whitespace and numbers fields from 1, and $NF is the last field regardless of how many there are.
# top source IPs of failed SSH loginsgrep -E 'sshd\[[0-9]+\]: Failed password' /var/log/auth.log \ | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
# biggest things in /var/logdu -sh /var/log/* | sort -h | tailThe failed-password one works because the line always ends from <ip> port <n> ssh2, so counting back from the end is stable whether or not the username was valid.
Five One-Liners That Get Copied Around and Return Nothing
An earlier version of this post recommended all of these. Every one of them was tested for this rewrite, and the output below is real.
1. The load average pipeline that prints nothing at all
awk '{print $1,$2,$NF}' /var/log/messages | grep "load average"Exit code 1, no output. awk runs first and reduces each line to three fields, which throws away the words “load average” before grep ever sees them. The pipeline can never match.
Order matters. Filter, then extract:
$ grep "load average" /var/log/messages | awk '{print $1, $2, $3, $(NF-2), $(NF-1), $NF}'Feb 21 10:24:01 0.42, 0.55, 0.612. The ping latency one-liner that prints an IP address
grep "ping statistics" /var/log/messages | awk '{print $1,$2,$7}'Sold as “isolates date, time, and the round-trip ping time in milliseconds”. What it actually prints:
Feb 21 8.8.8.8Field 7 of a --- 8.8.8.8 ping statistics --- header is the target address. The round-trip numbers live on the following line, which the grep never selected. On top of that, nothing sends ping output to syslog unless you built that yourself, so on a normal machine this matches zero lines regardless.
If you want latency trends, log them on purpose with a small script writing to its own file, or use a real monitoring agent.
3. Counting processes by a field that includes the PID
awk '{print $5}' /var/log/auth.log | sort | uniq -cField 5 is sshd[2101]:, PID included. Every line gets its own bucket:
1 sshd[2101]: 1 sshd[2102]: 1 sshd[2103]: 1 sshd[2104]: 1 sudo:A count where every value is 1 is not a count. Strip the PID first:
$ awk '{print $5}' /var/log/auth.log | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -rn 4 sshd: 1 sudo:4. Watching HTTP status codes in the wrong file
tail -f /var/log/apache2/error.log | grep -E --color '40[0-9]|50[0-9]'Two problems stacked. HTTP status codes are in access.log. The error.log records PHP warnings, config problems and segfaults, and has no status column.
The regex is also far too loose. Run it over a real error line and watch what it grabs:
$ echo '[php:error] [pid 405] uncaught exception, 5012 bytes' | grep -Eo '40[0-9]|50[0-9]'405501It matched a process ID and three digits out of the middle of a byte count. Neither is an HTTP status.
Anchor to the status field in the file that has one:
tail -f /var/log/apache2/access.log | awk '$9 ~ /^[45][0-9][0-9]$/ {print $9, $7}'5. Finding other people’s sudo usage with whoami
grep "sudo" /var/log/auth.log | grep -v "$(whoami)"Two failure modes and both are silent. Run it as yourself and it filters out the only account that is realistically in there, so you get nothing and conclude the box is clean. Run it under sudo and whoami returns root, so it filters on the wrong name entirely.
Name the account explicitly, and pull the command while you are there:
$ sed -n 's/.* sudo: *\([^ ]*\) : .*COMMAND=\(.*\)/\1 -> \2/p' /var/log/auth.logkingpin -> /usr/bin/apt updateOr on a journald system, skip the parsing:
journalctl _COMM=sudo -S "7 days ago" -o json | jq -r '.MESSAGE'Bonus: the port scan detector that detects brute force
The failed-password IP counter is a good command. It was labelled “Detecting Port Scans”, and it does not do that. Failed SSH passwords mean somebody reached your SSH port and guessed. A port scan is somebody knocking on ports where nothing answers, which never touches auth.log.
Port scans show up in your firewall log. On a UFW box that lands in the kernel journal:
journalctl -k -g 'UFW BLOCK' -S today | grep -oP 'SRC=\K[0-9a-f.:]+' | sort | uniq -c | sort -rn | headOne source hitting many distinct DPT= values in a short window is a scan. One source hammering a single port is brute force. Different problem, different response.
Web Server Logs Cannot Tell You About Latency by Default
This deserves its own warning because it comes up constantly. The combined log format Apache and Nginx ship with contains the client IP, the request, the status, and the response size. There is no duration field. Any one-liner claiming to show you “page load times” from a stock access log is extracting something else and mislabelling it.
Add the field yourself. In Nginx:
log_format timed '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" $request_time';
access_log /var/log/nginx/access.log timed;Then the query works, because the data exists:
$ awk '{print $NF, $7}' /var/log/nginx/access.log | sort -rn | head2.905 /slow0.412 /index.htmlApache’s equivalent is %D in LogFormat, which gives microseconds.
Container Logs Are a Third Place Entirely
Neither journald nor /var/log covers the thing most of us are actually debugging. Docker writes container stdout to JSON files under /var/lib/docker/containers/<id>/<id>-json.log, and you read them with docker logs, which supports the same time-boxing habit:
docker logs --since 15m --timestamps nginxdocker logs --tail 200 -f postgres 2>&1 | grep -i fatalTwo things bite people here. The 2>&1 matters, because docker logs writes the container’s stderr to your stderr, and a bare pipe to grep silently drops it. Most application errors are on stderr. That one omission is why “I grepped the logs and there was nothing” happens.
The other is retention. The default json-file driver has no size limit, so a chatty container can fill the disk, and when you restart the container to fix something, you lose nothing but you also gain nothing because the file keeps growing. Cap it per service:
services: app: image: myapp:latest logging: driver: json-file options: max-size: "10m" max-file: "3"Or better, hand the whole problem to systemd and get one query language for the entire box:
logging: driver: journaldThe driver writes CONTAINER_NAME, CONTAINER_ID and CONTAINER_TAG as indexed fields on every entry, so journalctl CONTAINER_NAME=app -S "10 min ago" works, -p err works, and -o json hands you the container name without parsing anything. On a host where you already live in journalctl, this is the single highest-value change in this article.
Working Through a Failure, In Order
Theory is cheap, so here is the sequence applied to a common shape of problem: a service is up, the health check is green, and users get intermittent 502s.
Start wide and time-boxed, not narrow:
journalctl -p warning -S "30 min ago"Priority filter first, because it cuts the volume hard without you having to guess at a keyword. Say that surfaces repeated nginx warnings about upstream timeouts. That already tells you the proxy is fine and the thing behind it is not.
Then pin the unit and widen the priority, because the interesting line is often info:
journalctl -u app.service -S "30 min ago" -o short-precise-o short-precise gives microsecond timestamps. Correlating two services to the second is not enough when the failure is a timeout.
Suppose the give-away is a cluster of connection-pool messages arriving in bursts. Confirming a pattern like that is a counting job, not a reading job:
journalctl -u app.service -S "2 hours ago" -o json \ | jq -r '.__REALTIME_TIMESTAMP' \ | awk '{print strftime("%H:%M", $1/1000000)}' | uniq -cThat prints events per minute. A flat line is normal load. A spike on a fixed interval is a scheduled job, and a backup script saturating the same database the app pools against will produce exactly that signature.
Three habits did the work there. Filter by priority before keyword. Time-box before anything. Count before reading. None of them require a tool you do not already have.
Habits Worth Building
Time-box before you filter. -S "10 min ago" on a journalctl query, or grep a timestamp prefix on a file. Reading the whole log and then narrowing is backwards, and on a busy server it is the difference between an answer and a coffee break.
Read the priority, not the word “error”. Applications write the string “error” into informational messages constantly. -p err filters on what the program actually declared.
Check -b -1 after a crash. The previous boot is where the reason lives. journalctl -b -1 -p err is the first command to run after an unexplained reboot.
Ship them somewhere before you need them. A log that rotated away is a log you do not have. Loki is the low-overhead choice for a home lab and keeps the query language close to what you already think in.
Common Questions
Where are Linux log files if /var/log/syslog does not exist?
In the systemd journal, at /var/log/journal/, in a binary format you read with journalctl. Arch, Fedora and minimal Debian 12 installs ship no rsyslog, so no flat text files are written. Install rsyslog if you specifically need /var/log/syslog for a tool that parses it.
How do I see logs from the previous boot?
Run journalctl -b -1, where -1 means one boot back. Add -p err to cut it down to errors. This only works when persistent storage is on: check that /var/log/journal/ exists, and create it with sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald if it does not.
Can I get response times out of my Nginx access log?
Not from the default format. The stock combined log has no duration field, so any one-liner promising page load times from it is reading the wrong column. Add $request_time to a custom log_format and point access_log at it. Apache uses %D for microseconds in LogFormat.
Is grep or journalctl faster on a big log?
journalctl, when you filter on indexed fields like -u, -p, _UID or _COMM, because it seeks rather than scanning. Plain -g regex search is a full scan and comparable to grep. For flat files, ripgrep beats grep on large directories by a wide margin and respects the same patterns.
How do I stop the journal from eating my disk?
Set SystemMaxUse=500M in /etc/systemd/journald.conf and restart systemd-journald. Without a cap, journald limits itself to 10% of the filesystem, which is a lot on a big disk. Check the current figure any time with journalctl --disk-usage, and reclaim space now with journalctl --vacuum-size=200M.
