System Logs

When something goes wrong, or something suspicious happens, the logs are where you look first. On a modern Ubuntu system that means the systemd journal, a structured binary log that captures output from every service, the kernel, and authentication events, all queryable through one tool. Knowing how to slice that data quickly, how to make it survive a reboot, and how to get a copy off the host is foundational to every investigation in this foundation.

This page covers three things: driving journalctl to answer real questions, recognizing what an attack looks like in the auth logs, and forwarding logs somewhere an attacker cannot reach.

journald and the older /var/log files

Ubuntu 24.04 runs systemd-journald as the primary log store, but plenty of tooling still writes plain-text files under /var/log. Understanding which is which saves confusion.

Source	Location	Query with
systemd journal	/run/log/journal (volatile) or /var/log/journal (persistent)	journalctl
Auth and sudo events	journal, and /var/log/auth.log if rsyslog installed	journalctl or grep
Kernel ring buffer	journal	journalctl -k or dmesg
Package manager	/var/log/apt/, /var/log/dpkg.log	grep, less

By default the journal is volatile: it lives in /run and is lost on reboot. For any host you care about, that is the first thing to fix.

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl kill --kill-who=main --signal=SIGUSR1 systemd-journald
journalctl --header | grep -m1 'File Path'

Expected output:

File Path: /var/log/journal/9f1c.../system.journal

A path under /var/log/journal (not /run) confirms the journal now persists across reboots.

Querying with journalctl

The power of journalctl is filtering. A few flags cover the majority of investigations.

# everything from the SSH service since boot, newest last
journalctl -u ssh.service -b --no-pager | tail -n 4

Expected output:

Jul 06 09:12:03 lab sshd[1420]: Accepted publickey for ubuntu from 10.0.0.5 port 51234 ssh2
Jul 06 09:12:03 lab sshd[1420]: pam_unix(sshd:session): session opened for user ubuntu
Jul 06 09:41:55 lab sshd[1601]: Received disconnect from 10.0.0.5 port 51234:11: disconnected
Jul 06 09:41:55 lab sshd[1601]: pam_unix(sshd:session): session closed for user ubuntu

Filter by time, by priority, and by field. Priority err and above surfaces problems quickly; time ranges narrow an investigation window.

journalctl --since "2026-07-06 09:00" --until "2026-07-06 10:00" -p err --no-pager

Expected output:

Jul 06 09:33:10 lab kernel: audit: type=1400 apparmor="DENIED" operation="open" ...

Use -o verbose to see every structured field on a matched entry, including _UID, _PID, and _SYSTEMD_UNIT, which is what makes the journal useful for attribution.

What a brute-force attempt looks like

Failed authentication is one of the clearest signals in the logs. A password-guessing attack against SSH produces a burst of Failed password lines, often from a single IP, cycling through usernames.

journalctl -u ssh.service --since "-1h" --no-pager | grep -c "Failed password"
journalctl -u ssh.service --since "-1h" --no-pager | grep "Failed password" | tail -n 3

Expected output:

847
Jul 06 10:02:11 lab sshd[2210]: Failed password for invalid user admin from 185.220.101.7 port 40122 ssh2
Jul 06 10:02:12 lab sshd[2212]: Failed password for invalid user root from 185.220.101.7 port 40188 ssh2
Jul 06 10:02:13 lab sshd[2214]: Failed password for invalid user oracle from 185.220.101.7 port 40201 ssh2

847 failures in an hour from one address, cycling admin, root, and oracle, is a textbook automated attack. Two things follow. First, key-only SSH auth makes this noise harmless. Second, a tool like fail2ban reads these same log lines and bans the source IP automatically. The point of recognizing the pattern is that the same signal shows up in the audit subsystem (see Auditd) with fuller detail, and a successful login after a burst of failures is the line worth alerting on.

Forwarding logs off the host

An attacker who gains root can edit or delete local logs, including the journal. The defense is to ship a copy elsewhere in near real time, so the evidence exists somewhere the attacker does not control. systemd-journal-upload sends the journal to a remote collector; rsyslog can forward over TCP or TLS to a central syslog server.

sudo apt-get install -y systemd-journal-remote
sudo systemctl edit systemd-journal-upload.service
# set: URL=https://logs.internal.example:19532
sudo systemctl enable --now systemd-journal-upload.service
systemctl is-active systemd-journal-upload.service

Expected output:

active

The collector should live on a separate host or network segment. Even a simple setup that copies auth events to another machine turns “the attacker erased the logs” into “the attacker erased the local copy but not the remote one”. This off-host copy is the integrity property the Overview called essential.

Practical Guidance

1. Make the journal persistent (/var/log/journal) on any host you care about, since the default volatile store is lost on every reboot.
2. Learn a handful of journalctl filters (-u, -b, --since, -p, -o verbose) so you can narrow to the relevant events in seconds.
3. Treat a burst of Failed password lines from one IP as an automated attack, and alert specifically on a success that follows such a burst.
4. Prefer key-only SSH so brute-force noise is harmless, and let fail2ban act on the same log lines automatically.
5. Forward logs to a separate host or network in near real time, because local logs are only as trustworthy as the (possibly compromised) machine holding them.
6. Verify off-host forwarding actually works by generating an event and confirming it arrives at the collector before you depend on it.