Linux Security Logging: journald, auditd and Centralized Logs

When something goes wrong on a Linux server, logs are how you find out what happened, and attackers know it: wiping or editing logs is one of the first things an intruder does. Make logs useful for security in five steps: keep the systemd journal persistent with a clear retention limit, keep every server's clock accurate with chrony, record sensitive changes with auditd (user accounts, sudo rules, SSH configuration), forward everything over TLS to a central log server the application servers cannot modify, and review a short list of events every week.

The commands below apply to current Debian 12/13, Ubuntu 24.04 and RHEL/AlmaLinux 9/10.

Which logs matter for security

You do not need to read everything. These sources answer most security questions:

• Authentication: SSH logins (successful and failed), sudo use, local console logins.
• Account and privilege changes: new users, password changes, group membership, edits to sudo rules.
• Configuration changes to SSH, the firewall, cron jobs and services.
• Services: web server, mail and application logs, especially errors and login attempts.
• Security tools: fail2ban bans, malware scanners, package updates.

Step 1: Make the journal persistent, with limits

On modern distributions, most of these events land in the systemd journal. If the journal is kept only in memory, a reboot erases it, which is exactly when you may need it.

Check what you have:

ls -ld /var/log/journal          # exists = persistent storage
journalctl --list-boots | head   # several boots listed = history survives reboots
journalctl --disk-usage

Upstream systemd documents persistent as the default today, while older versions used auto (persistent only if /var/log/journal exists). Note that Debian 12 and later no longer install rsyslog by default, so the journal may be the only local log. Set it explicitly with a drop-in file rather than editing journald.conf:

/etc/systemd/journald.conf.d/50-retention.conf:

[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=90day

sudo systemctl restart systemd-journald

SystemMaxUse caps disk use (by default 10% of the file system, up to 4 GB) and MaxRetentionSec removes entries older than the given age (disabled by default). Choose a retention period that fits your business and any regulatory or contractual requirements you have.

Useful queries:

journalctl -u ssh --since "24 hours ago"      # sshd.service on RHEL
journalctl _COMM=sudo --since today
journalctl -p warning --since yesterday

Step 2: Keep time in sync with chrony

Logs from several servers are only useful if their timestamps agree. RHEL and AlmaLinux use chrony by default; on Debian and Ubuntu, installing chrony replaces the simpler systemd-timesyncd.

sudo apt install chrony            # Debian / Ubuntu
sudo systemctl enable --now chronyd   # RHEL / AlmaLinux (service is "chrony" on Debian/Ubuntu)
chronyc tracking
chronyc sources -v
timedatectl

Point all servers at the same reliable time sources and alert if a server's offset drifts.

Step 3: Record sensitive changes with auditd

The Linux Audit system records events at the kernel level, including who changed a watched file, even when the change happened through a tool that does not log on its own. Install it on Debian/Ubuntu with sudo apt install auditd; it is installed by default on RHEL.

Put rules in a file under /etc/audit/rules.d/:

/etc/audit/rules.d/50-business.rules:

-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/ssh/sshd_config.d/ -p wa -k sshd_config
-w /root/.ssh/ -p wa -k root_ssh
-w /etc/crontab -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
## Commands run as root by real logged-in users (can be noisy; test first)
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=unset -k admin_cmds

-w watches a path, -p wa records writes and attribute changes, and -k adds a key you can search for. Load and check the rules:

sudo augenrules --load
sudo auditctl -l

On RHEL, Red Hat notes that the service command is the correct way to restart auditd (sudo service auditd restart); use systemctl only to enable it or check its status.

Search the results:

sudo ausearch -k identity -i --start today
sudo ausearch -k sudoers -i
sudo aureport --login --summary -i

Once the rules are stable, adding -e 2 as the last rule makes the audit configuration immutable until the next reboot, so an intruder cannot quietly switch auditing off. Red Hat also ships sample rule sets in /usr/share/audit/sample-rules/, with the warning that they are a starting point, not proof of compliance.

Step 4: Ship logs off the server

Logs that live only on the server can be deleted by anyone who gains root there. A central log server keeps a copy that the application servers can send to but not change. rsyslog is the standard tool and supports TLS.

Install it on each client and on the log server. On Debian 12 and later rsyslog is no longer installed by default; on RHEL it is installed and reads the journal through its imjournal module.

sudo apt install rsyslog rsyslog-gnutls     # Debian / Ubuntu
sudo dnf install rsyslog rsyslog-gnutls     # RHEL / AlmaLinux

You will need a small private certificate authority: one CA certificate, plus a certificate and key for the log server and each client.

Client: forward everything over TLS

/etc/rsyslog.d/90-forward.conf:

global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog-tls/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog-tls/web01-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog-tls/web01-key.pem"
)
*.* action(type="omfwd" target="logs.example.internal" port="6514" protocol="tcp"
      StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name"
      StreamDriverPermittedPeers="logs.example.internal"
      queue.type="LinkedList" queue.filename="fwd_central"
      queue.maxDiskSpace="1g" queue.saveOnShutdown="on"
      action.resumeRetryCount="-1")

The disk-assisted queue holds messages if the log server is unreachable and sends them when it returns. StreamDriverAuthMode="x509/name" makes the client verify it is talking to the real log server.

Log server: receive and store per host

/etc/rsyslog.d/10-receive.conf:

global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog-tls/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog-tls/logs-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog-tls/logs-key.pem"
)
module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1"
       StreamDriver.AuthMode="x509/name" PermittedPeer=["*.example.internal"])
template(name="PerHost" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
ruleset(name="remote") { action(type="omfile" dynaFile="PerHost") }
input(type="imtcp" port="6514" ruleset="remote")

Validate with rsyslogd -N1, restart rsyslog, open TCP 6514 only to your servers, then test with logger "test from web01" on a client. On RHEL with SELinux, confirm the port you use is allowed for rsyslog (semanage port -l | grep syslog). Lock down the log server itself: few admins, no shared accounts, and no SSH access from the application servers.

To get audit events to the log server too, enable the audit syslog plugin (active = yes in /etc/audit/plugins.d/syslog.conf; on some distributions it comes in the audispd-plugins package) or have rsyslog read /var/log/audit/audit.log.

Step 5: Rotate and retain logs

• Text logs are rotated by logrotate. On the central server, add a rule for the per-host files:

/etc/logrotate.d/remote-logs:

/var/log/remote/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl kill -s HUP rsyslog.service
    endscript
}

• auditd rotates its own log through max_log_file, num_logs and max_log_file_action in /etc/audit/auditd.conf.
• The journal is trimmed by the limits from Step 1.
• Back up the central log server, as you would any other important server.

What to review every week

A 15-minute weekly review catches most problems early:

1. Successful SSH logins from unexpected addresses or at unusual hours.
2. Spikes in failed logins (SSH, mail, admin panels) and fail2ban bans.
3. Any identity, sudoers, sshd_config or root_ssh audit events, matched against approved changes.
4. New cron jobs or systemd services nobody remembers creating.
5. Gaps: a server that stopped sending logs is a warning sign on its own.
6. Time drift reported by chrony.

If you find something you cannot explain, follow our guide on the first 24 hours after a breach and preserve the logs before changing anything. For the rest of the server baseline, see Linux server hardening essentials.

Need help making your logs useful?

Core Florida IT sets up logging, auditing and monitoring for Linux servers at businesses across Miami-Dade and South Florida, with remote support nationwide, as part of our cybersecurity and managed IT support services. If you are not sure your servers would tell you what happened after an incident, request a free IT assessment.

Sources

• systemd: journald.conf manual
• Debian 12 release notes: changes to the logging system
• Red Hat Enterprise Linux 9: Auditing the system
• Red Hat Enterprise Linux 9: Configuring a remote logging solution
• rsyslog documentation: omfwd forwarding output module
• rsyslog documentation: imtcp input module
• chrony documentation