Fail2ban Setup Guide for Linux Servers: Jails, nftables and Testing

Fail2ban watches your server's logs for repeated failures (bad SSH passwords, failed mail logins, web login guessing) and temporarily blocks the offending IP address in the firewall. On a Linux server without a control panel, the right setup is: put all settings in jail.local (never edit jail.conf), read SSH failures from the systemd journal, use nftables or firewalld actions, ban repeat offenders for longer, whitelist your own addresses, and test every filter with fail2ban-regex. Fail2ban reduces noise and slows down guessing, but it does not replace key-only SSH logins.

This guide covers current Debian 12/13, Ubuntu 24.04 and RHEL/AlmaLinux 9/10 servers. (cPanel servers have their own tools; we will cover those separately.)

What fail2ban does, and what it does not

Fail2ban runs "jails". Each jail combines a filter (regular expressions that recognize a failure in a log), a log source (a file or the systemd journal) and an action (usually a firewall ban). When one IP produces maxretry failures within findtime, it is banned for bantime.

It is useful against:

• Password guessing on SSH, mail (Postfix SMTP AUTH, Dovecot IMAP/POP3) and web logins.
• Noisy scanners that fill logs and waste resources.

It is weak against:

• Distributed attacks. A botnet that tries one password from each of thousands of addresses never trips a per-IP threshold.
• Attacks that do not log a failure, such as exploits of a vulnerable application.
• Attackers with valid credentials. A stolen password logs in on the first try.

That is why fail2ban is a layer, not the lock. The lock is SSH key authentication with passwords disabled, covered in our Linux server hardening essentials.

Step 1: Install fail2ban

## Debian / Ubuntu
sudo apt update
sudo apt install fail2ban python3-systemd

## RHEL / AlmaLinux / Rocky (fail2ban comes from EPEL)
sudo dnf install epel-release     # on RHEL, enable EPEL per the Fedora EPEL docs
sudo dnf install fail2ban
sudo systemctl enable --now fail2ban

The current upstream release is fail2ban 1.1.1 (August 2026); distributions ship their own packaged versions. On Ubuntu 24.04, confirm the service actually started with systemctl status fail2ban: the original 24.04 package failed under Python 3.12 (Launchpad bug 2055114), so make sure you have the latest updates installed.

Step 2: Never edit jail.conf, create jail.local

The header of jail.conf says it plainly: do not modify this file, because package updates can overwrite it. Put your changes in /etc/fail2ban/jail.local or in files under /etc/fail2ban/jail.d/ ending in .local. Settings in .local files override the defaults.

Many distribution packages already drop a file in jail.d/ (for example, Debian's defaults-debian.conf enables the sshd jail). Check what is there first with ls /etc/fail2ban/jail.d/.

A sensible starting jail.local:

[DEFAULT]
## Your office static IP, VPN range and monitoring server. Never ban these.
ignoreip = 127.0.0.1/8 ::1 203.0.113.10 10.8.0.0/24

bantime  = 1h
findtime = 10m
maxretry = 5

## Repeat offenders get progressively longer bans
bantime.increment = true
bantime.factor    = 1
bantime.maxtime   = 1w

## nftables actions (Debian 12/13, Ubuntu 24.04 with nftables)
banaction          = nftables-multiport
banaction_allports = nftables-allports

Replace the example addresses with your own. Forgetting ignoreip is the classic way to lock your own office out after someone mistypes a password a few times.

Which ban action to use

• Debian/Ubuntu with nftables: nftables-multiport and nftables-allports are shipped in /etc/fail2ban/action.d/. Recent Debian packages already default to nftables.
• RHEL/AlmaLinux with firewalld: use the firewalld actions (firewallcmd-rich-rules, firewallcmd-ipset or firewallcmd-multiport). The EPEL fail2ban-firewalld package configures this for you, so do not override banaction with nftables on those servers.
• Ubuntu with ufw: a ufw action exists, but mixing tools makes troubleshooting harder; pick one firewall front end per server.

How bantime.increment works

With bantime.increment = true, fail2ban remembers previous bans in its database and multiplies the ban time each time the same address returns, up to bantime.maxtime. The first ban is short, so a staff member who mistypes a password is not locked out for long, while a persistent scanner ends up blocked for a week. The [recidive] jail is an older alternative that bans addresses which keep reappearing in fail2ban's own log; you do not need both.

Step 3: The sshd jail with the systemd backend

Modern servers log SSH to the systemd journal. Debian 12 and later no longer install rsyslog by default, so /var/log/auth.log may not exist at all, and a file-based jail would silently watch nothing. Use the journal:

[sshd]
enabled  = true
backend  = systemd
port     = ssh
maxretry = 4
## Optional: also catch pre-auth probing and other noise
mode     = aggressive

The upstream filter matches the sshd.service unit and both the sshd and sshd-session process names (newer OpenSSH releases log authentication from sshd-session). On Debian and Ubuntu the unit is called ssh.service, so add the matching line there:

journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd + _COMM=sshd-session

If you moved SSH to another port, set port = 2222 (or your port); otherwise the ban will block the wrong port.

Step 4: Jails for web and mail services

Enable only jails for services that actually run on the server.

[nginx-http-auth]
enabled = true

[nginx-botsearch]
enabled = true

[apache-auth]
enabled = true

[postfix]
enabled = true
mode    = aggressive

[dovecot]
enabled = true

Notes:

• nginx-http-auth and apache-auth catch failed HTTP basic authentication. They do not understand your application's own login form; protect that with rate limits or a WAF.
• nginx-botsearch bans clients probing for common admin paths that do not exist.
• postfix and dovecot cover SMTP AUTH and IMAP/POP3 password guessing. On servers where mail logs only go to the journal (Debian 12 and later without rsyslog), set backend = systemd for these jails too and confirm with fail2ban-regex that the filter actually matches; otherwise point logpath at your mail log.

Step 5: Test before you trust it

fail2ban-regex runs a filter against real logs and reports how many lines matched. Always run it after enabling a jail:

## A file-based log
sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-http-auth.conf

## The systemd journal (Debian/Ubuntu unit name shown)
sudo fail2ban-regex systemd-journal sshd \
  -m "_SYSTEMD_UNIT=ssh.service + _COMM=sshd + _COMM=sshd-session"

A jail with zero matches on a server that is clearly being probed is misconfigured. Then reload and check that the jails are running:

sudo fail2ban-client -t            # test the configuration
sudo systemctl reload fail2ban
sudo fail2ban-client status        # list jails
sudo fail2ban-client status sshd   # failures and banned IPs for one jail

Step 6: Check and lift bans

sudo fail2ban-client status sshd                 # current bans
sudo fail2ban-client set sshd unbanip 198.51.100.7
sudo fail2ban-client unban 198.51.100.7          # remove from all jails
sudo nft list ruleset | less                     # see the firewall side

If a staff member is banned repeatedly, the problem is usually a saved old password in a mail client or a script, not an attack. Fix the client, then unban.

Pair it with key-only SSH

Fail2ban only reacts after failures. The configuration that makes SSH password guessing pointless is:

• PasswordAuthentication no and KbdInteractiveAuthentication no in sshd_config (unless you use keyboard-interactive for 2FA).
• PermitRootLogin no (or prohibit-password where root must log in with a key).
• SSH reachable only from your office or VPN where possible, through the firewall.
• Automatic security updates, so a vulnerable daemon is not the way in.

With those in place, fail2ban mostly keeps logs readable and reduces load. It remains valuable for mail and web logins, where passwords are still in use.

Quick checklist

• Settings in jail.local or jail.d/*.local, never jail.conf.
• ignoreip contains your office, VPN and monitoring addresses.
• sshd jail on the systemd backend with the right unit name.
• Ban action matches your firewall (nftables or firewalld).
• bantime.increment enabled with a sensible bantime.maxtime.
• Every jail tested with fail2ban-regex.
• Key-only SSH and patching in place, because fail2ban is not a substitute.

Need help locking down your Linux servers?

Core Florida IT configures and maintains Linux servers for businesses across Miami-Dade and South Florida, with remote support nationwide, including server setup and repair and cybersecurity reviews. If you are not sure what protects your servers today, request a free IT assessment and we will walk through it with you.

Sources

• Fail2ban project: jail.conf (default configuration and comments)
• Fail2ban project: sshd filter
• Fail2ban releases
• fail2ban-regex(1) manual page
• Debian 12 release notes: changes to the logging system
• Ubuntu Launchpad bug 2055114: fail2ban on 24.04