Policy-Based Routing (PBR) on Linux: Multiple Routing Tables, ip rule & fwmark

Created: 2026-11-08 · Last updated: 2026-11-08 · Nejat Hakan

Executive Summary: Standard IP routing makes decisions strictly based on the Destination IP address. When a server has multiple internet uplinks (Multi-WAN multihoming), isolated tenant subnets, or policy-driven VPN tunnels, destination-only routing inevitably causes asymmetric routing failures and connection drops. Policy-Based Routing (PBR) unlocks the Linux kernel's Routing Policy Database (RPDB), allowing engineers to route packets based on Source IP, Ingress Interface, or Netfilter Firewall Marks (fwmark) across multiple independent routing tables.

The RPDB Architecture

Understand the Routing Policy Database (ip rule), priority order evaluation (0–32767), and the default tables (local, main, default).
:material-table-split:{ .lg .middle } Custom Routing Tables

Register named routing tables in /etc/iproute2/rt_tables to isolate ISP-specific gateways and VPN interfaces from the main table.
Multi-WAN Multihoming

Solve the asymmetric return-path dilemma: ensure traffic arriving on WAN-2 always exits via WAN-2's gateway, regardless of the default route.
Firewall Marking (fwmark)

Classify traffic dynamically in nftables (by port, protocol, or systemd cgroup) and steer matching packets through dedicated routing tables.

1. The Dilemma of Destination-Only Routing

Under standard routing, the Linux kernel maintains a single primary routing table (main). When an application sends a packet to 93.184.216.34, the kernel looks only at the destination IP:

Destination: 93.184.216.34 -> Match default via 203.0.113.1 dev eth0 (ISP 1)

The Asymmetric Routing Trap in Multi-Homed Servers

Consider a server connected to two ISPs: * eth0 (ISP 1): 203.0.113.10 (Default Gateway: 203.0.113.1) * eth1 (ISP 2): 198.51.100.20 (Gateway: 198.51.100.1)

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    CLIENT["Internet Client"] ==>|"1. SYN Packet arrives on eth1 (ISP 2)"| SRV["Multi-Homed Debian Server"]
    SRV -.->|"2. Kernel consults default route (ISP 1)!<br/>SYN-ACK exits eth0 with Src IP 198.51.100.20"| ISP1["ISP 1 Upstream Router"]
    ISP1 -->|"3. DROPPED by ISP Anti-Spoofing (BCP 38)!"| BLACKHOLE["Packet Lost / Client Hangs"]

Because the kernel's default route points to ISP 1, the return packet (SYN-ACK) exits via eth0, carrying eth1's source IP. ISP 1's upstream router detects a spoofed IP and drops it, or stateful firewalls drop the asymmetric return flow.

2. The Routing Policy Database (RPDB)

Linux solves this via the Routing Policy Database (RPDB). Instead of immediately consulting the routing table, the kernel first evaluates a prioritized list of rules:

ip rule show

Default output on a fresh Debian system:

0:     from all lookup local
32766: from all lookup main
32767: from all lookup default

Priority 0 (local table): Reserved for loopback, local interface IPs, and broadcast addresses. Cannot be removed.
Priority 32766 (main table): The standard routing table populated by ip route add or DHCP.
Priority 32767 (default table): Previously used for fallback routes; rarely used today.

3. Registering Custom Routing Tables

Custom routing tables are identified internally by an integer ID (1 to 2147483647). To make administration human-readable, map table IDs to symbolic names in /etc/iproute2/rt_tables:

# Append custom named tables to /etc/iproute2/rt_tables
sudo bash -c 'echo "201 isp2" >> /etc/iproute2/rt_tables'
sudo bash -c 'echo "202 vpn_out" >> /etc/iproute2/rt_tables'

Verify contents:

cat /etc/iproute2/rt_tables

4. Multi-WAN Multihoming Solution

To solve the dual-ISP asymmetric routing problem, we enforce a strict policy: Traffic originating from ISP 2's IP must always use ISP 2's routing table.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    A["Outgoing Packet"] --> B{"Evaluate ip rules in priority order"}
    B -->|"Source IP == 198.51.100.20"| C["Lookup Table 'isp2'<br/>Default via 198.51.100.1 dev eth1"]
    B -->|"All other traffic"| D["Lookup Table 'main'<br/>Default via 203.0.113.1 dev eth0"]

Step 1: Populate Table `isp2`

Create a dedicated default route and local subnet route inside table isp2:

# Add default gateway for ISP 2 into table isp2
sudo ip route add default via 198.51.100.1 dev eth1 table isp2

# Add local subnet route so table isp2 knows how to reach directly connected hosts
sudo ip route add 198.51.100.0/24 dev eth1 src 198.51.100.20 table isp2

Step 2: Create the Policy Rule

# Any packet with source IP 198.51.100.20 must consult table isp2
sudo ip rule add from 198.51.100.20 table isp2 priority 1000

Step 3: Verify the Kernel Route Selection

Query the kernel routing engine to prove the rule works:

# Query route for an internet IP from ISP 2's source address
ip route get 8.8.8.8 from 198.51.100.20
# Output: 8.8.8.8 from 198.51.100.20 via 198.51.100.1 dev eth1 table isp2

# Query route without specifying source (falls back to main table)
ip route get 8.8.8.8
# Output: 8.8.8.8 via 203.0.113.1 dev eth0 table main

5. Advanced Steering: Firewall Marks (`fwmark`)

Source IP steering only works if the source IP is known before routing occurs. If you want to route traffic based on Destination Port (e.g., steer all BitTorrent or S3 backup traffic through a WireGuard VPN while web browsing uses WAN-1), use Netfilter Firewall Marks:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    A["Application generates S3 Backup traffic (TCP 9000)"] --> B["nftables PREROUTING / OUTPUT Hook<br/>Mark packet with fwmark 0x64 (100)"]
    B --> C["Kernel RPDB evaluates: ip rule fwmark 100 lookup vpn_out"]
    C --> D["Table vpn_out routes packet via wg0 tunnel"]

1. Populate Table `vpn_out`

sudo ip route add default dev wg0 table vpn_out

2. Add `fwmark` Policy Rule

# Match any packet marked with hex 0x64 (decimal 100)
sudo ip rule add fwmark 100 table vpn_out priority 2000

3. Mark Matching Packets in `nftables`

Add the mark rule in /etc/nftables.conf:

table inet pbr_mangle {
    chain output {
        type route hook output priority mangle; policy accept;

        # Mark all outbound TCP traffic destined for port 9000 (S3 Backup) with mark 100
        tcp dport 9000 meta mark set 100
    }
}

6. Persistent Configuration with `systemd-networkd`

In production, commands executed directly via ip rule are lost upon reboot. Configure persistent PBR in /etc/systemd/network/:

# File: /etc/systemd/network/20-wan-isp2.network
[Match]
Name=eth1

[Network]
Address=198.51.100.20/24

# 1. Add Default Route to Table 201 (isp2)
[Route]
Gateway=198.51.100.1
Table=201

# 2. Add Policy Rule to force traffic from this IP to consult Table 201
[RoutingPolicyRule]
From=198.51.100.20/32
Table=201
Priority=1000

Apply and reload:

sudo networkctl reload
ip rule show

7. Next Steps

With multi-table policy routing fully configured, explore physical link redundancy and traffic isolation using Layer-2 bonding and VLANs:

802.1Q VLAN Trunks & 802.3ad LACP Bonding

Aggregate multiple 10GbE NICs into high-availability active-active bonds and trunk isolated VLAN security zones.

LACP & VLAN Guide