Zum Inhalt

Linux Networking Diagnostics: iproute2, Socket Inspection (ss), nftables, tcpdump & mtr

Executive Summary: Legacy net-tools utilities (ifconfig, netstat, route, arp) have been deprecated for over fifteen years—they rely on obsolete /proc/net text parsing, hide secondary IPv4/IPv6 CIDR addresses assigned to an interface, and cannot inspect kernel TCP congestion window metrics (cwnd, rtt). This operational reference covers the Netlink-powered iproute2 suite (ip, ss, bridge, tc), DNS resolution tracing (dig +trace, resolvectl), bidirectional path analysis (mtr), kernel packet capture (tcpdump BPF filters), and connection timing dissection via curl -w.

  • Netlink iproute2 (ip & ss)

    Communicate directly with the kernel network stack via Netlink sockets (AF_NETLINK) to inspect links, routes, neighbors, and TCP socket internals.

  • BPF Packet Capture (tcpdump)

    Filter wire-level frames inside the kernel using Berkeley Packet Filter (BPF) expressions before copying packets to userspace.

  • Path & MTU Diagnostics (mtr)

    Combine ICMP/UDP/TCP traceroute with continuous packet-loss statistics and Path MTU Discovery (ping -M do -s 1472).

  • L7 Latency Breakdown (curl -w)

    Isolate DNS lookup, TCP 3-way handshake, TLS 1.3 negotiation, and Time-To-First-Byte (TTFB) latencies in a single HTTP request.

1. Architecture: How iproute2 & tcpdump Interact with the Kernel Stack

Unlike legacy ifconfig/netstat which read truncated text files from /proc/net/tcp, modern tools (ip, ss, nft) open a Netlink socket (AF_NETLINK) to exchange binary structs directly with the Linux kernel, while tcpdump attaches a BPF (Berkeley Packet Filter) bytecode program to the network tap (AF_PACKET) just above the NIC driver ring buffer:

%%{init: {'flowchart': {'wrappingWidth': 600, 'subGraphTitleMargin': {'top': 15, 'bottom': 15}}}}%%
flowchart TB
    subgraph UserspaceTools["Modern Linux Diagnostic Utilities"]
        IPSS["ip / ss / bridge<br/>(iproute2 Suite)"]
        TCPDump["tcpdump -nn -i eth0<br/>(BPF Bytecode Filter)"]
    end

    subgraph KernelNet["Linux Kernel Network Stack"]
        Netlink["AF_NETLINK Socket API<br/>(RTNETLINK / SOCK_DIAG)"]
        PacketTap["AF_PACKET / dev_add_pack<br/>(Before/After nftables)"]
        NICRing["NIC RX/TX DMA Ring<br/>(NAPI Poll + sk_buff)"]
    end

    IPSS <-->|"Binary Structs"| Netlink
    TCPDump <-->|"In-Kernel BPF Match"| PacketTap
    NICRing --> PacketTap
    PacketTap --> Netlink

Legacy net-tools to Modern iproute2 Migration Table

Deprecated Legacy Command Modern iproute2 Replacement Added Engineering Capability
ifconfig -a ip -br -c addr show (or ip -s link) Displays all secondary CIDRs, link states, and RX/TX drop counters.
route -n ip route show / ip route get <IP> Queries the kernel FIB to show the exact interface, source IP, and gateway chosen for any destination IP.
arp -an ip neigh show Displays both IPv4 ARP and IPv6 NDP neighbor cache states (REACHABLE, STALE, FAILED).
netstat -tulnp ss -tulnp (or ss -ti) Uses kernel tcp_diag Netlink to list sockets $10\times$ faster and expose BBR/CUBIC cwnd and rtt.

2. Layer 2/3 Inspection & Policy Routing with ip

When troubleshooting multi-homed servers, VPN tunnels (wg0), or Docker bridges, never guess which route a packet will take—ask the kernel's Forwarding Information Base (FIB) directly via ip route get:

Inspecting Links, Neighbor Tables & Kernel FIB Route Selection
# 1. Concise colorized overview of all interfaces, states, and IPv4/IPv6 CIDR addresses
ip -br -c addr show

# 2. Inspect interface RX/TX packet error, drop, and overrun counters at the NIC driver level
ip -s -s link show dev eth0

# 3. Ask the kernel which exact interface, gateway, and source IP it will use to reach 9.9.9.9
ip route get 9.9.9.9

# 4. Test Path MTU Discovery (PMTUD) with Don't-Fragment bit set (1472 payload + 20 IP + 8 ICMP = 1500 MTU)
ping -c 3 -M do -s 1472 1.1.1.1

# 5. Run live bidirectional TCP/ICMP path loss & jitter analysis across every hop via mtr
mtr --report --report-cycles 10 --tcp --port 443 nejat-hakan.de

3. Layer 4 Socket Forensics with ss (SOCK_DIAG)

ss queries the kernel's INET_DIAG Netlink module directly. Beyond listing listening ports (ss -tulnp), ss can filter sockets in kernel space by state (established, time-wait, syn-recv) and dump internal TCP stack metrics:

Auditing Listening Sockets, Active Connections & TCP Congestion Metrics
# 1. List all listening TCP and UDP sockets with owning process name, PID, and FD number
sudo ss -tulnp

# 2. Show all ESTABLISHED connections to/from port 443 with internal TCP timers (-o) and BBR/CUBIC RTT (-i)
sudo ss -tnoip state established '( dport = :443 or sport = :443 )'

# 3. Print instant summary of total sockets, TCP states (estab, closed, timewait, synrecv)
ss -s

4. Wire-Level Packet Capture with tcpdump

When firewall rules (nftables), asymmetric routing (rp_filter), or TLS handshakes fail silently, capture the wire traffic with tcpdump. Always pass -nn so tcpdump never blocks attempting reverse-DNS PTR lookups or /etc/services port translations during live capture:

High-Precision Production tcpdump BPF Filters
# 1. Watch live ICMP traffic or WireGuard UDP/51820 handshakes across ALL interfaces (-i any)
sudo tcpdump -ni any 'icmp or icmp6 or udp port 51820'

# 2. Capture ONLY TCP SYN and RST packets to diagnose connection refusals or firewall drops
sudo tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'

# 3. Capture DNS queries and responses on Port 53 with verbose packet details
sudo tcpdump -nvvvi any 'port 53'

# 4. Write a rotating ring-buffer of 5 x 100MB .pcap files (-C 100 -W 5) for Wireshark analysis
sudo tcpdump -ni eth0 -s 0 -C 100 -W 5 -w /tmp/capture.pcap 'host 192.168.10.25 and tcp port 5432'

Packet Capture Ordering: Incoming vs. Outgoing Firewall Hooks

Because tcpdump hooks into AF_PACKET just above the NIC driver: - On Inbound (RX) packets, tcpdump sees the packet BEFORE nftables/iptables filters it. (If tcpdump sees an incoming SYN on eth0, but the local service never replies with SYN-ACK, check your local firewall or rp_filter!) - On Outbound (TX) packets, tcpdump sees the packet AFTER nftables allows it.

5. Layer 7 DNS & HTTP Timing Dissection (dig & curl -w)

Isolate whether a slow web request is caused by DNS resolution, TCP handshake latency, TLS handshake overhead, or backend application processing (starttransfer):

Dissecting DNS Delegation & Millisecond HTTP/TLS Latency Phases
# 1. Trace complete iterative DNS delegation from Root (.) -> TLD -> Authoritative NS
dig +trace +nodnssec nejat-hakan.de A

# 2. Measure exact millisecond breakdown of DNS, TCP, TLS 1.3, and Backend TTFB with curl
curl -so /dev/null -w "\
DNS Lookup:     %{time_namelookup}s\n\
TCP Connect:    %{time_connect}s\n\
TLS Handshake:  %{time_appconnect}s\n\
TTFB (Backend): %{time_starttransfer}s\n\
Total Time:     %{time_total}s (HTTP %{http_code})\n" https://nejat-hakan.de