Skip to content

DNS Resolution Architecture & Bottom-Up Layer 1–7 Troubleshooting on Linux

Created: 2026-11-08  ·  Last updated: 2026-11-08  ·  Nejat Hakan

Executive Summary: When an application throws Connection timed out or Temporary failure in name resolution, inexperienced administrators randomly tweak firewall rules and edit configuration files. Professional systems engineering demands a deterministic methodology: tracing how the GNU C Library (glibc) processes domain names through /etc/nsswitch.conf, auditing modern caching daemons like systemd-resolved, tracing DNS hierarchies with dig +trace, and following a strict Bottom-Up Layer 1 to Layer 7 diagnostic runbook.

  • The Linux DNS Subsystem

    Understand the exact lookup sequence: getaddrinfo() system calls, NSS order in /etc/nsswitch.conf, /etc/hosts overrides, and /etc/resolv.conf.

  • systemd-resolved Mastery

    Demystify the local stub resolver 127.0.0.53, inspect interface-specific search domains, flush caches, and query records via resolvectl.

  • Deep DNS Tracing with dig

    Perform authoritative root-zone traversals (+trace), inspect EDNS0 buffer limits, test reverse DNS (PTR), and parse structured answers.

  • Layer 1–7 Runbook

    Systematic bottom-up troubleshooting matrix: from physical carrier link and ARP/NDP to routing, firewall drops, and TLS handshake failures.

1. The Linux DNS Resolution Chain: From Code to Wire

When an application calls getaddrinfo("api.internal.corp"), the Linux kernel is not directly involved in DNS parsing. Name resolution is a user-space function handled by the GNU C Library (glibc), structured by the Name Service Switch (NSS):

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    APP["User-Space Application<br/>Calls getaddrinfo() / gethostbyname()"] --> NSS["1. Consult /etc/nsswitch.conf<br/>hosts: files mymachines mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns"]
    NSS -->|"First: files"| HOSTS["2. Inspect /etc/hosts<br/>Static IP-to-Hostname mappings"]
    HOSTS -->|"Found"| SUCCESS["Return IP to Application"]
    HOSTS -->|"Not Found -> Next: dns"| RESOLV["3. Consult /etc/resolv.conf<br/>nameserver 127.0.0.53 or Upstream IP"]
    RESOLV --> RES["4. Local Stub Resolver<br/>systemd-resolved / Unbound"]
    RES -->|"Cache Miss"| UPSTREAM["5. Upstream Recursive Resolver<br/>9.9.9.9, 1.1.1.1, or Enterprise DNS"]
    UPSTREAM --> SUCCESS

Inspecting NSS Configuration: /etc/nsswitch.conf

The hosts: directive dictates the precedence of name sources:

# File: /etc/nsswitch.conf (Typical Debian with systemd-resolved)
hosts:          files resolve [!UNAVAIL=return] dns myhostname
  • files: Checks /etc/hosts first. If an entry exists, the lookup terminates immediately without network traffic.
  • resolve: Hands queries over to the D-Bus API of systemd-resolved.
  • [!UNAVAIL=return]: If systemd-resolved is running and reports that the domain does not exist, stop here (do not fall back to plain DNS).
  • dns: Falls back to legacy DNS queries using the servers defined in /etc/resolv.conf.
  • myhostname: Automatically resolves the local hostname and loopback addresses.

Verifying Resolution Order via getent

Never test DNS purely with ping or nslookup. Use getent, which uses the exact same glibc NSS library routines as production applications: 

# Queries the full NSS stack (respects /etc/hosts + systemd-resolved + DNS)
getent ahosts nejat-hakan.de

2. Managing DNS with systemd-resolved on Debian

Modern Debian installations use systemd-resolved to provide local DNS caching, DNSSEC validation, and interface-specific split-DNS routing.

The 127.0.0.53 Stub Resolver Demystified

If /etc/resolv.conf points to nameserver 127.0.0.53, your system is using systemd-resolved's local loopback stub listener:

# Verify the symlink configuration
ls -la /etc/resolv.conf
# Output: /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

Daily Diagnostics with resolvectl

# Display global and per-interface DNS servers, search domains, and DNSSEC status
resolvectl status

# Query a domain through the resolved daemon and view latency + DNSSEC validation
resolvectl query nejat-hakan.de

# Flush the local DNS cache
sudo resolvectl flush-caches

# Inspect cache statistics (hit/miss ratio)
resolvectl statistics

3. Authoritative DNS Diagnostics with dig

The dig (Domain Information Groper) utility is the gold standard for network engineers.

1. Essential Queries & Clean Output

# Compact query for IPv4 (A) and IPv6 (AAAA) records
dig +noall +answer nejat-hakan.de A nejat-hakan.de AAAA

# Query specific record types (MX, TXT, CNAME, SOA)
dig +short nejat-hakan.de MX
dig +short nejat-hakan.de TXT

# Reverse DNS lookup (PTR) for an IP address
dig +short -x 93.184.216.34

2. Hierarchical Root-to-Leaf Tracing: dig +trace

When debugging propagation delays or cache poisoning, bypass local resolvers and traverse the DNS hierarchy from the root servers downward:

dig nejat-hakan.de +trace

The output illustrates the complete recursive resolution process: 1. Queries the 13 Root Name Server clusters ([a-m].root-servers.net) for .de. 2. Receives the authoritative DENIC nameservers for .de ([a-z].denic.de). 3. Queries DENIC to locate the Authoritative Nameservers for nejat-hakan.de. 4. Retrieves the final A / AAAA record directly from the authoritative source.

4. The Layer 1 to Layer 7 Bottom-Up Troubleshooting Runbook

When a service fails to communicate, diagnosing problems from the top down (editing application configurations or restarting services) introduces chaos. Always diagnose strictly bottom-up, starting from physical link integrity.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    L1["Layer 1: Physical Link<br/>ethtool eth0 -> Link detected: yes?"] -->|Pass| L2["Layer 2: Local Neighbor<br/>ip neigh -> Gateway MAC resolved?"]
    L2 -->|Pass| L3["Layer 3: IP Routing<br/>ip route get -> Valid gateway route? ping Gateway"]
    L3 -->|Pass| L4["Layer 4: Transport & Firewall<br/>nc -zv Target Port / ss -tulpn -> Connection refused vs. Timed out?"]
    L4 -->|Pass| L5["Layer 5/6: TLS / Security<br/>openssl s_client -> Valid certificate & cipher?"]
    L5 -->|Pass| L7["Layer 7: Application & DNS<br/>curl -v / dig -> HTTP 200 OK & Proper payload?"]

5. Comprehensive Diagnostic Matrix

Layer Step & Scope Diagnostic Command Failure Symptom Root Cause & Remediation
Layer 1 Physical Link sudo ethtool eth0 Link detected: no or Speed: Unknown! Unplugged cable, damaged fiber patch, dead switch port, or transceiver failure.
Layer 2 Link State & Errors ip -s link show dev eth0 state DOWN or massive RX/TX errors / dropped Interface is administratively down (sudo ip link set eth0 up) or duplex mismatch on switch port.
Layer 2 Neighbor ARP/NDP ip neigh show 192.168.1.1 dev eth0 FAILED or INCOMPLETE Wrong VLAN tag on switch port, incorrect subnet mask configured, or gateway is powered off.
Layer 3 Default Gateway ip route show & ping -c 2 192.168.1.1 Network is unreachable or Destination Host Unreachable Missing default route (sudo ip route add default via 192.168.1.1) or IP address conflict on LAN.
Layer 3 WAN Routing traceroute -n 1.1.1.1 or mtr -n 1.1.1.1 Hops stop at upstream ISP router (* * *) ISP outage, upstream BGP failure, or missing NAT masquerade on WAN egress.
Layer 4 Local Port Binding sudo ss -tulpn \| grep :5432 Process is not listed in ss output Service daemon is crashed, disabled, or bound exclusively to 127.0.0.1 instead of 0.0.0.0.
Layer 4 Remote Port / Firewall nc -zv 10.10.20.11 5432 Case A: Connection timed out
Case B: Connection refused		 Timed out: Dropped silently by nftables/ufw or cloud security group.
Refused: Port reached, but no daemon is listening.
Layer 6 TLS Handshake openssl s_client -connect api.example.com:443 -servername api.example.com verify error:num=10:certificate has expired Expired TLS certificate, missing intermediate CA certificates, or SNI mismatch.
Layer 7 DNS Resolution dig +short api.example.com Temporary failure in name resolution Invalid nameserver in /etc/resolv.conf, firewall dropping UDP port 53, or upstream DNS outage.
Layer 7 HTTP Application curl -Iv https://api.example.com HTTP/1.1 502 Bad Gateway or 504 Gateway Time-out Reverse proxy (Nginx/HAProxy) is operational, but backend application pool is down.

6. Where to Go Next: Cluster 2

Congratulations! You have completed Cluster 1: Foundations & Kernel Protocols. Now proceed to hardening network services, building encrypted jump hosts, and crafting stateful firewalls:

  • Cluster 2: Cryptographic OpenSSH Architecture & Bastions

    Master ED25519 key management, hardened sshd_config, ProxyJump bastions, and encrypted SOCKS5 port forwarding.

    OpenSSH Hardening Guide