Network diagnostic tools

ping, traceroute, mtr, ss, nc, curl -v, tcpdump. What to use for which symptom.

Why

"The site won't load" can mean a dozen different failures: no address, no route, the host is down, the port is closed, the name doesn't resolve, the proxy can't reach the backend. Each has its own tool. If you know what question each tool answers, diagnosis takes minutes, not an evening with tcpdump.

Approach: treat by symptom

The network works in layers (see the lesson on OSI and TCP/IP). When something breaks, don't reach for tcpdump right away. Go bottom-up (or top-down, but systematically):

1. Is my address right? (IP, route)
2. Does it reach the host? (ping, ICMP)
3. Is the port open? (nc, telnet)
4. Does the name resolve? (dig)
5. Is the HTTP response what you expect? (curl -v)
6. Where does it get lost? (mtr, traceroute)
7. What's actually going over the wire? (tcpdump)

Most of the time diagnosis ends at steps 1-4.

What's local: ip

ip a            # all interfaces and their IPs
ip r            # routing table (where each packet will go)
ip n            # neighbor table: IP -> MAC on the local network (ARP for IPv4)

ip a replaced the old ifconfig. What to check in it:

• did the interface get an address from DHCP;
• which IP is assigned to which interface;
• no 169.254.x.x address: it means DHCP didn't answer and the system assigned itself a link-local address.

ip r shows the default gateway (the default via ... line). If it's missing, nothing gets out, even with a correct IP.

The ip commands exist on Linux and WSL2. macOS doesn't have them; use ifconfig and netstat -rn there.

Who's listening on ports: ss

ss -tulnp

Flags:

• -t TCP, -u UDP;
• -l only sockets in the LISTEN state;
• -n numeric output, don't translate ports into service names;
• -p show the process.

Output (trimmed):

Netid State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp   LISTEN 0      128          0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=789,fd=3))
tcp   LISTEN 0      244        127.0.0.1:5432      0.0.0.0:*    users:(("postgres",pid=1024,fd=6))
tcp   LISTEN 0      511                *:80              *:*    users:(("nginx",pid=2001,fd=6))

0.0.0.0:22 means sshd listens on all IPv4 interfaces. And 127.0.0.1:5432 means postgres is reachable only from this same machine. It's not exposed externally, and for a database that's usually correct.

The Process column fills in only for processes you're allowed to see. Without sudo you'll only see your own processes, other people's have an empty field. So on a server run sudo ss -tulnp.

ss replaced netstat. The familiar netstat -tulnp from articles does the same thing, but lives in the net-tools package. Debian deprecated it long ago in favor of iproute2, so on a fresh server expect ss. In minimal images built on BusyBox (Alpine, busybox) it's the opposite: no ss, but a stripped-down netstat is there, and netstat -tln works fine. macOS has no ss either; check ports there with lsof -nP -iTCP -sTCP:LISTEN.

Host reachability: ping

ping -c 4 example.com

-c N sets how many packets to send. Without -c, ping on both Linux and macOS keeps sending until you press Ctrl+C.

How to read the result:

• replies arrive, latency is stable: the network to the host is fine;
• 100% loss: either the host is unreachable, or ICMP is blocked (often on cloud firewalls);
• partial loss: a problem somewhere on the path.

No reply to ping doesn't mean the service is down. Many hosts block ICMP while the TCP ports are open. Check the service with nc.

Port reachability: nc

The simplest TCP check without a dedicated client:

nc -zv example.com 443

Connection to example.com 443 port [tcp/https] succeeded!

nc -zv example.com 5432

nc: connect to example.com port 5432 (tcp) failed: Connection refused

This is the output of OpenBSD nc (the netcat-openbsd package on Debian and Ubuntu). macOS phrases it a bit differently, same meaning.

Flags:

• -z zero-I/O: don't send data, just check that the connection establishes;
• -v verbose output.

Three possible results:

• succeeded: the port is open, something is listening on it;
• Connection refused: the packet reached the host, but the port is closed (nothing is listening);
• timeout: no reply at all (a firewall is silently dropping packets, or the host is down). To avoid waiting long, add -w 3.

nc has several implementations, and they behave differently. In BusyBox (Alpine, busybox, and many minimal images) -z exists, but the output is sparse: on success a host (address:port) open line, on failure nothing, just exit code 1. Check the code with echo $? if there's no line.

If nc is missing entirely, bash has /dev/tcp built in:

(echo > /dev/tcp/example.com/443) >/dev/null 2>&1 && echo open || echo closed

This is a bash-only feature; BusyBox's sh doesn't have it. And if packets are silently dropped, the command will wait for the system connection timeout.

The old telnet host port is still useful on legacy servers.

Where it's lost: traceroute and mtr

traceroute shows the packet's path through nodes (hops):

traceroute example.com

 1  192.168.1.1  0.5 ms
 2  10.0.0.1     2.1 ms
 3  * * *
 4  203.0.113.10  18 ms

The output is just an example, the addresses on your path will differ. * * * means an intermediate node isn't replying. Often that's intentional and not a problem by itself, if the next nodes reply.

mtr is traceroute that continuously pings every node and shows the loss and latency for each. Invaluable when a connection is slow between specific points:

mtr -r -c 100 example.com

-r turns on report mode, -c 100 sets a hundred probes per node. Look for the node where Loss% turns nonzero and stays that way all the way to the end. Loss on one intermediate node with zero loss on the following ones usually just means that node doesn't like answering ICMP. On Linux mtr comes from the mtr package (or mtr-tiny), on macOS via brew install mtr, and there you need to run it with sudo.

HTTP diagnostics: curl -v

curl -v https://example.com

Shows:

• which IP the name resolved to and where the connection went;
• the TLS handshake for https:// (TLS version, certificate);
• the request and its headers (lines with >), including Host;
• the response and its headers (lines with <).

Useful variants:

curl -I https://example.com                     # response headers only
curl -L https://example.com                     # follow redirects
curl --resolve host:443:1.2.3.4 https://host/   # override name resolution
curl -w "%{time_total}\n" -o /dev/null -s https://example.com  # measure time

--resolve is especially useful during migrations: DNS still points at the old server, and you already want to check the new one. You set the IP by hand, and the override applies only to this command. Format is name:port:address, the port must match the port in the URL, otherwise the override doesn't apply. Unlike -H 'Host: ...', the name stays in the URL, so for HTTPS both SNI and the certificate check work correctly.

One IP and port often serves several sites. The web server picks the site by the Host header, which curl takes from the URL. A request by IP without a name gets the default site, not the one you need.

DNS: dig

Covered in detail in the previous lesson. A quick reminder:

dig +short example.com           # just the IP
dig example.com MX               # MX records
dig @8.8.8.8 example.com         # ask a specific resolver

What's on the wire: tcpdump

When nothing else has helped. tcpdump shows packets:

sudo tcpdump -i any -nn 'host example.com'        # everything to and from the host
sudo tcpdump -i any -nn 'port 5432'               # everything on a port
sudo tcpdump -i any -nn -A 'port 80'              # with packet contents as text (-A)
sudo tcpdump -i any -nn -w trace.pcap 'host x'    # dump to a file (open in Wireshark)

-A prints the packet contents in ASCII. For plain HTTP you'll see requests and headers, while HTTPS is encrypted and -A shows garbage. -i any exists on Linux; on macOS specify a concrete interface, for example -i en0.

This isn't an everyday tool anymore. It's useful when you need to answer:

• does the packet leave at all;
• I sent it, but the other side never got it;
• where do strange delays or retransmissions come from.

For a real analysis, open the .pcap in Wireshark. It's far more informative than raw tcpdump text.

Cheat sheet by symptom

Symptom	What to check
"server not responding"	ping, then nc -zv host port
"we're getting a 502 from nginx"	the nginx log (upstream address and error), ss -tlnp on the backend, curl http://localhost:PORT from there
"DNS returns the wrong IP"	dig, dig @8.8.8.8, check the TTL
"new server is ready, DNS not switched yet"	curl -v --resolve name:port:IP
"ssh hangs on connecting"	ssh -v, then nc -zv host 22, then mtr
"connection is slow"	mtr -r host, curl -w '%{time_connect} %{time_total}\n'
"request went out, response never came"	tcpdump
"service should be running, but the port doesn't respond"	ss -tulnp locally

Common mistakes

• Assuming that no reply to ping means "the service is down". ICMP gets blocked often, check the port.
• Confusing Connection refused with a timeout. The first means "the host is alive, the port is closed", the second means "packets aren't getting through or are being dropped". These are different lines of investigation.
• Looking for the cause of a 502 in the browser. The proxy writes the response, the cause is in its log: the upstream address and the connection error.
• Running ss -tulnp without sudo and concluding nobody's holding the port, because the Process column is empty.
• Checking a new server by IP without a name: you'll get the default site, not the one you need.
• Editing /etc/hosts for a one-off check and forgetting to revert it.

What you'll need for the tasks

• curl -v and curl -I: the request lines > and response lines <, the Host header;
• curl --resolve name:port:address: override resolution in a single command;
• docker compose -f <file> logs <service>: the proxy log;
• docker compose -f <file> exec <service> <command>: run a command inside a lab container;
• netstat -tln inside the container (BusyBox images don't have ss);
• nc -zv host port and the exit code via echo $?;
• nginx -s reload: apply a new nginx config without restarting.

In the next lesson we'll cover firewalls and routing: why a service is alive but unreachable from outside, and why ping and traceroute go silent.