TCP: handshake and flags

Three-way handshake, sequence numbers, the SYN/ACK/FIN/RST flags, and what makes TCP reliable.

Why this matters

Connection refused, a curl that hangs, thousands of TIME-WAIT on a server, SYN-RECV climbing in monitoring. You can only read these symptoms once you understand how TCP opens and closes a connection. This lesson covers the handshake, the flags, and the states, and in the labs you'll catch them in live traffic.

Why a connection needs a handshake

TCP guarantees that bytes arrive whole and in the order they were sent. To do that, each side numbers its data: every byte in the stream has a sequence number. Before sending data, the two sides must exchange their initial numbers and confirm them. That exchange is the three-way handshake.

The initial number is called the ISN (initial sequence number). It doesn't start at zero: each side picks it fresh for every connection, and it looks like a random 32-bit number.

client                                server
  SYN, seq=x          ------------->      "want a connection, my number is x"
  <-------------  SYN-ACK, seq=y, ack=x+1  "got it, my number is y"
  ACK, ack=y+1        ------------->      "got your number"

Three steps: SYN from the client, SYN-ACK from the server, ACK from the client. After that the connection is established (ESTABLISHED) and data flows both ways.

Why ack=x+1 and not x? The ack field means "I'm expecting the next byte with this number." The SYN itself occupies one number in the stream, so the acknowledgment is one higher.

Flags

Flags are bits in the TCP segment header that set its meaning:

• SYN: synchronize sequence numbers, open a connection;
• ACK: the acknowledgment field is meaningful (set on almost every segment);
• FIN: no more data from the sender, a polite close;
• RST: reset, the connection is torn down immediately with no goodbyes;
• PSH: hand the data to the application right away instead of buffering it. You'll see it on almost every packet that carries data.

Closing comes in two flavors. Normal: FIN exchanged in both directions, everything unsent still arrives. Abnormal: RST, buffers are dropped. An RST in response to a SYN means "port closed," which is why curl fails instantly with Connection refused instead of hanging. If a firewall silently drops the SYN instead, there's no response at all, and the client waits until it times out (see the lesson on firewalls).

Connection states

Every connection in the kernel sits in some state. You can see them in ss:

$ ss -tan
State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port
LISTEN     0      511           0.0.0.0:80          0.0.0.0:*
ESTAB      0      0        10.0.0.5:80       203.0.113.7:51742
SYN-RECV   0      0        10.0.0.5:80       198.51.100.9:40118
TIME-WAIT  0      0        10.0.0.5:80       203.0.113.7:51700

What the most common ones mean:

• LISTEN: the socket is waiting for incoming connections on the port;
• SYN-SENT: the client sent a SYN and is waiting for a SYN-ACK;
• SYN-RECV: the server got a SYN, replied with a SYN-ACK, and is waiting for the final ACK;
• ESTAB: the handshake is done, data is flowing (in ss, ESTABLISHED is shortened to ESTAB);
• TIME-WAIT: the connection is closed, but the kernel still remembers it for a while so that late packets don't land in a new connection. It stays with the side that closed first.

SYN-RECV usually lasts a fraction of a millisecond. Many stuck SYN-RECV connections are a sign that clients send a SYN and then vanish without sending the third step. That's what a SYN flood looks like. A client that never receives the SYN-ACK produces the same picture. The server retries the SYN-ACK for a while (5 times by default on Linux, net.ipv4.tcp_synack_retries), then drops the entry.

If an address in ss is written as [::ffff:10.0.0.7]:51742, that's a plain IPv4 address 10.0.0.7: it looks like that for a socket that listens on both IPv4 and IPv6.

How to see the handshake: tcpdump

tcpdump prints the packets that pass through an interface. Its output writes flags in square brackets as single letters: S is SYN, F is FIN, R is RST, P is PSH, and a dot . means ACK. For example, [S.] is SYN-ACK, and [F.] is FIN together with ACK.

A handshake from host 192.0.2.20 to 192.0.2.10:80:

$ tcpdump -n -i eth0 tcp port 80
IP 192.0.2.20.40100 > 192.0.2.10.80: Flags [S], seq 3355443210, win 64240, ...
IP 192.0.2.10.80 > 192.0.2.20.40100: Flags [S.], seq 1048576001, ack 3355443211, win 65160, ...
IP 192.0.2.20.40100 > 192.0.2.10.80: Flags [.], ack 1, win 502, ...

The third line shows ack 1, even though the server's ISN is a huge number. That's not a mistake: after the first packets, tcpdump by default prints numbers relative to the start of the connection, to make them easier to read. The -S option shows the real (absolute) numbers:

$ tcpdump -nS -i eth0 tcp port 80
IP 192.0.2.20.40100 > 192.0.2.10.80: Flags [S], seq 3355443210, ...
IP 192.0.2.10.80 > 192.0.2.20.40100: Flags [S.], seq 1048576001, ack 3355443211, ...
IP 192.0.2.20.40100 > 192.0.2.10.80: Flags [.], ack 1048576002, ...

-n turns off resolving addresses to names, -r file reads a previously recorded capture instead of a live interface, -w file writes one.

Reliability

The receiver acknowledges received bytes with the ack number. If the acknowledgment doesn't arrive in time, the sender retransmits the segment (retransmission). A lost packet turns into a delay, not lost data.

You can see retransmissions in the statistics:

$ nstat -az TcpRetransSegs
TcpRetransSegs                  63441              0.0

-a shows the counter's absolute value, -z prints it even when it's zero. Without -a, nstat prints the increase since the last run.

nstat from the iproute2 package is the modern replacement for netstat -s from net-tools: net-tools hasn't been developed in a long time, and minimal images simply don't have it.

A steadily growing counter points to a bad link: packets are getting lost, TCP will redeliver them, but at a cost in speed. SYN-ACK retries count toward this counter too.

Common mistakes

• Reading ack 1 and seq 1:94 in tcpdump as real numbers. These are relative numbers; the real ones are visible with -S.
• Treating TIME-WAIT as a problem. It's a normal tail of a closed connection. The real warning sign is SYN-RECV or SYN-SENT growing and not going away.
• Confusing Connection refused with a timeout. Refused means the host answered with RST and the port is closed. A timeout means there's no response at all: the packet was lost or dropped.
• Expecting the ISN to start at zero. It's random for every connection.

What to remember

Handshake: SYN -> SYN-ACK -> ACK, after which the connection is ESTABLISHED.

Every byte in the stream is numbered (sequence number), receipt is acknowledged with ack, and anything unacknowledged gets resent.

FIN closes a connection politely, RST tears it down instantly; RST in response to a SYN means the port is closed.

ss -tan shows connection states; stuck SYN-RECV entries signal a problem.

What you'll need for the tasks

• docker exec <container> <command>: run a command inside a lab container;
• tcpdump -n -r <file> and the -S option for absolute numbers;
• tcpdump's flag notation: [S], [S.], [.], [P.], [F.], [R.];
• ss -tan and the state filter ss -tan state <state>;
• nstat -az TcpRetransSegs.

The next lesson covers UDP: a transport with no connection and no guarantees.