Symptoms:
Hanging connections, stalling data transfers. Partially established connections.
Cause:
Connections over interfaces with an MTU (Max. Transmit Unit) smaller
than 1500 bytes, such as PPPOE, (PPP) dial-up routers, MS Windows as router,
CIPE or OpenVPN tunnels, non-Ethernet-II frames, etc.,
require somewhat smaller packets than usual.
If a packet is too large for an interface, it is fragemented and sent in several
smaller packets. The fragmented packets are reassembled at the target computer's
interface, which is a fully transparant process.
Fragmentation is a cpu intensive task, too expensive for backbone routers,
and is often avoided by setting the "Don't Fragment" (DF) bit.
When the DF-bit is set to '1' in an IP packet, the router will not fragment
it if too large, but sends an ICMP packet (type 3, subtype 4: unreachable,
fragmentation needed) back to the sender, requesting smaller packets, and (if
implemented by the router) also the MTU to use in a 16-bits data field.
This process is called Path MTU (PMTU) discovery and is described
in RFC 1191.
When a router or firewall doesn't allow ICMP packets and drops them,
the ICMP notification will not be received by the sender and the PMTU process
fails: the handshake is lost and the connection hangs or the file transfer stalls.
Solutions:
One solution is to allow fragmentation by (if possible) preventing that the
DF
-bit is set:
- Disable PMTU discovery for the failing host (so packets with
DF=0
are sent)
to allow fragmentation. If the DF
bit is set by a router in the path, this solution fails.
Prevent transmission of packets larger than the smallest MTU
along the path to
the failing host, by specifying a smaller MTU
for the interface or a smaller
MSS
for the route to that host.
- Reduce the
MTU
of the interface used for the trafic with the failing host:
# /sbin/ifconfig eth0 mtu 1400
- Set the
MSS
(Max. Segment Size) to 1400 using the route command with the
'mss ' option for the route to the problematic network,
or set the MSS
for the route to the problematic host
# route add -host aaa.bbb.ccc.ddd gw 10.10.0.8 eth0 mss 1400
- If the interface over which the tunnel travels to the peer has a smaller MTU, the MTU of CIPE should be adjusted accordingly.
- Set (if possible) the MRU (Max. Receive Unit) of the router to the maximum value, e.g. 1492 for PPPOE.
Notes:
- The Maximum Segment Size (MSS) is the size of the payload
without the overhead (40 bytes for TCP). A MSS of 1460 matches a MTU of 1500.
- CIPE adds 44 bytes (protocol=3), or 58 bytes (protocol=4) overhead to the
payload. The MTU CIPE can send over an Ethernet interface with an MTU of 1500
is 1500-44-14=1442 and 1500-58-14=1428 respectively. (14 bytes for the ethernet
header).
- Because CipeX compresses large packets into smaller ones, the
PMTU problem (in the traffic to/from the peer) becomes visible only when
non-compressible (e.g. already compressed or random data) files are transferred.
- Be aware of the asymmetrical behaviour that is to be expected: in one direction
PMTU discovery does not encounter ICMP 'obstacles' and packets will be adjusted in size,
while in the other direction an 'obstacle' between the router and the source prevents
the PMTU discovery.
- Be also aware that this problem only occurs with maximum size packets with the 'DF'
flag set, e.g. with file transfer. Or even with packets without the 'DF' flag when a
router in the traject is configured to set this flag on all packets (to prevent overhead
caused by fragmentation).
Ref: This is a summary of information available at the following locations:
http://sdb.suse.de/en/sdb/html/cg_pmtu.html
http://www.netheaven.com/pmtu.html
http://blue-labs.org/howto/mtu-mss.php
RFC 1191: Path MTU Discovery 1990 (obsoletes RFC 1063)
Below the layout of the ICMP - Type 3: "Destination Unreachable",Code 4: "Fragmentation needed and DF set"
packet as described in RFC 1191.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 3 | Code = 4 | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unused = 0 | Next-Hop MTU |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Internet Header + 64 bits of Original Datagram Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+