CipeX - PMTU problems with IP-over-IP tunneling

Symptoms:

Hanging connections, stalling data transfers. Partially established connections.

Cause:

Connections over interfaces with an MTU (Max. Transmit Unit) smaller than 1500 bytes, such as PPPOE, (PPP) dial-up routers, MS Windows as router, CIPE or OpenVPN tunnels, non-Ethernet-II frames, etc., require somewhat smaller packets than usual. If a packet is too large for an interface, it is fragemented and sent in several smaller packets. The fragmented packets are reassembled at the target computer's interface, which is a fully transparant process.
Fragmentation is a cpu intensive task, too expensive for backbone routers, and is often avoided by setting the "Don't Fragment" (DF) bit. When the DF-bit is set to '1' in an IP packet, the router will not fragment it if too large, but sends an ICMP packet (type 3, subtype 4: unreachable, fragmentation needed) back to the sender, requesting smaller packets, and (if implemented by the router) also the MTU to use in a 16-bits data field. This process is called Path MTU (PMTU) discovery and is described in RFC 1191.
When a router or firewall doesn't allow ICMP packets and drops them, the ICMP notification will not be received by the sender and the PMTU process fails: the handshake is lost and the connection hangs or the file transfer stalls.

Solutions:

One solution is to allow fragmentation by (if possible) preventing that the DF-bit is set:

Disable PMTU discovery for the failing host (so packets with DF=0 are sent) to allow fragmentation. If the DF bit is set by a router in the path, this solution fails.

Prevent transmission of packets larger than the smallest MTU along the path to the failing host, by specifying a smaller MTU for the interface or a smaller MSS for the route to that host.

Reduce the MTU of the interface used for the trafic with the failing host:
# /sbin/ifconfig eth0 mtu 1400
Set the MSS (Max. Segment Size) to 1400 using the route command with the 'mss ' option for the route to the problematic network, or set the MSS for the route to the problematic host
# route add -host aaa.bbb.ccc.ddd gw 10.10.0.8 eth0 mss 1400
If the interface over which the tunnel travels to the peer has a smaller MTU, the MTU of CIPE should be adjusted accordingly.
Set (if possible) the MRU (Max. Receive Unit) of the router to the maximum value, e.g. 1492 for PPPOE.

Notes:

The Maximum Segment Size (MSS) is the size of the payload without the overhead (40 bytes for TCP). A MSS of 1460 matches a MTU of 1500.
CIPE adds 44 bytes (protocol=3), or 58 bytes (protocol=4) overhead to the payload. The MTU CIPE can send over an Ethernet interface with an MTU of 1500 is 1500-44-14=1442 and 1500-58-14=1428 respectively. (14 bytes for the ethernet header).
Because CipeX compresses large packets into smaller ones, the PMTU problem (in the traffic to/from the peer) becomes visible only when non-compressible (e.g. already compressed or random data) files are transferred.
Be aware of the asymmetrical behaviour that is to be expected: in one direction PMTU discovery does not encounter ICMP 'obstacles' and packets will be adjusted in size, while in the other direction an 'obstacle' between the router and the source prevents the PMTU discovery.
Be also aware that this problem only occurs with maximum size packets with the 'DF' flag set, e.g. with file transfer. Or even with packets without the 'DF' flag when a router in the traject is configured to set this flag on all packets (to prevent overhead caused by fragmentation).

Ref: This is a summary of information available at the following locations:

http://sdb.suse.de/en/sdb/html/cg_pmtu.html

http://www.netheaven.com/pmtu.html

http://blue-labs.org/howto/mtu-mss.php

RFC 1191: Path MTU Discovery 1990 (obsoletes RFC 1063)

Below the layout of the ICMP - Type 3: "Destination Unreachable",Code 4: "Fragmentation needed and DF set" packet as described in RFC 1191.


       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 3    |   Code = 4    |           Checksum            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           unused = 0          |         Next-Hop MTU          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Internet Header + 64 bits of Original Datagram Data      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+