Forum - MSS/PMTU issue? :: SixXS - IPv6 Deployment & Tunnel Broker

MSS/PMTU issue?

Shadow Hawkins on Tuesday, 17 January 2012 11:46:12

Hi, I seem to be having an odd issue with MSS or PMTU. My network setup is as follows: - a server, with a sixxs tunnel, a subnet, and some hosts behind it, at home - a (mobile) client, at work, with a sixxs tunnel - the server tunnel has MTU 1470 - the client tunnel has MTU 1280 I realize these different MTUs are a bit odd, but the setup should just work. For some reason it doesn't, however! What works: - tracepath6 shows a pmtu between server and mobile client of 1280, both ways - tracepath6 also shows pmtu of 1280 between mobile client and a host behind the server - scp between mobile client and server over ipv6 works as expected - ping6 with packet size of 1400 works between mobile and server (both ways) - ping6 with packet size of 1400 also works between mobile and host behind the server (both ways) But, the problem: scp between the mobile client and host behind the server does not work. I observe the following traffic (host = host behind server, client = mobile client): 4.473781 host -> client SSH Encrypted response packet len=2416 4.473863 server -> host ICMPv6 Too big 4.680007 host -> client SSH [TCP Retransmission] Encrypted response packet len=1208 4.693465 client -> host TCP 49524 > ssh [ACK] Seq=1 Ack=26577 Win=502 Len=0 TSV=13989133 TSER=1139708 And this is repeated for every packet sent, resulting in abysmal throughput. So, the question is: why does the host behind the firewall insist on sending packets with size 2416? Why is this even possible, the interface has MTU of 1500! If anyone has any insights or pointers on how to debug this problem furter, I'd be most grateful!

MSS/PMTU issue?

Jeroen Massar SixXS Staff

on Tuesday, 17 January 2012 12:04:21

I realize these different MTUs are a bit odd, but the setup should just work. For some reason it doesn't, however!

Should just work.

- tracepath6 shows a pmtu between server and mobile client of 1280, both ways

- tracepath6 also shows pmtu of 1280 between mobile client and a host behind the server

That should be correct as that is the lowest MTU there is on that path. (and the lowest for IPv6)

- scp between mobile client and server over ipv6 works as expected

If that works, then where does it break? :)

- ping6 with packet size of 1400 works between mobile and server (both ways)

- ping6 with packet size of 1400 also works between mobile and host behind the server (both ways)

That is because you don't set the "Don't Fragment" flag, set that and it will fail. Then again, it working shows that the packet makes it after defragmentation.

So, the question is: why does the host behind the firewall insist on sending packets with size 2416?

Because your firewall is dropping the ICMP messages for Packet Too Big?

Why is this even possible, the interface has MTU of 1500!

Because it depends on where you are looking: before or after fragmentation. If you can provide the configuration on the client (interface & routing tables etc) maybe someone can see a flaw there.

MSS/PMTU issue?

Shadow Hawkins on Tuesday, 17 January 2012 12:40:50

Because your firewall is dropping the ICMP messages for Packet Too Big?

That's the weird thing: the firewall itself sends the ICMP Too Big message. The host behind firewall sends a packet of size 2416, gets a reply 'Too Big', and then sends a packet of size 1208 (as indicated by the MSS in the Too Big error), which makes it. Then, the next packet is again size 2416! So, this almost looks like a bug in the host. Link MTU is 1500, so it should never send a packet of size 2416, right? Host behind the router: ip -6 addr: 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 2001:610:6f9:0:21b:21ff:fe23:3d3e/64 scope global dynamic valid_lft 296sec preferred_lft 116sec inet6 fe80::21b:21ff:fe23:3d3e/64 scope link valid_lft forever preferred_lft forever ip -6 route: 2001:610:6f9::/64 dev eth1 proto kernel metric 256 expires 299sec fe80::/64 dev eth1 proto kernel metric 256 default via fe80::dad3:85ff:feaf:7e77 dev eth1 proto kernel metric 1024 expires 29sec hoplimit 64 The router: ip -6 addr (relevant portions) 3: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 inet6 2001:610:6f9:ffff::1/128 scope global valid_lft forever preferred_lft forever inet6 2001:610:6f9::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::dad3:85ff:feaf:7e77/64 scope link valid_lft forever preferred_lft forever ip -6 route 2001:610:600:429::/64 via :: dev sixxs proto kernel metric 256 mtu 1470 advmss 1410 hoplimit 4294967295 2001:610:6f9::/64 dev eth0.1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295 2001:610:6f9:ffff::1 dev eth0.1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295 2001:610:6f9:ffff::/64 dev eth0.1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295 default via 2001:610:600:429::1 dev sixxs metric 1024 mtu 1470 advmss 1410 hoplimit 4294967295

MSS/PMTU issue?

Shadow Hawkins on Tuesday, 17 January 2012 15:12:47

Some further digging: the host behind the firewall does seem to pick up the mtu correctly (I replaced addresses with names here): michel@host:~$ ip -6 route show cached mobile-client via fe80::dad3:85ff:feaf:7e77 dev eth1 metric 0 cache expires 515sec mtu 1280 rtt 262ms rttvar 250ms cwnd 5 hoplimit 64 server via server dev eth1 metric 0 cache mtu 1470 So it knows the route to the mobile client has a mtu of 1280. scp just seems to ignores this, and sends out packets which are too big. Bigger than the link MTU even!

MSS/PMTU issue?

Shadow Hawkins on Tuesday, 17 January 2012 21:03:11

Okay ignore all of the above. At home, more weird issues were happening, also with ipv4 and nfs. I took the microsoft way and rebooted the server. Problem gone.

Please note Posting is only allowed when you are logged in.