|
Ticket ID: SIXXS #13362789 Ticket Status: User PoP: gblon02 - Goscomb Technologies (London)
IPv6 MTU Path discovery does not work to www.netflix.com
![[gb]](/s/countries/gb.gif) Shadow Hawkins on Saturday, 21 March 2015 23:49:47
Hello SixXS,
about 3 weeks ago I started to have a problem with traffic to www.netflix.com. Debugging trafix using tshark, it looks like some issues with ICMP on the way to www.netflix.com and it seems to only affect them.
My tunnel is T155025/R254189. Below is done from host on internal network (2a01:348:6:87a3:20c:29ff:fee6:d614). 2a01:348:6:87a3::1 is internal side of the router, which then has IPv6 tunnel to gblon02. MTU on internal network is 1500, MTU of the tunnel currently set to 1428, but it worked exactly the same with default value of 1280.
Here is tshark for www.netflix.com:
150.696157 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a01:578:3::36e4:ea94 TCP 80 53998 > https [SYN] Seq=0 Win=14400 Len=0 MSS=1440 SACK_PERM=1 TSval=6022010 TSecr=0 WS=128
150.714902 2a01:578:3::36e4:ea94 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 80 https > 53998 [SYN, ACK] Seq=0 Ack=1 Win=14080 Len=0 MSS=1420 SACK_PERM=1 TSval=570636358 TSecr=6022010 WS=256
150.718551 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a01:578:3::36e4:ea94 TCP 72 53998 > https [ACK] Seq=1 Ack=1 Win=14464 Len=0 TSval=6022033 TSecr=570636358
150.719008 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a01:578:3::36e4:ea94 SSL 345 Client Hello
150.740435 2a01:578:3::36e4:ea94 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 72 https > 53998 [ACK] Seq=1 Ack=274 Win=15360 Len=0 TSval=570636364 TSecr=6022033
150.743448 2a01:578:3::36e4:ea94 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 SSL 1352 [TCP Previous segment lost] Continuation Data
150.743602 2a01:578:3::36e4:ea94 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 SSL 316 Continuation Data
150.744313 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a01:578:3::36e4:ea94 TCP 84 [TCP Dup ACK 63#1] 53998 > https [ACK] Seq=274 Ack=1 Win=14464 Len=0 TSval=6022059 TSecr=570636364 SLE=2817 SRE=4097
150.744568 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a01:578:3::36e4:ea94 TCP 84 [TCP Dup ACK 63#2] 53998 > https [ACK] Seq=274 Ack=1 Win=14464 Len=0 TSval=6022059 TSecr=570636364 SLE=2817 SRE=4341
and here traceroute:
[root@localhost ~]# traceroute6 -I -n www.netflix.com
traceroute to www.netflix.com (2a01:578:3::2e89:b535), 30 hops max, 80 byte packets
1 2a01:348:6:87a3::1 2.860 ms 2.508 ms 2.209 ms
2 2a01:348:6:7a3::1 8.849 ms 9.765 ms 10.854 ms
3 2a01:348:0:4:0:3:1:1 12.180 ms 13.386 ms 15.104 ms
4 2a01:348:0:4:0:3:0:1 16.719 ms 17.867 ms 19.595 ms
5 2a01:348::65:0:1 20.980 ms 22.209 ms 35.116 ms
6 2a01:348::80:0:1 24.249 ms 24.454 ms 25.772 ms
7 2001:728:0:5000::7a5 27.671 ms 23.615 ms 22.780 ms
8 2001:728:0:5000::72a 22.875 ms 23.601 ms 23.529 ms
9 * * *
10 * * *
11 2a01:578::1f 32.896 ms 32.950 ms 32.844 ms
12 2a01:578::13 31.875 ms 31.988 ms 33.926 ms
13 2a01:578::b 33.662 ms 32.868 ms 33.913 ms
14 2a01:578:3::2e89:b535 34.600 ms 19.606 ms 19.444 ms
[root@localhost ~]# traceroute6 -I -n 2a01:578:3::36e4:ea94
traceroute to 2a01:578:3::36e4:ea94 (2a01:578:3::36e4:ea94), 30 hops max, 80 byte packets
1 2a01:348:6:87a3::1 2.897 ms 2.499 ms 2.234 ms
2 2a01:348:6:7a3::1 7.710 ms 8.493 ms 9.552 ms
3 2a01:348:0:4:0:3:1:1 10.671 ms 11.824 ms 12.940 ms
4 2a01:348:0:4:0:3:0:1 41.985 ms 42.671 ms 42.689 ms
5 2a01:348::65:0:1 21.370 ms 21.484 ms 22.150 ms
6 2a01:348::80:0:1 23.693 ms 22.997 ms 24.922 ms
7 2001:728:0:5000::7a5 26.922 ms 22.952 ms 23.635 ms
8 2001:728:0:5000::72a 23.396 ms 23.298 ms 23.401 ms
9 * * *
10 2a01:578::1f 30.008 ms 29.938 ms 30.255 ms
11 2a01:578::13 29.325 ms 28.451 ms 28.859 ms
12 2a01:578::b 29.197 ms 29.313 ms 30.080 ms
13 2a01:578:3::36e4:ea94 29.865 ms 24.185 ms 25.149 ms
When I try www.facebook.com all seems to work as expected:
229.201199 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 80 60723 > https [SYN] Seq=0 Win=14400 Len=0 MSS=1440 SACK_PERM=1 TSval=6100518 TSecr=0 WS=128
229.213645 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 80 https > 60723 [SYN, ACK] Seq=0 Ack=1 Win=13980 Len=0 MSS=1410 SACK_PERM=1 TSval=888621813 TSecr=6100518 WS=256
229.214577 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 72 60723 > https [ACK] Seq=1 Ack=1 Win=14464 Len=0 TSval=6100531 TSecr=888621813
229.214945 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 SSL 346 Client Hello
229.239353 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 72 https > 60723 [ACK] Seq=1 Ack=275 Win=15104 Len=0 TSval=888621838 TSecr=6100531
229.239823 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 SSL 497 [TCP Previous segment lost] Continuation Data
229.240481 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 84 [TCP Dup ACK 80#1] 60723 > https [ACK] Seq=275 Ack=1 Win=14464 Len=0 TSval=6100557 TSecr=888621838 SLE=2797 SRE=3222
229.246862 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 1428 [TCP Retransmission] Server Hello
229.247017 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 114 [TCP Retransmission] [TCP segment of a reassembled PDU]
229.247384 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 1428 [TCP Retransmission] [TCP segment of a reassembled PDU]
229.247526 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 114 [TCP Retransmission] [TCP segment of a reassembled PDU]
229.247664 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 84 60723 > https [ACK] Seq=275 Ack=1399 Win=17280 Len=0 TSval=6100564 TSecr=888621845 SLE=2797 SRE=3222
229.247968 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 72 60723 > https [ACK] Seq=275 Ack=3222 Win=20224 Len=0 TSval=6100565 TSecr=888621845
229.252692 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TLSv1.2 198 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
229.265787 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 72 https > 60723 [ACK] Seq=3222 Ack=401 Win=15104 Len=0 TSval=888621865 TSecr=6100569
229.266280 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 330 Encrypted Handshake Message, Change Cipher Spec, Encrypted Handshake Message
229.267752 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TLSv1.2 215 Application Data
229.281299 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 72 https > 60723 [ACK] Seq=3480 Ack=544 Win=16128 Len=0 TSval=888621880 TSecr=6100584
229.330759 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 654 Application Data
229.333105 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TLSv1.2 272 Application Data
229.347820 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 72 https > 60723 [ACK] Seq=4062 Ack=744 Win=17408 Len=0 TSval=888621946 TSecr=6100650
229.455070 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 1428 [TCP segment of a reassembled PDU]
229.455344 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 533 Application Data, Application Data
229.455685 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 1428 [TCP segment of a reassembled PDU]
229.455919 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 1428 Application Data
229.456140 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 1428 Application Data
229.456362 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 1428 Application Data
229.456530 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 72 60723 > https [ACK] Seq=744 Ack=5879 Win=31104 Len=0 TSval=6100772 TSecr=888622047
229.456740 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TLSv1.2 679 Application Data
229.456850 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 72 60723 > https [ACK] Seq=744 Ack=11303 Win=37888 Len=0 TSval=6100772 TSecr=888622047
229.457142 2a03:2880:f01a:1:face:b00c:0:1 -> 2a01:348:6:87a3:20c:29ff:fee6:d614 TCP 1428 [TCP segment of a reassembled PDU]
229.459705 2a01:348:6:87a3:20c:29ff:fee6:d614 -> 2a03:2880:f01a:1:face:b00c:0:1 TCP 72 60723 > https [ACK] Seq=744 Ack=13266 Win=35968 Len=0 TSval=6100776 TSecr=888622047
[root@localhost ~]# traceroute6 -I -n www.facebook.com
traceroute to www.facebook.com (2a03:2880:f01a:1:face:b00c:0:1), 30 hops max, 80 byte packets
1 2a01:348:6:87a3::1 2.049 ms 1.375 ms 1.283 ms
2 2a01:348:6:7a3::1 7.361 ms 8.393 ms 9.765 ms
3 2a01:348:0:4:0:3:1:1 10.887 ms 12.511 ms 14.790 ms
4 2a01:348:0:4:0:3:0:1 24.752 ms 24.678 ms 24.617 ms
5 2a01:348::66:1:1 24.315 ms 25.383 ms 25.337 ms
6 2a01:348::27:0:1 23.567 ms 25.805 ms 25.759 ms
7 2001:7f8:4::80a6:2 35.444 ms 31.628 ms 31.023 ms
8 2620:0:1cff:dead:bef0::235 29.678 ms 29.538 ms 29.757 ms
9 2a03:2880:f01a:ffff::b 30.427 ms 22.655 ms 23.361 ms
10 2a03:2880:f01a:1:face:b00c:0:1 22.672 ms 23.502 ms 24.700 ms
I tried to contact NTT (owner of 2a01:348:6:87a3::1), but they say, that they will only deal with their direct customers, i.e. not me. What is the way to get this resolved?
As temporary solution I have enforced smaller TCP MSS on packets leaving my network to get it to work, but would think that nudging whoever is dropping the ICMPv6 packets to fix that, would be better solution.
Any way you can help, please?
Kind Regards
Vladimir
State change: remoteproblem
The state of this ticket has been changed to remoteproblem
IPv6 MTU Path discovery does not work to www.netflix.com
MTU on internal network is 1500, MTU of the tunnel currently set to 1428, but it worked exactly the same with default value of 1280.
You did change the MTU on both your side of the tunnel AND on the Website (and thus the PoP) I hope?
- Check if your tunnel uses the right MTU
- Check with tracepath6 if ICMP PTBs come through properly.
This could just be Netflix dropping ICMP PTB, at least Google is doing the same thing.
I tried to contact NTT (owner of 2a01:348:6:87a3::1), but they say, that they will only deal with their direct customers, i.e. not me. What is the way to get this resolved?
"2a01:348:6:87a3::1" has no relation to NTT in any way. 2a01:348:6:87a3::1 is in SixXS address space of the gblon02 PoP.
That noted, Netflix works fine for me.
Though checking tracepath6:
$ tracepath6 www.netflix.com
1?: [LOCALHOST] 0.107ms pmtu 1500
2: .... 3.050ms pmtu 1480
[..]
9: pni-amazon.lon2.init7.net 36.448ms asymm 8
10: no reply
11: no reply
12: 2a01:578::13 46.143ms asymm 15
13: 2a01:578::13 46.556ms asymm 15
14: 2a01:578::b 47.346ms asymm 13
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
That is a lot of "no replies". Thus something is dropping ICMP on that path.
IPv6 MTU Path discovery does not work to www.netflix.com
Shadow Hawkins on Monday, 23 March 2015 23:34:56
Jeroen Massar wrote:
> MTU on internal network is 1500, MTU of the tunnel currently set to 1428, but it worked exactly the same with default value of 1280. You did change the MTU on both your side of the tunnel AND on the Website (and >thus the PoP) I hope?
Yes, my tunnel is AYIYA, I changed it on website, restarted aiccu and on my side of tunnel it says:
4: sixxs: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1428 qdisc pfifo_fast state UNKNOWN qlen 500
- Check with tracepath6 if ICMP PTBs come through properly.
This is tracepath6:
[root@localhost ~]# tracepath6 -n www.netflix.com
1?: [LOCALHOST] 0.130ms pmtu 1500
1: 2a01:348:6:87a3::1 1.526ms
1: 2a01:348:6:87a3::1 2.075ms
2: 2a01:348:6:87a3::1 1.338ms pmtu 1428
2: 2a01:348:6:7a3::1 18.615ms
3: 2a01:348:0:4:0:3:1:1 18.919ms asymm 2
4: 2a01:348:0:4:0:3:0:1 21.187ms asymm 3
5: 2a01:348::65:0:1 19.539ms asymm 4
6: 2a01:348::80:0:1 19.074ms asymm 5
7: 2001:728:0:5000::7a5 20.148ms asymm 6
8: 2001:728:0:5000::72a 19.540ms asymm 7
9: no reply
10: 2a01:578::8 32.752ms asymm 18
11: 2a01:578::7 31.019ms asymm 18
12: 2a01:578::13 31.449ms asymm 17
13: 2a01:578::b 31.611ms asymm 15
14: no reply
15: no reply
16: no reply
...
This could just be Netflix dropping ICMP PTB, at least Google is doing the same thing.
Hmm. With regard of Google I noticed that they seem to restrict their outgoing MTU to 1280.
> I tried to contact NTT (owner of 2a01:348:6:87a3::1), but they say, that they will only > deal with their direct customers, i.e. not me. What is the way to get this resolved? "2a01:348:6:87a3::1" has no relation to NTT in any way. 2a01:348:6:87a3::1 is in SixXS address space of the gblon02 PoP.
My bad here, copied wrong address, should have been 2001:728:0:5000::72a.
That noted, Netflix works fine for me. Though checking tracepath6:
$ tracepath6 www.netflix.com 1?: [LOCALHOST] 0.107ms pmtu 1500 2: .... 3.050ms pmtu 1480 [..] 9: pni-amazon.lon2.init7.net 36.448ms asymm 8 10: no reply 11: no reply 12: 2a01:578::13 46.143ms asymm 15 13: 2a01:578::13 46.556ms asymm 15 14: 2a01:578::b 47.346ms asymm 13 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply That is a lot of "no replies". Thus something is dropping ICMP on that path.
The end is the same as mine. I guess it would be easier to try it somehow from Netflix side, but to get through their support is tricky. Last time I tried, the lady was two minutes searching what IPv6 is :-(
Anyway thanks for help.
IPv6 MTU Path discovery does not work to www.netflix.com
Yes, my tunnel is AYIYA, I changed it on website[..]
But is your IPv4 MTU also really 1500 that you have maxed the MTU to 1428?
Hmm. With regard of Google I noticed that they seem to restrict their outgoing MTU to 1280.
Not even the MTU, they restrict the TCP MSS. They do not care about non-TCP non-HTTP(s) ports.
They still have to hit themselves in the face when they finally fully understand that their own QUIC protocol does not support packet sizes other than 1500
IPv6 MTU Path discovery does not work to www.netflix.com
Shadow Hawkins on Wednesday, 25 March 2015 22:47:25
Jeroen Massar wrote:
> Yes, my tunnel is AYIYA, I changed it on website[..] But is your IPv4 MTU also really 1500 that you have maxed the MTU to 1428?
yes I am sure my IPv4 MTU is 1500. I checked it before maxing MTU to 1428. And even with MTU of 1428 https://www.facebook.com works without issue. Got contact to Amazon networking people (Netflix is using Amazon AWS), will try to contact them to see if they can help.
IPv6 MTU Path discovery does not work to www.netflix.com
Carmen Sandiego on Friday, 10 April 2015 21:29:02
Vladimir Michl wrote:
Jeroen Massar wrote:
Any luck with speaking with amazon? I'm having problems with http://www.wimbornefirst.dorset.sch.uk/ not working and the common theme is that this site and netflix are hosted by amazon ELB in Ireland. (I'm using gblon02 also.)
> Yes, my tunnel is AYIYA, I changed it on website[..] But is your IPv4 MTU also really 1500 that you have maxed the MTU to 1428?
yes I am sure my IPv4 MTU is 1500. I checked it before maxing MTU to 1428. And even with MTU of 1428 https://www.facebook.com works without issue. Got contact to Amazon networking people (Netflix is using Amazon AWS), will try to contact them to see if they can help.
IPv6 MTU Path discovery does not work to www.netflix.com
Any luck with speaking with amazon? I'm having problems with http://www.wimbornefirst.dorset.sch.uk/ not working and the common theme is that this site and netflix are hosted by amazon ELB in Ireland. (I'm using gblon02 also.)
$ tracepath6 www.wimbornefirst.dorset.sch.uk
1?: [LOCALHOST] 0.006ms pmtu 1500
[..]
15: AMAZON.COM.car1.Dublin.Level3.net 33.140ms asymm 14
16: 2a01:578::15 35.834ms asymm 14
17: 2a01:578::b 35.931ms asymm 13
18: no reply
19: no reply
20: no reply
Obviously ICMPv6 is being dropped. Amazon--....
State change: user
The state of this ticket has been changed to user
|
![[ch]](/s/countries/ch.gif)
on Sunday, 22 March 2015 11:28:54
Posting is only allowed when you are