Forum - Microsoftonline (Office365) :: SixXS - IPv6 Deployment & Tunnel Broker

Microsoftonline (Office365)

Shadow Hawkins on Monday, 29 July 2013 20:56:41

I am looking to find out if anyone has had problems with microsoftonline.com services. I have a Mikrotik router running 5.25 and we have had problems with services since they upgraded some portions of the Office 365 to IPV6. If you use the services for email (outlook) or the webapps, we have problems with connectivity. The configuration for the tunnel was setup exactly as the faq stated for a Mikrotik router except we had bumped the mtu up to 1480 to resolve some problems we were having with government websites. If anyone could point to something that may helpe me out I would appreciate it. I had also posted on the Miktrotik forums some information related to this. IPV6 and Microsoft's webapps and outlook.com From what I can tell we really have not had problems with other IPV6 enabled sites. Oddly enough, I have a Linux router at home using sixxs and the sites load just fine for me there, I am using the same POP at work as I do at home. Thanks

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 05:35:20

we have had problems with services

What kind of "problems"? You might want to do a tcpdump/wireshark on the client machine or any intermediate host to see what packets you see and what not. MTU sizes can indeed be problematic, some people drop ICMPv6 as they clearly do not know what it is used for. Wireshark will tell you about these issues though.

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 13:56:28

Jeroen Massar wrote:

> we have had problems with services What kind of "problems"? You might want to do a tcpdump/wireshark on the client machine or any intermediate host to see what packets you see and what not. MTU sizes can indeed be problematic, some people drop ICMPv6 as they clearly do not know what it is used for. Wireshark will tell you about these issues though.

I will get a wireshark dump shortly today, but to describe the "problems" Our location has been using IPV6 for about 2 years now, we have had little trouble overall. We have had some problems with Microsoft's online services after Microsoft upgraded them to Office365. Which included there hosts started supporting IPV6. We have had some problems with outlook.com's connection for outlook now that IPV6 was enabled on there end. Also webapps will never connect unless we disable ipv6 on the hosts web browser. The host for some of the apps is usc-word-edit.officeapps.live.com The lookup looks like this. usc-word-edit.officeapps.live.com is an alias for usc-word-edit.officeapps.live.com.akadns.net. usc-word-edit.officeapps.live.com.akadns.net has address 65.54.54.55 usc-word-edit.officeapps.live.com.akadns.net has IPv6 address 2a01:111:f406:3000::55 So the web browser by default will use the ipv6 address to connect. I will begin a request to open a document from off of the Live Drive site, during that time I will look in my logs and see this below: firewall,info DROP FORWARD forward: in:sixxs out:vrrp4_ipv6, proto TCP (ACK,RST), 2a01:111:f406:3000::55:443->2604:8800:100:81a0:6506:975e:92f8:30f1:50497, len 20 The 8800:100:81a0:6506:975e:92f8:30f1 is my machine. After some time the broweser will say that the site could not be connected to and stop trying. If I disable IPV6 in the browser (via a plugin for Firefox or Chrome) I am able to connect. If we disable IPV6 on the host the email client and the web browser will connect fine. With IPV6 enabled the host email client will take a very long time updating and in some case will never connect. I recently added a rule for dropping invalid before the drop rule and these drop connections now are showing up like below 09:47:04 firewall,info DROP Invalid forward: in:sixxs out:10_10_NET, proto TCP (ACK,FIN,PSH), 2600:1407:9:2:8100::236:443->2604:8800:161:4:30f9:e145:a0f1:789e:46541, len 59 Where should I put up tcpdump files? I will mirror a port off of the router and capture from there, the file may be pretty large?

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 14:03:04

So the web browser by default will use the ipv6 address to connect.

Unfortunately due to the "Happy Eyeballs" it is not that simple anymore. Proper programs will use the 'best' or 'first' available transport.... which makes debugging a lot of fun! ;)

firewall,info DROP FORWARD forward: in:sixxs out:vrrp4_ipv6, proto TCP (ACK,RST), 2a01:111:f406:3000::55:443->2604:8800:100:81a0:6506:975e:92f8:30f1:50497, len 20

The 8800:100:81a0:6506:975e:92f8:30f1 is my machine.

If you are dropping packets, then of course things will not work.

I recently added a rule for dropping invalid before the drop rule and these drop connections now are showing up like below

Maybe you should not drop these packets!? What kind of rules do you have?

Where should I put up tcpdump files?

Nowhere, yet at least. I suggest fixing your firewall first.

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 15:37:51

Jeroen Massar wrote:

> So the web browser by default will use the ipv6 address to connect. Unfortunately due to the "Happy Eyeballs" it is not that simple anymore. Proper programs will use the 'best' or 'first' available transport.... which makes debugging a lot of fun! ;)

firewall,info DROP FORWARD forward: in:sixxs out:vrrp4_ipv6, proto TCP (ACK,RST), 2a01:111:f406:3000::55:443->2604:8800:100:81a0:6506:975e:92f8:30f1:50497, len 20

The 8800:100:81a0:6506:975e:92f8:30f1 is my machine.

If you are dropping packets, then of course things will not work.

I recently added a rule for dropping invalid before the drop rule and these drop connections now are showing up like below

Maybe you should not drop these packets!? What kind of rules do you have?

Where should I put up tcpdump files?

Nowhere, yet at least. I suggest fixing your firewall first.

First of all we have tested this without any filtering turned on, just an open firewall. We still have the same problems with the connections. The addition of the filter tells me based on the configuration we have that allows related and established connections to return, we should not be seeing any drops or invalid connections. The packets should just pass back through. But for some reason the firewall does not see the connections as related or established. I followed a connection via wireshark and during the whole connection to the server we ended up with a large number TCP Retransmissions (client Hello) and TCP Dup ACK's.

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 15:58:12

First of all we have tested this without any filtering turned on, just an open firewall. We still have the same problems with the connections.

Rechecking, the above is a ACK,RST, thus it is effectively denying connections. That address seems to be either heavily firewalled or not active:

$ traceroute6 2a01:111:f406:3000::55 traceroute to 2a01:111:f406:3000::55 (2a01:111:f406:3000::55), 30 hops max, 80 byte packets [..] 3 pao-76e-3.ntwk.msn.net (2001:504:d::98) 0.491 ms 0.491 ms 0.483 ms 4 2a01:111:2000:2::655 (2a01:111:2000:2::655) 33.344 ms 33.334 ms 33.318 ms 5 * * 2a01:111:2000:2::646 (2a01:111:2000:2::646) 33.299 ms 6 * * * 7 * * *

The addition of the filter tells me based on the configuration we have that allows related and established connections to return, we should not be seeing any drops or invalid connections.

Are you initiating a connection toward that address? Which DNS label is associated to it?

But for some reason the firewall does not see the connections as related or established

What kind (model/version) of firewall is this? Your line has "out:vrrp4_ipv6" which sounds like you are doing some kind of VRRP, are you sure that is working properly along with your connection tracking etc?

I followed a connection via wireshark and during the whole connection to the server we ended up with a large number TCP Retransmissions (client Hello) and TCP Dup ACK's.

Just sounds like more debugging might be required.

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 16:17:28

Jeroen Massar wrote:

> First of all we have tested this without any filtering turned on, just an open firewall. We still have the same problems with the connections. Rechecking, the above is a ACK,RST, thus it is effectively denying connections. That address seems to be either heavily firewalled or not active:

The addition of the filter tells me based on the configuration we have that allows related and established connections to return, we should not be seeing any drops or invalid connections.

Are you initiating a connection toward that address? Which DNS label is associated to it?

But for some reason the firewall does not see the connections as related or established

I followed a connection via wireshark and during the whole connection to the server we ended up with a large number TCP Retransmissions (client Hello) and TCP Dup ACK's.

Just sounds like more debugging might be required.

We have networks that are not setup with VRRP and the same results. For the above address that is Microsoft and they block all traffic for ICMP. Here is another address of theirs to work with. 2a01:111:f400:400::9 this address is outlook mail. The host name is outlook.office365.com Another hostname is usc-word-edit.officeapps.live.com We use a Mikrotik router and we had configured the router for the tunnel based on the howtos from sixxs as I had stated in my post earlier. To let you know I have another tunnel at home that is running a LINUX firewall that works just fine with this. That is why the confusion I am having with this setup not working.

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 16:23:14

2a01:111:f400:400::9 this address is outlook mail. The host name is outlook.office365.com

Another hostname is usc-word-edit.officeapps.live.com

$ telnet usc-word-edit.officeapps.live.com 443 Trying 2a01:111:f406:1006::87... Connected to usc-word-edit.officeapps.live.com.akadns.net. Escape character is '^]'. ^]q

2a01:111:f400:400::9 is live too. Note that I am getting a different address thanks to the magic of Akamai, which always makes a lot of fun out of debugging, as you might be hitting a path that is actually broken...

To let you know I have another tunnel at home that is running a LINUX firewall that works just fine with this. That is why the confusion I am having with this setup not working.

Same network path, same tunnel type, parameters etc?

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 16:25:41

Jeroen Massar wrote:

> 2a01:111:f400:400::9 this address is outlook mail. The host name is outlook.office365.com Another hostname is usc-word-edit.officeapps.live.com

$ telnet usc-word-edit.officeapps.live.com 443 Trying 2a01:111:f406:1006::87... Connected to usc-word-edit.officeapps.live.com.akadns.net. Escape character is '^]'. ^]q

To let you know I have another tunnel at home that is running a LINUX firewall that works just fine with this. That is why the confusion I am having with this setup not working.

Same network path, same tunnel type, parameters etc?

Yes, it has the same POP anhd the same configuration for MTU set to 1480 and is using sixxs. The firewall is using iptables, but has the same rules for allowing related and established.

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 16:38:04

Yes, it has the same POP and the same configuration for MTU set to 1480 and is using sixxs.

Is the underlying path 1480 capable?

$ tracepath 207.250.246.98 1: uschi03.sixxs.net 0.378ms pmtu 1500 1: gw01-VL206.ord07.cymru.com 1.382ms 1: gw01-VL206.ord07.cymru.com 0.562ms 2: tge3-1.fr3.ord4.llnw.net 6.731ms 3: ve8.fr3.ord.llnw.net 1.944ms 4: ve6.fr4.ord.llnw.net 2.293ms 5: chi2-pr1-ae5-102.us.twtelecom.net 2.117ms asymm 6 6: ind1-ar4-xe-0-1-0-0.us.twtelecom.net 9.153ms asymm 8 7: no reply 8: 207-250-246-98.static.twtelecom.net 11.583ms reached Resume: pmtu 1500 hops 8 back 57 $ tracepath 216.37.13.67 1: uschi03.sixxs.net 0.342ms pmtu 1500 1: gw01-VL206.ord07.cymru.com 0.434ms 1: gw01-VL206.ord07.cymru.com 0.845ms 2: tge3-1.fr3.ord4.llnw.net 1.933ms 3: ve8.fr3.ord.llnw.net 4.407ms 4: 144.223.139.101 1.894ms 5: 144.232.1.105 3.701ms 6: 144.232.20.186 9.260ms 7: 144.232.18.58 9.693ms 8: sl-st21-pit-0-0-0.sprintlink.net 23.465ms asymm 10 9: 63.160.2.10 29.839ms asymm 12 10: no reply 11: no reply 12: no reply 13: te2-7-4.4045.701-core.expedient.com 15.796ms asymm 7 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply

One of them seems to have iffy filtering going on.... which might indicate odd filters. Is that the host that is causing issues?

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 17:00:39

Jeroen Massar wrote:

> Yes, it has the same POP and the same configuration for MTU set to 1480 and is using sixxs. Is the underlying path 1480 capable?

One of them seems to have iffy filtering going on.... which might indicate odd filters. Is that the host that is causing issues?

207.250.246.98 this is the host we are working on now. The other system is really out of my control for the firewall. I lowered the MTU on the tunnel to 1280 and it seems to be acting the same as before.

Microsoftonline (Office365)

Jeroen Massar SixXS Staff

on Tuesday, 30 July 2013 17:03:43

I lowered the MTU on the tunnel to 1280 and it seems to be acting the same as before.

Changes to tunnel parameters are not instant, you have to wait a little longer for it take effect...

Microsoftonline (Office365)

Shadow Hawkins on Tuesday, 30 July 2013 17:58:09

Jeroen Massar wrote:

> I lowered the MTU on the tunnel to 1280 and it seems to be acting the same as before. Changes to tunnel parameters are not instant, you have to wait a little longer for it take effect...

I determined that the first hop out of our network, that device could only do 1470 for an mtu, all other devices in the hops to sixxs could as well. I set the tunnel to 1450 mtu and tested the connections again after about 15 minutes. Same results.

Please note Posting is only allowed when you are logged in.