|
Static tunnel dead after power failure
![[nl]](/s/countries/nl.gif) Shadow Hawkins on Wednesday, 01 November 2006 22:57:20
I've been busy with this problem for the last 3 hours, but to no avail. This morning, the power went down in my room, so I took my server to another room and compiled linux-2.6.17-beyond4 (previous kernel was 2.6.16-beyond2). I did a make oldconfig and doublechecked the values in make menuconfig. Also, I put in a new gigabit NIC.
When the power came back on, I started my server again and found there was no IPv6 connectivity. The sixxs device did get created by aiccu, and it also discovered my tunnel IP. An strace (and later tcpdump) on aiccu showed me that it did actually connect to the PoP to fetch the information, and it set my interface afterwards. My routes also looked fine, so I suspected my kernel.
After some more testing I noticed my IPv6 stack was actually working properly, at least for my internal network. I could also use the AYIYA-tunnel on my laptop, so it's not a problem with my internet connection.
I upgraded aiccu to 2006.07.25, that didn't work.
I manually created the connection using the ip-commands, that didn't work.
I did aiccu test:
####### [1/8] Ping the IPv4 Local/Your Outer Endpoint (62.131.46.165) - PASS
####### [2/8] Ping the IPv4 Remote/PoP Outer Endpoint (192.87.102.107) - PASS
####### [3/8] Traceroute to the PoP (192.87.102.107) over IPv4 - PASS
###### [4/8] Checking if we can ping IPv6 localhost (::1) - PASS
###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:610:600:28d::2) - PASS
###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:610:600:28d::1) - FAIL
PING 2001:610:600:28d::1(2001:610:600:28d::1) 56 data bytes
From 2001:610:600:28d::2 icmp_seq=1 Destination unreachable: Address unreachable
###### [7/8] Traceroute6 to the central SixXS machine (noc.sixxs.net) - FAIL
traceroute to noc.sixxs.net (2001:838:1:1:210:dcff:fe20:7c7c) from 2001:610:600:28d::2, 30 hops max, 16 byte packets
1 cl-654.ams-05.nl.sixxs.net (2001:610:600:28d::2) 0.145 ms !H 0.081 ms !H 0.036 ms !H
###### [8/8] Traceroute6 to (www.kame.net) - FAIL
traceroute to www.kame.net (2001:200:0:8002:203:47ff:fea5:3085) from 2001:610:600:28d::2, 30 hops max, 16 byte packets
1 cl-654.ams-05.nl.sixxs.net (2001:610:600:28d::2) 0.08 ms !H 0.079 ms !H 0.029 ms !H
So it looks like only the IPv6-part is not working. Now I decided to test my AYIYA-tunnel. After editing my tunnel ID, I restarted aiccu and I had IPv6-connectivity! Ping, browsing, everything worked fine. I had a good look at the routing table and ifconfig, reset my config to the static tunnel, started aiccu again and modified the route settings (only 1 route was a little bit different). That didn't work, I have no connectivity with my static route.
Removing all other IPv6-addresses (on eth0 eth1 eth2) didn't help either.
Details:
My server is in DMZ, worked like a dream before the power failure. Aiccu was recompiled (upgraded), no apparent filesystem corruptions found. eth0 (internet interface) wasn't changed by the hardware upgrade.
Static tunnel: T8273 nlams05 2001:610:600:28d::2
AYIYA tunnel: T9685 nlams04 2001:960:2:53f::2
helios ~ # ip -6 ro
2001:610:600:28d::/64 via :: dev sixxs metric 256 expires -1sec mtu 1280 advmss 1220 metric 10 4294967295
2001:610:683::/64 dev eth2 metric 256 expires -7435sec mtu 1500 advmss 1220 metric 10 4294967295
2001:610:683:1::/64 dev eth0 metric 256 expires -7472sec mtu 1500 advmss 1220 metric 10 4294967295
2001:610:683:2::/64 dev eth1 metric 256 expires -7467sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 dev eth0 metric 256 expires -7475sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 dev eth1 metric 256 expires -7467sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 dev eth2 metric 256 expires -7436sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 dev vmnet8 metric 256 expires -6674sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 dev vmnet1 metric 256 expires -6674sec mtu 1500 advmss 1220 metric 10 4294967295
fe80::/64 via :: dev sixxs metric 256 expires -1sec mtu 1280 advmss 1220 metric 10 4294967295
ff00::/8 dev eth0 metric 256 expires -7475sec mtu 1500 advmss 1220 metric 10 4294967295
ff00::/8 dev eth1 metric 256 expires -7467sec mtu 1500 advmss 1220 metric 10 4294967295
ff00::/8 dev eth2 metric 256 expires -7436sec mtu 1500 advmss 1220 metric 10 4294967295
ff00::/8 dev vmnet8 metric 256 expires -6674sec mtu 1500 advmss 1220 metric 10 4294967295
ff00::/8 dev vmnet1 metric 256 expires -6674sec mtu 1500 advmss 1220 metric 10 4294967295
ff00::/8 dev sixxs metric 256 expires -1sec mtu 1280 advmss 1220 metric 10 4294967295
default via 2001:610:600:28d::1 dev sixxs metric 1024 expires -1sec mtu 1280 advmss 1220 metric 10 4294967295
sixxs Link encap:IPv6-in-IPv4
inet6 addr: 2001:610:600:28d::2/64 Scope:Global
inet6 addr: fe80::3e83:2ea5/128 Scope:Link
inet6 addr: fe80::410:600:28d:2/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Note 0 packets. I don't know why. In iptraf I only see responses from lo when I attempt to ping something. With my AYIYA-tunnel I get at least 10 packets instantly. I also asked someone I know which PoP he is on, he said nlams05. He reconnected this very evening without any problem.
I'm out of options, I don't know what else I can try. I hope someone can help.
Static tunnel dead after power failure
###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:610:600:28d::2) - PASS
This should generate packets on the interface already.
###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:610:600:28d::1) - FAIL PING 2001:610:600:28d::1(2001:610:600:28d::1) 56 data bytes From 2001:610:600:28d::2 icmp_seq=1 Destination unreachable: Address unreachable
This clearly indicates that proto-41 packets can't get out as your local host is already replying that the address is unreachable.
My server is in DMZ,
If you mean with this behind a NAT then you should definitely check your NAT device and maybe MAC tables (as you changed your NIC) in that thing. A reboot of the NAT thingy is usually a good try too ;) It's a Microsoft solution but it works sometimes. Also check your local and remote address of the tunnel.
I also asked someone I know which PoP he is on, he said nlams05. He reconnected this very evening without any problem.
Define "reconnect", static tunnels always exist on the PoP when they are enabled. Or did you mean 'restarting AICCU' ?
Note that the PoP can't IPv4 ping 62.131.46.165. I suggest you check your firewall rules.
I'm out of options, I don't know what else I can try. I hope someone can help.
Read "Reporting problems" on the contact page, it contains a long list of things you can test on. The tcpdump portion is most likely the part you should focus on. Also important is to check 'ip tun sho' which shows the local and remote tunnel IPv4 addresses.
On another note, I notice that you have vmware interfaces, I recall that vmware sometimes has some strange interactions with the kernel causing IPv6 to break. As I do have a couple of boxes with vmware running though, I know that it can work but I recall that it required a couple of workarounds.
(PS: I like the fact that AYIYA simply works ;)
Static tunnel dead after power failure
Shadow Hawkins on Thursday, 02 November 2006 08:07:28###### [5/8] Ping the IPv6 Local/Your Inner Tunnel Endpoint (2001:610:600:28d::2) - PASS This should generate packets on the interface already.
Ah, yes. I see it does. Only thing is I get packets on lo.
###### [6/8] Ping the IPv6 Remote/PoP Inner Tunnel Endpoint (2001:610:600:28d::1) - FAIL PING 2001:610:600:28d::1(2001:610:600:28d::1) 56 data bytes From 2001:610:600:28d::2 icmp_seq=1 Destination unreachable: Address unreachable This clearly indicates that proto-41 packets can't get out as your local host is already replying that the address is unreachable.
And do you happen to know some sort of workaround, or a reason why this is happening? I noticed that these pings show up as errors on the sixxs device.
sixxs Link encap:IPv6-in-IPv4
...
TX packets:0 errors:1378 dropped:0 overruns:0 carrier:1378
...
I don't even see the light on my switch blink when I ping6 -i 0.
My server is in DMZ, If you mean with this behind a NAT then you should definitely check your NAT device and maybe MAC tables (as you changed your NIC) in that thing. A reboot of the NAT thingy is usually a good try too It's a Microsoft solution but it works sometimes. Also check your local and remote address of the tunnel.
I rebooted my router, that didn't help. I don't use MAC-addresses for network access, and I didn't change that NIC. It's still the same as it was before. I can also reach my webserver over IPv4 (from another server), so the DMZ is working properly, because I don't have any portforwarding rules for port 80.
I also asked someone I know which PoP he is on, he said nlams05. He reconnected this very evening without any problem. Define "reconnect", static tunnels always exist on the PoP when they are enabled. Or did you mean 'restarting AICCU' ?
Yes, restarting AICCU probably. I'm not sure if he even uses AICCU, but his he 'restarted' his entire connection.
Note that the PoP can't IPv4 ping 62.131.46.165. I suggest you check your firewall rules.
Hm, indeed. That's probably normal with this router. I can't find any option to change it.
I'm out of options, I don't know what else I can try. I hope someone can help. Read "Reporting problems" on the contact page, it contains a long list of things you can test on. The tcpdump portion is most likely the part you should focus on. Also important is to check 'ip tun sho' which shows the local and remote tunnel IPv4 addresses.
ip tun sho told me the following:
helios ~ # ip tun sho
sit0: ipv6/ip remote any local any ttl 64 nopmtudisc
sixxs: ipv6/ip remote 192.87.102.107 local 62.131.46.165 ttl inherit
which looks fine to me. I'll have a closer look at man tcpdump later. I only know some basic tcpdump options.
On another note, I notice that you have vmware interfaces, I recall that vmware sometimes has some strange interactions with the kernel causing IPv6 to break. As I do have a couple of boxes with vmware running though, I know that it can work but I recall that it required a couple of workarounds.
Stopping vmware (and unloading those modules from the kernel) and restarting aiccu didn't fix anything either. I wish it were that simple :)
helios ~ # lsmod
Module Size Used by
vmnet 29292 13
vmmon 98892 0
tun 7552 0
e1000 106240 0
via686a 12936 0
eeprom 5136 0
i2c_isa 3200 1 via686a
i2c_viapro 6612 0
i2c_core 14800 4 via686a,eeprom,i2c_isa,i2c_viapro
3c59x 34280 0
dmfe 15580 0
This is the list of loaded modules at the moment. I don't see anything which might even be related to iptables, so I still don't know why it's not working. I'll try building the tunnel on my laptop tomorrow, maybe it can clarify the problem a bit.
Static tunnel dead after power failure
This clearly indicates that proto-41 packets can't get out as your local host is already replying that the address is unreachable. And do you happen to know some sort of workaround, or a reason why this is happening? I noticed that these pings show up as errors on the sixxs device.
Because the tunneled packet already fails to be delivered. The kernel clearly knows this and thus does not even attempt to try and transmit the packet any further. -> Check the configuration of the tunnel (thus source + remote IP address)
It shows up as a carrier error as the carrier (the tunnel) is failing.
ip tun sho told me the following: helios ~ # ip tun sho sit0: ipv6/ip remote any local any ttl 64 nopmtudisc sixxs: ipv6/ip remote 192.87.102.107 local 62.131.46.165 ttl inherit
And is 62.131.46.165 locally configured on your machine? (check 'ip addr sho' or 'ifconfig'). I guess it is not, as that would very clearly explain why the tunnel is failing, as when the tunnel sends packets outbound it can't select that IP address as a source and thus it fails -> carrier error.
Static tunnel dead after power failure
Shadow Hawkins on Friday, 03 November 2006 12:19:40
Heh. I thought the current configuration was exactly the same as when it still worked. Apparently not. After adding my external IP to eth0 I stopped getting the errors, but I didn't get any packets returned either. Probably my router doesn't allow packets from that source IP, if they reach it at all. After manually configuring my tunnel like this:
helios ~ # cat sixxs
ip tunnel add sixxs mode sit local 10.0.0.153 remote 192.87.102.107
ip link set sixxs up
ip link set mtu 1280 dev sixxs
ip tunnel change sixxs ttl 64
ip -6 addr add 2001:610:600:28d::2/64 dev sixxs
ip -6 ro add default via 2001:610:600:28d::1 dev sixxs
I got this:
helios ~ # ping6 2001:610:600:28d::1
PING 2001:610:600:28d::1(2001:610:600:28d::1) 56 data bytes
64 bytes from 2001:610:600:28d::1: icmp_seq=1 ttl=64 time=11.1 ms
64 bytes from 2001:610:600:28d::1: icmp_seq=2 ttl=64 time=11.1 ms
64 bytes from 2001:610:600:28d::1: icmp_seq=3 ttl=64 time=11.2 ms
64 bytes from 2001:610:600:28d::1: icmp_seq=4 ttl=64 time=10.6 ms
helios ~ # ping6 www.kame.net
PING www.kame.net(orange.kame.net) 56 data bytes
64 bytes from orange.kame.net: icmp_seq=1 ttl=50 time=267 ms
64 bytes from orange.kame.net: icmp_seq=2 ttl=50 time=266 ms
64 bytes from orange.kame.net: icmp_seq=3 ttl=50 time=268 ms
:) Thank you for the help
Static tunnel dead after power failure
I'll put an entry on the AICCU todo list to add a 'local_ipv4' option which overrides the parameter it receives using the TIC protocol. Then AICCU can setup the above type of tunnel automatically.
Static tunnel dead after power failure
Shadow Hawkins on Friday, 03 November 2006 19:04:12
Oh wait, that might also explain why it did work before. It's awfully close but I think I changed my tunnel from heartbeat to static AFTER the last server reboot, so it was never rebuilt. Now it is static and reads the IP from the TIC protocol, and fails because that IP belongs to my router. A local_ipv4 entry would be great indeed. Would you mind sending me an e-mail whenever that feature is built into aiccu? I'll help testing it then.
|
![[ch]](/s/countries/ch.gif)
on Thursday, 02 November 2006 00:16:50
Posting is only allowed when you are