Tried to UCARP my tunnel endpoint
Shadow Hawkins on Wednesday, 01 October 2008 07:54:53
To amuse myself, and because I have two linux boxes on all the time, I decided to try to make my NAT'ed SixXS heartbeat tunnel automatically failover to the other linux box if one were to go down.
Using UCARP and following this example I was able to set up a virtual IPv4 IP that would successfully failover between nodes. I set my router to forward protocol 41 (6in4) to the virtual IP. Then I removed the scripts that start my aiccu and radvd daemons and put the startup commands in scripts that would start and stop them when the local node became the master node. Per the linked example I used arping to notify other ethernet nodes that the virtual IP changed hardware addresses. And the software side of things worked wonderfully.
The network side was spotty. Within my NAT things went mostly as expected, but sometimes I wouldn't be able to access the tunnel. I could see pings go out but not come back. It seemed to work after a while, but not reliably, and the SixXS Tunnel Statistics showed I was losing all of my pings most of the time.
I finally checked my router's security log, and it was blocking tons of outbound protocol 41 (tunnel) packets because of duplicate NAT mappings. So my router vehemently objected to my moving the tunnel endpoint around like that. I tried tweaking a number of things but couldn't make it work. I had to go back to the way it was.
Ultimately I think my problem had nothing to do with IPv6 or tunneling. I think trying to run any type of failover clustering behind my NAT would fail similarly. (Maybe if I spoofed the hardware address and ensured the service used the virtual IP instead of the node IP...but I've already spent too many hours on this.)
UCARP worked quite well and was easy to use, though. I thought that would be the hard part. I had a different version, different kernel and different distribution on each box, so I'm even further impressed with it. One of my boxes has OpenVZ, and I originally tried to put the gateway in a VE but was unable to get aiccu to create a tun/tap tunnel even with the dev nodes created and the proper (I think) capabilities configured for the VE.
Here is some of the code snippets I used if anyone's intereted:
Ubuntu Hardy Heron server edition has a newer ifupdown which allows some slick options in /etc/network/interfaces. This one has a bridge and OpenVZ, but apparently neither of those was the problem:
auto vzbr0
iface vzbr0 inet static
address 192.168.1.124
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports eth1 regex veth*
ucarp-vid 2
ucarp-vip 192.168.1.2
ucarp-password 1234567890123456
ucarp-advskew 50
ucarp advbase 1
ucarp-upscript /etc/ucarp/vip-2-up.sh
ucarp-downscript /etc/ucarp/vip-2-down.sh
My other box was Debian Etch, and the snazzy options above didn't work with its older version of ifupdown, but this following snippet worked fine as the other node for the above snippet:
auto eth1
iface eth1 inet static
address 192.168.1.123
netmask 255.255.255.0
gateway 192.168.1.1
up /usr/sbin/ucarp \
--interface=$IFACE \
--srcip=192.168.1.123 \
--daemonize \
--shutdown \
--vhid=2 \
--pass=1234567890123456 \
--addr=192.168.1.2 \
--upscript=/etc/ucarp/vip-2-up.sh \
--downscript=/etc/ucarp/vip-2-down.sh \
--advskew=50 \
|| true
# really should use a start-stop-daemon or something instead of pkill
down /usr/bin/pkill ucarp
The ucarp up/down scripts I made are the same on both boxes. I commented them heavily for my own benefit.
/etc/ucarp/vip-2-up.sh:
#! /bin/sh
exec 2> /dev/null
# Add virtual IP to interface
/sbin/ip -4 addr add 192.168.1.2/24 dev "$1"
/sbin/ip -6 addr add 2001:4978:192::1/64 dev "$1"
# I was having trouble with this default route overriding the tunnel default route
/sbin/ip -6 route del ::/0 via 2001:4978:192::1 dev "$1"
# Add null route for my /48 subnet so unused /64 prefixes don't route back to internet
/sbin/ip -6 route add 2001:4978:192::/48 dev lo
# Start aiccu tunnel and radvd router daemons
/usr/sbin/invoke-rc.d aiccu start
/usr/sbin/invoke-rc.d radvd start
# Send some ARP update requests so subnet knows where the virtual IP is now
arping -c 5 -U -I "$1" 192.168.1.2
/etc/ucarp/vip-2-down.sh:
#! /bin/sh
exec 2> /dev/null
# Stop aiccu tunnel and radvd router daemons
/usr/sbin/invoke-rc.d radvd stop
/usr/sbin/invoke-rc.d aiccu stop
# Remove null route for my /48 subnet
/sbin/ip -6 route del 2001:4978:192::/48 dev lo
# Restore the ...:1 default route since my static address doesn't seem to pick up the advertised route
/sbin/ip -6 route add ::/0 via 2001:4978:192::1 dev "$1"
# Remove virtual IP from interface
/sbin/ip -6 addr del 2001:4978:192::1/64 dev "$1"
/sbin/ip -4 addr del 192.168.1.2/24 dev "$1"
Note: My script manually assigns the virtual IP. The ucarp docs say that if I use the ucarp-upscript and ucarp-downscript entries I have to assign the IP myself, otherwise ifupdown would handle it. It worked out, anwyay, since my Etch box needed the manual scripts and I had a few extra steps to take, anyway.
I am moving the prefix::1 address back and forth, too, just for those two nodes. I found I had to manually assign them a gateway. At least I think I did. Some tweaking of sysctl might have eliminated the need for that.
Tried to UCARP my tunnel endpoint
Shadow Hawkins on Wednesday, 01 October 2008 08:23:58
I just had some additional insight. I think the main problem may be that the 6in4 packets are being sent from the node IP instead of the virtual IP.
Does anyone know offhand how I can force AICCU to use a certain IPv4 IP?
I'm guessing I may have to configure the tunnel manually and use AICCU just for the heartbeat.
And no, I don't really need that kind of reliability. This is just my idea of fun.
Tried to UCARP my tunnel endpoint
Shadow Hawkins on Wednesday, 01 October 2008 09:15:12
rewrite the ipv4 address bind directive in the aiccu.conf from the scripts you use to hook the rest in with?. using some sed magic or similar.
Posting is only allowed when you are logged in. |