CARP and NIC-teaming on ESXi

[Sebastiaan van Erk]

Hi all,

I have a problem using CARP for IP failover on a FreeBSD 7.1-amd64
virtual machine on ESXi 3.5. The problem can be reproduced using a
single FreeBSD virtual mmachine.

When I put the FreeBSD VM on a virtual switch on ESXi with one physical
network card associated to it, CARP works fine, the carp interface
becomes MASTER.

However, as soon as I associate 2 physical network cards with the
virtual switch so that I have NIC failover, the FreeBSD machine
continuously becomes the BACKUP even though there are no other machines
with the same vhid. In the logs I get this message repeated ad infinitum:

carp0: MASTER -> BACKUP (more frequent advertisement received)
arp_rtrequest: bad gateway 192.168.1.1 (!AF_LINK)
carp0: MASTER -> BACKUP (more frequent advertisement received)
arp_rtrequest: bad gateway 192.168.1.1 (!AF_LINK)

When I ping the shared IP addres 192.168.1.1 it replies VERY
infrequently (a few times ever 50000 pings).

When I do a tcpdump to see the carp advertisements then when it is
working (only one network card), I see something like this (with carp
advertisements approximately every second):

10:08:11.241213 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4
(0x0800), length 70: (tos 0x10, ttl 255, id 28744, offset 0, flags [DF],
proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2,
Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36,
addrs(7):
156.231.51.105,143.57.8.21,37.96.2.205,99.70.81.166,146.170.184.206,103.60.18.123,240.32.224.52
10:08:12.651052 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4
(0x0800), length 70: (tos 0x10, ttl 255, id 28757, offset 0, flags [DF],
proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2,
Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36,
addrs(7):
156.231.51.105,143.57.8.22,34.65.134.219,132.160.185.229,103.202.156.249,174.25.227.190,231.95.30.57
10:08:14.061257 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4
(0x0800), length 70: (tos 0x10, ttl 255, id 28769, offset 0, flags [DF],
proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2,
Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36,
addrs(7):
156.231.51.105,143.57.8.23,91.0.105.24,240.239.51.204,21.9.216.6,232.26.58.127,73.8.235.226

However, when I do the NIC teaming on ESXi, I constantly see 2 packets
arrive at the same time and I get the 'more frequent advertisement
received' message in the logs:

10:10:47.527982 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4
(0x0800), length 70: (tos 0x10, ttl 255, id 30136, offset 0, flags [DF],
proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2,
Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36,
addrs(7):
156.231.51.105,143.57.8.129,248.175.196.213,178.47.150.234,200.203.153.156,219.129.15.78,19.136.6.207
10:10:47.529163 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4
(0x0800), length 70: (tos 0x10, ttl 255, id 30136, offset 0, flags [DF],
proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2,
Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36,
addrs(7):
156.231.51.105,143.57.8.129,248.175.196.213,178.47.150.234,200.203.153.156,219.129.15.78,19.136.6.207

Does anybody know what could be going on, and whether it's possible to
get this working?

Thanks in advance,
Sebastiaan

[Brian A. Seklecki]

On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote:
> I see something like this (with carp
> advertisements approximately every second):
>
> 10:08:11.241213 00:00:5e:00:01:4a

There are a handful of settings buried deep in ESXi related to virtual
switch promiscuous mode, multicast, and MAC spoofing (transmitting from
the VRRP/HSRP/CARP address).  You have to configure them in about 3-4
places.  ~BAS




_______________________________________________
freebsd-cluster@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-cluster
To unsubscribe, send any mail to "freebsd-cluster-unsubscribe@..."

[Sebastiaan van Erk]

Hi,

Thanks for the reply.

I had already found the promiscuous mode of the virtual switch that the
boxes are on, otherwise CARP does not work at all. CARP itself works (I
have it set up with 2 real ESXi machines each with a virtual FreeBSD
machine which is doing CARP, so that if either physical machine fails,
the CARP kicks in).

The only problem is that together with NIC teaming on ESXi it breaks;
even when I just set the second NIC to standby mode. Do you have any
idea where I can find the settings and which ones to tune?

Regards,
Sebastiaan

Brian A. Seklecki wrote:

> On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote:
>> I see something like this (with carp
>> advertisements approximately every second):
>>
>> 10:08:11.241213 00:00:5e:00:01:4a
>
> There are a handful of settings buried deep in ESXi related to virtual
> switch promiscuous mode, multicast, and MAC spoofing (transmitting from
> the VRRP/HSRP/CARP address).  You have to configure them in about 3-4
> places.  ~BAS
>
>
>
>