carp + no ip address on iface (only master can receive acks)

[Alex Strawman]

one small problem with carp and ip-less interfaces..

scenario: you have no ip address bound to each of the real interfaces,
and carp is sharing the one address for you (isp only gives you 1
address).

only the master can craft packets out (assuming this shared carp'ed
address is the external).

ok, now this makes sense, how is the next hop meant to send packets
back? it sends them to the mac address the carp0 is broadcasting,
which the master happily accepts, only to see its not in its state
table, and drops it.

the backup system doesn't get it's acks back..

is there currently a way around this?

i bashed a quick thing to route via the other system (via pfsync
interface), and if the host is down or this box (the backup) becomes a
master, then remove the route and resort back to the default (via the
carp0 interface, which the next hop will now reply too, or should i
say, the carp0 will now accept to/from)

buts thats fair hokey


Alex

[Ryan McBride-3]

On Thu, Nov 17, 2005 at 03:02:56PM +1100, Alex Strawman wrote:
> ok, now this makes sense, how is the next hop meant to send packets
> back? it sends them to the mac address the carp0 is broadcasting,
> which the master happily accepts, only to see its not in its state
> table, and drops it.
>
> the backup system doesn't get it's acks back..

Traffic shouldn't even be getting OUT on the backup in this situation.

> is there currently a way around this

Get more IP addresses.

[Alex Strawman]

> Traffic shouldn't even be getting OUT on the backup in this situation.

i agree - there is no correct solution without using an ip addr for
each real interface.

would be nice to for example use an external ntp server to sync with,
but unless it uses another route (rather than ip-less carp'd
interface), it cannot (without dodgy work-arounds).

[Jon Hart-2]

On Thu, Nov 17, 2005 at 10:02:46PM +1100, Alex Strawman wrote:
> > Traffic shouldn't even be getting OUT on the backup in this situation.
>
> i agree - there is no correct solution without using an ip addr for
> each real interface.

> would be nice to for example use an external ntp server to sync with,
> but unless it uses another route (rather than ip-less carp'd
> interface), it cannot (without dodgy work-arounds).

Assuming you are also using pfsync, that means you've got another
interface that directly connects both firewalls.  Assign normal,
non-multicast addresses to those physical pfsync interfaces and ensure
that you can pass traffic between the two.  Configure pf on both boxes
to NAT traffic out over its external carp'd IP address when it is coming
in on $pfsync_if from $pfsync_net.

This allows your carp backup to still have outbound net so things like
NTP, mail and external DNS lookups still work.  Yes, there are ways you
could run these and other various services internally but there may be
reasons you cannot do this.  In the end, having outbound connectivity
for an otherwise unreachable host is a good thing, IMO.  

The catch here is that when the backup carp host is a backup, its
routing table must be aware that its route is no longer out over the
carp'd interface, but rather over your pfsync interface and the
receiving end is going to nat for you.  There may be other ways to
accomplish this (ospf, perhaps), but I went with ifstated.  

In my case, carp5/em5 on each box points upstream and each box has
a single external IP address assigned to it.  The two boxes each have
the same two failover upstream gateways (not under my control -- a.b.c.d
and w.x.y.z).  Yes, this setup is different than yours, but it should
give you enough to help you figure out the routing bit.

The config below is for the backup carp host.  A similar one exists for
the master (all that is different is that the primary/secondary routes
are different and the pfsync IP to route to is different (.3 vs .2)).

Pretty?  Depends who you ask.  The right solution?  Likely not, but it
got me out of a hole that I needed a way out of quickly,  and it may
help you too.

-jon


#####

init-state wan_master
wan_carp_up = "carp5.link.up"
wan_carp_init = "carp5.link.unknown"
wan_iface_up = "em5.link.up"
wan_primary_route_up = '"ping -q -c1 -w1 a.b.c.d 2> /dev/null 1> /dev/null" every 2'
wan_secondary_route_up = '"ping -q -c1 -w1 w.x.y.z 2> /dev/null 1> /dev/null" every 2'



state wan_master {
   init {
      run "echo WAN master at `date` | mail -s 'WAN master change' \
      someone@..."
   }

   # probably just came up.  give things a chance to sync
   if ($wan_carp_init)
      run "sleep 5"

   if (! $wan_carp_up)
      set-state wan_slave
   if ($wan_primary_route_up)
      set-state primary_route
   if (! $wan_primary_route_up) && ($wan_secondary_route_up)
      set-state secondary_route
   if (! $wan_primary_route_up) && (! $wan_secondary_route_up)
      run "echo THIS SHOULD NEVER HAPPEN!"
}

state wan_slave {
   init {
      # simple.  drop the default route and route over $SYNC_IF
      run "route change default 192.168.0.2"
      run "echo WAN slave at `date`  | mail -s 'WAN slave change' \
      someone@..."
   }

   # if our link(s) come up, become the master
   if ($wan_carp_up)
      set-state wan_master
}

state primary_route {
   init {
      run "route change default a.b.c.d"
      run "echo Using primary route at `date`  | mail -s 'Primary route change' someone@..."
   }

   # if our link(s) go down, become the slave
   if (! $wan_carp_up)
      set-state wan_slave

   if (! $wan_primary_route_up)
      set-state secondary_route
}

state secondary_route {
   init {
      run "route change default w.x.y.z"
      run "echo Using secondary route at `date`  | mail -s 'Secondary route change' someone@..."
   }
   # if our link(s) go down, become the slave
   if (! $wan_carp_up)
      set-state wan_slave

   if (! $wan_secondary_route_up)
      set-state primary_route
}