networking won't come back up until reboot after ISP outage

[umage]

Hello. I have had 3 network outages this week, and everytime it
happened, I found that my freebsd8rc1 machine/router was unable to
communicate over the WAN interface even after the outages ended. A
reboot helps, but that's not an acceptable solution.

The symptoms are very weird: even though dhclient successfully receives
a dhcp lease and sets up the interface, I am unable to ping the gateway.
The ifconfig utility says the interface is up and configured correctly;
netstat -arn says routes are set up as they should be; and tcpdump
reports random network traffic arriving on the interface (so the network
itself is up and running). Taking the interface down and back up doesn't
help, reloading ipfw rules doesn't help.

I have some more things to test - whether tcpdump reports any attempts
at outgoing traffic, and whether disconnecting/reconnecting the network
cable will do anything. And maybe reverting ipfw rules to what I've been
using before. Other than this, I have no idea what else to do...
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[Jason Helfman-2]

Have you tried restarting routing?

/etc/rc.d/routing restart

I have found the same symptoms with other outages and not performing the
above.

I have done "/etc/rc.d/netif restart" and "/etc/rc.d/routing restart".

Wtih using these commands, I have found this to be successful in restoring
network interfaces communication, without having to reboot.

Hope this helps.


On Sat, Nov 07, 2009 at 03:05:40PM +0100, umage thus spake:

>Hello. I have had 3 network outages this week, and everytime it
>happened, I found that my freebsd8rc1 machine/router was unable to
>communicate over the WAN interface even after the outages ended. A
>reboot helps, but that's not an acceptable solution.
>
>The symptoms are very weird: even though dhclient successfully
>receives a dhcp lease and sets up the interface, I am unable to ping
>the gateway. The ifconfig utility says the interface is up and
>configured correctly; netstat -arn says routes are set up as they
>should be; and tcpdump reports random network traffic arriving on the
>interface (so the network itself is up and running). Taking the
>interface down and back up doesn't help, reloading ipfw rules doesn't
>help.
>
>I have some more things to test - whether tcpdump reports any
>attempts at outgoing traffic, and whether disconnecting/reconnecting
>the network cable will do anything. And maybe reverting ipfw rules to
>what I've been using before. Other than this, I have no idea what
>else to do...
>_______________________________________________
>freebsd-questions@... mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."
>

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[umage]

On 7. 11. 2009 19:07, Jason wrote:

> Have you tried restarting routing?
>
> /etc/rc.d/routing restart
>
> I have found the same symptoms with other outages and not performing the
> above.
>
> I have done "/etc/rc.d/netif restart" and "/etc/rc.d/routing restart".
>
> Wtih using these commands, I have found this to be successful in
> restoring
> network interfaces communication, without having to reboot.
>
> Hope this helps.
>
>
> On Sat, Nov 07, 2009 at 03:05:40PM +0100, umage thus spake:
>> Hello. I have had 3 network outages this week, and everytime it
>> happened, I found that my freebsd8rc1 machine/router was unable to
>> communicate over the WAN interface even after the outages ended. A
>> reboot helps, but that's not an acceptable solution.
>>
>> The symptoms are very weird: even though dhclient successfully
>> receives a dhcp lease and sets up the interface, I am unable to ping
>> the gateway. The ifconfig utility says the interface is up and
>> configured correctly; netstat -arn says routes are set up as they
>> should be; and tcpdump reports random network traffic arriving on the
>> interface (so the network itself is up and running). Taking the
>> interface down and back up doesn't help, reloading ipfw rules doesn't
>> help.
>>
>> I have some more things to test - whether tcpdump reports any
>> attempts at outgoing traffic, and whether disconnecting/reconnecting
>> the network cable will do anything. And maybe reverting ipfw rules to
>> what I've been using before. Other than this, I have no idea what
>> else to do...
>> _______________________________________________
>> freebsd-questions@... mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to
>> "freebsd-questions-unsubscribe@..."
>>

Thank you for the hint, I will try it when this happens again.
Note: the output of netstat -r was identical to what it is currently...
FYI: I've been using freebsd 6.2 -> 7.2 until now, and I never had to
intervene - the system resumed networking as usual. It might have
something to do with migrating to 8rc1 (most likely not), or that I'm
now using DHCP and there's a glitch somewhere (maybe).
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[Michael Powell-8]

umage wrote:

> On 7. 11. 2009 19:07, Jason wrote:
>> Have you tried restarting routing?
>>
>> /etc/rc.d/routing restart
>>
>> I have found the same symptoms with other outages and not performing the
>> above.
>>
>> I have done "/etc/rc.d/netif restart" and "/etc/rc.d/routing restart".
>>

[snip]
>>>
> Thank you for the hint, I will try it when this happens again.
> Note: the output of netstat -r was identical to what it is currently...
> FYI: I've been using freebsd 6.2 -> 7.2 until now, and I never had to
> intervene - the system resumed networking as usual. It might have
> something to do with migrating to 8rc1 (most likely not), or that I'm
> now using DHCP and there's a glitch somewhere (maybe).

My configuration is most likely different from yours in that my DSL modem-
router is configured for split-bridge. This allows the DSL modem to handle
the PPPoE connection and login but passes the WAN IP to my FreeBSD gateway
box via DHCP. So the NIC on my gateway is getting it's lease from the DSL
modem instead of directly from Verizon. Your DHCP lease is probably coming
directly from the ISP I would presume.

When the connection goes down and comes back up it will take 5 minutes
before my FreeBSD gateway box checks the lease and decides if a renewal is
in order. This is automatic. If I am sitting in front of my computer and I
want to speed this up I issue /etc/rc.d/netif restart on the gateway and it
will come up and be happy in about 10 seconds, rather than waiting out the 5
minute time out.

-Mike



_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[umage]

 > When the connection goes down and comes back up it will take 5 minutes
 > before my FreeBSD gateway box checks the lease and decides if a
renewal is
 > in order. This is automatic. If I am sitting in front of my computer
and I
 > want to speed this up I issue /etc/rc.d/netif restart on the gateway
and it
 > will come up and be happy in about 10 seconds, rather than waiting
out the 5
 > minute time out.

In my case the router does get the renewed ip, as I described earlier.
However, even after waiting 8+ hours, the system will not recover from
the outage properly (reason unknown). That's what this thread is all about.
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[umage]

On 9. 11. 2009 1:27, umage wrote:

> > When the connection goes down and comes back up it will take 5 minutes
> > before my FreeBSD gateway box checks the lease and decides if a
> renewal is
> > in order. This is automatic. If I am sitting in front of my computer
> and I
> > want to speed this up I issue /etc/rc.d/netif restart on the gateway
> and it
> > will come up and be happy in about 10 seconds, rather than waiting
> out the 5
> > minute time out.
>
> In my case the router does get the renewed ip, as I described earlier.
> However, even after waiting 8+ hours, the system will not recover from
> the outage properly (reason unknown). That's what this thread is all
> about.

When I started the system today, I found that again it had no
connectivity. I did some checks and then found that 'natd' was not
running. But this is not happening that frequently, and seems to only
have started after the last system update. Could be some sort of race
condition. Is there a logfile that natd writes to, so that I may
investigate the reason why it is exiting?
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[Michael Powell-8]

umage wrote:

[snip]

>> In my case the router does get the renewed ip, as I described earlier.
>> However, even after waiting 8+ hours, the system will not recover from
>> the outage properly (reason unknown). That's what this thread is all
>> about.
> When I started the system today, I found that again it had no
> connectivity. I did some checks and then found that 'natd' was not
> running. But this is not happening that frequently, and seems to only
> have started after the last system update. Could be some sort of race
> condition. Is there a logfile that natd writes to, so that I may
> investigate the reason why it is exiting?

My first gut instinct about your problem was to blame dhclient first. But no
NATD would definitely be a problem. I am assuming we are talking about IPFW
and NATD here, and it has been many years since I've used it. I migrated to
IPFILTER and then on to PF quite some time ago.

Most logging related to IPFW is already present, but IIRC to log NATD you
need to turn it on, and possibly configure it in syslog.conf should you
desire the output somewhere other than /var/log/alias.log.

Keep in mind there are two ways to pass options. You can use something like
natd_flags="-l" in /etc/rc.conf. Man natd will provide a list. The second
method is to place the options in a file such as natd.conf and pull them in
like natd_flags="-f /etc/natd.conf".

I looked in my notes and here is a snippet from an old /etc/rc.conf:

natd_enable="YES"
natd_interface="ppp0"
natd_flags="-f /etc/natd.conf"

My /etc/natd.conf:

interface ppp0
use_sockets yes
same_ports yes
dynamic yes

You could add a 'log yes' line here; it does the same as the -l described
above. Note that you might need the 'dynamic yes' switch for an interface
that changes. In my case I was using it for a ppp dial-up connection, change
interface as needed.

Sounds like you are narrowing down the culprit(s). Also note that it could
possibly be a timing issue related to the order things start up. If the NATD
is attempting to start before the interface has come up it will die.
Shouldn't happen, but...   YMMV

-Mike
 

_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."

[umage]

 >Sounds like you are narrowing down the culprit(s). Also note that it could
 >possibly be a timing issue related to the order things start up. If
the NATD
 >is attempting to start before the interface has come up it will die.
/etc/rc.d/natd has no REQUIRE section, so it is indeed possible for it
to activate right at the very start. It also has a 'precmd' that checks
for dhcp interfaces and sets up the -dynamic flag. Might be related. I
have no way of knowing though, unless I add some debug messages and
figure out to log them. I added "REQUIRE: NETWORKING" and now I'm
waiting to see if the issue appears again. It's a tedious procedure...
and the fact that sometimes, this causes named to not work isn't helping
either.

 >You can use something like natd_flags="-l" in /etc/rc.conf.
I have tried this, and the only thing it logs are nat rules that get set
up at startup time. I could not find anything that would turn on actual
status messages for this process...

PS: Is there a way to turn on logging on the entire rc startup
procedure? There are a lot of messages that get printed onto the
physical screen, but none of them actually end up in /var/log/messages.
None of the three 'rc_debug', 'rc_info' or 'rc_startmsgs' do it.
_______________________________________________
freebsd-questions@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@..."