Problem with DHCP

[uprinz]

Hi!

I am using some Nut/OS based systems that need to work in different
networks with DHCP. Some of them are moved several times a week into
totally different networks.
Every time the DHCP reassign of the saved address fails, as the network
is a different one, the device hangs inside the dhcp routines.

A quick hack was to erase the saved address from the network settings.
Therefore I use a button that is checked on power-up and if it is
pressed the saved dhcp settings are erased. But that should not be the
solution. AFAIK the process fails at the point, where the DHCP get a
NACK from the dhcp server after trying to get the saved address reassigned.

Did someone already change that procedure and can provide some modified
code?

Best regards, Ulrich
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Michael Jones-11]

Ulrich Prinz Wrote:

>Hi!
>I am using some Nut/OS based systems that need to work in different
>networks with DHCP. Some of them are moved several times a week into
>totally different networks.
>Every time the DHCP reassign of the saved address fails, as the network
>is a different one, the device hangs inside the dhcp routines.
>A quick hack was to erase the saved address from the network settings.
>Therefore I use a button that is checked on power-up and if it is
>pressed the saved dhcp settings are erased. But that should not be the
>solution. AFAIK the process fails at the point, where the DHCP get a
>NACK from the dhcp server after trying to get the saved address reassigned.
>Did someone already change that procedure and can provide some modified
>code?
>
>Best regards, Ulrich

Hi,
I'm currently investigating some network related issues and the behavior you
described above it one of them. The current schedule suggests that this will
be solved within the next two weeks...

Cu,
Michael

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[uprinz]

Hi!

Thanks, for beeing active in that area. I discovered, that the DHCP is
not the Problem in the latest EIR software. It simply dies cause the
system dies elswhere.

But there are different problems in the DHCP system. I try to resume
from my mind...

I had a system configured for a class B network by dhcp. It got an
address like 10.201.49.129/255.255.0.0
Then I plugged it into the voip network and did not get a new address.
It tried for hours before I got back to the system and checked it's ip.

To solve the problem quickly I simply used a second setup that never got
an ip as it was used for another project. It was configured by dhcp
without a problem.

Then we needed several systems in a network and all of them that never
where set up by dhcp where working, from the others only the ones, that
already had an address from that network. None of them could be
connected to another network again. So I implemented a quick hack as
these systems ha buttons and a display. So I added a menu point for
reflashing the netconf sector to 0xFF. After rebooting it starts and
uses dhcp without a problem.

I hope, that these information may help you. If you have any things to
test, just pass the code over and I try it.

Best regards, Ulrich

Michael Jones schrieb:

> Ulrich Prinz Wrote:
>
>> Hi!
>> I am using some Nut/OS based systems that need to work in different
>> networks with DHCP. Some of them are moved several times a week into
>> totally different networks.
>> Every time the DHCP reassign of the saved address fails, as the network
>> is a different one, the device hangs inside the dhcp routines.
>> A quick hack was to erase the saved address from the network settings.
>> Therefore I use a button that is checked on power-up and if it is
>> pressed the saved dhcp settings are erased. But that should not be the
>> solution. AFAIK the process fails at the point, where the DHCP get a
>> NACK from the dhcp server after trying to get the saved address reassigned.
>> Did someone already change that procedure and can provide some modified
>> code?
>>
>> Best regards, Ulrich
>
> Hi,
> I'm currently investigating some network related issues and the behavior you
> described above it one of them. The current schedule suggests that this will
> be solved within the next two weeks...
>
> Cu,
> Michael
>
> _______________________________________________
> http://lists.egnite.de/mailman/listinfo/en-nut-discussion

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Michael Jones-11]

Hi,

I've been spending the last days pulling out my already sparse hair over
this issue.

The environment is a ETHERNUT2 compatible board which receives data from a
server. This server sends bursts of information in packets between 8 to 200
bytes. These bursts might contain between 1 to 1000+ packets. Using
Wireshark I can see 800+ sent within one second. The Nut/OS has about 27000
Bytes free memory when idle (which equals about 300 minimal size (19 + 67)
netbuf allocs).

Speed is not so much a problem, it's totally OK that the board is not able
to handle the load at the speed that it is sent.

Now to the observations:

1)  The SMCS LAN91C111 will start losing packets once the OS loses the race
to pick the packets up in time.
    (The chip only has 4 packets buffer, so this is by design and too be
expected).

2)  The lost packets cause the "later" packets to be queued into so_rx_nbq
for "future" use.
    (TCPSM.c, NutTcpStateEstablished, ~1190)

3)  The Memory fills up with these future use packets.
    (AFAICS the windows size is not adjusted and there is no limit on these
packets)

4a) The system finally recovers the missing packet and recovers from above
situation.

4b) The future packets use up all free memory till no more packets can be
received.
    (lanc111.c, NicGetPacket, ~1065)

5)  TCP State Machine finally gets fed up and timeouts the Socket and
pending e.g. NutTcpReceive.
    (haven't found the exact point where it does this yet - but it does)

6)  the App calls NutTcpCloseSocket.

7)  NutTcpCloseSocket calls NutTcpStateCloseEvent.

8)  NutTcpStateCloseEvent sets state to TCPS_FIN_WAIT_1.

9)  The state TCPS_FIN_WAIT_1 causes NutTcpOutput(sock, 0, 0) with SO_FIN
and SO_ACK.

9a) NutTcpOutput sends packet to host,
    host answers causing NutTcpStateProcess to call NutTcpStateFinWait1,
    which calls NutTcpDestroySocket,
    which frees all used memory and system recovers...

9b) NutTcpOutput FAILS to send packet to host because there is no memory
left,
    Host never answers,
    Memory never gets freed,
    System dies horrible death...

Now there are two aspects in this:

a) How to stop the packets flooding the memory.
b) How to assure that the Socket gets destroyed even in this grim situation.

Comments and thought are welcome.

Cu,
Michael



_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Michael Jones-11]

Hi,

This, proof of concept change seems to work:

Tcpsm.c, NutTcpStateEstablished, 1190

    if (htonl(th->th_seq) > sock->so_rx_nxt) {

        fputc('!', stdout);  //@DEBUG

        NETBUF *nbq;
        NETBUF **nbqp;
        TCPHDR *thq;
        uint32_t th_seq;
        uint32_t thq_seq;

        uint16_t Count = 0;  // <----
        uint32_t Size = 0;   // <----

        if (nb->nb_ap.sz) {
            nbq = sock->so_rx_nbq;
            nbqp = &sock->so_rx_nbq;
            while (nbq) {

                Count++; // <----
                Size += nbq->nb_ap.sz; // <----

                thq = (TCPHDR *) (nbq->nb_tp.vp);
                th_seq = htonl(th->th_seq);
                thq_seq = htonl(thq->th_seq);
                if (th_seq < thq_seq && nb->nb_ap.sz <= thq->th_win) {
                    *nbqp = nb;
                    nb->nb_next = nbq;
                    break;
                }
                if (th_seq == thq_seq || nb->nb_ap.sz > thq->th_win) {
                    NutNetBufFree(nb);
                    sock->so_tx_flags |= SO_ACK | SO_FORCE;
                    NutTcpOutput(sock, 0, 0);
                    return;
                }
                nbqp = &nbq->nb_next;
                nbq = nbq->nb_next;
            }
            if (nbq == 0) {
                   
                printf("[%d, %ld]", Count, Size);

                /*
                 * Limit the number of packets in the look-a-head list.
                 *
                 * Using a constant number of packets is only a hack and
won't be
                 * sufficient in different environments.
                 *
                 * Alternatives:
                 *
                 * a) the size of stored packets vs free memory
                 * b) the size of stored packets vs window size
                 * c) ???
                 *
                 * Also, Size does not reflect the effectively allocated
amount of memory,
                 * as all packets smaller 67 are actually stored as 67
bytes.
                 *
                 */

                if(Count > 20) {                              
                    NutNetBufFree(nb);
                    sock->so_tx_flags |= SO_ACK | SO_FORCE;
                    NutTcpOutput(sock, 0, 0);
                    return;
                }

                *nbqp = nb;
                nb->nb_next = 0;
            }
        } else
            NutNetBufFree(nb);

        sock->so_tx_flags |= SO_ACK | SO_FORCE;
        NutTcpOutput(sock, 0, 0);
        return;
    }




-----Original Message-----
From: en-nut-discussion-bounces@...
[mailto:en-nut-discussion-bounces@...] On Behalf Of Michael Jones
Sent: Donnerstag, 15. Oktober 2009 23:16
To: 'Ethernut User Chat (English)'
Subject: [En-Nut-Discussion] TCP vs. Memory (RFC)

Hi,

I've been spending the last days pulling out my already sparse hair over
this issue.

The environment is a ETHERNUT2 compatible board which receives data from a
server. This server sends bursts of information in packets between 8 to 200
bytes. These bursts might contain between 1 to 1000+ packets. Using
Wireshark I can see 800+ sent within one second. The Nut/OS has about 27000
Bytes free memory when idle (which equals about 300 minimal size (19 + 67)
netbuf allocs).

Speed is not so much a problem, it's totally OK that the board is not able
to handle the load at the speed that it is sent.

Now to the observations:

1)  The SMCS LAN91C111 will start losing packets once the OS loses the race
to pick the packets up in time.
    (The chip only has 4 packets buffer, so this is by design and too be
expected).

2)  The lost packets cause the "later" packets to be queued into so_rx_nbq
for "future" use.
    (TCPSM.c, NutTcpStateEstablished, ~1190)

3)  The Memory fills up with these future use packets.
    (AFAICS the windows size is not adjusted and there is no limit on these
packets)

4a) The system finally recovers the missing packet and recovers from above
situation.

4b) The future packets use up all free memory till no more packets can be
received.
    (lanc111.c, NicGetPacket, ~1065)

5)  TCP State Machine finally gets fed up and timeouts the Socket and
pending e.g. NutTcpReceive.
    (haven't found the exact point where it does this yet - but it does)

6)  the App calls NutTcpCloseSocket.

7)  NutTcpCloseSocket calls NutTcpStateCloseEvent.

8)  NutTcpStateCloseEvent sets state to TCPS_FIN_WAIT_1.

9)  The state TCPS_FIN_WAIT_1 causes NutTcpOutput(sock, 0, 0) with SO_FIN
and SO_ACK.

9a) NutTcpOutput sends packet to host,
    host answers causing NutTcpStateProcess to call NutTcpStateFinWait1,
    which calls NutTcpDestroySocket,    
    which frees all used memory and system recovers...

9b) NutTcpOutput FAILS to send packet to host because there is no memory
left,
    Host never answers,
    Memory never gets freed,
    System dies horrible death...

Now there are two aspects in this:

a) How to stop the packets flooding the memory.
b) How to assure that the Socket gets destroyed even in this grim situation.

Comments and thought are welcome.

Cu,
Michael



_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

Michael Jones wrote:
> This, proof of concept change seems to work:

before considering to apply your change I'd like to test it here.
However, I failed to reproduce the problem.

As you wrote, the remote peer will fill up the our local memory with
packets. But how can this be? Ethernut announces its TCP window size and
the remote will never send more than this amount without further ACK.

Nut/OS 4.9 announces a TCP window size of 3216 bytes. The remote (PC
running Windows 2003) runs something like

for(n = 0; ; n++) {
 sprintf(buffer, "%lu\n", n);
 send(sock, buffer, (int)strlen(buffer), 0);
}

and Nut/OS uses

for (;;) {
 NutTcpReceive(sock, buffer, sizeof(buffer));
}

Both loops are simplified. Actually I'm checking results and doing some
debug output.

Wireshark shows

PC: 2 bytes
Nut: ACK (win=3214)
PC: 132 bytes
PC: 6 bytes
Nut: ACK (win=3076)

...and so on. After a short time, as expected

Nut: ACK (win=570)
PC: 228 bytes
Nut: ACK (win=0)
Nut: ACK (win=598)
PC: 536 bytes
Nut: ACK (win=0)

Nut's memory never drops by more than 5k.

May be I have to find a way to force the PC to send smaller packets. But
how? As soon as Nut closes the window, the remote will collect all data
into one packet. When Nut opens the window again, the remote will send
the maximum in one packet. This maximum is limited by the newly
announced window size and the MSS.

Any idea how to reproduce the problem?

Harald

P.S.: I do remember the discussion about the zero window size problem
eating up all memory, which appeared here earlier and I'll look into
this as well.


_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

OK, OK! I got it. :-)

I added NutSleep(5000) into Nut's loop.

On the remote I sent the first buffer byte only and added a Sleep(1) to
give TCP more time to process the small packet.

After a short time Nut/OS heap space went down to 82 bytes. Wow!

Interestingly, after a few more packets the situation becomes more
stable and heap allocation didn't increase more than 10k. I agree, that
this is still a lot.

Harald


Harald Kipp wrote:

> Michael Jones wrote:
>> This, proof of concept change seems to work:
>
> before considering to apply your change I'd like to test it here.
> However, I failed to reproduce the problem.
>
> As you wrote, the remote peer will fill up the our local memory with
> packets. But how can this be? Ethernut announces its TCP window size and
> the remote will never send more than this amount without further ACK.
>
> Nut/OS 4.9 announces a TCP window size of 3216 bytes. The remote (PC
> running Windows 2003) runs something like
>
> for(n = 0; ; n++) {
>  sprintf(buffer, "%lu\n", n);
>  send(sock, buffer, (int)strlen(buffer), 0);
> }
>
> and Nut/OS uses
>
> for (;;) {
>  NutTcpReceive(sock, buffer, sizeof(buffer));
> }
>
> Both loops are simplified. Actually I'm checking results and doing some
> debug output.
>
> Wireshark shows
>
> PC: 2 bytes
> Nut: ACK (win=3214)
> PC: 132 bytes
> PC: 6 bytes
> Nut: ACK (win=3076)
>
> ...and so on. After a short time, as expected
>
> Nut: ACK (win=570)
> PC: 228 bytes
> Nut: ACK (win=0)
> Nut: ACK (win=598)
> PC: 536 bytes
> Nut: ACK (win=0)
>
> Nut's memory never drops by more than 5k.
>
> May be I have to find a way to force the PC to send smaller packets. But
> how? As soon as Nut closes the window, the remote will collect all data
> into one packet. When Nut opens the window again, the remote will send
> the maximum in one packet. This maximum is limited by the newly
> announced window size and the MSS.
>
> Any idea how to reproduce the problem?
>
> Harald
>
> P.S.: I do remember the discussion about the zero window size problem
> eating up all memory, which appeared here earlier and I'll look into
> this as well.
>
>
> _______________________________________________
> http://lists.egnite.de/mailman/listinfo/en-nut-discussion

--
egnite GmbH
Erinstr. 9
44575 Castrop-Rauxel
Germany

Fon [phone]
Fax [phone]

http://www.egnite.de/
http://www.ethernut.de/

Handelsregister: Amtsgericht Dortmund HRB 19783
USt-IdNr.: DE 189520047
Geschäftsführung: Harald Kipp

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

Harald Kipp wrote:

> After a short time Nut/OS heap space went down to 82 bytes. Wow!

I even got it down to 60 bytes. Without crashing, btw.

Actually Michael's suggestion was only part of the story. Limiting the
buffer for segments following a lost segment was the first task. I made
it global, so it will limit the buffer for all connections.

The second part was the buffer for packets in sequence. During a phone
call with Michael we discussed several strategies. I strongly rejected
his idea to use the filler of the Ethernet frame and suggested a real
window buffer instead. After hanging up I thought about Michael's idea
again and it finally led me to the current solution. If there are at
least 8 packets in the queue with at least one small packet it front,
they will be collected into 1 packet. The great thing here is, that only
a few changes were required.

I committed the code to SVN trunk.

Harald
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Michael Jones-11]

Harald Kipp wrote:

> I committed the code to SVN trunk.

I've check Harald's changes and find them very neat and my tests here show
only positive results!

In my opinion this change should resolve a lot of strange und random
crashes, disconnects and other connection problem. (I know this has been
haunting *me* for quite some while).

Thanks!

Cu,
Michael

-----Original Message-----
From: en-nut-discussion-bounces@...
[mailto:en-nut-discussion-bounces@...] On Behalf Of Harald Kipp
Sent: Dienstag, 20. Oktober 2009 20:51
To: Ethernut User Chat (English)
Subject: Re: [En-Nut-Discussion] TCP vs. Memory - a solution

Harald Kipp wrote:

> After a short time Nut/OS heap space went down to 82 bytes. Wow!

I even got it down to 60 bytes. Without crashing, btw.

Actually Michael's suggestion was only part of the story. Limiting the
buffer for segments following a lost segment was the first task. I made
it global, so it will limit the buffer for all connections.

The second part was the buffer for packets in sequence. During a phone
call with Michael we discussed several strategies. I strongly rejected
his idea to use the filler of the Ethernet frame and suggested a real
window buffer instead. After hanging up I thought about Michael's idea
again and it finally led me to the current solution. If there are at
least 8 packets in the queue with at least one small packet it front,
they will be collected into 1 packet. The great thing here is, that only
a few changes were required.

I committed the code to SVN trunk.

Harald
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion