ICMP echo request response failure on ARP cache timeout

[Ethernut]

Hi all,

This long post deals with inner details of Nut/OS. If you are not
interested or not able to follow, you can simply ignore it.

The issue I'm describing here exists at least since the ARP thread had
been removed from Nut/OS (Feb. 2005). It is actually no big deal. By
default an ARP entry times out after 10 minutes. This time is adjustable
in the Configurator. When an ARP cache entry times out, sending an ICMP
(Ping) response fails.

ICMP responses are somewhat special in Nut/OS. Let's look to the normal
sequence when pinging Nut/OS from a PC:

1. Ping application is started on the PC with the IP address of a Nut/OS
node.

2. The PC broadcasts an ARP request to find out the Ethernet MAC address
for the specified IP address

3. The receiver thread in the Nut/OS NIC driver receives the ARP packet
and calls
3.1 NutEtherInput
3.2 NutArpInput
3.3 NutArpCacheUpdate to add the PC's IP to the ARP cache.
3.4 NutArpOutput
3.5 Send routine of the NIC driver

4. PC receives the ARP response and stores the MAC/IP relation in its
ARP cache.

5. PC sends out the ICMP echo request.

5. The receiver thread in the Nut/OS NIC driver receives the ARP packet
and calls
5.1 NutEtherInput
5.2 NutIpInput
5.3 NutIcmpInput
5.4 NutIcmpReflect
5.5 NutIcmpOutput
5.6 NutIpOutput
5.7 NutArpCacheQuery to retrieve the ARP entry cached in step 3.3.
5.8 Send routine of the NIC driver

6. PC receives the response and continues at step 5.

The interesting step is 3.3, where Nut/OS caches the MAC/IP relation of
the PC. Note, that this is done on ARP requests, because it is most
likely, that a remote hosted sending an ARP request intends to talk to
us. By caching requests we do not need to send out an ARP query ourself
when responding to IP requests.

However, entries in the ARP cache must be removed if they reach a
specific age, with Nut/OS after 10 minutesby default, as mentioned
above. In this case step 5.7 fails to get the MAC address from the
cache. Instead Nut/OS sends out an ARP request to re-new the PCs IP/MAC
relation. It will then wait for the response, but nothing happens. Why?

Simply because the receiver thread in the NIC driver itself is blocked
in NutArpCacheQuery. After 500ms (default) NutArpCacheQuery gives up and
returns an error, being unable to send out the ICMP echo. Back in the
NIC receiver loop it now receives the ARP response from the PC, adding
it to the ARP cache. The next ICMP echo request will be answered again
without any problem.

Why is this ICMP specific? Well, sending out responses internally
(without application intervention) is only done by ICMP and TCP. The TCP
state machine runs its own thread. All incoming packets are stored in a
queue. The TCP state machine thread processes this queue and calls the
NIC send routine, if required. The same is true for other protocols,
where the application thread actually calls the send routine. Only in
ICMP the NIC receiver itself calls the send routine. If the send routine
is blocked, no incoming packets are processed.

How to solve this? Not sure. I can think of the following:

A. Re-implementing the ARP thread is for sure not the best idea, because
threads consume a lot of RAM for their stack.

B. In many other implementations all outgoing packets waiting for ARP
responses are queued. They are sent out as soon as the related ARP
response arrives. In certain situations this may consume even more RAM
than the threaded solution. Furthermore, the queue must be checked in
regular time intervals to remove packets, for which no response was
received within a specific time.

To state it once again: Losing a single ICMP packet should not be
considered an error. If, for example, an ARP packet gets lost, the same
will happen in any implementation that follows the specifications.  By
definition, the upper layer is responsible for retries, not ARP. In the
initial implementation Nut/OS did ARP retries, which was miserably wrong.

However, losing a ping reply every 10 minutes doesn't look good.

Harald





_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Nathan Moore-5]

Hey Harald,It's interesting you ran across this now.  I've had issues with
the same architecture of
the IP code.  With Ethernet it's almost unnoticeable that the rx thread is
sending something even
with an ARP request and reply in the middle of the mix, but with PPP it
becomes more noticable
on AVR (rx and tx buffers are small for ahdlc dev and the pipe is much
easier to saturate).
While PPP doesn't use ARP you're still blocking the reception of more than
256 bytes after the
reception of the ping request.
I have another serially accessible network interface which behaves much
worse under this type of
activity (rx thread sending).

I toyed with the idea of introducing tx threads per network device (didn't
this exist a long time ago?)
, but then threads are unlimited in the speed with which they may add to the
tx queue.  To get
around this I thought about adding an event to netbufs which the thread that
generated the packet
could (but wouldn't have to) wait to be signaled by the tx thread.  This
event would be broadcast
posted before a netbuf was freed.  This method would have required a lot of
changes to the way
large portions of the networking code and drivers worked, the programming
interfaces, and also
would introduce issues with the possibility of having a NETBUF* that you
didn't know if it had been
freed or not (btw, the "who frees a netbuf" within nut/net is very
confusing).
I liked this idea for the ability to keep any and all network interfaces
sending at top speed as long as
there was data to send (well, if the arp code were reworked to allow an IP
packets to be tied to
a list of outstanding arp requests and either moved to a ready to send queue
when the request was
filled or terminated if it wasn't so that the eth_tx thread wouldn't have to
wait for arp).
I think that there are other subtle implementation issues with this as well.

Another idea was for general purpose helper threads which could either be
generated on demand and
do the work needed when blocking the current thread is unacceptable.  The
interface to this got tricky,
and I never have done it.  Also I was worried about how to throttle this
while not overly hindering it's
usefulness.

So far the need to ping has not been great enough to put that much effort
into it.

Networking code is hard.

Nathan

On Fri, May 22, 2009 at 11:37 AM, Harald Kipp <harald.kipp@...> wrote:

> Hi all,
>
> This long post deals with inner details of Nut/OS. If you are not
> interested or not able to follow, you can simply ignore it.
>
> The issue I'm describing here exists at least since the ARP thread had
> been removed from Nut/OS (Feb. 2005). It is actually no big deal. By
> default an ARP entry times out after 10 minutes. This time is adjustable
> in the Configurator. When an ARP cache entry times out, sending an ICMP
> (Ping) response fails.
>
> ICMP responses are somewhat special in Nut/OS. Let's look to the normal
> sequence when pinging Nut/OS from a PC:
>
> 1. Ping application is started on the PC with the IP address of a Nut/OS
> node.
>
> 2. The PC broadcasts an ARP request to find out the Ethernet MAC address
> for the specified IP address
>
> 3. The receiver thread in the Nut/OS NIC driver receives the ARP packet
> and calls
> 3.1 NutEtherInput
> 3.2 NutArpInput
> 3.3 NutArpCacheUpdate to add the PC's IP to the ARP cache.
> 3.4 NutArpOutput
> 3.5 Send routine of the NIC driver
>
> 4. PC receives the ARP response and stores the MAC/IP relation in its
> ARP cache.
>
> 5. PC sends out the ICMP echo request.
>
> 5. The receiver thread in the Nut/OS NIC driver receives the ARP packet
> and calls
> 5.1 NutEtherInput
> 5.2 NutIpInput
> 5.3 NutIcmpInput
> 5.4 NutIcmpReflect
> 5.5 NutIcmpOutput
> 5.6 NutIpOutput
> 5.7 NutArpCacheQuery to retrieve the ARP entry cached in step 3.3.
> 5.8 Send routine of the NIC driver
>
> 6. PC receives the response and continues at step 5.
>
> The interesting step is 3.3, where Nut/OS caches the MAC/IP relation of
> the PC. Note, that this is done on ARP requests, because it is most
> likely, that a remote hosted sending an ARP request intends to talk to
> us. By caching requests we do not need to send out an ARP query ourself
> when responding to IP requests.
>
> However, entries in the ARP cache must be removed if they reach a
> specific age, with Nut/OS after 10 minutesby default, as mentioned
> above. In this case step 5.7 fails to get the MAC address from the
> cache. Instead Nut/OS sends out an ARP request to re-new the PCs IP/MAC
> relation. It will then wait for the response, but nothing happens. Why?
>
> Simply because the receiver thread in the NIC driver itself is blocked
> in NutArpCacheQuery. After 500ms (default) NutArpCacheQuery gives up and
> returns an error, being unable to send out the ICMP echo. Back in the
> NIC receiver loop it now receives the ARP response from the PC, adding
> it to the ARP cache. The next ICMP echo request will be answered again
> without any problem.
>
> Why is this ICMP specific? Well, sending out responses internally
> (without application intervention) is only done by ICMP and TCP. The TCP
> state machine runs its own thread. All incoming packets are stored in a
> queue. The TCP state machine thread processes this queue and calls the
> NIC send routine, if required. The same is true for other protocols,
> where the application thread actually calls the send routine. Only in
> ICMP the NIC receiver itself calls the send routine. If the send routine
> is blocked, no incoming packets are processed.
>
> How to solve this? Not sure. I can think of the following:
>
> A. Re-implementing the ARP thread is for sure not the best idea, because
> threads consume a lot of RAM for their stack.
>
> B. In many other implementations all outgoing packets waiting for ARP
> responses are queued. They are sent out as soon as the related ARP
> response arrives. In certain situations this may consume even more RAM
> than the threaded solution. Furthermore, the queue must be checked in
> regular time intervals to remove packets, for which no response was
> received within a specific time.
>
> To state it once again: Losing a single ICMP packet should not be
> considered an error. If, for example, an ARP packet gets lost, the same
> will happen in any implementation that follows the specifications.  By
> definition, the upper layer is responsible for retries, not ARP. In the
> initial implementation Nut/OS did ARP retries, which was miserably wrong.
>
> However, losing a ping reply every 10 minutes doesn't look good.
>
> Harald
>
>
>
>
>
> _______________________________________________
> http://lists.egnite.de/mailman/listinfo/en-nut-discussion
>

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

Hi Nathan,

Sorry for not coming back to this earlier and thanks for your valuable
input.

Nathan Moore wrote:

> I toyed with the idea of introducing tx threads per network device (didn't
> this exist a long time ago?)

I'm almost sure that this never existed. At least I remember that the
ICMP/ARP problem existed in the very beginning.

> , but then threads are unlimited in the speed with which they may add to the
> tx queue.

Right. On the other hand the same is true for the current
implementation, where the receiver may overflow the TCP message queue.
Currently an ugly hack is used to limit TCP traffic: If the available
memory drops below a certain value, then segments are discarded.

The reason for using a dedicated thread for the TCP state machine is,
that TCP needs to initiate transfers based on timing (re-transmission,
zero window polling).

If we do the main TCP state processing in the network receiver thread
and do all transmits in another thread, like you suggested, we are safe
from TCP segment overflows. If more segments arrive than we can process,
then the overflow occurs in the Ethernet interface and will not eat up
our valuable RAM.

We'd still need a TCP state machine thread to handle re-transmissions.
That one could be simpler than the combined timing/handler loop used
right now.

> To get
> around this I thought about adding an event to netbufs which the thread that
> generated the packet
> could (but wouldn't have to) wait to be signaled by the tx thread.

We may use a simpler mechanism to limit the output queue. We can, for
example, track the total size of waiting packets and block the sender
only if the queue is full.

This will not completely solve the ICMP echo problem, but it will now
occur only, if the tx queue is filled up.

Another question is: Which part of the layer should be processed by the
transmit thread?

Doing this in the driver may not be the best idea. It requires to
rewrite several drivers and may make writing new drivers more complex
than it needs to be.

The IP layer is attractive, because only a singe thread is required to
handle all interfaces including ARP. However, ARP and other non-IP
protocols (we use DLC in one application) must be handled special and we
may end up with similar problems than we have now.

So, it looks to me, that the physical layer is the one to handle in the
transmit thread. We'd need one thread per interface type, currently
Ethernet and PPP.

> (btw, the "who frees a netbuf" within nut/net is very
> confusing)

You're not alone. The initial idea was to release the NETBUF as early as
possible to make it available to the system for other requirements.
Thus, in case of a fatal error, it is released by the lower level
routine. In all other cases it is kept, because the top level may re-use
it (TCP re-transmits).

In early days we had a lot of trouble with the RTL8019. One reason was
that even Realtek was not aware, that only half of the chip's memory is
available in 8-bit mode.

Today fatal errors occur almost never. May be it's the right time to
change this weird NETBUF handling now.

Harald


_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Michael Jones-11]

Hi Harald,


You wrote:

...Today fatal errors occur almost never. May be it's the right time to
change this weird NETBUF handling now...


I'm so with you on this...


Also, I want to point out that we use an extra thread for DHCP in our
systems. The current implementation does not handle DHCP in large
environments very well, does not cope with delayed network startup and does
not handle DNS and TimeServer updates via DHCP at all. Maybe we could build
an extendable Network thread that could have various aspects plugged into it
on demand? That would lower the overall "cost" of this thread.



Cu,
Michael

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Nathan Moore-5]

>
>
> Nathan Moore wrote:
>
> > I toyed with the idea of introducing tx threads per network device
> (didn't
> > this exist a long time ago?)
>
> I'm almost sure that this never existed. At least I remember that the
> ICMP/ARP problem existed in the very beginning.

I never actually saw code that had that in it, but I seem to remember some
old documentation
listing tx and rx threads for Ethernet.  I think it was showing off listing
of the threads rather than
anything specifically network related.  It's been months since then now and
I can't remember what
document that was.  Anyway, I just assumed that at the time that document
was created tx threads
were in use.

>
>
> > , but then threads are unlimited in the speed with which they may add to
> the
> > tx queue.
>
> Right. On the other hand the same is true for the current
> implementation, where the receiver may overflow the TCP message queue.
> Currently an ugly hack is used to limit TCP traffic: If the available
> memory drops below a certain value, then segments are discarded.

Every time I try to do much that involves the TCP state machine thread my
head
explodes, so I don't pretend to understand all of it's internals.

>
> The reason for using a dedicated thread for the TCP state machine is,
> that TCP needs to initiate transfers based on timing (re-transmission,
> zero window polling).


Yeah.  Oh, there may be an error in the TCP thead when using a very slow
network
interface.  A thread calling NutTcpSend() will indirectly call NutIpSend()
if the socket
is ready to send.  If the interface is slow enough the TCP thread may try to
retransmit
before the other thread has finished sending.  This probably never happens
on "normal"
interfaces.


> If we do the main TCP state processing in the network receiver thread
> and do all transmits in another thread, like you suggested, we are safe
> from TCP segment overflows. If more segments arrive than we can process,
> then the overflow occurs in the Ethernet interface and will not eat up
> our valuable RAM.


I tend to favor optimizing getting things sent over receiving because it
encourages
more free RAM.

>
> We'd still need a TCP state machine thread to handle re-transmissions.
> That one could be simpler than the combined timing/handler loop used
> right now.


Since you now do protocol registration you could make part of that
registration
a callback to a periodically called function and a pointer to it's data --
the TCP
thread could turn into one of these functions, and a more generic IP
transmission
thread would call it.

>
> > To get
> > around this I thought about adding an event to netbufs which the thread
> that
> > generated the packet
> > could (but wouldn't have to) wait to be signaled by the tx thread.
>
> We may use a simpler mechanism to limit the output queue. We can, for
> example, track the total size of waiting packets and block the sender
> only if the queue is full.

PPP fills up way too easily.  Well, on AVR it does, and that's the
only architecture I
use.

>
> This will not completely solve the ICMP echo problem, but it will now
> occur only, if the tx queue is filled up.


Well, in that case rather than blocking you could drop the echo reply.

>
> Another question is: Which part of the layer should be processed by the
> transmit thread?


This is a tough problem.  ARP requests complicate this a lot if you shoot
for maximum throughput since if you did it at the IP layer then you're tx
thread will either get blocked (and block the queue) when an arp request is
made (with the current arp code, I think) or would have to keep track of
which packets waiting for arp replies.  Changing the events in ARP entries
to netbuf* might get you halfway there.

>
>
> Doing this in the driver may not be the best idea. It requires to
> rewrite several drivers and may make writing new drivers more complex
> than it needs to be.


I wouldn't do it in the lowest level (chip specific) but maybe for PPP and
for ETH.
In my version of Nut (a fork) I have altered NutIpOutput to not know
anything about
ARP, ethernet, or PPP.  I replaced the drivers' generic output routine
pointers with
IP output routine pointers which do interface specific stuff (ARP) and then
directly call
the generic routines.  I did this because of the the nonstandard network
device I am
working with, but it has other benifits.  (similar things done for
NutIfConfig).

I can't think of any differences that could exist below the device family
layer that would impact
this.  I haven't used wlan, though and could conceive that it may behave
differently
from either PPP or Eth.


>
> The IP layer is attractive, because only a singe thread is required to
> handle all interfaces including ARP. However, ARP and other non-IP
> protocols (we use DLC in one application) must be handled special and we
> may end up with similar problems than we have now.


The more general purpose the solution the slower/bigger it gets.
I prefer a thread per device because it allows each device to be kept busy
as long
as there is work to do, but that's a moot point for most people b/c they
aren't using
multiple network interfaces.
(Oh yeah, in the NutIfConfig changes I mentioned above I changed it so that
PPP
didn't record it's IP to EEPROM, which it did in the same place as Eth.)

>
> So, it looks to me, that the physical layer is the one to handle in the
> transmit thread. We'd need one thread per interface type, currently
> Ethernet and PPP.


For ethernet you could have a chain of protocols which each had a queue of
packets (netbufs).
These chain links could be made of (raw, IPv4).  The Ethernet TX thread
would do different
things based on which link a packet was under.  Raw packets would just be
sent, while IP packets
would be fed through ARP.  If an arp request needed to be made then that arp
request would be
appended to the RAW queue (unless there was an outstanding request already).
 If the IP packet
was ready to send then it's ethernet header would be completed and that
packet would become a
RAW packet, and be placed in the raw queue.  Essentially only things in the
RAW queue can be
sent to the device.

>
> > (btw, the "who frees a netbuf" within nut/net is very
> > confusing)
>
> You're not alone. The initial idea was to release the NETBUF as early as
> possible to make it available to the system for other requirements.
> Thus, in case of a fatal error, it is released by the lower level
> routine. In all other cases it is kept, because the top level may re-use
> it (TCP re-transmits).


Messing with this made me wish for garbage collection.

>
> In early days we had a lot of trouble with the RTL8019. One reason was
> that even Realtek was not aware, that only half of the chip's memory is
> available in 8-bit mode.


I hate when that happens.

>
> Today fatal errors occur almost never. May be it's the right time to
> change this weird NETBUF handling now.


This would make it much easier for other people to work on networking code.

I had thought about using a linked list or tree-like structure for netbufs
(maybe even
with reference counting and garbage collection).  That would be a huge
undertaking
but would allow for more than the current 4 layer model, and cloning them
could
just be aliasing them at the layer that you were interested in --
NutIpOutput's
handling of broadcast would just pass the same netbuf head to all the
interfaces
and they would all generate their own headers for the same IP packet.


Nathan
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

Hi Michael,

Michael Jones wrote:

> Also, I want to point out that we use an extra thread for DHCP in our
> systems. The current implementation does not handle DHCP in large
> environments very well, does not cope with delayed network startup and does
> not handle DNS and TimeServer updates via DHCP at all. Maybe we could build
> an extendable Network thread that could have various aspects plugged into it
> on demand? That would lower the overall "cost" of this thread.

This is one reason why DHCP needs a major rewrite. The other one is,
that it is a monster, code-size wise. I assume that the main loop is too
complicated for the compiler's optimizer. Furthermore, DHCP
automatically links to DNS, which is not needed by many applications.
Modularization is the magic word.

Also Nathan suggested some kind of general purpose network thread.
Sounds attractive, but quoting Nathan: There may be "other subtle
implementation issues with this as well."

Don't panic! I should have added to my previous post, that I do not have
the intention to redesign the whole network stuff right now. But we need
to think about _all_ possible improvements early to know which way to
go. My idea is to create a new set of network libraries, living
concurrently with the existing ones. It should be up to the application
developer whether he wants to change or keep the old version.

I'll start playing around with a "per-interface tx thread" and see, how
it develops.

Harald


_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Ethernut]

Hi Nathan,

Nathan Moore wrote:

> Every time I try to do much that involves the TCP state machine thread my
> head
> explodes, so I don't pretend to understand all of it's internals.

TCP is a complex beast. In opposite to other parts of Nut/OS, the TCP
state machine had been written in the first place with the human reader
in mind, using a specific function for each state. Some parts could have
been optimized to create less code. But none of the contributors ever
had the courage to work it out. There are many things that may go
miserably wrong. And many things did go wrong in the past. I remember
some very nasty and long living bugs in this area.


> is ready to send.  If the interface is slow enough the TCP thread may try to
> retransmit
> before the other thread has finished sending.

Right, TCP timeout processing had been crimped into the state machine
loop, which made it less flexible and controllable.


> I tend to favor optimizing getting things sent over receiving because it
> encourages
> more free RAM.

That's a very important guideline.

>> We'd still need a TCP state machine thread to handle re-transmissions.
>> That one could be simpler than the combined timing/handler loop used
>> right now.
>
> Since you now do protocol registration you could make part of that
> registration
> a callback to a periodically called function and a pointer to it's data --
> the TCP
> thread could turn into one of these functions, and a more generic IP
> transmission
> thread would call it.

I can't comment on this right now, without having a clearer picture of
the general structure.

>> Doing this in the driver may not be the best idea. It requires to
>> rewrite several drivers and may make writing new drivers more complex
>> than it needs to be.
>
> In my version of Nut (a fork) I have altered NutIpOutput to not know
> anything about
> ARP, ethernet, or PPP.  I replaced the drivers' generic output routine
> pointers with
> IP output routine pointers which do interface specific stuff (ARP) and then
> directly call
> the generic routines.

My fault. Of course, using a device specific thread does not mean, that
the thread must be implemented in the driver.


> The more general purpose the solution the slower/bigger it gets.
> I prefer a thread per device because it allows each device to be kept busy
> as long
> as there is work to do,

This is a good argument and follows the way other parts of the system
are designed. Otherwise a slow device may slow down the transfer of
faster devices or require a complicated scheduling.


> (Oh yeah, in the NutIfConfig changes I mentioned above I changed it so that
> PPP
> didn't record it's IP to EEPROM, which it did in the same place as Eth.)

I recently removed CONFNET saving from the system routines. I wasn't
aware of the PPP problem, but recognized, that many people got confused
by this "feature". If the application uses the wrong function call
sequence, they may end up with the hard coded values overwriting a valid
EEPROM contents (I think the ftpd sample has or had this problem too).
Now the application must explicitly call NutSaveConfig to store new
settings.

Not fully sure, if this is valid for PPP too. Furthermore, the system
may still save the current IP address, in which case the PPP problem
remains. Actually each interface should have its own EEPROM area, but
running two interfaces concurrently is something that never worked out
of the box so far.

> For ethernet you could have a chain of protocols which each had a queue of
> packets (netbufs).
> These chain links could be made of (raw, IPv4).  The Ethernet TX thread
> would do different
> things based on which link a packet was under.  Raw packets would just be
> sent, while IP packets
> would be fed through ARP.  If an arp request needed to be made then that arp
> request would be
> appended to the RAW queue (unless there was an outstanding request already).
>  If the IP packet
> was ready to send then it's ethernet header would be completed and that
> packet would become a
> RAW packet, and be placed in the raw queue.  Essentially only things in the
> RAW queue can be
> sent to the device.

Sounds interesting enough to give it a try.

> I had thought about using a linked list or tree-like structure for netbufs
> (maybe even
> with reference counting and garbage collection).  That would be a huge
> undertaking
> but would allow for more than the current 4 layer model, and cloning them
> could
> just be aliasing them at the layer that you were interested in --
> NutIpOutput's
> handling of broadcast would just pass the same netbuf head to all the
> interfaces
> and they would all generate their own headers for the same IP packet.

Garbage collection is a controversial feature in the area of tiny
embedded systems. That doesn't mean that it wouldn't offer advantages.
For many embedded developers dynamic memory allocation is a no-no as
well. I personally never experienced heap fragmentation problems on
Nut/OS (though the algorithm used here is quite slow).

Well, there are many things to try before thinking about garbage
collection. Indeed, re-using packet headers on multiple packets is
something I thought about too. It doesn't actually require garbage
collection, but just the reference counter.


Harald
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion

[Nathan Moore-5]

Hey Harald,

>
> > Every time I try to do much that involves the TCP state machine thread my
> > head
> > explodes, so I don't pretend to understand all of it's internals.
>
> TCP is a complex beast. In opposite to other parts of Nut/OS, the TCP
> state machine had been written in the first place with the human reader
> in mind, using a specific function for each state. Some parts could have
> been optimized to create less code. But none of the contributors ever
> had the courage to work it out. There are many things that may go
> miserably wrong. And many things did go wrong in the past. I remember
> some very nasty and long living bugs in this area.

It's like reading lisp -- very easy to get lost in, even if you understand
it in theory.
PPP is the same way.

>
> >> We'd still need a TCP state machine thread to handle re-transmissions.
> >> That one could be simpler than the combined timing/handler loop used
> >> right now.
> >
> > Since you now do protocol registration you could make part of that
> > registration
> > a callback to a periodically called function and a pointer to it's data
> --
> > the TCP
> > thread could turn into one of these functions, and a more generic IP
> > transmission
> > thread would call it.
>
> I can't comment on this right now, without having a clearer picture of
> the general structure.

This is similar to the chain of protocols I mentioned for Ethernet.  It
would
walk a list of transport layer structures and decide if it should call each
transport layer's periodic/timer handler function and pass that function
a pointer to something (maybe this entry in the protocol chain) that would
allow it to do something.  For TCP this would be call a function which is
essentially the TCP state machine thread function and it's argument would
allow it to access the socket list.  For UDP, ICMP, and IGMP this might be
a NULL pointer (and not called).
I hope this is clearer.
If you extend this concept when you register a transport layer protocol over
IP the structure could have pointers to the input routine and periodic/state
machine
routine.  Then introducing new protocols would be very modularized.
struct ip_transport {
   struct ip_transport *next;
   input(...);
   periodic(...);
   uint32_t periodic_next_time; // next time to call the periodic()
   ... whatever else generic transport stuff ...
   ... transport protocol specific stuff (either by void* or by tacking it
onto the tail of this struct) ...
};
extern struct ip_transport * ip_protocols;

>
>
>
>
> I recently removed CONFNET saving from the system routines. I wasn't
> aware of the PPP problem, but recognized, that many people got confused
> by this "feature". If the application uses the wrong function call
> sequence, they may end up with the hard coded values overwriting a valid
> EEPROM contents (I think the ftpd sample has or had this problem too).
> Now the application must explicitly call NutSaveConfig to store new
> settings.
>
> I'll have a look at the current code.

>
> > For ethernet you could have a chain of protocols which each had a queue
> of
> > packets (netbufs).
> > These chain links could be made of (raw, IPv4).  The Ethernet TX thread
> > would do different
> > things based on which link a packet was under.  Raw packets would just be
> > sent, while IP packets
> > would be fed through ARP.  If an arp request needed to be made then that
> arp
> > request would be
> > appended to the RAW queue (unless there was an outstanding request
> already).
> >  If the IP packet
> > was ready to send then it's ethernet header would be completed and that
> > packet would become a
> > RAW packet, and be placed in the raw queue.  Essentially only things in
> the
> > RAW queue can be
> > sent to the device.
>
> Sounds interesting enough to give it a try.
>
>
> > I had thought about using a linked list or tree-like structure for
> netbufs
> > (maybe even
> > with reference counting and garbage collection).  That would be a huge
> Well, there are many things to try before thinking about garbage
> collection. Indeed, re-using packet headers on multiple packets is
> something I thought about too. It doesn't actually require garbage
> collection, but just the reference counter.

Yeah -- I was just thinking about GC using reference counting.  Something
like mark and sweep would interfere with use of the heap by everything else,
and I'm pretty sure that would never work on AVR because pointers are
multibyte.  You'd have to make sure that pointers were completely stored
in at least one location before you could do GC or you might reclaim RAM
only for some code to later move the 2 bytes of the pointer back together
and dereference it later.  Plus it takes a good bit of resources to do GC.

Nathan
_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion