ncat: using UDP with --chat

[clemens fischer-3]

Hi,

it said:

  $ ncat -u -l 9999 --chat
  Ncat: UDP mode does not support connection brokering.
  If this feature is important to you, write nmap-dev@... with a
  description of how you intend to use it, as an aid to deciding how UDP
  connection brokering should work. QUITTING.

My idea is to use UDP chat mode as a kind of emergency IPC mechanism
together with "--allow" between cooperating processes on machines
without eg. posix mesage queues.

Connecting clients would tag their messages themselves, like:

  $ echo "temperature1: 33C" | ncat -u localhost 9999

Other clients would see and evaluate the entire line and respond in
a similiar way.


regards, clemens fischer

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[David Fifield]

On Tue, Oct 13, 2009 at 07:08:32PM +0200, clemens fischer wrote:

> it said:
>
>   $ ncat -u -l 9999 --chat
>   Ncat: UDP mode does not support connection brokering.
>   If this feature is important to you, write nmap-dev@... with a
>   description of how you intend to use it, as an aid to deciding how UDP
>   connection brokering should work. QUITTING.
>
> My idea is to use UDP chat mode as a kind of emergency IPC mechanism
> together with "--allow" between cooperating processes on machines
> without eg. posix mesage queues.
>
> Connecting clients would tag their messages themselves, like:
>
>   $ echo "temperature1: 33C" | ncat -u localhost 9999
>
> Other clients would see and evaluate the entire line and respond in
> a similiar way.

Cool, thanks for the note. One of the problems we've had in defining
connection brokering for UDP is knowing which clients to relay data to.
When the Ncat broker receives the "temperature1: 33C" message, where
does it send it? Presumably it should be sent to all interested
processes, but the broker has to know about those somehow.

One idea we've had is to consider any host that has ever sent UDP data
to the broker to be a "connected" client, perhaps with timeouts for
inactive hosts. This would require each participating process to "sign
in" by sending a message, to make the broker aware of them.

Is that something that would work for you? It would help if you would
sketch more of the IPC architecture. You've shown how the clients will
send to each other; what do they do to receive?

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org

[clemens fischer-3]

> David Fifield:

> On Tue, Oct 13, 2009 at 07:08:32PM +0200, clemens fischer wrote:
> > it said:
> >
> >   $ ncat -u -l 9999 --chat
> >   Ncat: UDP mode does not support connection brokering.  If this
> >   feature is important to you, write nmap-dev@... with a
> >   description of how you intend to use it, as an aid to deciding how
> >   UDP connection brokering should work. QUITTING.
> >
> > My idea is to use UDP chat mode as a kind of emergency IPC mechanism
> > together with "--allow" between cooperating processes on machines
> > without eg. posix mesage queues.
> >
> > Connecting clients would tag their messages themselves, like:
> >
> >   $ echo "temperature1: 33C" | ncat -u localhost 9999
> >
> > Other clients would see and evaluate the entire line and respond in
> > a similiar way.
>
> Cool, thanks for the note. One of the problems we've had in defining
> connection brokering for UDP is knowing which clients to relay data
> to.  When the Ncat broker receives the "temperature1: 33C" message,
> where does it send it? Presumably it should be sent to all interested
> processes, but the broker has to know about those somehow.
>
> One idea we've had is to consider any host that has ever sent UDP data
> to the broker to be a "connected" client, perhaps with timeouts for
> inactive hosts. This would require each participating process to "sign
> in" by sending a message, to make the broker aware of them.
>
> Is that something that would work for you? It would help if you would
> sketch more of the IPC architecture. You've shown how the clients will
> send to each other; what do they do to receive?

I was thinking about this a bit.  Currently I lean towards this idea:

Messages are of two categories.  Every message starts with a tag,
ending in a possibly configurable tag-separator.  After the
tag-separator can be any text.  If that text is empty, the process is
indicating interest in further messages of the "type" with this tag.

Textual tag-separators in the tag have to be transmitted either escaped
with the customary backslash or doubled, ie. given a colon as the
tag-separator, "temp\:1:" might be a valid tag, or "temp::1:"
(implementors choice).

An implementation may choose to allow querying the messages available so
far by sending an empty tag.  So, sending ":" alone would make the
server send back the tags learned so far.  This is for monitoring or for
"special clients".  It would be very nice if the hosts allowed to
monitor such a network could be made configurable, but the --allow* and
--deny* options should do fine.

I like the idea of the timeouts for inactive hosts, but I fear that
selecting by "connected on this UDP-port" alone could lead an explosion
of message-load.


regards, clemens
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Thu, Oct 22, 2009 at 01:21:11PM +0200, clemens fischer wrote:

> > On Tue, Oct 13, 2009 at 07:08:32PM +0200, clemens fischer wrote:
> > > it said:
> > >
> > >   $ ncat -u -l 9999 --chat
> > >   Ncat: UDP mode does not support connection brokering.  If this
> > >   feature is important to you, write nmap-dev@... with a
> > >   description of how you intend to use it, as an aid to deciding how
> > >   UDP connection brokering should work. QUITTING.
> > >
> > > My idea is to use UDP chat mode as a kind of emergency IPC mechanism
> > > together with "--allow" between cooperating processes on machines
> > > without eg. posix mesage queues.
> > >
> > > Connecting clients would tag their messages themselves, like:
> > >
> > >   $ echo "temperature1: 33C" | ncat -u localhost 9999
> > >
> > > Other clients would see and evaluate the entire line and respond in
> > > a similiar way.
> >
> > Cool, thanks for the note. One of the problems we've had in defining
> > connection brokering for UDP is knowing which clients to relay data
> > to.  When the Ncat broker receives the "temperature1: 33C" message,
> > where does it send it? Presumably it should be sent to all interested
> > processes, but the broker has to know about those somehow.
> >
> > One idea we've had is to consider any host that has ever sent UDP data
> > to the broker to be a "connected" client, perhaps with timeouts for
> > inactive hosts. This would require each participating process to "sign
> > in" by sending a message, to make the broker aware of them.
> >
> > Is that something that would work for you? It would help if you would
> > sketch more of the IPC architecture. You've shown how the clients will
> > send to each other; what do they do to receive?
>
> I was thinking about this a bit.  Currently I lean towards this idea:
>
> Messages are of two categories.  Every message starts with a tag,
> ending in a possibly configurable tag-separator.  After the
> tag-separator can be any text.  If that text is empty, the process is
> indicating interest in further messages of the "type" with this tag.
>
> Textual tag-separators in the tag have to be transmitted either escaped
> with the customary backslash or doubled, ie. given a colon as the
> tag-separator, "temp\:1:" might be a valid tag, or "temp::1:"
> (implementors choice).
>
> An implementation may choose to allow querying the messages available so
> far by sending an empty tag.  So, sending ":" alone would make the
> server send back the tags learned so far.  This is for monitoring or for
> "special clients".  It would be very nice if the hosts allowed to
> monitor such a network could be made configurable, but the --allow* and
> --deny* options should do fine.
>
> I like the idea of the timeouts for inactive hosts, but I fear that
> selecting by "connected on this UDP-port" alone could lead an explosion
> of message-load.

So, this is really the heart of the issue. Whatever the format of the
messages, the server has to somehow know what addresses to send messages
to. The wouldn't be an explosion of messages because we would still
enforce --max-conns, but we would end up sending messages to addresses
that no longer care about them. But so far no one has offered an
alternative implementation idea.

But from your description of the server, I think that simple connection
brokering is not enough for what you want. Things like having the server
send back a list of recorded tags in response to a ":" will require some
domain-specific knowledge in the server. That is, unless the "server" of
this protocol is different from the Ncat server, and connects to the
Ncat server as a normal client. But then you still have the problem of
know who to relay messages to.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[clemens fischer-3]

> David Fifield:

> So, this is really the heart of the issue. Whatever the format of
> the messages, the server has to somehow know what addresses to send
> messages to. The wouldn't be an explosion of messages because we
> would still enforce --max-conns, but we would end up sending messages
> to addresses that no longer care about them. But so far no one has
> offered an alternative implementation idea.
>
> But from your description of the server, I think that simple
> connection brokering is not enough for what you want. Things like
> having the server send back a list of recorded tags in response to a
> ":" will require some domain-specific knowledge in the server. That
> is, unless the "server" of this protocol is different from the Ncat
> server, and connects to the Ncat server as a normal client. But then
> you still have the problem of know who to relay messages to.

I'm not sure how to handle this.  In my original concept the entity now
to be realized by Ncat was only a broker, and it as the only one.  Each
participant could serve and receive messages, and one could have made
one of them an observer just logging and replaying (parts of) messages
on request.  I needed a simple and robust system monitoring facility
back then, without any domain or topology knowledge.

Could Ncat in UDP brokering mode record all connecting clients together
with the their last message and a sequence number?  What I originally
wanted was like this:

                      sender/
  client message      response        Ncat behaviour
  ------------------------------------------------------------
  c-1    temp1: 33C   <N><S1><ok>     register c-1 as sender
  c-2    :            <N><S2><ok>     register c-2 as observer
  c-3    sick: yes    <N><S3><ok>     register c-3 as sender
  c-4    temp1:       <C1><S1>33C     register c-4 as
                                      interested in temp1
  c-5    :            <C2><S2>temp1,  don't register c-5,
                      sick            just make it aware
  c-2    ::           <N><S4><ok>     unregister c-2
  c-6    :            <N><S5><ok>     register c-6 as observer
  c-7    :            <timeout>       nothing, expects c-6 to answer
  c-6    ::           <N><S5><ok>     unregister c-6

  legend:

  <N>       sending party is Ncat
  <Cn>      sending party is client-n
  <timeout> nothing is sent, no state change, an error
  <Sn>      sequence number, relates messages and clients
  <ok>      no response text, but no error either

In this example, c-2 and c-6 are special in that either could become an
observer.  c-2 can handle it for some time and does so, c-6 can not and
unregisters immediately.  There could be a client (c-7) asking for tags
between the time c-6 asks, recognizes the new role it should fulfill and
unregisters.  By that time Ncat has c-6 observer registered and expects
it to answer, but this doesn't happen, as c-6 is not wired as an
observer.

The sequence numbers are meant for clients to know if the value they got
sent is new or an old one.

Clients registering successfully as senders  should store the sequence
number from the response sent by Ncat, because they are responsible for
sending it back once other clients ask for the information.

Does all this make sense?  How does the dbus message bus work with
regard to brokering messages?


gruss, clemens
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Mon, Nov 09, 2009 at 12:28:21AM +0100, clemens fischer wrote:

> > So, this is really the heart of the issue. Whatever the format of
> > the messages, the server has to somehow know what addresses to send
> > messages to. The wouldn't be an explosion of messages because we
> > would still enforce --max-conns, but we would end up sending messages
> > to addresses that no longer care about them. But so far no one has
> > offered an alternative implementation idea.
> >
> > But from your description of the server, I think that simple
> > connection brokering is not enough for what you want. Things like
> > having the server send back a list of recorded tags in response to a
> > ":" will require some domain-specific knowledge in the server. That
> > is, unless the "server" of this protocol is different from the Ncat
> > server, and connects to the Ncat server as a normal client. But then
> > you still have the problem of know who to relay messages to.
>
> I'm not sure how to handle this.  In my original concept the entity now
> to be realized by Ncat was only a broker, and it as the only one.  Each
> participant could serve and receive messages, and one could have made
> one of them an observer just logging and replaying (parts of) messages
> on request.  I needed a simple and robust system monitoring facility
> back then, without any domain or topology knowledge.
>
> Could Ncat in UDP brokering mode record all connecting clients together
> with the their last message and a sequence number?  What I originally
> wanted was like this:
>
>                       sender/
>   client message      response        Ncat behaviour
>   ------------------------------------------------------------
>   c-1    temp1: 33C   <N><S1><ok>     register c-1 as sender
>   c-2    :            <N><S2><ok>     register c-2 as observer
>   c-3    sick: yes    <N><S3><ok>     register c-3 as sender
>   c-4    temp1:       <C1><S1>33C     register c-4 as
>                                       interested in temp1
>   c-5    :            <C2><S2>temp1,  don't register c-5,
>                       sick            just make it aware
>   c-2    ::           <N><S4><ok>     unregister c-2
>   c-6    :            <N><S5><ok>     register c-6 as observer
>   c-7    :            <timeout>       nothing, expects c-6 to answer
>   c-6    ::           <N><S5><ok>     unregister c-6
>
>   legend:
>
>   <N>       sending party is Ncat
>   <Cn>      sending party is client-n
>   <timeout> nothing is sent, no state change, an error
>   <Sn>      sequence number, relates messages and clients
>   <ok>      no response text, but no error either
>
> In this example, c-2 and c-6 are special in that either could become an
> observer.  c-2 can handle it for some time and does so, c-6 can not and
> unregisters immediately.  There could be a client (c-7) asking for tags
> between the time c-6 asks, recognizes the new role it should fulfill and
> unregisters.  By that time Ncat has c-6 observer registered and expects
> it to answer, but this doesn't happen, as c-6 is not wired as an
> observer.
>
> The sequence numbers are meant for clients to know if the value they got
> sent is new or an old one.
>
> Clients registering successfully as senders  should store the sequence
> number from the response sent by Ncat, because they are responsible for
> sending it back once other clients ask for the information.

Thank you. This helps me understand better what you want to accomplish.
I think this kind of thing, which amounts to a new network protocol, is
more than Ncat brokering should do. I think it should, as much as
possible, be a transparent pipe that doesn't have to process data above
the TCP or UDP layer.

Have you thought about using TCP brokering mode for this? What you've
described above could be implemented with all the intelligence in the
clients. The : and :: messages would just be the beginning and end of
TCP connections. The hard part of all this is managing connections, and
TCP does that.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[clemens fischer-3]

(Sorry for being late again, I am busy with something else.)

> David Fifield:

> On Mon, Nov 09, 2009 at 12:28:21AM +0100, clemens fischer wrote:
>
> >                       sender/
> >   client message      response        Ncat behaviour
> >   ------------------------------------------------------------
> >   c-1    temp1: 33C   <N><S1><ok>     register c-1 as sender
> >   c-2    :            <N><S2><ok>     register c-2 as observer
> >   c-3    sick: yes    <N><S3><ok>     register c-3 as sender
> >   c-4    temp1:       <C1><S1>33C     register c-4 as
> >                                       interested in temp1
> >   c-5    :            <C2><S2>temp1,  don't register c-5,
> >                       sick            just make it aware
> >   c-2    ::           <N><S4><ok>     unregister c-2
> >   c-6    :            <N><S5><ok>     register c-6 as observer
> >   c-7    :            <timeout>       nothing, expects c-6 to answer
> >   c-6    ::           <N><S5><ok>     unregister c-6
> >
> >   legend:
> >
> >   <N>       sending party is Ncat
> >   <Cn>      sending party is client-n
> >   <timeout> nothing is sent, no state change, an error
> >   <Sn>      sequence number, relates messages and clients
> >   <ok>      no response text, but no error either
>
> Thank you. This helps me understand better what you want to
> accomplish.  I think this kind of thing, which amounts to a new
> network protocol, is more than Ncat brokering should do. I think it
> should, as much as possible, be a transparent pipe that doesn't have
> to process data above the TCP or UDP layer.
>
> Have you thought about using TCP brokering mode for this? What you've
> described above could be implemented with all the intelligence in the
> clients. The : and :: messages would just be the beginning and end of
> TCP connections. The hard part of all this is managing connections,
> and TCP does that.

Sounds reasonable.  But.  Can brokering be made to work over unix local
sockets in reliable datagram mode, like TCP?  What I mean is mentioned
in unix(7):  "unix_socket = socket(AF_UNIX, type, 0);" where type would
preferably be SOCK_STREAM or even SOCK_SEQPACKET (portability?) and
allow nice tricks with passing SO_PASSCRED!  I had experimented with
"socat" much earlier than 1.7.x and never got anything useful to work.

In short:  the TCP brokering is fine, but I'd like it to be local to the
machine, without the overhead and the security-problem (firewalling just
for this application).


regards, clemens
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Wed, Nov 18, 2009 at 10:29:54PM +0100, clemens fischer wrote:

> (Sorry for being late again, I am busy with something else.)
>
> > On Mon, Nov 09, 2009 at 12:28:21AM +0100, clemens fischer wrote:
> >
> > >                       sender/
> > >   client message      response        Ncat behaviour
> > >   ------------------------------------------------------------
> > >   c-1    temp1: 33C   <N><S1><ok>     register c-1 as sender
> > >   c-2    :            <N><S2><ok>     register c-2 as observer
> > >   c-3    sick: yes    <N><S3><ok>     register c-3 as sender
> > >   c-4    temp1:       <C1><S1>33C     register c-4 as
> > >                                       interested in temp1
> > >   c-5    :            <C2><S2>temp1,  don't register c-5,
> > >                       sick            just make it aware
> > >   c-2    ::           <N><S4><ok>     unregister c-2
> > >   c-6    :            <N><S5><ok>     register c-6 as observer
> > >   c-7    :            <timeout>       nothing, expects c-6 to answer
> > >   c-6    ::           <N><S5><ok>     unregister c-6
> > >
> > >   legend:
> > >
> > >   <N>       sending party is Ncat
> > >   <Cn>      sending party is client-n
> > >   <timeout> nothing is sent, no state change, an error
> > >   <Sn>      sequence number, relates messages and clients
> > >   <ok>      no response text, but no error either
> >
> > Thank you. This helps me understand better what you want to
> > accomplish.  I think this kind of thing, which amounts to a new
> > network protocol, is more than Ncat brokering should do. I think it
> > should, as much as possible, be a transparent pipe that doesn't have
> > to process data above the TCP or UDP layer.
> >
> > Have you thought about using TCP brokering mode for this? What you've
> > described above could be implemented with all the intelligence in the
> > clients. The : and :: messages would just be the beginning and end of
> > TCP connections. The hard part of all this is managing connections,
> > and TCP does that.
>
> Sounds reasonable.  But.  Can brokering be made to work over unix local
> sockets in reliable datagram mode, like TCP?  What I mean is mentioned
> in unix(7):  "unix_socket = socket(AF_UNIX, type, 0);" where type would
> preferably be SOCK_STREAM or even SOCK_SEQPACKET (portability?) and
> allow nice tricks with passing SO_PASSCRED!  I had experimented with
> "socat" much earlier than 1.7.x and never got anything useful to work.

Ncat can't do that, I'm sure. It is an interesting idea.

> In short:  the TCP brokering is fine, but I'd like it to be local to the
> machine, without the overhead and the security-problem (firewalling just
> for this application).

If you give the Ncat server a source address, it will listen only on
that interface. It will be local to the machine if you do this:

ncat --broker 127.0.0.1

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/