Re: Add support infrastructure for Named Pipe

[Steve French-2]

> the only user of this support that has ever been identified is WINE.
I have had requests over the years for adding the few missing calls,
from a lot more than from WINE.  Various distros have to ship multiple
versions of Samba with conflicting licenses to work around problems in
desktops, and the libsmbclient is not appropriate for everyone.  For a
while it looked like Centeris/Likewise would address some of the need
(presumably not an option for most distros though) but even with that
it is messy to have userspace making connections when kernel code
already has a connection to the same server.

> I don't think this stuff has any place in the kernel.  Just implement the named pipes directly in wine.
I agree that all code that is above the SMB/CIFS layer could (and
should) be put in user space (probably in a new helper library, since
Samba currently doesn't have anything which matches exactly, and I
have seen requests from others over the last few years for something
like what Centeris was trying to do with librpc for dcom over pipes),
but it doesn't make much sense to reimplement another cifs client in
user space, when pipe operations use (mostly) read and write SMBs.
As far as I can tell, there are only a couple of additional SMBs that
this patch suggests adding and it looks straightforward, and much much
easier than adding a whole new SMB/CIFS client in user space.   CIFS
already had support for mounting to IPC$ shares and opening a pipe
(pipes are similar to file objects), but it turns out that for a
userspace library to take advantage of that it is helpful to add
transact and two other ioctls.   Seems much easier to do it this way,
and will avoid having to renegotiate multiple tcp sockets from the
same client (which conserves server resources).

> You'll need infrastructure too to actually handle doing some
> sort of implicit "mount" of an IPC$ share when ...
> new host What's going to trigger the "mount" the IPC$ share? What if I have
> multiple users on the same host that want to talk to the same server?

We have to do that (automatic session setup if we don't have an smb uid
for this uid) anyway in cifs.ko (just as NFS has to autoconnect).
Today (in cifs), we use the credentials that match the pid (if we know
them, and use the default if we don't).   This needs to be fixed
to allow obtaining credentials from userspace (whether or not
pipe support is added) so is a different issue.

Since this patch is fairly small, and the smbs fit with what cifs already does,
I think it makes sense.   The userspace DCE/RPC code (which Shirish has
prototyped and tried various versions of) is the majority of the code, but
doesn't have to go in kernel - would go in a new Samba lib presumably.



--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Thu, 15 Oct 2009 18:55:57 -0500
Steve French <smfrench@...> wrote:

> > the only user of this support that has ever been identified is WINE.
> I have had requests over the years for adding the few missing calls,
> from a lot more than from WINE.  Various distros have to ship multiple
> versions of Samba with conflicting licenses to work around problems in
> desktops, and the libsmbclient is not appropriate for everyone. For a
> while it looked like Centeris/Likewise would address some of the need
> (presumably not an option for most distros though) but even with that
> it is messy to have userspace making connections when kernel code
> already has a connection to the same server.
>

Shoving code into the kernel to work around licensing problems is not a
good solution. All of those problems can equally be solved by a daemon
that runs in userspace.

I'm not of the opinion that an extra connection to the server is a huge
deal. At least, not a big enough problem to warrant doing this in
kernel rather than in userspace.

> > I don't think this stuff has any place in the kernel.  Just implement the named pipes directly in wine.
> I agree that all code that is above the SMB/CIFS layer could (and
> should) be put in user space (probably in a new helper library, since
> Samba currently doesn't have anything which matches exactly, and I
> have seen requests from others over the last few years for something
> like what Centeris was trying to do with librpc for dcom over pipes),
> but it doesn't make much sense to reimplement another cifs client in
> user space, when pipe operations use (mostly) read and write SMBs.
> As far as I can tell, there are only a couple of additional SMBs that
> this patch suggests adding and it looks straightforward, and much much
> easier than adding a whole new SMB/CIFS client in user space.   CIFS
> already had support for mounting to IPC$ shares and opening a pipe
> (pipes are similar to file objects), but it turns out that for a
> userspace library to take advantage of that it is helpful to add
> transact and two other ioctls.   Seems much easier to do it this way,
> and will avoid having to renegotiate multiple tcp sockets from the
> same client (which conserves server resources).
>
> > You'll need infrastructure too to actually handle doing some
> > sort of implicit "mount" of an IPC$ share when ...
> > new host What's going to trigger the "mount" the IPC$ share? What if I have
> > multiple users on the same host that want to talk to the same server?
>
> We have to do that (automatic session setup if we don't have an smb uid
> for this uid) anyway in cifs.ko (just as NFS has to autoconnect).
> Today (in cifs), we use the credentials that match the pid (if we know
> them, and use the default if we don't).   This needs to be fixed
> to allow obtaining credentials from userspace (whether or not
> pipe support is added) so is a different issue.
>
> Since this patch is fairly small, and the smbs fit with what cifs already does,
> I think it makes sense.   The userspace DCE/RPC code (which Shirish has
> prototyped and tried various versions of) is the majority of the code, but
> doesn't have to go in kernel - would go in a new Samba lib presumably.
>
>
>

Just because CIFS has part of the code needed to this already doesn't
mean that it's the right place to do this. I'm sure that libsmbclient
or the Likewise stuff has a good portion of what is needed already too.

One thing I haven't yet heard is how you expect WINE to actually use
this stuff. As I noted before, there is no guarantee that you will know
what hosts WINE wants to talk to before you start it up. Therefore you
can't know beforehand what IPC$ mounts you will need.

With what you have so far, WINE will need to mount the IPC$
share in order to use it. I personally am not comfortable with a
solution that requires giving WINE the elevated privileges that it will
need in order to actually do this mount. Doing this in userspace is
better and safer for that reason alone.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Shirish Pargaonkar]

On Fri, Oct 16, 2009 at 7:06 AM, Jeff Layton <jlayton@...> wrote:

> On Thu, 15 Oct 2009 18:55:57 -0500
> Steve French <smfrench@...> wrote:
>
>> > the only user of this support that has ever been identified is WINE.
>> I have had requests over the years for adding the few missing calls,
>> from a lot more than from WINE.  Various distros have to ship multiple
>> versions of Samba with conflicting licenses to work around problems in
>> desktops, and the libsmbclient is not appropriate for everyone. For a
>> while it looked like Centeris/Likewise would address some of the need
>> (presumably not an option for most distros though) but even with that
>> it is messy to have userspace making connections when kernel code
>> already has a connection to the same server.
>>
>
> Shoving code into the kernel to work around licensing problems is not a
> good solution. All of those problems can equally be solved by a daemon
> that runs in userspace.
>
> I'm not of the opinion that an extra connection to the server is a huge
> deal. At least, not a big enough problem to warrant doing this in
> kernel rather than in userspace.
>
>> > I don't think this stuff has any place in the kernel.  Just implement the named pipes directly in wine.
>> I agree that all code that is above the SMB/CIFS layer could (and
>> should) be put in user space (probably in a new helper library, since
>> Samba currently doesn't have anything which matches exactly, and I
>> have seen requests from others over the last few years for something
>> like what Centeris was trying to do with librpc for dcom over pipes),
>> but it doesn't make much sense to reimplement another cifs client in
>> user space, when pipe operations use (mostly) read and write SMBs.
>> As far as I can tell, there are only a couple of additional SMBs that
>> this patch suggests adding and it looks straightforward, and much much
>> easier than adding a whole new SMB/CIFS client in user space.   CIFS
>> already had support for mounting to IPC$ shares and opening a pipe
>> (pipes are similar to file objects), but it turns out that for a
>> userspace library to take advantage of that it is helpful to add
>> transact and two other ioctls.   Seems much easier to do it this way,
>> and will avoid having to renegotiate multiple tcp sockets from the
>> same client (which conserves server resources).
>>
>> > You'll need infrastructure too to actually handle doing some
>> > sort of implicit "mount" of an IPC$ share when ...
>> > new host What's going to trigger the "mount" the IPC$ share? What if I have
>> > multiple users on the same host that want to talk to the same server?
>>
>> We have to do that (automatic session setup if we don't have an smb uid
>> for this uid) anyway in cifs.ko (just as NFS has to autoconnect).
>> Today (in cifs), we use the credentials that match the pid (if we know
>> them, and use the default if we don't).   This needs to be fixed
>> to allow obtaining credentials from userspace (whether or not
>> pipe support is added) so is a different issue.
>>
>> Since this patch is fairly small, and the smbs fit with what cifs already does,
>> I think it makes sense.   The userspace DCE/RPC code (which Shirish has
>> prototyped and tried various versions of) is the majority of the code, but
>> doesn't have to go in kernel - would go in a new Samba lib presumably.
>>
>>
>>
>
> Just because CIFS has part of the code needed to this already doesn't
> mean that it's the right place to do this. I'm sure that libsmbclient
> or the Likewise stuff has a good portion of what is needed already too.
>
> One thing I haven't yet heard is how you expect WINE to actually use
> this stuff. As I noted before, there is no guarantee that you will know
> what hosts WINE wants to talk to before you start it up. Therefore you
> can't know beforehand what IPC$ mounts you will need.
>
> With what you have so far, WINE will need to mount the IPC$
> share in order to use it. I personally am not comfortable with a
> solution that requires giving WINE the elevated privileges that it will
> need in order to actually do this mount. Doing this in userspace is
> better and safer for that reason alone.
>

Do we need to have a mount i.e. would a tree connection to IPC$ share suffice
for a WINE app user id by obtaining credentials for that user id via cifs upcall
using a arbitrary security mechanism when that user very first time accesses
a named pipe (e.g. during CreateFile)?

> --
> Jeff Layton <jlayton@...>
>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Volker Lendecke]

On Fri, Oct 16, 2009 at 07:26:20AM -0500, Shirish Pargaonkar wrote:
> Do we need to have a mount i.e. would a tree connection to
> IPC$ share suffice for a WINE app user id by obtaining
> credentials for that user id via cifs upcall using a
> arbitrary security mechanism when that user very first
> time accesses a named pipe (e.g. during CreateFile)?

Regarding the credentials: Either you have Kerberos, or for
NTLM winbind can provide a credential store. If anything
does not work sufficiently with winbind, we will fix it.

Volker


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Fri, Oct 16, 2009 at 7:06 AM, Jeff Layton <jlayton@...> wrote:
> On Thu, 15 Oct 2009 18:55:57 -0500
> Steve French <smfrench@...> wrote:

> Shoving code into the kernel to work around licensing problems is not a
> good solution. All of those problems can equally be solved by a daemon
> that runs in userspace.

Not sure I would describe adding two or three SMB calls "shoving code into
the kernel" especially as this in kernel in most other OS and fits with
the kernel layer.   Remember that no one is suggesting adding DCOM
and RPC to kernel - just the ability to do a couple missing primitives
(e.g. transact and setnmpstate) which makes the user space code eaiser.

> I'm not of the opinion that an extra connection to the server is a huge
> deal. At least, not a big enough problem to warrant doing this in
> kernel rather than in userspace.

It seems insane to creating a dependency on a userspace daemon
which has to hang around just to avoid adding a couple of wire requests
(especially since these transact calls aren't really written in userspace
this way).  Even with  userspace IPC to a GPL daemon I have heard concerns
that that is frowned upon (IANAL).

> With what you have so far, WINE will need to mount the IPC$
> share in order to use it.

I am not suggesting that we give WINE elevated priv.


--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Fri, Oct 16, 2009 at 7:36 AM, Volker Lendecke
<Volker.Lendecke@...> wrote:

> On Fri, Oct 16, 2009 at 07:26:20AM -0500, Shirish Pargaonkar wrote:
>> Do we need to have a mount i.e. would a tree connection to
>> IPC$ share suffice for a WINE app user id by obtaining
>> credentials for that user id via cifs upcall using a
>> arbitrary security mechanism when that user very first
>> time accesses a named pipe (e.g. during CreateFile)?
>
> Regarding the credentials: Either you have Kerberos, or for
> NTLM winbind can provide a credential store. If anything
> does not work sufficiently with winbind, we will fix it.

Yes.  That is excellent.   I think the issue of getting
credentials is independent though of whether to add the missing
SMB to the kernel though.



--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Fri, 16 Oct 2009 07:26:20 -0500
Shirish Pargaonkar <shirishpargaonkar@...> wrote:

> On Fri, Oct 16, 2009 at 7:06 AM, Jeff Layton <jlayton@...> wrote:
> > On Thu, 15 Oct 2009 18:55:57 -0500
> > Steve French <smfrench@...> wrote:
> >
> >> > the only user of this support that has ever been identified is WINE.
> >> I have had requests over the years for adding the few missing calls,
> >> from a lot more than from WINE.  Various distros have to ship multiple
> >> versions of Samba with conflicting licenses to work around problems in
> >> desktops, and the libsmbclient is not appropriate for everyone. For a
> >> while it looked like Centeris/Likewise would address some of the need
> >> (presumably not an option for most distros though) but even with that
> >> it is messy to have userspace making connections when kernel code
> >> already has a connection to the same server.
> >>
> >
> > Shoving code into the kernel to work around licensing problems is not a
> > good solution. All of those problems can equally be solved by a daemon
> > that runs in userspace.
> >
> > I'm not of the opinion that an extra connection to the server is a huge
> > deal. At least, not a big enough problem to warrant doing this in
> > kernel rather than in userspace.
> >
> >> > I don't think this stuff has any place in the kernel.  Just implement the named pipes directly in wine.
> >> I agree that all code that is above the SMB/CIFS layer could (and
> >> should) be put in user space (probably in a new helper library, since
> >> Samba currently doesn't have anything which matches exactly, and I
> >> have seen requests from others over the last few years for something
> >> like what Centeris was trying to do with librpc for dcom over pipes),
> >> but it doesn't make much sense to reimplement another cifs client in
> >> user space, when pipe operations use (mostly) read and write SMBs.
> >> As far as I can tell, there are only a couple of additional SMBs that
> >> this patch suggests adding and it looks straightforward, and much much
> >> easier than adding a whole new SMB/CIFS client in user space.   CIFS
> >> already had support for mounting to IPC$ shares and opening a pipe
> >> (pipes are similar to file objects), but it turns out that for a
> >> userspace library to take advantage of that it is helpful to add
> >> transact and two other ioctls.   Seems much easier to do it this way,
> >> and will avoid having to renegotiate multiple tcp sockets from the
> >> same client (which conserves server resources).
> >>
> >> > You'll need infrastructure too to actually handle doing some
> >> > sort of implicit "mount" of an IPC$ share when ...
> >> > new host What's going to trigger the "mount" the IPC$ share? What if I have
> >> > multiple users on the same host that want to talk to the same server?
> >>
> >> We have to do that (automatic session setup if we don't have an smb uid
> >> for this uid) anyway in cifs.ko (just as NFS has to autoconnect).
> >> Today (in cifs), we use the credentials that match the pid (if we know
> >> them, and use the default if we don't).   This needs to be fixed
> >> to allow obtaining credentials from userspace (whether or not
> >> pipe support is added) so is a different issue.
> >>
> >> Since this patch is fairly small, and the smbs fit with what cifs already does,
> >> I think it makes sense.   The userspace DCE/RPC code (which Shirish has
> >> prototyped and tried various versions of) is the majority of the code, but
> >> doesn't have to go in kernel - would go in a new Samba lib presumably.
> >>
> >>
> >>
> >
> > Just because CIFS has part of the code needed to this already doesn't
> > mean that it's the right place to do this. I'm sure that libsmbclient
> > or the Likewise stuff has a good portion of what is needed already too.
> >
> > One thing I haven't yet heard is how you expect WINE to actually use
> > this stuff. As I noted before, there is no guarantee that you will know
> > what hosts WINE wants to talk to before you start it up. Therefore you
> > can't know beforehand what IPC$ mounts you will need.
> >
> > With what you have so far, WINE will need to mount the IPC$
> > share in order to use it. I personally am not comfortable with a
> > solution that requires giving WINE the elevated privileges that it will
> > need in order to actually do this mount. Doing this in userspace is
> > better and safer for that reason alone.
> >
>
> Do we need to have a mount i.e. would a tree connection to IPC$ share suffice
> for a WINE app user id by obtaining credentials for that user id via cifs upcall
> using a arbitrary security mechanism when that user very first time accesses
> a named pipe (e.g. during CreateFile)?
>

Well...all of this code seems to presuppose that there will be actual
filesystem objects on which to operate. IOW, you can't call an ioctl on
a non-existent filehandle.

Again, it's not at all clear from this patchset how userspace will
actually use this code. How do you envision userspace programs actually
making use of this?

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Fri, 16 Oct 2009 07:35:33 -0500
Steve French <smfrench@...> wrote:

> On Fri, Oct 16, 2009 at 7:06 AM, Jeff Layton <jlayton@...> wrote:
> > On Thu, 15 Oct 2009 18:55:57 -0500
> > Steve French <smfrench@...> wrote:
>
> > Shoving code into the kernel to work around licensing problems is not a
> > good solution. All of those problems can equally be solved by a daemon
> > that runs in userspace.
>
> Not sure I would describe adding two or three SMB calls "shoving code into
> the kernel" especially as this in kernel in most other OS and fits with
> the kernel layer.   Remember that no one is suggesting adding DCOM
> and RPC to kernel - just the ability to do a couple missing primitives
> (e.g. transact and setnmpstate) which makes the user space code eaiser.
>
> > I'm not of the opinion that an extra connection to the server is a huge
> > deal. At least, not a big enough problem to warrant doing this in
> > kernel rather than in userspace.
>
> It seems insane to creating a dependency on a userspace daemon
> which has to hang around just to avoid adding a couple of wire requests
> (especially since these transact calls aren't really written in userspace
> this way).  Even with  userspace IPC to a GPL daemon I have heard concerns
> that that is frowned upon (IANAL).
>

I don't think it's insane at all. It seems perfectly reasonable to have
a standalone CIFS named pipe service daemon rather than doing this in
kernel.

IANAL either, but I don't see how this would be any different
than talking to any other general-purpose service (syslog or the
resolver, for instance). To my understanding, whether this code is in
kernel or userspace is somewhat irrelevant as long as you're not
linking to it.

Either way...I think we should focus on the best technical solution for
this and worry about licensing once we settle on that.

> > With what you have so far, WINE will need to mount the IPC$
> > share in order to use it.
>
> I am not suggesting that we give WINE elevated priv.
>

But you are. Something has to mount this IPC$ share, right? CIFS only
open sockets to servers when someone does an actual mount. This
patchset doesn't change that.

Again, it's very difficult to discuss this without understanding how
you intend for userspace to actually use this support. Can you outline
how you envision userspace programs using this new code?

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Christoph Hellwig]

On Fri, Oct 16, 2009 at 09:05:18AM -0400, Jeff Layton wrote:
> Well...all of this code seems to presuppose that there will be actual
> filesystem objects on which to operate. IOW, you can't call an ioctl on
> a non-existent filehandle.
>
> Again, it's not at all clear from this patchset how userspace will
> actually use this code. How do you envision userspace programs actually
> making use of this?

Yeah.  I have no problem adding kernel support for the named pipes to
the kernel if we really need to, but:

 - please odn't shove it into a filesystem as that's a totally unrelated
   abstraction
 - write some proper code for it, not the thing that was submitted
   adding random ioctls dereferecing user pointers and similar.
   Incidentally moving it out of cifs will make that a lot easier as
   bad code doesn't really stick out in cifs.

_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Sun, Oct 18, 2009 at 7:12 PM, Christoph Hellwig <hch@...> wrote:

> On Fri, Oct 16, 2009 at 09:05:18AM -0400, Jeff Layton wrote:
>> Well...all of this code seems to presuppose that there will be actual
>> filesystem objects on which to operate. IOW, you can't call an ioctl on
>> a non-existent filehandle.
>>
>> Again, it's not at all clear from this patchset how userspace will
>> actually use this code. How do you envision userspace programs actually
>> making use of this?
>
> Yeah.  I have no problem adding kernel support for the named pipes to
> the kernel if we really need to, but:
>
>  - please odn't shove it into a filesystem as that's a totally unrelated
>   abstraction
>  - write some proper code for it, not the thing that was submitted
>   adding random ioctls dereferecing user pointers and similar.
>   Incidentally moving it out of cifs will make that a lot easier as
>   bad code doesn't really stick out in cifs.

Problem with making them via something other than an ioctl is -
they are handle based ops and operations on pipes
are the same on the wire as open/close/read/write
operations on files so the majority of this already works (these
are the client not server side of pipe operations Shirish
is talking about).   The missing operations
are handle based (e.g. transact and get/set named pipe info)
so ioctls seem reasonable and  I don't know of an obvious
alternative to ioctl, to do handle based operations.   Certainly it is much
smaller to do a few ioctls  than the alternatives and the ioctls can easily
be hidden by a user space library as apparently
Shirish did in a prototype and saves having to write a new cifs client
from scratch.

A quick search shows that the idea of having ioctls for special
purpose pipe operations (Unix pipes and Windows pipes) is
not new, it seems to have been done before on multiple platforms.
(includ


--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Sun, 18 Oct 2009 20:22:41 -0500
Steve French <smfrench@...> wrote:

> On Sun, Oct 18, 2009 at 7:12 PM, Christoph Hellwig <hch@...> wrote:
> > On Fri, Oct 16, 2009 at 09:05:18AM -0400, Jeff Layton wrote:
> >> Well...all of this code seems to presuppose that there will be actual
> >> filesystem objects on which to operate. IOW, you can't call an ioctl on
> >> a non-existent filehandle.
> >>
> >> Again, it's not at all clear from this patchset how userspace will
> >> actually use this code. How do you envision userspace programs actually
> >> making use of this?
> >
> > Yeah.  I have no problem adding kernel support for the named pipes to
> > the kernel if we really need to, but:
> >
> >  - please odn't shove it into a filesystem as that's a totally unrelated
> >   abstraction
> >  - write some proper code for it, not the thing that was submitted
> >   adding random ioctls dereferecing user pointers and similar.
> >   Incidentally moving it out of cifs will make that a lot easier as
> >   bad code doesn't really stick out in cifs.
>
> Problem with making them via something other than an ioctl is -
> they are handle based ops and operations on pipes
> are the same on the wire as open/close/read/write
> operations on files so the majority of this already works (these
> are the client not server side of pipe operations Shirish
> is talking about).   The missing operations
> are handle based (e.g. transact and get/set named pipe info)
> so ioctls seem reasonable and  I don't know of an obvious
> alternative to ioctl, to do handle based operations.   Certainly it is much
> smaller to do a few ioctls  than the alternatives and the ioctls can easily
> be hidden by a user space library as apparently
> Shirish did in a prototype and saves having to write a new cifs client
> from scratch.
>

You don't need to write a new cifs client from scratch, there are
libraries that can take most of that work off of your hands
(libsmbclient, as an example though there are others).

You've just articulated the main reason that I feel that a
filesystem-based interface to this code is a poor fit. Rather than
doing this:

WINE -> VFS interface -> Wire Protocol

...it makes much more sense to do this:

WINE -> Pipe Daemon -> Wire Protocol

The inteface between WINE and the pipe daemon can look however you
wish, and you're not going to have to implement ioctls here.

Even if you do this, and hide all of the ioctl ugliness behind a
userspace library it's still not at all clear to me how you intend to
use this code. The question I still have is this:

"In the likely event that WINE wants to talk to a host to which you have
no preexisting CIFS mount, what's going to set up the mount to the IPC$
share?"

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Shirish Pargaonkar]

On Mon, Oct 19, 2009 at 9:04 AM, Jeff Layton <jlayton@...> wrote:

> On Sun, 18 Oct 2009 20:22:41 -0500
> Steve French <smfrench@...> wrote:
>
>> On Sun, Oct 18, 2009 at 7:12 PM, Christoph Hellwig <hch@...> wrote:
>> > On Fri, Oct 16, 2009 at 09:05:18AM -0400, Jeff Layton wrote:
>> >> Well...all of this code seems to presuppose that there will be actual
>> >> filesystem objects on which to operate. IOW, you can't call an ioctl on
>> >> a non-existent filehandle.
>> >>
>> >> Again, it's not at all clear from this patchset how userspace will
>> >> actually use this code. How do you envision userspace programs actually
>> >> making use of this?
>> >
>> > Yeah.  I have no problem adding kernel support for the named pipes to
>> > the kernel if we really need to, but:
>> >
>> >  - please odn't shove it into a filesystem as that's a totally unrelated
>> >   abstraction
>> >  - write some proper code for it, not the thing that was submitted
>> >   adding random ioctls dereferecing user pointers and similar.
>> >   Incidentally moving it out of cifs will make that a lot easier as
>> >   bad code doesn't really stick out in cifs.
>>
>> Problem with making them via something other than an ioctl is -
>> they are handle based ops and operations on pipes
>> are the same on the wire as open/close/read/write
>> operations on files so the majority of this already works (these
>> are the client not server side of pipe operations Shirish
>> is talking about).   The missing operations
>> are handle based (e.g. transact and get/set named pipe info)
>> so ioctls seem reasonable and  I don't know of an obvious
>> alternative to ioctl, to do handle based operations.   Certainly it is much
>> smaller to do a few ioctls  than the alternatives and the ioctls can easily
>> be hidden by a user space library as apparently
>> Shirish did in a prototype and saves having to write a new cifs client
>> from scratch.
>>
>
> You don't need to write a new cifs client from scratch, there are
> libraries that can take most of that work off of your hands
> (libsmbclient, as an example though there are others).
>
> You've just articulated the main reason that I feel that a
> filesystem-based interface to this code is a poor fit. Rather than
> doing this:
>
> WINE -> VFS interface -> Wire Protocol
>
> ...it makes much more sense to do this:
>
> WINE -> Pipe Daemon -> Wire Protocol
>
> The inteface between WINE and the pipe daemon can look however you
> wish, and you're not going to have to implement ioctls here.
>
> Even if you do this, and hide all of the ioctl ugliness behind a
> userspace library it's still not at all clear to me how you intend to
> use this code. The question I still have is this:
>
> "In the likely event that WINE wants to talk to a host to which you have
> no preexisting CIFS mount, what's going to set up the mount to the IPC$
> share?"
>
> --
> Jeff Layton <jlayton@...>
>

Jeff,

Couple ot fhings.

1. There would/could be other apps besides WINE which interface with named pipes
2. Will work on getting the ioctls code coded better
3. Not sure yet, but this is what could happen (still nebulous about
the implementation)
   We have a /cifsipc directory created when cifs is loaded
    When a user using an app (such as WINE) opens a pipe, lookup
triggers cifs upcall
      to get credentials for session setup and tree connect.
    So can then have pseudofiles like /cifsipc/servername/user_id as a
result of nt create andx

Basically what a user space library does for utlities like smbclient,
can same not be done in
the kernel? Do not think there will be too much new code here.
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Volker Lendecke]

On Mon, Oct 19, 2009 at 09:42:52AM -0500, Shirish Pargaonkar wrote:
> Basically what a user space library does for utlities like smbclient,
> can same not be done in
> the kernel? Do not think there will be too much new code here.

Let me ask another question: Why do you insist on putting
this into the kernel? What's the deeper problem that you can
not solve in user space?

Volker


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Mon, 19 Oct 2009 09:42:52 -0500
Shirish Pargaonkar <shirishpargaonkar@...> wrote:

> On Mon, Oct 19, 2009 at 9:04 AM, Jeff Layton <jlayton@...> wrote:
> > On Sun, 18 Oct 2009 20:22:41 -0500
> > Steve French <smfrench@...> wrote:
> >
> >> On Sun, Oct 18, 2009 at 7:12 PM, Christoph Hellwig <hch@...> wrote:
> >> > On Fri, Oct 16, 2009 at 09:05:18AM -0400, Jeff Layton wrote:
> >> >> Well...all of this code seems to presuppose that there will be actual
> >> >> filesystem objects on which to operate. IOW, you can't call an ioctl on
> >> >> a non-existent filehandle.
> >> >>
> >> >> Again, it's not at all clear from this patchset how userspace will
> >> >> actually use this code. How do you envision userspace programs actually
> >> >> making use of this?
> >> >
> >> > Yeah.  I have no problem adding kernel support for the named pipes to
> >> > the kernel if we really need to, but:
> >> >
> >> >  - please odn't shove it into a filesystem as that's a totally unrelated
> >> >   abstraction
> >> >  - write some proper code for it, not the thing that was submitted
> >> >   adding random ioctls dereferecing user pointers and similar.
> >> >   Incidentally moving it out of cifs will make that a lot easier as
> >> >   bad code doesn't really stick out in cifs.
> >>
> >> Problem with making them via something other than an ioctl is -
> >> they are handle based ops and operations on pipes
> >> are the same on the wire as open/close/read/write
> >> operations on files so the majority of this already works (these
> >> are the client not server side of pipe operations Shirish
> >> is talking about).   The missing operations
> >> are handle based (e.g. transact and get/set named pipe info)
> >> so ioctls seem reasonable and  I don't know of an obvious
> >> alternative to ioctl, to do handle based operations.   Certainly it is much
> >> smaller to do a few ioctls  than the alternatives and the ioctls can easily
> >> be hidden by a user space library as apparently
> >> Shirish did in a prototype and saves having to write a new cifs client
> >> from scratch.
> >>
> >
> > You don't need to write a new cifs client from scratch, there are
> > libraries that can take most of that work off of your hands
> > (libsmbclient, as an example though there are others).
> >
> > You've just articulated the main reason that I feel that a
> > filesystem-based interface to this code is a poor fit. Rather than
> > doing this:
> >
> > WINE -> VFS interface -> Wire Protocol
> >
> > ...it makes much more sense to do this:
> >
> > WINE -> Pipe Daemon -> Wire Protocol
> >
> > The inteface between WINE and the pipe daemon can look however you
> > wish, and you're not going to have to implement ioctls here.
> >
> > Even if you do this, and hide all of the ioctl ugliness behind a
> > userspace library it's still not at all clear to me how you intend to
> > use this code. The question I still have is this:
> >
> > "In the likely event that WINE wants to talk to a host to which you have
> > no preexisting CIFS mount, what's going to set up the mount to the IPC$
> > share?"
> >
> > --
> > Jeff Layton <jlayton@...>
> >
>
> Jeff,
>
> Couple ot fhings.
>
> 1. There would/could be other apps besides WINE which interface with named pipes

Sure, ok. WINE is the obvious user of the named pipe code, but I
suppose there can be others. Still, you have the same problem of how to
handle this "mounting".

> 2. Will work on getting the ioctls code coded better

I don't have a problem with the ioctl coding itself (honestly, I
haven't reviewed that part of it). My issue is rather with the entire
approach of trying to shoehorn this stuff into the kernel's VFS API.
It's a kludgey solution that hamstrings you later if you decide that it
needs changes.

> 3. Not sure yet, but this is what could happen (still nebulous about
> the implementation)
>    We have a /cifsipc directory created when cifs is loaded
>     When a user using an app (such as WINE) opens a pipe, lookup
> triggers cifs upcall
>       to get credentials for session setup and tree connect.
>     So can then have pseudofiles like /cifsipc/servername/user_id as a
> result of nt create andx
>
> Basically what a user space library does for utlities like smbclient,
> can same not be done in
> the kernel? Do not think there will be too much new code here.

You can pretty much do anything you want in the kernel, in principle.
The question is whether you should. Given that this code is going to be
essentially useless without a userspace library, putting this in the
kernel seems to have more disadvantages than advantages.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Mon, Oct 19, 2009 at 9:56 AM, Volker Lendecke
<Volker.Lendecke@...> wrote:
> On Mon, Oct 19, 2009 at 09:42:52AM -0500, Shirish Pargaonkar wrote:
>> Basically what a user space library does for utlities like smbclient,
>> can same not be done in
>> the kernel? Do not think there will be too much new code here.
>
> Let me ask another question: Why do you insist on putting
> this into the kernel? What's the deeper problem that you can
> not solve in user space?

a few quick points:
1) Easier: Even without extending cifs.ko with anny new ioctls, we
already can do simple RPC operations over existing cifs.ko today (with
prototype user space code that does the DCE part bind etc.).
Currently there is no user space SMB client library that could be used
broadly by Linux applications and tools, although libsmbclient can be
used by those which are gpl.  This has been a huge headache over the
years (and causes distros to ship conflicting levels of samba
libraries, and still not address the problem).  After long discussions
with jra, gd, and others, I don't see any way that we can convert
libsmbclient and libnet to lgpl.  At least two developers told me they
didn't like doing userspace IPC to a new gpl libsmbclient daemon to
avoid GPL (even if it could be made stable).   Easiest to simply use
the existing kernel cifs code to use cifs file/pipe support - and just
work on the user space rpc pieces by adding a new lgpl library for rpc
to Samba client (user space).  Shirish prototyped this for - it uses
existing cifs.ko, but user space code for RPC.  For example - if you
want to list what servers or shares are available (e.g. in
mount.cifs), why use different code to list the shares and connect to
the shares?  Those pipe examples already basically work with current
cifs code (without the ioctls) but Luke's comments mentioned that the
PeekNamedPIpe and GetHandleNmPipe SMBs (which were part of Shirish's
small ioctl patch) would be useful for others so thus the motivation
for Shirish's patch (after he read the Wiki on the topic).

2) Affinity with other file objects: Although not available in Linux -
named pipes are treated as file objects on every other OS since they
were invented (in OS/2), and so the client side is implemented in
kernel in all the cases I can think of (Windows, OS/2 etc.).  Other
pseudofiles (such as fifos, char devices etc. created in Linux via
mknod) are treated as file objects today in kernel.  The client side
named pipe code is of course small, since it is just file operations
over SMB/CIFS (or SMB2).   The server side is already implemented in
Samba server.

3) Usability: Since the file system client does not use the server's
(or smbclient's) smb.conf file for configuration settings (security
levels, signing etc.), it can be confusing e.g. to list shares with
smbclient (with looser security settings) and not be able to connect
to the share (with cifs.ko using stricter security settings).

4) Performance: The cifs kernel client can connect to a server and
keep a session - but with smbclient and the libsmbclient model in
general you can end up reconnecting dozens of times to the server from
the same client (even though a session to the server already exists in
kernel).  In addition connecting via to the same server from via
smbclient from (when you already e.g. have a kernel mount to it) can
cause the "SMB NAT problem" in some versions of Windows.

--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Gerald Carter-3]

Steve,

Just a few points.  Feel free to ignore or respond.  The only
point I'm making here is that other solutions do exist already.
I'm not advocating our code as a solution for this debate.
Just pointing out that this is a well traveled problem (as you
know).


Steve French wrote:

> a few quick points:
> 1) Easier: Even without extending cifs.ko with anny new ioctls, we
> already can do simple RPC operations over existing cifs.ko today (with
> prototype user space code that does the DCE part bind etc.).
> Currently there is no user space SMB client library that could be used
> broadly by Linux applications and tools, although libsmbclient can be
> used by those which are gpl.

The Likewise rdr driver inside lwiod already does all this. [1]
The client library is LGPLv2.1+.  However, it does require IPC
to a locally running daemon so it does handle the credential/connection
caching.


> 2) Affinity with other file objects: Although not available
> in Linux -
...
> 3) Usability: Since the file system client does not use
> the server's (or smbclient's) smb.conf file for configuration
> settings (security levels, signing etc.), it can be confusing
> e.g. to list shares with smbclient (with looser security
> settings) and not be able to connect to the share (with cifs.ko
> using stricter security settings).

Ditto.

> 4) Performance: The cifs kernel client can connect to a server and
> keep a session - but with smbclient and the libsmbclient model in
> general you can end up reconnecting dozens of times to the server from
> the same client (even though a session to the server already exists in
> kernel).  In addition connecting via to the same server from via
> smbclient from (when you already e.g. have a kernel mount to it) can
> cause the "SMB NAT problem" in some versions of Windows.

Connection caching I will agree on which I mentioned in point #1.
Although I don't believe throughput is really an issue on named
pipes so that part of performance is not valid IMO.




cheers, jerry
--
[1]  http://git.likewiseopen.org/?p=likewise-open.git;a=summary






_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Mon, Oct 19, 2009 at 1:54 PM, Gerald Carter <jerry@...> wrote:
> Steve,
>
> Just a few points.  Feel free to ignore or respond.  The only
> point I'm making here is that other solutions do exist already.
> I'm not advocating our code as a solution for this debate.
> Just pointing out that this is a well traveled problem (as you
> know).

I agree that Likewise librpc would address some of this,
but haven't seen any progress in getting likewise into other distros
directly (or indirectly by moving it's librpc into samba).  Any progress
on getting this into more major distros?

I would much prefer to map an existing lgpl or bsd "librpc" (preferably to
existing cifs code when rpc is not going over tcp), but for the purposes
of a "system library" for common tasks, we need something in all of
the distros.

> Steve French wrote:
>
>> a few quick points:
>> 1) Easier: Even without extending cifs.ko with anny new ioctls, we
>> already can do simple RPC operations over existing cifs.ko today (with
>> prototype user space code that does the DCE part bind etc.).
>> Currently there is no user space SMB client library that could be used
>> broadly by Linux applications and tools, although libsmbclient can be
>> used by those which are gpl.
>
> The Likewise rdr driver inside lwiod already does all this. [1]
> The client library is LGPLv2.1+.  However, it does require IPC
> to a locally running daemon so it does handle the credential/connection
> caching.

IPC to Winbind or to Likewise daemon for credentials (once) is fine,
although it would be better IMO to cache credentials using Dave
Howell's existing kernel mechanism for caching keys.

>> 2) Affinity with other file objects: Although not available
>> in Linux -
> ...
>> 3) Usability: Since the file system client does not use
>> the server's (or smbclient's) smb.conf file for configuration
>> settings (security levels, signing etc.), it can be confusing
>> e.g. to list shares with smbclient (with looser security
>> settings) and not be able to connect to the share (with cifs.ko
>> using stricter security settings).
>
> Ditto.
>
>> 4) Performance: The cifs kernel client can connect to a server and
>> keep a session - but with smbclient and the libsmbclient model in
>> general you can end up reconnecting dozens of times to the server from
>> the same client (even though a session to the server already exists in
>> kernel).  In addition connecting via to the same server from via
>> smbclient from (when you already e.g. have a kernel mount to it) can
>> cause the "SMB NAT problem" in some versions of Windows.
>
> Connection caching I will agree on which I mentioned in point #1.
> Although I don't believe throughput is really an issue on named
> pipes so that part of performance is not valid IMO.

I have seen examples where the redundant session setups etc. does make
a difference (server to server migration moves a lot of rpc traffic,
and using an existing session can cut the traffic by more than 1/2).
Certainly the load on Samba server (in particular) by running
extra processes per client can be significant when multiplied
across many clients to same server.

--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Gerald Carter-3]

Steve French wrote:

> I agree that Likewise librpc would address some of this,
> but haven't seen any progress in getting likewise into other distros
> directly (or indirectly by moving it's librpc into samba).  Any
> progress on getting this into more major distros?

My understanding is that Ubuntu will be moving to 5.3 in Karmic+1.
We're working on the Likewiee 4.x to 5.x package upgrade issues now.
I believe that packages for openSUSE and Fedora are being prepared
currently (wuth the intent of pushing them to the distro upstream
for inclusion) but I'm not on that detail anymore so I don't
an ETA.

> IPC to Winbind or to Likewise daemon for credentials (once) is fine,
> although it would be better IMO to cache credentials using Dave
> Howell's existing kernel mechanism for caching keys.

We each have our own issues to solve.  I need a generic way for
all applications to use gss-ntlm without having to require kernel
upgrades on Linux and on non-linux platforms.  I understand
you point, but I'll concede that that reusing an existing kernel
interface for creds is not as important in my mind (at the
moment) as a standard interface that is widely available cross
platform.

> I have seen examples where the redundant session setups etc. does make
> a difference (server to server migration moves a lot of rpc traffic,
> and using an existing session can cut the traffic by more than 1/2).
> Certainly the load on Samba server (in particular) by running
> extra processes per client can be significant when multiplied
> across many clients to same server.

Yeah.  I agree about connection caching.  I was sayiong that I don't
believe I.O through on named pipes to be a real issue.   But if
someone has a real world example where I/O on a named pipe is a deal
killer, I'll happily reconsider.

But back to the issue at hand, I was only pointing out that a
user space named pipe client library exists in various forms.
Neither Samba nor Likewise are alone in this regard, and a new
client implementation is not hard to write.

What I've not really heard here, which I think is what Jeff is
trying to get answered, is why a user space client lib implementation
(that does connection caching) is not acceptable for Shirish's
use case.




cheers, jerry
--
=====================================================================
http://www.plainjoe.org/
"What man is a man who does not make the world better?"      --Balian



_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Mon, Oct 19, 2009 at 2:40 PM, Gerald Carter <jerry@...> wrote:
> Steve French wrote:
>
>> I agree that Likewise librpc would address some of this,

> What I've not really heard here, which I think is what Jeff is
> trying to get answered, is why a user space client lib implementation
> (that does connection caching) is not acceptable for Shirish's
> use case.

I think that is the point - the kernel implementation in effect does
cifs connections and connection caching already (in Windows,
in Linux cifs.ko, even as far back as OS/2).   There are
various userspace rpc libraries (each with their own problems
which have been mentioned), but any or all of them could
with trivial change work today with existing cifs kernel sessions.

There is a distinct point Jeff and others have made about
credentials, and automatic use of different credentials than the
logged on user, and what Jeff would like to eventually see
for cifs.ko and others pulling credentials from Winbind, but
that is not really related to the pipe discussion.  LikewiseOpen
as you note has a different caching infrastructure for credentials
but as long as kernel mounts and user tools can get at the
same credentials, its upcall addresses similar credential
problem - but this is not specific to pipes but general
problem with cifs tools using the right credentials when
contacting a remote servr.

--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Steve French-2]

On Mon, Oct 19, 2009 at 2:25 PM, Steve French <smfrench@...> wrote:

> On Mon, Oct 19, 2009 at 1:54 PM, Gerald Carter <jerry@...> wrote:
>> Steve,
>>
>> Just a few points.  Feel free to ignore or respond.  The only
>> point I'm making here is that other solutions do exist already.
>> I'm not advocating our code as a solution for this debate.
>> Just pointing out that this is a well traveled problem (as you
>> know).
>
> I agree that Likewise librpc would address some of this,

After looking at Jerry's pointer, and to clarify my earlier comment
about the librpc.  The likewise code doesn't look it would help,
especially since it has dependencies on pieces that conflict with
other parts of Linux. The Likewise librpc doesn't appear to map to
Samba's libsmbclient (as Rafal's pitch showed it working in the past
through npmpx), and so doesn't solve the problem for Samba, and it
can't map pipe open/close/read/write through the cifs vfs (so can't
address the problem I mentioned where "NetShareEnum" shows shares on
servers which mount can't reach).   Looking at lwio in more detail, it
is a FUSE module with an upcall to the smb handling code, so can't
handle high stress workloads (a file system which allocates memory in
userspace, or upcalls to userspace, especially in the write patch can
deadlock as we have easily shown in stress testing in the past with
various FS).  FUSE (and thus based user space file systems like
lwio)ares also missing various kernel entry points and opens up bigger
race conditions on calls which are atomic in libc, but not in the vfs
layer.   It also looks like the lw librpc is tightly coupled to the
other pieces of Likewise (which we don't want to get in the way of
Samba and winbind in particular).


--
Thanks,

Steve
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client