CIFS slow on gigabit, doesn't support sockopt=TCP_NODELAY ?

[Timothy Normand Miller]

Hi, everyone.

I've noticed that the cifs client for Linux is slow over gigabit
ethernet.  It seems to max out at about 10 megs/sec, while the drives
can go a lot faster.  Also, if I mount the same network share from a
Mac, it's a lot faster.  When _sharing_ via samba, one can set the
TCP_NODELAY option (among others), which fixes the problem.  But with
the cifs client, I find that there appears to be no way to set the
option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
and you can also put that into /etc/fstab.  Either way, the option
appears to be ignored.

I filed this bug with Gentoo:  http://bugs.gentoo.org/265183

Am I doing this wrong?  Is there a work-around?  Or plans to fix it?

It was actually suggested that I ask on samba-technical, but I thought
I'd start here.  Should I ask on that list?


Thanks!

--
Timothy Normand Miller
http://www.cse.ohio-state.edu/~millerti
Open Graphics Project
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Sun, 23 Aug 2009 20:21:10 -0400
Timothy Normand Miller <theosib@...> wrote:

> Hi, everyone.
>
> I've noticed that the cifs client for Linux is slow over gigabit
> ethernet.  It seems to max out at about 10 megs/sec, while the drives
> can go a lot faster.  Also, if I mount the same network share from a
> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
> TCP_NODELAY option (among others), which fixes the problem.  But with
> the cifs client, I find that there appears to be no way to set the
> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
> and you can also put that into /etc/fstab.  Either way, the option
> appears to be ignored.
>

Yeah. The sockopt= option is completely ignored. I proposed a patch
to remove it not too long ago...

I'm unclear...so you're saying here that turning on TCP_NODELAY on the
server it helps throughput when using the Linux CIFS client?

It seems to be that there are probably bottlenecks you'll hit sooner
before Nagle's algorithm becomes an issue (lack of parallelism in
writes for one thing), but it may be worth experimenting with that flag
if you've seen a measurable throughput increase.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Volker Lendecke]

On Mon, Aug 24, 2009 at 09:50:20AM -0400, Jeff Layton wrote:

> Yeah. The sockopt= option is completely ignored. I proposed a patch
> to remove it not too long ago...
>
> I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> server it helps throughput when using the Linux CIFS client?
>
> It seems to be that there are probably bottlenecks you'll hit sooner
> before Nagle's algorithm becomes an issue (lack of parallelism in
> writes for one thing), but it may be worth experimenting with that flag
> if you've seen a measurable throughput increase.

While there... Any updates on the parallel reads and writes
we discussed a couple of months ago?

Volker


_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Mon, 24 Aug 2009 16:02:48 +0200
Volker Lendecke <Volker.Lendecke@...> wrote:

> On Mon, Aug 24, 2009 at 09:50:20AM -0400, Jeff Layton wrote:
> > Yeah. The sockopt= option is completely ignored. I proposed a patch
> > to remove it not too long ago...
> >
> > I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> > server it helps throughput when using the Linux CIFS client?
> >
> > It seems to be that there are probably bottlenecks you'll hit sooner
> > before Nagle's algorithm becomes an issue (lack of parallelism in
> > writes for one thing), but it may be worth experimenting with that flag
> > if you've seen a measurable throughput increase.
>
> While there... Any updates on the parallel reads and writes
> we discussed a couple of months ago?
>

Not from me. Would still definitely be a nice-to-have.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Timothy Normand Miller]

On Mon, Aug 24, 2009 at 9:50 AM, Jeff Layton<jlayton@...> wrote:

> On Sun, 23 Aug 2009 20:21:10 -0400
> Timothy Normand Miller <theosib@...> wrote:
>
>> Hi, everyone.
>>
>> I've noticed that the cifs client for Linux is slow over gigabit
>> ethernet.  It seems to max out at about 10 megs/sec, while the drives
>> can go a lot faster.  Also, if I mount the same network share from a
>> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
>> TCP_NODELAY option (among others), which fixes the problem.  But with
>> the cifs client, I find that there appears to be no way to set the
>> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
>> and you can also put that into /etc/fstab.  Either way, the option
>> appears to be ignored.
>>
>
> Yeah. The sockopt= option is completely ignored. I proposed a patch
> to remove it not too long ago...
>
> I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> server it helps throughput when using the Linux CIFS client?

No.  I think I'll give you excess detail just to make sure that I
don't leave out something important.  :)

I have a Linux server, an Apple AEBS with disks hanging off of it, and
various clients (some Macs).  On the Linux server, I can configure the
SMB server to enable those options so that when a client accesses the
server, it's fast.  The AEBS is also a server, but I cannot configure
it with those options.  I can get a Mac to get fast access to the AEBS
by configuring those TCP options on the client side.

The problem pertains to having the Linux box access the disks on the
AEBS.  In that case, I can't configure the TCP options on either the
client (the Linux box in this case) or the server (the AEBS), so
transfers are slow.  This is important, because a large part of my
large file copies are between the Linux box and the AEBS.

On the Linux server, SMB configuration has no impact on CIFS performance.

>
> It seems to be that there are probably bottlenecks you'll hit sooner
> before Nagle's algorithm becomes an issue (lack of parallelism in
> writes for one thing), but it may be worth experimenting with that flag
> if you've seen a measurable throughput increase.

Yeah.  I've noticed that multiple independent writers are serialized.
I can't tell if reading and writing interfere.

Thanks!


--
Timothy Normand Miller
http://www.cse.ohio-state.edu/~millerti
Open Graphics Project
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Mon, 24 Aug 2009 14:36:37 -0400
Timothy Normand Miller <theosib@...> wrote:

> On Mon, Aug 24, 2009 at 9:50 AM, Jeff Layton<jlayton@...> wrote:
> > On Sun, 23 Aug 2009 20:21:10 -0400
> > Timothy Normand Miller <theosib@...> wrote:
> >
> >> Hi, everyone.
> >>
> >> I've noticed that the cifs client for Linux is slow over gigabit
> >> ethernet.  It seems to max out at about 10 megs/sec, while the drives
> >> can go a lot faster.  Also, if I mount the same network share from a
> >> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
> >> TCP_NODELAY option (among others), which fixes the problem.  But with
> >> the cifs client, I find that there appears to be no way to set the
> >> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
> >> and you can also put that into /etc/fstab.  Either way, the option
> >> appears to be ignored.
> >>
> >
> > Yeah. The sockopt= option is completely ignored. I proposed a patch
> > to remove it not too long ago...
> >
> > I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> > server it helps throughput when using the Linux CIFS client?
>
> No.  I think I'll give you excess detail just to make sure that I
> don't leave out something important.  :)
>
> I have a Linux server, an Apple AEBS with disks hanging off of it, and
> various clients (some Macs).  On the Linux server, I can configure the
> SMB server to enable those options so that when a client accesses the
> server, it's fast.  The AEBS is also a server, but I cannot configure
> it with those options.  I can get a Mac to get fast access to the AEBS
> by configuring those TCP options on the client side.
>
> The problem pertains to having the Linux box access the disks on the
> AEBS.  In that case, I can't configure the TCP options on either the
> client (the Linux box in this case) or the server (the AEBS), so
> transfers are slow.  This is important, because a large part of my
> large file copies are between the Linux box and the AEBS.
>
> On the Linux server, SMB configuration has no impact on CIFS performance.
>

If you wanted to experiment with it, it doesn't sound too hard to hack
up a patch to do so. You'd just need to add a kernel_setsockopt call to
the ipv4_connect (and/or ipv6_connect) functions. The place where the
sk_rcvtimeo/sk_sndtimeo get set is probably around the right spot. In
fact, those should probably be changed to kernel_setsockopt calls too.

If that shows notable improvement then we could discuss the best way to
set up an option for it. Probably we'd want to flesh out the sockopt=
option.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Timothy Normand Miller]

On Mon, Aug 24, 2009 at 3:42 PM, Jeff Layton<jlayton@...> wrote:

> On Mon, 24 Aug 2009 14:36:37 -0400
> Timothy Normand Miller <theosib@...> wrote:
>
>> On Mon, Aug 24, 2009 at 9:50 AM, Jeff Layton<jlayton@...> wrote:
>> > On Sun, 23 Aug 2009 20:21:10 -0400
>> > Timothy Normand Miller <theosib@...> wrote:
>> >
>> >> Hi, everyone.
>> >>
>> >> I've noticed that the cifs client for Linux is slow over gigabit
>> >> ethernet.  It seems to max out at about 10 megs/sec, while the drives
>> >> can go a lot faster.  Also, if I mount the same network share from a
>> >> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
>> >> TCP_NODELAY option (among others), which fixes the problem.  But with
>> >> the cifs client, I find that there appears to be no way to set the
>> >> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
>> >> and you can also put that into /etc/fstab.  Either way, the option
>> >> appears to be ignored.
>> >>
>> >
>> > Yeah. The sockopt= option is completely ignored. I proposed a patch
>> > to remove it not too long ago...
>> >
>> > I'm unclear...so you're saying here that turning on TCP_NODELAY on the
>> > server it helps throughput when using the Linux CIFS client?
>>
>> No.  I think I'll give you excess detail just to make sure that I
>> don't leave out something important.  :)
>>
>> I have a Linux server, an Apple AEBS with disks hanging off of it, and
>> various clients (some Macs).  On the Linux server, I can configure the
>> SMB server to enable those options so that when a client accesses the
>> server, it's fast.  The AEBS is also a server, but I cannot configure
>> it with those options.  I can get a Mac to get fast access to the AEBS
>> by configuring those TCP options on the client side.
>>
>> The problem pertains to having the Linux box access the disks on the
>> AEBS.  In that case, I can't configure the TCP options on either the
>> client (the Linux box in this case) or the server (the AEBS), so
>> transfers are slow.  This is important, because a large part of my
>> large file copies are between the Linux box and the AEBS.
>>
>> On the Linux server, SMB configuration has no impact on CIFS performance.
>>
>
> If you wanted to experiment with it, it doesn't sound too hard to hack
> up a patch to do so. You'd just need to add a kernel_setsockopt call to
> the ipv4_connect (and/or ipv6_connect) functions. The place where the
> sk_rcvtimeo/sk_sndtimeo get set is probably around the right spot. In
> fact, those should probably be changed to kernel_setsockopt calls too.
>
> If that shows notable improvement then we could discuss the best way to
> set up an option for it. Probably we'd want to flesh out the sockopt=
> option.

I don't have much experience with Linux kernel drivers, so maybe you
can give me some hints.

(1) There's a userspace component (mount.cifs).  Is the problem in there?
(2) Or is it in the kernel driver?
(3) How are the options parsed?  Is this done for us?  That is, do I
need to look at existing data structures, or do I need to add some
parsing code?
(4) Is it okay to discuss this on this list, or should I move to a
different one?

Thanks.

--
Timothy Normand Miller
http://www.cse.ohio-state.edu/~millerti
Open Graphics Project
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Mon, 24 Aug 2009 16:29:52 -0400
Timothy Normand Miller <theosib@...> wrote:

> On Mon, Aug 24, 2009 at 3:42 PM, Jeff Layton<jlayton@...> wrote:
> > On Mon, 24 Aug 2009 14:36:37 -0400
> > Timothy Normand Miller <theosib@...> wrote:
> >
> >> On Mon, Aug 24, 2009 at 9:50 AM, Jeff Layton<jlayton@...> wrote:
> >> > On Sun, 23 Aug 2009 20:21:10 -0400
> >> > Timothy Normand Miller <theosib@...> wrote:
> >> >
> >> >> Hi, everyone.
> >> >>
> >> >> I've noticed that the cifs client for Linux is slow over gigabit
> >> >> ethernet.  It seems to max out at about 10 megs/sec, while the drives
> >> >> can go a lot faster.  Also, if I mount the same network share from a
> >> >> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
> >> >> TCP_NODELAY option (among others), which fixes the problem.  But with
> >> >> the cifs client, I find that there appears to be no way to set the
> >> >> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
> >> >> and you can also put that into /etc/fstab.  Either way, the option
> >> >> appears to be ignored.
> >> >>
> >> >
> >> > Yeah. The sockopt= option is completely ignored. I proposed a patch
> >> > to remove it not too long ago...
> >> >
> >> > I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> >> > server it helps throughput when using the Linux CIFS client?
> >>
> >> No.  I think I'll give you excess detail just to make sure that I
> >> don't leave out something important.  :)
> >>
> >> I have a Linux server, an Apple AEBS with disks hanging off of it, and
> >> various clients (some Macs).  On the Linux server, I can configure the
> >> SMB server to enable those options so that when a client accesses the
> >> server, it's fast.  The AEBS is also a server, but I cannot configure
> >> it with those options.  I can get a Mac to get fast access to the AEBS
> >> by configuring those TCP options on the client side.
> >>
> >> The problem pertains to having the Linux box access the disks on the
> >> AEBS.  In that case, I can't configure the TCP options on either the
> >> client (the Linux box in this case) or the server (the AEBS), so
> >> transfers are slow.  This is important, because a large part of my
> >> large file copies are between the Linux box and the AEBS.
> >>
> >> On the Linux server, SMB configuration has no impact on CIFS performance.
> >>
> >
> > If you wanted to experiment with it, it doesn't sound too hard to hack
> > up a patch to do so. You'd just need to add a kernel_setsockopt call to
> > the ipv4_connect (and/or ipv6_connect) functions. The place where the
> > sk_rcvtimeo/sk_sndtimeo get set is probably around the right spot. In
> > fact, those should probably be changed to kernel_setsockopt calls too.
> >
> > If that shows notable improvement then we could discuss the best way to
> > set up an option for it. Probably we'd want to flesh out the sockopt=
> > option.
>
> I don't have much experience with Linux kernel drivers, so maybe you
> can give me some hints.
>
> (1) There's a userspace component (mount.cifs).  Is the problem in there?

Nope. Sockets are opened and configured within the kernel.

> (2) Or is it in the kernel driver?

Yep.

> (3) How are the options parsed?  Is this done for us?  That is, do I
> need to look at existing data structures, or do I need to add some
> parsing code?

I wouldn't worry about option parsing at the moment. I'd just hardcode
those functions (ipv4_connect and/or ipv6_connect) to call
kernel_setsockopt for TCP_NODELAY. Once you've tested it, if you find
that it's helpful then you can worry about cleaning it up and adding a
mount option (or doing something else).

kernel_setsockopt is somewhat similar to its userspace counterpart.
There are examples of its use in other areas of the kernel.

> (4) Is it okay to discuss this on this list, or should I move to a
> different one?
>

Yes. This is the place to discuss the linux-cifs client development.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Timothy Normand Miller]

I have another question.  Setting TCP_NODELAY and IPTOS_LOWDELAY seem
to affect different levels of the protocol stack.  Consider this bit
of code I googled:

#ifdef IPTOS_LOWDELAY
      int lowdelay = IPTOS_LOWDELAY;
      if (setsockopt(connection_in, IPPROTO_IP, IP_TOS, (void *)&lowdelay,
                     sizeof(lowdelay)) < 0)
        error("setsockopt IPTOS_LOWDELAY: %.100s", strerror(errno));
#endif /* IPTOS_LOWDELAY */
#if defined(TCP_NODELAY) && defined(ENABLE_TCP_NODELAY)
      if (setsockopt(connection_in, IPPROTO_TCP, TCP_NODELAY, (void *)&on,
                     sizeof(on)) < 0)
        error("setsockopt TCP_NODELAY: %.100s", strerror(errno));
#endif /* TCP_NODELAY */

I figured out how to set TCP_NODELAY in the kernel, because I saw an
example.  So I do this in ipv4_connect:

        val = 1;
        socket->ops->setsockopt(socket, SOL_TCP, TCP_NODELAY,
                                            (char __user *)&val, sizeof(val));

Thus, for IPTOS_LOWDELAY, would I do this?

        val = IPTOS_LOWDELAY;
        socket->ops->setsockopt(socket, SOL_IP, IP_TOS,
                                            (char __user *)&val, sizeof(val));


Thanks.
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Mon, 24 Aug 2009 23:10:15 -0400
Timothy Normand Miller <theosib@...> wrote:

> I have another question.  Setting TCP_NODELAY and IPTOS_LOWDELAY seem
> to affect different levels of the protocol stack.  Consider this bit
> of code I googled:
>
> #ifdef IPTOS_LOWDELAY
>       int lowdelay = IPTOS_LOWDELAY;
>       if (setsockopt(connection_in, IPPROTO_IP, IP_TOS, (void *)&lowdelay,
>                      sizeof(lowdelay)) < 0)
>         error("setsockopt IPTOS_LOWDELAY: %.100s", strerror(errno));
> #endif /* IPTOS_LOWDELAY */
> #if defined(TCP_NODELAY) && defined(ENABLE_TCP_NODELAY)
>       if (setsockopt(connection_in, IPPROTO_TCP, TCP_NODELAY, (void *)&on,
>                      sizeof(on)) < 0)
>         error("setsockopt TCP_NODELAY: %.100s", strerror(errno));
> #endif /* TCP_NODELAY */
>
> I figured out how to set TCP_NODELAY in the kernel, because I saw an
> example.  So I do this in ipv4_connect:
>
>         val = 1;
>         socket->ops->setsockopt(socket, SOL_TCP, TCP_NODELAY,
>                                             (char __user *)&val, sizeof(val));
>

That'll probably work in most cases but it's more consistent and
correct to use kernel_setsockopt (it'll make sure the %fs register gets
set correctly for the call). The args are the same and it calls the
socket's setsockopt operation after doing some setup.


> Thus, for IPTOS_LOWDELAY, would I do this?
>
>         val = IPTOS_LOWDELAY;
>         socket->ops->setsockopt(socket, SOL_IP, IP_TOS,
>                                             (char __user *)&val, sizeof(val));
>
>

Hmm...never used setsockopt to set TOS bits, but I suppose it would
work. You could also do that with iptables rules too.

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Timothy Normand Miller]

On Tue, Aug 25, 2009 at 7:09 AM, Jeff Layton<jlayton@...> wrote:

>
>> Thus, for IPTOS_LOWDELAY, would I do this?
>>
>>         val = IPTOS_LOWDELAY;
>>         socket->ops->setsockopt(socket, SOL_IP, IP_TOS,
>>                                             (char __user *)&val, sizeof(val));
>>
>>
>
> Hmm...never used setsockopt to set TOS bits, but I suppose it would
> work. You could also do that with iptables rules too.

Sounds complicated.  Any hints you can give me about how to do it?




--
Timothy Normand Miller
http://www.cse.ohio-state.edu/~millerti
Open Graphics Project
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Timothy Normand Miller]

Here's my "patch" to fs/cifs/connect.c:

        /*
         * Eventually check for other socket options to change from
         *  the default. sock_setsockopt not used because it expects
         *  user space buffer
         */
        socket->sk->sk_rcvtimeo = 7 * HZ;
        socket->sk->sk_sndtimeo = 5 * HZ;

+        rc = kernel_setsockopt(socket, SOL_TCP, TCP_NODELAY, (char
__user *)&val, sizeof(val));
+        printk(KERN_WARNING "CIFS: TCP_NODELAY returns %d\n", rc);
+
+        val = IPTOS_LOWDELAY;
+        rc = kernel_setsockopt(socket, SOL_IP, IP_TOS, (char __user
*)&val, sizeof(val));
+        printk(KERN_WARNING "CIFS: IPTOS_LOWDELAY returns %d\n", rc);


Both calls return zero, possibly suggesting that they succeeded.  When
NODELAY alone, I didn't notice any improvement.  With both, I notice
roughly an 11% speedup, with improved _consistency_ in throughput as
measured by iozone.  This is still about 1/2 the speed I can get
accessing the same share from my Mac.

There are several possibilities that come to mind:

(1) [most probable] I'm doing it wrong.
(2) There's a bug or design defficiency in CIFS that's causing this
(3) The bottleneck(s) in CIFS are somewhere other than the networking
(4) The SMB protocol itself is deficient, and the Mac is using some
other protocol to access the share


The reason I suspected that TCP_NODELAY and IPTOS_LOWDELAY were
involved was that because I enabled those options on the Linux box's
SMB server, 10 megs/sec was the limit I could get between the Mac and
the Linux box over SMB.  When enabling those options, I then became
limited by the disks (typically about 50 megs/sec for writes, faster
for reads from the RAID1).  With the AEBS share, the speed was also
limited to about 10 megs/sec, which I didn't think was a coincidence.
Since the Mac can move more like 20 megs/sec to/from the AEBS (limited
by the fact that these are USB drives), my suspicion is that I simply
haven't done it right.

How can I be sure I'm setting these options correctly and on the right
socket and that it applies to both directions, etc?
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client

[Jeff Layton-2]

On Tue, 25 Aug 2009 15:45:19 -0400
Timothy Normand Miller <theosib@...> wrote:

> Here's my "patch" to fs/cifs/connect.c:
>
>         /*
>          * Eventually check for other socket options to change from
>          *  the default. sock_setsockopt not used because it expects
>          *  user space buffer
>          */
>         socket->sk->sk_rcvtimeo = 7 * HZ;
>         socket->sk->sk_sndtimeo = 5 * HZ;
>
> +        rc = kernel_setsockopt(socket, SOL_TCP, TCP_NODELAY, (char
> __user *)&val, sizeof(val));
> +        printk(KERN_WARNING "CIFS: TCP_NODELAY returns %d\n", rc);
> +
> +        val = IPTOS_LOWDELAY;
> +        rc = kernel_setsockopt(socket, SOL_IP, IP_TOS, (char __user
> *)&val, sizeof(val));
> +        printk(KERN_WARNING "CIFS: IPTOS_LOWDELAY returns %d\n", rc);
>
>
> Both calls return zero, possibly suggesting that they succeeded.  When
> NODELAY alone, I didn't notice any improvement.  With both, I notice
> roughly an 11% speedup, with improved _consistency_ in throughput as
> measured by iozone.  This is still about 1/2 the speed I can get
> accessing the same share from my Mac.
>

What exactly are you measuring here? Reads? Writes?

What about with just IPTOS_LOWDELAY alone? To answer your earlier
question, see the TOS mangle target in the iptables manpage, or just
google around for "iptables TOS bits".

> There are several possibilities that come to mind:
>
> (1) [most probable] I'm doing it wrong.

No, looks like you're setting the socket options correctly.

> (2) There's a bug or design defficiency in CIFS that's causing this
> (3) The bottleneck(s) in CIFS are somewhere other than the networking

That's what I'd suspect. Lack of parallelism in reads/writes is
probably the main bottleneck here, and not network throughput. Nagle's
algorithm in particular shouldn't be an issue at all if you're
keeping the socket stuffed with enough outgoing data.

> (4) The SMB protocol itself is deficient, and the Mac is using some
> other protocol to access the share
>

I doubt this is the case, but you should probably get familiar with
wireshark if you're going to poke around in this area.

>
> The reason I suspected that TCP_NODELAY and IPTOS_LOWDELAY were
> involved was that because I enabled those options on the Linux box's
> SMB server, 10 megs/sec was the limit I could get between the Mac and
> the Linux box over SMB.  When enabling those options, I then became
> limited by the disks (typically about 50 megs/sec for writes, faster
> for reads from the RAID1).  With the AEBS share, the speed was also
> limited to about 10 megs/sec, which I didn't think was a coincidence.
> Since the Mac can move more like 20 megs/sec to/from the AEBS (limited
> by the fact that these are USB drives), my suspicion is that I simply
> haven't done it right.
>
> How can I be sure I'm setting these options correctly and on the right
> socket and that it applies to both directions, etc?

--
Jeff Layton <jlayton@...>
_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@...
https://lists.samba.org/mailman/listinfo/linux-cifs-client