SMBD CPU climbs sky high when writing DPX files

[Andy Liebman]

Hello,

I'm sorry of this gets posted twice.  I submitted the same post about 12
hours ago and it seems it didn't go through.

I am experiencing a strange problem when writing (capturing) DPX video
files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd
process go from 9 percent CPU utilization to 100 percent CPU utilization
over the course of about 40 minutes.  When smbd reaches 100 percent, the
capture stops (drops frames). I have tested this with three different
filesystem formats (XFS, ext3 and ext4) with similar -- although
slightly different -- results.

My test systems have been running the following:

2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2
2.6.28-11 Ubuntu 9.04 with Samba 3.3.2
2.6.27.38 kernel with Samba 3.0.23d
2.6.20.15 kernel with Samba 3.0.23d

For those of you who may not be familiar with the DPX format, unlike
QuickTime or other "streaming" media formats, DPX is "one file per
frame". So, a 20 minute DPX movie in the US "NTSC TV" format would
consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all
written into one directory.

I have a very capable networked storage system. The underlying hardware
RAID is capable of reading and writing at least 700 MB/sec sustained.
The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with
4 or 6 GBs of RAM (depending on the system). I don't have any problem
writing uncompressed high-definition QuickTime files to Samba shares at
160 MB/sec over 10 Gigabit Ethernet.  However, trying to write standard
definition DPX files over Samba -- only 40 MB/sec -- fails consistently
after about 60,000 files have been written, plus or minus about 5000 files.

In the screenshots shown in the link below, you can this happening when
I was writing to a 6 TB ext4 filesystem. This was a real shocker.

 http://sites.google.com/site/andrewl733info/ext4_and_samba_2


On the above system (which had 6 GB of RAM), it seems that most cpu
activity was restricted to a single core of a Quad Core CPU.  SMBD cpu
utilization climbed steadily throughout the capture.  CPU began at 7
percent for one core and by the time I got to 66000 frames, I was at 100
percent cpu for one core. I started taking screen shots a third of the
way through. Then I went back and started again.

I also tested XFS as the filesystem.  In fact, that's my preferred
filesystem.  I formatted the same 6 TB volume using the suggestions of
experienced XFS developers (specifying "lazy-count=1" for the "log" and
properly specifying the "sunit" and "swidth" for the data portion) and I
mounted it with the important flags "-o,noatime,nobarrier,inode64".  
Here's a record of what happened under these circumstances.

 http://sites.google.com/site/andrewl733info/xfs_and_dpx-1

Depending on the kernel tested (and in some cases whether an irqbalance
daemon was running), either most activity was concentrated on a single
core, or it was spread out evenly among all 4 cores.  But in either
case, smbd cpu utilization would climb steadily from under 10 percent at
the beginning to 10 times that at the end. And when "top" showed that my
single active smbd was at 100 percent, the capture would fail. (To
clarify this last point, when cpu activity is spread evenly among all
cores, top still shows the total activity for smbd. When it says 100
percent, that would be about 25 percent on EACH core, or 100 percent out
of 400 percent total.)

My questions are:

1)  why is smbd cpu utilization climbing so high in just 40 minutes?
2)  could the cause be related to creating, opening and closing so many
files so quickly?  After all, we are creating, opening and closing 30
files each second.
3)  is there anything I can do about it (besides give up)? Samba doesn't
seem to be running out of memory. There is still 3-cores worth of CPU
being unused at the time the capture fails.

I have collected Level 10 logs of the capture to the XFS volume.  Please
feel free to look at them. I'm sorry if the format isn't totally
convenient.  The actual log file turned out to be 1.6 GB.  I made a
single bz2 file out of this -- 27 MB, but Google Sites only lets me
upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with
the "split" program. I believe you will have to download all of them,
and then put them back together with something like:

  "cat sambalogpart_a* sambalog.bz2"

       
http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1 


       
http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1 


       
http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1 



Andy
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Volker Lendecke]

On Thu, Nov 05, 2009 at 10:02:45AM -0500, Andy Liebman wrote:
> I'm sorry of this gets posted twice.  I submitted the same post about 12  
> hours ago and it seems it didn't go through.

Contact AOL support. I have already replied:

http://lists.samba.org/archive/samba/2009-November/151677.html

Volker


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Andy Liebman]

> I have already replied:
>
> http://lists.samba.org/archive/samba/2009-November/151677.html
>
> Volker
>  
Thanks Volker,

Your reply did vanish into thin air.  Not in my Spam or Trash.  So, it's
a good thing you sent me the new link. As it turns out, doing some
Googling I found a similar suggestion yesterday from Jeremy.

    http://us1.samba.org/samba/ftp/HOWTO/Samba-LargeDirectory-HOWTO

Jeremy's suggestion was:

        case sensitive = True
        default case = upper
        preserve case = no
        short preserve case = no


Your suggestion was:

    preserve case = no
    short preserve case = no
    default case = lower
    case sensitive = yes


     

The difference being, you say "lower" and he said "upper". Is there a
reason to use one versus the other? I'll try out your suggestion and
report back to say if that solves the problem or not.  But reading
Jeremy's posting, I'm not sure why it would make a difference.  It seems
the case [in]sensitive issue was about reading directories.  In the case
of capturing DPX files, I am writing to them.  But are you saying the
problem really applies to both getting directory listings and to writing
to a directory?

If this does fix the problem, what will happen if the application or
user creates files with a mixture of upper and lower case?  I don't have
control over whether the application writes "Media" versus "media" in
the default path to files.  Or won't it really matter, as Windows sees
them as the same anyway?

Thanks again,

Andy
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Volker Lendecke]

On Thu, Nov 05, 2009 at 10:58:07AM -0500, Andy Liebman wrote:
> The difference being, you say "lower" and he said "upper". Is there a  
> reason to use one versus the other? I'll try out your suggestion and  

That's just taste.

> report back to say if that solves the problem or not.  But reading  
> Jeremy's posting, I'm not sure why it would make a difference.  It seems  
> the case [in]sensitive issue was about reading directories.  In the case  
> of capturing DPX files, I am writing to them.  But are you saying the  
> problem really applies to both getting directory listings and to writing  
> to a directory?

The problem is the following: When creating a file we have
to prove that the file does not exist in a different
upper/lower case combination. Under Unix, the only way to
prove this is to list the whole directory and do a case
insensitive comparison on each existing file.

> If this does fix the problem, what will happen if the application or  
> user creates files with a mixture of upper and lower case?  I don't have  
> control over whether the application writes "Media" versus "media" in  
> the default path to files.  Or won't it really matter, as Windows sees  
> them as the same anyway?

Doesn't matter. Windows does not distinguish between both.

Volker


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Andy Liebman]

Hi Volker,

>
> The problem is the following: When creating a file we have
> to prove that the file does not exist in a different
> upper/lower case combination. Under Unix, the only way to
> prove this is to list the whole directory and do a case
> insensitive comparison on each existing file.
>
>  

Thank you SO much. Your suggestion totally solved the problem. It makes
sense when you think about it.

Results:  When capturing standard definition DPX files, CPU utilization
is now down at 7 percent (out of 400 percent) and holds steady forever
basically.  I was able to capture 150,000 frames (83 minutes) before I
got bored and stopped my test.

In addition, I'm now capturing HD-444 DPX files which run at about 260
MB/sec and that's doing fine as well -- running at 37 percent total CPU
(out of 400 percent).

Cheers,
Andy
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Volker Lendecke]

On Thu, Nov 05, 2009 at 01:43:59PM -0500, Andy Liebman wrote:

> >The problem is the following: When creating a file we have
> >to prove that the file does not exist in a different
> >upper/lower case combination. Under Unix, the only way to
> >prove this is to list the whole directory and do a case
> >insensitive comparison on each existing file.
> >
> >  
>
> Thank you SO much. Your suggestion totally solved the problem. It makes
> sense when you think about it.
>
> Results:  When capturing standard definition DPX files, CPU utilization
> is now down at 7 percent (out of 400 percent) and holds steady forever
> basically.  I was able to capture 150,000 frames (83 minutes) before I
> got bored and stopped my test.
>
> In addition, I'm now capturing HD-444 DPX files which run at about 260
> MB/sec and that's doing fine as well -- running at 37 percent total CPU
> (out of 400 percent).

If you want to read them out at wire speed, you might want
to look at the vfs_preopen module, part of Samba 3.4. In
case you have latency problems, this module forks helper
processes that open and read the next n files in sequence.
It was written for exactly your application, out of a
cluster file system with pretty high open latency we got
almost 400MBytes/second with that module well tuned,
regardless of the broken access pattern :-)

Volker


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba