load issues due to sanesecurity signatures

[Avinash-11]

Hi everyone,

We are using Sanesecurity signatures in clamd for scanning mails. Recently
we are seeing some load issues on clamd server due to sanesecurity
signatures (load is automatically decreasing when the sanesecurity sigs are
removed)

Does anyone face this issue before? Sanesecurity sigs are much needed to
catch spam, is these anyway that i can fix this issue? Please help me.

Thanks in advance,
Avinash
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Tom Shaw-2]

At 4:10 PM -0600 11/2/09, Noel Jones wrote:

>On 11/2/2009 1:42 PM, Avinash wrote:
>>Hi everyone,
>>
>>We are using Sanesecurity signatures in clamd for scanning mails. Recently
>>we are seeing some load issues on clamd server due to sanesecurity
>>signatures (load is automatically decreasing when the sanesecurity sigs are
>>removed)
>>
>>Does anyone face this issue before? Sanesecurity sigs are much needed to
>>catch spam, is these anyway that i can fix this issue? Please help me.
>>
>
>Likely just one of the signature files is causing problems. Try
>disabling them one at a time until load comes down to an acceptable
>level.  I'd start with winnow.complex.patterns.ldb.

Just a question. Why disable a file that currently has only 2 rules
in it? Wouldn't you want to 1) determine what he has enabled? After
all safebrowsing is humongous, 2) what hardware configuration and
scan volume he is using and 3) what else is running on the machine?

After all there are a lot of us using all sansecurity files and
safebrowsing with no issues which would lead one to believe that
there is not a signature file that is causing problems but more
probably the interaction of light hardware, higher data volume and
other processes running on the server coupled with a large number of
signatures.

Lets first look at what Avinash wrote. He said all was well with
ClamAV and SaneSecurity signatures until recently.

It would be nice to know what changed. If it is that the volume of
email has increased then he needs to look at his entire setup - what
else is running on his machine and what it contributes to the load.
I doubt its a signature file causing problems per se.

Just my 2 cents,

Tom


Tom
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Noel Jones-2]

On 11/2/2009 1:42 PM, Avinash wrote:

> Hi everyone,
>
> We are using Sanesecurity signatures in clamd for scanning mails. Recently
> we are seeing some load issues on clamd server due to sanesecurity
> signatures (load is automatically decreasing when the sanesecurity sigs are
> removed)
>
> Does anyone face this issue before? Sanesecurity sigs are much needed to
> catch spam, is these anyway that i can fix this issue? Please help me.
>
>

Likely just one of the signature files is causing problems.
Try disabling them one at a time until load comes down to an
acceptable level.  I'd start with winnow.complex.patterns.ldb.

   -- Noel Jones
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Freddie Cash-8]

On Mon, Nov 2, 2009 at 1:45 PM, Tom Shaw <tshaw@...> wrote:

> At 4:10 PM -0600 11/2/09, Noel Jones wrote:
>
>> On 11/2/2009 1:42 PM, Avinash wrote:
>>
>>> Hi everyone,
>>>
>>> We are using Sanesecurity signatures in clamd for scanning mails.
>>> Recently
>>> we are seeing some load issues on clamd server due to sanesecurity
>>> signatures (load is automatically decreasing when the sanesecurity sigs
>>> are
>>> removed)
>>>
>>> Does anyone face this issue before? Sanesecurity sigs are much needed to
>>> catch spam, is these anyway that i can fix this issue? Please help me.
>>>
>>>
>> Likely just one of the signature files is causing problems. Try disabling
>> them one at a time until load comes down to an acceptable level.  I'd start
>> with winnow.complex.patterns.ldb.
>>
>
> Just a question. Why disable a file that currently has only 2 rules in it?
> Wouldn't you want to 1) determine what he has enabled? After all
> safebrowsing is humongous, 2) what hardware configuration and scan volume he
> is using and 3) what else is running on the machine?
>
> After all there are a lot of us using all sansecurity files and
> safebrowsing with no issues which would lead one to believe that there is
> not a signature file that is causing problems but more probably the
> interaction of light hardware, higher data volume and other processes
> running on the server coupled with a large number of signatures.
>
> Lets first look at what Avinash wrote. He said all was well with ClamAV and
> SaneSecurity signatures until recently.
>

clamd on our mail server started hogging 100% of both CPUs, and mail started
backing up like crazy.  This started last Thursday evening.  I played with
the Postfix, Amavisd-new, and Clamd settings all Friday morning trying to
figure this out and clear out the backlog of messages.

On a whim, I renamed the clamav database directory, ran freshclam to get
just the basic signatures, and restarted clamd.  Number of signatures went
from 925,000+ to under 600,000, and CPU usage dropped to below 20%.  Cleared
out 1200 messages from the queue in under 15 minutes.  Reran the script to
download all the extra signature databases, putting the total back up over
700,000, and still the CPU usage is under 20%.

Haven't had any issues since then, so can't really say if it was a corrupted
database, a bad signature, or exactly what the issue was.  Don't have any
plans to test the old copies of the database files, as I don't want to mess
with things now that they are working again.  :)

Something strange happened to the database files last week.  This week,
everything is fine.
--
Freddie Cash
fjwcash@...
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Jason Haar]

On 11/03/2009 12:21 PM, Freddie Cash wrote:
>
> On a whim, I renamed the clamav database directory, ran freshclam to get
> just the basic signatures, and restarted clamd.  Number of signatures went
> from 925,000+ to under 600,000, and CPU usage dropped to below 20%.  Cleared
> out 1200 messages from the queue in under 15 minutes.  Reran the script to
> download all the extra signature databases, putting the total back up over
> 700,000, and still the CPU usage is under 20%.
>  

Do you still have that renamed directory? Can you see what is different
between the working and non-working dirs? The sansecurity folk would
probably be interested...

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: [phone] Fax: [phone]
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Steve Basford]

> Hi everyone,
>
> We are using Sanesecurity signatures in clamd for scanning mails. Recently
> we are seeing some load issues on clamd server due to sanesecurity
> signatures (load is automatically decreasing when the sanesecurity sigs
> are
> removed)

Hi Avinash,

I guess as others have already asked, what databases were you using?

These two databases are the largest:

jurlbla.ndb
INetMsg-SpamDomains-2m.ndb

This one has the most "logic" in it, so perhaps this is the one causing
you problems:

scamnailer.ndb

If you are using INetMsg-SpamDomains-2m.ndb and INetMsg-SpamDomains-2w.ndb
together, you'll be using duplicate sigs.

Hopefully we'll be able to help, once we get a database list from you.

Thanks for the report.

Cheers,

Steve
Sanesecurity


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[aCaB]

Steve,

I see more and more custom db related issues on this list...

Last week I offered some help to early diagnose possible problems before
they hit the end users and I was trying to establish some cooperation
with you and the other db providers in order to improve your QA process.

Just in case you missed that mail...

-aCaB
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Steve Basford]

> Last week I offered some help to early diagnose possible problems before
> they hit the end users and I was trying to establish some cooperation
> with you and the other db providers in order to improve your QA process.

Hi.... sorry for not replying earlier... I'll email off-list with a few
thoughts.. just need to sort a few things out first.

Cheers,

Steve
Sanesecurity

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Avinash-11]

Hi everyone,

Thanks for the quick response.

We are using the below 6 sanesecurity files.

junk.ndb
phish.ndb
scam.ndb
spear.ndb
lott.ndb
spam.ldb

Some more info:

I tried with adding these files one by one to clamd database, junk.ndb is
causing more load among all. Phish.ndb, scam.ndb and spear.ndb are also
contributing to the load.

Just to note, only the 50k sanesecurity sigs are causing load (among all
other 0.7 million sigs).
Is there anyway that we can convert sanesecurity sigs to .cld (or .cvd) with
a sigtool? (ignore if not relevant)

We are running only clamd process on a Linux x86_64 server.

Thanks,
Avinash

PS: My last reply was not updated in the thread :-(  please ignore if it
gets posted.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Tom Shaw-2]

At 9:32 PM +0530 11/3/09, Avinash wrote:

>Hi everyone,
>
>Thanks for the quick response.
>
>We are using the below 6 sanesecurity files.
>
>junk.ndb
>phish.ndb
>scam.ndb
>spear.ndb
>lott.ndb
>spam.ldb
>
>Some more info:
>
>I tried with adding these files one by one to clamd database, junk.ndb is
>causing more load among all. Phish.ndb, scam.ndb and spear.ndb are also
>contributing to the load.
>
>Just to note, only the 50k sanesecurity sigs are causing load (among all
>other 0.7 million sigs).
>Is there anyway that we can convert sanesecurity sigs to .cld (or .cvd) with
>a sigtool? (ignore if not relevant)
>
>We are running only clamd process on a Linux x86_64 server.
>

Avinash

I think you need to tell us more.  We run clamd (0.95.2 and 3) on a
small, old PPC machine under unix with all official and unofficial
signatures with mail and other apps with no issues.

Initially you said "We are using Sanesecurity signatures in clamd for
scanning mails. Recently we are seeing some load issues on clamd
server due to sanesecurity signatures"

Can you explain what changed between the time all was fine and your
recent "load" issues? Can you explain what are the "load issues"?
What version of OS and clamd?

The more information the easier it will be for us to help.

Tom

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Avinash-11]

Hi Tom,

We are using clamav version 0.95.2 with both official and unofficial
signatures.

Last week we observed clamd is taking more time for scanning mails due to
high load on the server. To fix the issue, installed older version 0.95.1,
but there was no use (later came back to 0.95.2). After removing all
unofficial signatures, we came to know that sanesecurity sigs are causing
the problem.

Will let you know the OS version asap (away from my pc now)

Thanks,
Avinash
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Avinash-11]

$$ uname -a
Linux 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64
x86_64 GNU/Linux
$$

Thanks,
Avinash
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Avinash-11]

Hi Steve,

Are you able to find anything on this?  An early fix could be more helpful,
currently we are letting spam through.

Can i get older versions of Sanesecurity database files (junk.ndb, lott.ndb,
spear.ndb, spam.ldb, scam.ndb, phish.ndb) you've released earlier in this
month ?

Thanks,
Avinash
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Freddie Cash-8]

On Mon, Nov 2, 2009 at 5:35 PM, Jason Haar <Jason.Haar@...> wrote:

> On 11/03/2009 12:21 PM, Freddie Cash wrote:
>> On a whim, I renamed the clamav database directory, ran freshclam to get
>> just the basic signatures, and restarted clamd.  Number of signatures went
>> from 925,000+ to under 600,000, and CPU usage dropped to below 20%.  Cleared
>> out 1200 messages from the queue in under 15 minutes.  Reran the script to
>> download all the extra signature databases, putting the total back up over
>> 700,000, and still the CPU usage is under 20%.
>
> Do you still have that renamed directory? Can you see what is different
> between the working and non-working dirs? The sansecurity folk would
> probably be interested...

Yes, I still have this directory.  If anyone is interested in it, I
can tar it up and make it available.  Can also tar up the working
directory is needed.

The same list of database files are in both directories.  The number
of backup files created by Bill Landry's download script are different
between the two directories (some showing in only one or the other).
And the number of signatures loaded initially was different (the
number is back up to over 940,000 now).

Haven't had any issues since.  System load is under 1.0, CPU usage is
under 20%, mail is flowing through nice and quick.

--
Freddie Cash
fjwcash@...
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Steve Basford]

Freddie Cash wrote:
>
> Yes, I still have this directory.  If anyone is interested in it, I
> can tar it up and make it available.  Can also tar up the working
> directory is needed.
>
>
>  
Hi,

Yep, I'll take a look and see if I can see anything this end.

Cheers,

Steve
Sanesecurity
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Freddie Cash-8]

On Thu, Nov 5, 2009 at 11:46 AM, Steve Basford
<steveb_clamav@...> wrote:

> Freddie Cash wrote:
>>
>> Yes, I still have this directory.  If anyone is interested in it, I
>> can tar it up and make it available.  Can also tar up the working
>> directory is needed.
>
> Yep, I'll take a look and see if I can see anything this end.
>
> Cheers,
> Steve
> Sanesecurity

http://www.sd73.bc.ca/downloads/clamav-libdir-broken.tbz2
http://www.sd73.bc.ca/downloads/clamav-libdir-working.tbz2

Enjoy!  :)

--
Freddie Cash
fjwcash@...
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml