[Fwd: [sanesecurity] x86_64 users: possible malformed database problems]

[Steve Basford]

Hi All,

Some users (mainly x86_64 so far) noticed database errors (malformed  
database) when loading signatures.

As signature integrity is checked before upload to the mirrors and the
download scripts check integrity before use, this issue should not arise.

With help from various people on the Sanesecurity list, the problem was
narrowed down to users on x86_64 os versions eg:  CentOS 5.4 on x86_64,
who were using nearly all the available Third Party databases.

The typical errors were:

LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes.
Please report to http://bugs.clamav.net
LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable
LibClamAV Error: cli_parse_add():

Thanks to the ClamAV team, the bug was fixed in the clamav-devel version:

clamav-devel:

+Sat Oct 24 15:06:50 CEST 2009 (acab)
+ * libclamav/mpool.c: increase max pool to 8M to allow loading huge
custom dbs

I realise that people may not be able to move to the devel version in  
production environments, so the only work-around is to try and limit the
number of databases that you are using....

for example:

Largest size signature databases:

25/10/2009  15:53         2,526,656 jurlbl.ndb
24/10/2009  16:53         3,082,316 junk.ndb
25/10/2009  15:38         3,327,576 INetMsg-SpamDomains-2w.ndb
25/10/2009  15:29         3,886,074 scamnailer.ndb
25/10/2009  15:53         6,967,926 jurlbla.ndb
28/08/2009  12:10         9,393,566 securiteinfo.hdb
25/10/2009  15:47        12,645,831 INetMsg-SpamDomains-2m.ndb

As a reminder if you are using InetMsg signatures, you need to select:

*either* INetMsg-SpamDomains-2w.ndb *or* INetMsg-SpamDomains-2m.ndb
*not* both to save a bit of memory.

Hopefully once the devel version bugfix makes it's way into the stable
version, this problem should go away.

Sorry for any problems this has caused.

Cheers,

Steve
Sanesecurity
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[RobertLopez]

On Sun, Oct 25, 2009 at 11:27 AM, Steve Basford
<steveb_clamav@...> wrote:

> Hi All,
>
> Some users (mainly x86_64 so far) noticed database errors (malformed
>  database) when loading signatures.
>
> As signature integrity is checked before upload to the mirrors and the
> download scripts check integrity before use, this issue should not arise.
>
> With help from various people on the Sanesecurity list, the problem was
> narrowed down to users on x86_64 os versions eg:  CentOS 5.4 on x86_64,
> who were using nearly all the available Third Party databases.
> The typical errors were:
>
> LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes.
> Please report to http://bugs.clamav.net
> LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable
> LibClamAV Error: cli_parse_add():
>
> Thanks to the ClamAV team, the bug was fixed in the clamav-devel version:
>
> clamav-devel:
>
> +Sat Oct 24 15:06:50 CEST 2009 (acab)
> + * libclamav/mpool.c: increase max pool to 8M to allow loading huge
> custom dbs
>
> I realise that people may not be able to move to the devel version in
>  production environments, so the only work-around is to try and limit the
> number of databases that you are using....
>
> for example:
>
> Largest size signature databases:
>
> 25/10/2009  15:53         2,526,656 jurlbl.ndb
> 24/10/2009  16:53         3,082,316 junk.ndb
> 25/10/2009  15:38         3,327,576 INetMsg-SpamDomains-2w.ndb
> 25/10/2009  15:29         3,886,074 scamnailer.ndb
> 25/10/2009  15:53         6,967,926 jurlbla.ndb
> 28/08/2009  12:10         9,393,566 securiteinfo.hdb
> 25/10/2009  15:47        12,645,831 INetMsg-SpamDomains-2m.ndb
>
> As a reminder if you are using InetMsg signatures, you need to select:
>
> *either* INetMsg-SpamDomains-2w.ndb *or* INetMsg-SpamDomains-2m.ndb
> *not* both to save a bit of memory.
>
> Hopefully once the devel version bugfix makes it's way into the stable
> version, this problem should go away.
>
> Sorry for any problems this has caused.
>
> Cheers,
>
> Steve
> Sanesecurity
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml
>

We are using Ubuntu 9.04 x86_64. What symptoms were observed when they
"noticed database errors"?

--
Robert Lopez
Unix Systems Administrator
Central New Mexico Community College (CNM)
525 Buena Vista SE
Albuquerque, New Mexico 87106
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[aCaB]

Steve Basford wrote:

> LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes.
> Please report to http://bugs.clamav.net
> LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable
> LibClamAV Error: cli_parse_add():
>
> Thanks to the ClamAV team, the bug was fixed in the clamav-devel version:
>
> clamav-devel:
>
> +Sat Oct 24 15:06:50 CEST 2009 (acab)
> + * libclamav/mpool.c: increase max pool to 8M to allow loading huge
> custom dbs

Hi Steve,

The (now) increased pool size is around 16 times bigger than the largest
pool used by the offical db, so it'll probably be ok for a while.


That said, we should still figure out a way to avoid this kind of
troubles in the future (same goes for the infamous "clamd crashes while
loading 3rd party db's" bug which plagued the early 0.95's).

On our side we do a lot of QA over our own signatures to make sure
things like that won't happen, but of course we can't guarantee the same
for 3rd party databases.
At the end of the day, any service disruption, even if caused by the use
custom databases, is problematic and affects the entire ClamAV user
community.

I'm wondering if it would make sense for us to open up the QA side of
our infrastructure to you guys, in order to minimize this kind of
inconvenence.

I really believe something needs to happen here so that these type of
bugs can be caught quickly before they affect a number of users.

Thoughts?

aCaB

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[G.W. Haywood]

Hi there,

On Thu, 29 Oct 2009 aCaB wrote:

> On our side we do a lot of QA
> ...
> I really believe something needs to happen here so that these type of
> bugs can be caught quickly before they affect a number of users.
>
> Thoughts?

I suspect that rather than QA, what you do is just a lot of hap-hazard
testing.  That's why, whenever I see a new release of ClamAV, first I
will suppress a groan and then, before I risk it on any of my servers,
I'll wait a while and watch the users' list to see how much trouble it
causes.  This approach serves me well, although I can't say I'm proud
of the fact that I'm letting a lot of poor innocents do my acceptance
testing for me.

To be brutally frank, if I upgraded ClamAV on the basis of the
release announcements, ClamAV would in my installations cause a lot
more trouble than the things which it's designed to combat.

You need to develop a proper QA system.  Briefly, that means you need
to document exactly how you're going to avoid this sort of chaos, and
then you have to do what it says in the documentation that you're
going to do.  There's a fairly famous transcript of some evidence to
a House of Commons committee around here somewhere... ah, here it is:

[excerpt]
I would like to tell you something that you will not believe but which
I think it is important that you hear, and that is that almost every
IT supplier in the world today is incompetent. I have worked in the IT
industry almost all my working life for large and small organisations,
and I know what I speak. For example, the typical rate of delivered
faults after full user acceptance testing from the maker suppliers in
the industry over many years has been steady at around 20 faults per
thousand lines of code. We know how to deliver software with a fault
rate that is down around 0.1 faults per thousand lines of code and the
industry does not adopt these techniques. We are as an industry very
much in the early stages. The industry is only 50 years old. If you
compare that with civil engineering, which is several thousand years
old, we are tackling some of the most complex engineering designs and
building some of the most complex engineering systems that the world
has ever seen, essentially using craft technology. If you looked at
the methods that are employed in most companies you would come to the
conclusion that actually IT system development is a fashion business,
not an engineering business, because they jump from one methodology to
another year after year so long as it has a whizzy name, "Agile this"
or "Intensive that". The underlying engineering disciplines that every
mature engineering discipline has learnt it needs to use in order to
be able to show that the system it is building has the required
properties have not yet been employed in software and systems
engineering, and that is at the heart of why these things do not work.
[/excerpt]

Professor Martyn Thomas,

Minutes of Evidence taken before Home Affairs Committee Inquiry into
Identity Cards, Tuesday 24 February 2004.

http://www.parliament.the-stationery-office.co.uk/pa/cm200304/cmselect/cmhaff/uc130-iv/uc13002.htm

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[aCaB]

G.W. Haywood wrote:
> I suspect that rather than QA, what you do is just a lot of hap-hazard
> testing.  That's why, whenever I see a new release of ClamAV, first I
> will suppress a groan and then, before I risk it on any of my servers,
> I'll wait a while and watch the users' list to see how much trouble it
> causes.  This approach serves me well, although I can't say I'm proud
> of the fact that I'm letting a lot of poor innocents do my acceptance
> testing for me.

Hi G.W. Haywood,

My mail was about custom databases provided by 3rd parties, not about
ClamAV release cycles.

Besides, you miss another point: ClamAV is an open source software,
consisting of roughly 150K lines of C code and 650000 signatures,
currently maintained by three full time developers, one and a half full
time sigmakers and a system administrator.

We ALWAYS ask our users to test the development head and provide
feedbacks because we cannot do it all on our own: we lack the man power
and we lack the infrastructure, but, most importantly we lack YOUR
setup, YOUR deployment and YOUR envirnonment.

With some very notable exceptions (which I would really like to thank),
it is a fact that, despite the repeated requests, not many people test
the code. You can look at the bugzilla being all quiet for weeks, then,
as soon as we release a new version, it suddently gets flooded with tickets.

So, to conclude, if you want to get better releases, do your bit.

The only alternative is that we release what WE think is ok and we
re-release when YOU tell us it's not.


Thanks for the lesson,
-aCaB

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[G.W. Haywood]

Hello again,

On Wed, 4 Nov 2009 aCaB wrote:

> G.W. Haywood wrote:
> > I suspect that rather than QA, what you do is just a lot of hap-hazard
> > testing.
>
> My mail was about custom databases provided by 3rd parties, not about
> ClamAV release cycles.
>
> Besides, you miss another point: ClamAV is an open source software,

I don't think the points were lost on me.

> We ALWAYS ask our users to test the development head

That's exactly what I said.

> ... bugzilla being all quiet for weeks, then, as soon as we release
> a new version, it suddently gets flooded with tickets.

That's exactly what I said.

> So, to conclude, if you want to get better releases, do your bit.

That's exactly what I am doing now.  The trouble is you aren't in
a very receptive frame of mind.  I'm not going to do what you want
me to do because what you want me to do is more hap-hazard testing,
and that's what I'm telling you is almost a complete waste of time.

> The only alternative is that we release what WE think is ok and we
> re-release when YOU tell us it's not.

Not so.  One alternative is to do something like I'm suggesting.

> Thanks for the lesson,

Please leave the sarcasm at home tomorrow.

It was YOU who asked for thoughts.  Didn't you say something about
doing my bit?  That's what I'm trying to do now.  It's a shame you're
not listening when people give you the thoughts that you asked for.

I'm telling you that your take on how software should be developed is
very seriously lacking.  Until you start to listen to that you'll get
absolutely nowhere.  This really does demand a completely different
approach to the way software is built, taking it out of the realms of
what Professor Thomas calls 'craft technology' and into the mainstream
of engineering.

In a word, it's called 'quality'.  Most people haven't the faintest
idea what it's about.  Read about it, and about what it can achieve.
You'll be surprised.

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[George R. Kasica]

>On Mon, 2 Nov 2009 16:32:57 +0000 (GMT), you wrote:

>Hi there,
>
>On Thu, 29 Oct 2009 aCaB wrote:
>
>> On our side we do a lot of QA
>> ...
>> I really believe something needs to happen here so that these type of
>> bugs can be caught quickly before they affect a number of users.
>>
>> Thoughts?
>
>I suspect that rather than QA, what you do is just a lot of hap-hazard
>testing.  That's why, whenever I see a new release of ClamAV, first I
>will suppress a groan and then, before I risk it on any of my servers,
>I'll wait a while and watch the users' list to see how much trouble it
>causes.  This approach serves me well, although I can't say I'm proud
>of the fact that I'm letting a lot of poor innocents do my acceptance
>testing for me.

OK...I'm probably going to PO a few folks here but you asked for
opinions so I'll share one....

We run about 50 Unix/Linux servers here...almost all are 32-bit RHEL4
not an unknown distro by any means and what I'd suspect is easy to
test on. Several more are Solaris 9, again a well known OS, and
several more are Fedora Core 10, again a well known common OS build.

This latest release had basic make issues in ALL THREE OS versions
that should have been caught by simply typing make as it wouldn't even
finish the make step without various patches....in my mind that
doesn't qualify it as even ALPHA software much less BETA or a Release
Candidate and surely not a Production Release version...please don't
embarrass yourselves by saying you do alot of QA....that obviously
didn't happen in this release and has been a problem in the last few
releases....

I expect better from a commercial company like SourceFire now that
they own the product as a commercial concern...its the same codebase
for free or paid users as far as I know, the differences is paying
users get to call for support.

George
--
===[George R. Kasica]===        [phone]
President                       [phone] FAX
Netwrx Consulting Inc.          Jackson, WI USA
http://www.netwrx1.com
georgek@...
ICQ #12862186
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Török Edwin]

On 2009-11-05 18:27, George R. Kasica wrote:

>> On Mon, 2 Nov 2009 16:32:57 +0000 (GMT), you wrote:
>>    
>
>  
>> Hi there,
>>
>> On Thu, 29 Oct 2009 aCaB wrote:
>>
>>    
>>> On our side we do a lot of QA
>>> ...
>>> I really believe something needs to happen here so that these type of
>>> bugs can be caught quickly before they affect a number of users.
>>>
>>> Thoughts?
>>>      
>> I suspect that rather than QA, what you do is just a lot of hap-hazard
>> testing.  That's why, whenever I see a new release of ClamAV, first I
>> will suppress a groan and then, before I risk it on any of my servers,
>> I'll wait a while and watch the users' list to see how much trouble it
>> causes.  This approach serves me well, although I can't say I'm proud
>> of the fact that I'm letting a lot of poor innocents do my acceptance
>> testing for me.
>>    
>
> OK...I'm probably going to PO a few folks here but you asked for
> opinions so I'll share one....
>
> We run about 50 Unix/Linux servers here...almost all are 32-bit RHEL4
> not an unknown distro by any means and what I'd suspect is easy to
> test on. Several more are Solaris 9, again a well known OS, and
> several more are Fedora Core 10, again a well known common OS build.
>  

Hi,

We test ClamAV on several architectures and OSes:
Debian (etch, lenny, unstable) on various architectures (x86_64,
powerpc, sparc64, armv5tel, ...),
RHEL5.2 (x86-64), Solaris 10 (sparc), and Mac OS X (ppc).
Some of these tests are running on the GCC compile farm, some on our own
boxes, some on machines we have been kindly offered access for testing.
This of course can always be improved.

One of the build problems was an error when 'git' was not installed.
Unfortunately all of the machines we test ClamAV on have git in $PATH
(it is how it checks out the latest version).
For the next release we'll setup a buildhost that doesn't have git
installed.
The other problem (missing include unistd.h) only occurs on old
distributions, which we lack in our buildfarm.

We're currently investigating the possibility of using the OpenSUSE
build service to test the next ClamAV release on multiple Linux
distributions, including many old ones:
openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

> This latest release had basic make issues in ALL THREE OS versions
> that should have been caught by simply typing make as it wouldn't even
> finish the make step without various patches....in my mind that
> doesn't qualify it as even ALPHA software much less BETA or a Release
> Candidate and surely not a Production Release version...please don't
> embarrass yourselves by saying you do alot of QA....that obviously
> didn't happen in this release and has been a problem in the last few
> releases....
>  

We do more than just build tests for QA, but as you've realized the
build tests for old(er) OSes are currently insufficient.

> I expect better from a commercial company like SourceFire now that
> they own the product as a commercial concern...its the same codebase
> for free or paid users as far as I know, the differences is paying
> users get to call for support.
>  

There are also certified binary packages available.

Best regards,
--Edwin

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[George R. Kasica]

>We're currently investigating the possibility of using the OpenSUSE
>build service to test the next ClamAV release on multiple Linux
>distributions, including many old ones:
>openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
>Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

OK...I'm not going to debate "old" vs. "new" here but I'm fairly sure
the installed base of Fedora Core 10 and RHEL4 and Solaris 9 which are
all actively supported by the various Vendors/groups would disagree
with your assessment as would alot of businesses that are running them
in a day to day production setting for front line work.

Frankly, Don't think I would be able to get a "new" OS such as FC-11
or RHEL5 OK 'd to go into production at our company due to lack of
sufficient background from a security standpoint etc.

In any case, if you're looking for a test spot for FC10, Solaris 9,
RHEL4 I'd be happy to try to run some stuff here on a box - I'm not a
programmer but I can do basic things if given clear steps or test the
ability to at least get it to make etc in our QA/Test environment.

Let me know.

George
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Nathan Gibbs]

* aCaB wrote:

> G.W. Haywood wrote:
>> I suspect that rather than QA, what you do is just a lot of hap-hazard
>> testing.  That's why, whenever I see a new release of ClamAV, first I
>> will suppress a groan and then, before I risk it on any of my servers,
>> I'll wait a while and watch the users' list to see how much trouble it
>> causes.  This approach serves me well, although I can't say I'm proud
>> of the fact that I'm letting a lot of poor innocents do my acceptance
>> testing for me.
>
> Hi G.W. Haywood,
>
> My mail was about custom databases provided by 3rd parties, not about
> ClamAV release cycles.
>
> Besides, you miss another point: ClamAV is an open source software,
> consisting of roughly 150K lines of C code and 650000 signatures,
> currently maintained by three full time developers, one and a half full
> time sigmakers and a system administrator.
>

So less than 6 people maintain Clamav.

I'm impressed.

Cut these guys some slack.
They can't do everything.

Thats the beauty of Open Source.
If you don't like it FIX IT.
If you can't fix it, report that its broken.
If possible Where is it broken, why is it borken, and how it might be fixed.
If you can't do that, Get a commercial solution.
Freedom isn't free. ( from effort or errors )


--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com




_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[aCaB]

George R. Kasica wrote:
> In any case, if you're looking for a test spot for FC10, Solaris 9,
> RHEL4 I'd be happy to try to run some stuff here on a box - I'm not a
> programmer but I can do basic things if given clear steps or test the
> ability to at least get it to make etc in our QA/Test environment.

Hi George,
That would be cool!

There are basically two options.
The least intrusive is a small shell script to be run daily or so from
cron which posts resuts available here: http://farm.0xacab.net/
This only requires git, a compatible compiler and an ftp client.

The other one is to run a buildbot slave. Results are available  at
http://www.0xacab.net:8010/waterfall

If you want to help with either, please mail Edwin or me off list.

Thanks,
-acab
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Fajar A. Nugraha-3]

2009/11/6 Török Edwin <edwintorok@...>:
> One of the build problems was an error when 'git' was not installed.
> Unfortunately all of the machines we test ClamAV on have git in $PATH
> (it is how it checks out the latest version).

I usually do software compilation/packaging on specially-prepared
chroot environment, which I can tune according to that software's
special needs. For most packages it's simply a new directory created
with "yum --installroot=... install ..."

Fedora has Mock, which does the similar thing.

> For the next release we'll setup a buildhost that doesn't have git
> installed.
> The other problem (missing include unistd.h) only occurs on old
> distributions, which we lack in our buildfarm.
>
> We're currently investigating the possibility of using the OpenSUSE
> build service to test the next ClamAV release on multiple Linux
> distributions, including many old ones:
> openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
> Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

My build host is actually just a Xen domU running RHEL4 64bit, with
chroot environments for RHEL4 and 5 32 and 64bit (which is what we
use). It saves the number of host and resources needed for testing.

For (open)Solaris, you could probably make use of branded zones which
can give you a Linux 2.4 (e.g. Centos3), Solaris 8, Solaris 9, and
Solaris 10/Opensolaris build environment.

--
Fajar
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Ian Eiloart]

--On 5 November 2009 20:44:51 +0200 Török Edwin <edwintorok@...>
wrote:

> and Mac OS X (ppc).

But, not Intel? I'm surprised, but not complaining since we still use PPC
mail servers - but not for much longer - they're a bit long in the tooth
now.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Tom Shaw-2]

At 2:58 PM +0000 11/6/09, Ian Eiloart wrote:
>--On 5 November 2009 20:44:51 +0200 Török Edwin <edwintorok@...> wrote:
>
>>and Mac OS X (ppc).
>
>But, not Intel? I'm surprised, but not
>complaining since we still use PPC mail servers
>- but not for much longer - they're a bit long
>in the tooth now.

Works fine on intel and 64 bit as well under OSX.
Have universal build and installer ready shortly.

Tom
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[G.W. Haywood]

Hi there,

On Fri, 6 Nov 2009 Nathan Gibbs wrote:

> Cut these guys some slack.
> They can't do everything.

This isn't about cutting slack, or for that matter pointing fingers.
It's about contributing.  I'm contributing.  Sometimes people don't
want to hear what needs to be said.  That can be painful all round,
believe me.

> Thats the beauty of Open Source.

Let's just say I'm familiar with the concept and leave it at that.
We don't want you to get your foot completely stuck. :)

> ... report that its broken.  If possible Where is it broken, why is
> it borken, and how it might be fixed.

That's exactly what I'm doing here.  Unfortunately my message is not
one which people often want to hear.  They want to hear what a great
job they do, not how they could improve on how they're doing things by
doing things for which they really have little sympathy.  There isn't
some piece of code somewhere that has to be tweaked.  If it were so it
would be great, they'd just go off and tweak it and all would be well.
But it isn't like that.  The issue here is that if there is any system
which is supposed to prevent problems of the sort that we see so often
in releases of ClamAV, then demonstrably it isn't working.

My contention is that there is no system, and that's what needs to be
fixed.  But as I don't employ the developers, all I can do is point to
some of the benefits of using modern (i.e. developed in the past fifty
years or so) quality systems.  We no longer have to promise that the
architect will be executed if his building falls down.  But there are
still people in governments talking about penalty clauses in software
procurement contracts because they don't know any better.  Sorry, any
better way to get the systems delivered on time, under budget and with
acceptable levels of faults and performance.

Quality systems can and do result in improvements of a couple of
orders of magnitude in fault density in software (measured for example
in coding errors per 1000 lines of code).  The user's experience will
inevitably improve as a result - and let's not forget that one coding
error can easily cost several hours of the time of each of thousands
of the code's users - but in addition, the developers will be able to
get on with their developing instead of spending large chunks of time
tracking down the faults which they themselves put there because they
didn't have some kind of system.

Obviously I don't claim that all errors can be eliminated, but I have
seen the results of using and failing to use quality systems and the
differences can be stunning.  It's no exaggeration to say that the
cost savings in industry are phenomenal.  I've seen manufacturing cost
reduced by more than 60% and reliability improved by 500% as a result
of proper quality control procedures being applied.  I've also had to
fly to thirty of the United States to replace four 10 cent components
in a couple of hundred examples of a fifty-thousand dollar product
because the quality control procedures were ignored.  The cost to one
company of ignoring a few simple quality procedures which would have
found the problem at the sub-assembly manufacturing stage was tens of
thousands of dollars in repair bills and very much more than that in
loss of customer goodwill.  The cost of finding and fixing the problem
if the procedures had been followed would have been a few dollars.
This is a natural result of the logarithmic relationship between the
cost per fix and the stage in the product's life at which the fix is
applied.  It doesn't really matter what the product is, and despite
what you're thinking right now it is _very_ relevant to software, if
the software is built in anything like a modular fashion as probably
it should be.

Here's an example of an implementation of some of the ideas:

http://www.praxis-his.com/news/sparkGPL.asp

But you don't have to implement the whole thing as an automated
toolset, a lot of it can be paper-based or some other manual method
such as just writing in a comment at the top of every function what
the expected inputs and outputs are, then making sure (by inspection
or by test procedures in the code itself, and preferably both) that
those are indeed the things that happen in practice.

As I said in my first post, quality systems are fundamentally about
documentation.  We all know what a popular subject that is. :(  One
aspect of the documentation in this case would be writing down how to
prevent these things from happening.  That's the first problem.  If
you can't do that then you're in real trouble, it probably means that
you don't know _why_ the things are happening.  But at least now you
know what the real problem is.  It has little to do with the code.
It will take time and a little effort, but once it's done, then all
you have to do is what you wrote down you would do.  That just takes
discipline.  Admittedly this is another relatively unpopular concept
in the programming community. :)

There are very many places to start looking at quality systems, but
as we are in the very early stages I suggest that background reading
about the issues would be more appropriate than diving into projects
developed using modified ADA.

Here's one document which covers many of the issues:

http://satc.gsfc.nasa.gov/assure/agbsec3.txt

It's written in fairly general terms, and at first sight it can be a
bit daunting.  But what appears at first to be onerous, confusing and
even irrelevant can in fact be a great saver of labour, and when you
begin to understand where it's 'coming from', as they say in the USA,
it will all start to make sense.

There are many more sources of information about software quality
assurance on the Web, but the first step must be to recognize that
there is a problem to be solved.  When that is done, the developers
are in a position to apply the techniques needed to solve it.

<footnote>
Here in England we used to see "Made in Japan" on the base of lots of
cheap, tacky, plastic products.  The phrase became almost synonymous
with mass-produced, poor quality products.  About the time that I was
born, the Japanese government passed a law which forced all Japanese
manufacturers to work to the new 'quality systems'.  The rest, as they
say, is history.  For most of my childhood I was immersed in a culture
which held that "Made in Britain" meant "Good", while "Made in Japan"
meant "Rubbish".  In my teens, I began to wonder why my British made
motor-cycle leaked more oil and broke down more often than my friend's
Yamaha.  Later, I wondered why the Japanese seemed to be able to bring
new products to market so much faster than the British, and sell them
cheaper, and in such large volumes.  None of the British motor-cycle
manufacturers could compete with the Japanese onslaught and every one
of them went bankrupt.  The few motor-car manufacturers here which did
survive, if they can be said to have survived, have only done so by
getting huge handouts from the tax payer.  Our consumer electronics
industry all but disappeared and the term "Made in Japan" is no longer
something which provokes derision.  All this was the direct result of
the implementation of quality systems - elsewhere.  It's the stuff of
industrial revolutions, like when they moved from horses to steam, or
from armies of clerks with their quill pens, to computers.  And it's
really rather easy to do, and cheaper, and quicker, than not doing it.
</footnote>

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

[Nathan Gibbs]

* G.W. Haywood wrote:

> Hi there,
>
> On Fri, 6 Nov 2009 Nathan Gibbs wrote:
>
>> Cut these guys some slack. They can't do everything.
>
> This isn't about cutting slack, or for that matter pointing fingers.
> It's about contributing.  I'm contributing.  Sometimes people don't
> want to hear what needs to be said.  That can be painful all round,
> believe me.
>

I'm starting to see what you mean.

>> Thats the beauty of Open Source.
>
> Let's just say I'm familiar with the concept and leave it at that. We
>  don't want you to get your foot completely stuck. :)
>

I appreciate that, it won't be the first time or the last.
Agree to disagree ?
:-)

>> ... report that its broken.  If possible Where is it broken, why is
>>  it borken, and how it might be fixed.
>
> That's exactly what I'm doing here.  Unfortunately my message is not
> one which people often want to hear.  They want to hear what a great
> job they do, not how they could improve on how they're doing things
> by doing things for which they really have little sympathy.

Right.

> There isn't some piece of code somewhere that has to be tweaked.  If
>  it were so it would be great, they'd just go off and tweak it and
> all would be well. But it isn't like that.  The issue here is that if
>  there is any system which is supposed to prevent problems of the
> sort that we see so often in releases of ClamAV, then demonstrably it
>  isn't working.
>

I'll admin that the 0.94 and 0.95 releases have been a bit more
interesting than just install it and move on.  The initial clamav 0.95
upgrade broke some internal code here. The EOL note for 0.94 seems to
indicate that after 4-15-2010 anything that isn't upgraded will blow.
Personally, ( switching feet here ) I don't fault the Clamav team for
these decisions, although I feel it is a little rough on the user base.
I feel that the code is improving, and these are just some things we
will have to deal with.

> My contention is that there is no system,

I'll disagree with you there.  One man's chaos is another mans order.
I'm not missing your point that the "system" could improve.

> and that's what needs to be fixed.  But as I don't employ the
> developers, all I can do is point to some of the benefits of using
> modern (i.e. developed in the past fifty years or so) quality
> systems.

Snip

> Quality systems can and do result in improvements of a couple of
> orders of magnitude in fault density in software (measured for
> example in coding errors per 1000 lines of code).  The user's
> experience will inevitably improve as a result - and let's not forget
>  that one coding error can easily cost several hours of the time of
> each of thousands of the code's users - but in addition, the
> developers will be able to get on with their developing instead of
> spending large chunks of time tracking down the faults which they
> themselves put there because they didn't have some kind of system.
>

Been there, done that, no fun.

> Obviously I don't claim that all errors can be eliminated, but I have
>  seen the results of using and failing to use quality systems and the
>  differences can be stunning.

Snip
Agreed.

> The cost to one company of ignoring a few simple quality procedures
> ... was ... loss of customer goodwill.

Which in open source is one thing you don't want to lose.
Snip

> It doesn't really matter what the product is, and despite what you're
> thinking right now it is _very_ relevant to software, if the software
> is built in anything like a modular fashion as probably it should be.
>

Agreed, its far less expensive to fix an error before release than after.

>
> But you don't have to implement the whole thing as an automated
> toolset, a lot of it can be paper-based or some other manual method
> such as just writing in a comment at the top of every function what
> the expected inputs and outputs are, then making sure (by inspection
> or by test procedures in the code itself, and preferably both) that
> those are indeed the things that happen in practice.
>

Eight.

> As I said in my first post, quality systems are fundamentally about
> documentation.  We all know what a popular subject that is. :(  One
> aspect of the documentation in this case would be writing down how to
>  prevent these things from happening.  That's the first problem.  If
> you can't do that then you're in real trouble, it probably means that
> you don't know _why_ the things are happening.  But at least now you
> know what the real problem is.  It has little to do with the code. It
> will take time and a little effort, but once it's done, then all you
> have to do is what you wrote down you would do.  That just takes
> discipline.  Admittedly this is another relatively unpopular concept
> in the programming community. :)
>

You are SO RIGHT on that point.  A massive benefit can be obtained by
documenting ( even informally ) what you are doing.

This precisely hits some of the issues around a recent bug I put into
the tracker. Whether they add the feature or not isn't important what is
there isn't documented very well. As i said in the bug report.

"Please consider documenting all of the Environment variables that are
set on the external events.
When I read the manual & don't find them, as far as I'm concerned they
don't exist."

<Note>
When I set up our network, I kept it all "in my head".
Did it that way until late 2005.
Started documenting basic stuff.
Increased the detail slowly over time.
Now I spend less time troubleshooting and more time trouble fixing or
doing more productive work.
<End Note>

Snip
> There are many more sources of information about software quality
> assurance on the Web, but the first step must be to recognize that
> there is a problem to be solved.  When that is done, the developers
> are in a position to apply the techniques needed to solve it.
>
Agreed.

Seriously, you may want to rethink your style of approach.
I feel that I understand where you are coming from now, after lengthy
explanation.

I, possibly incorrectly, understood you first post to mean.

"Won't you stupid Clamav developers get it right already."

What I believe you meant was.

"The Clamav developers should design & consistently execute their QA
process."

I reserve the right to be wrong here (switching feet again).

Since the sourcefire acquisition, I've had higher expectations for
Clamav.  Solution wise I don't feel I have been disappointed.

However, from my seat in the user base, I seriously wonder if they give
a care about the users.


--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com




_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml