Vulnerability and Patch-Management in Linux (and other Unix)

[Rainer Duffner]

Hi,

we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
FreeBSD, Ubuntu and lately Solaris.
We use these for a variety of reasons and each system does its job quite
well.

However, patch-management seems to be a weak spot in most cases.
RedHat offers "RedHat Network", but it costs a lot of money (and they
charge more if you want to put your servers in groups in the RHN - WTF?)
FreeBSD offers the portaudit database - we should be able to hack
together something with that.
But what about CentOS? If you have an array of CentOS servers - how do
you track which vulnerabilities each one has?
Running yum update every night is no option.

Does CentOS also maintain a vulnerability database along the lines of
FreeBSD?
How about Solaris?
Ubuntu?

How do you track vulnerabilities across your datacenter?


Regards,

Rainer

[druid-2]

So, if you have the money you can use Opsware Server Automation System
(SAS) which will patch and manage all of those OSes and more. Opsware was
bought by HP so the product is now called HP Server Automation (HPSA).

To be honest, this is a GREAT solution, but costs a lot. for medium to
large enterprises totally worth it and actually kind of necassary, for
small business, welcome to the wonderful world of scripting :P.

http://en.wikipedia.org/wiki/Opsware
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-271-273^14711_4000_100__

I know this will probably be out of your price range, but it is sometimes
enlightening to see how large corporations handle this sort of thing.

On Thu, 19 Jun 2008, Rainer Duffner wrote:

> Hi,
>
> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5, FreeBSD,
> Ubuntu and lately Solaris.
> We use these for a variety of reasons and each system does its job quite
> well.
>
> However, patch-management seems to be a weak spot in most cases.
> RedHat offers "RedHat Network", but it costs a lot of money (and they charge
> more if you want to put your servers in groups in the RHN - WTF?)
> FreeBSD offers the portaudit database - we should be able to hack together
> something with that.
> But what about CentOS? If you have an array of CentOS servers - how do you
> track which vulnerabilities each one has?
> Running yum update every night is no option.
>
> Does CentOS also maintain a vulnerability database along the lines of
> FreeBSD?
> How about Solaris?
> Ubuntu?
>
> How do you track vulnerabilities across your datacenter?
>
>
> Regards,
>
> Rainer
>
>
>

[jacob-47]

Security plugin for YUM (which might also handle Redhat)

http://wiki.linux.duke.edu/YumUtils/Plugins/Security?highlight=(Category
Yum)

I haven't tried it but we are just in the process of evaluating/moving
to centos and it's on the todo list.

With Debian I usually just used the "stable" tree for apt which only
updates packages for security. It was never supposed to update the major
version number of a package (i.e. php-4 to php-5). There should be a way
to make Ubuntu do the same thing but I haven't used Ubuntu as a server
platform yet.

-----Original Message-----
From: listbounce@... [mailto:listbounce@...]
On Behalf Of druid@...
Sent: Thursday, June 19, 2008 1:09 PM
To: Rainer Duffner
Cc: focus-linux@...;
focus-linux-return-3196@...
Subject: Re: Vulnerability and Patch-Management in Linux (and other
Unix)

So, if you have the money you can use Opsware Server Automation System
(SAS) which will patch and manage all of those OSes and more. Opsware
was
bought by HP so the product is now called HP Server Automation (HPSA).

To be honest, this is a GREAT solution, but costs a lot. for medium to
large enterprises totally worth it and actually kind of necassary, for
small business, welcome to the wonderful world of scripting :P.

http://en.wikipedia.org/wiki/Opsware
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto
&cp=1-11-271-273^14711_4000_100__

I know this will probably be out of your price range, but it is
sometimes
enlightening to see how large corporations handle this sort of thing.

On Thu, 19 Jun 2008, Rainer Duffner wrote:

> Hi,
>
> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
FreeBSD,
> Ubuntu and lately Solaris.
> We use these for a variety of reasons and each system does its job
quite
> well.
>
> However, patch-management seems to be a weak spot in most cases.
> RedHat offers "RedHat Network", but it costs a lot of money (and they
charge
> more if you want to put your servers in groups in the RHN - WTF?)
> FreeBSD offers the portaudit database - we should be able to hack
together
> something with that.
> But what about CentOS? If you have an array of CentOS servers - how do
you

> track which vulnerabilities each one has?
> Running yum update every night is no option.
>
> Does CentOS also maintain a vulnerability database along the lines of
> FreeBSD?
> How about Solaris?
> Ubuntu?
>
> How do you track vulnerabilities across your datacenter?
>
>
> Regards,
>
> Rainer
>
>
>

No virus found in this incoming message.
Checked by AVG.
Version: 8.0.100 / Virus Database: 270.4.0/1509 - Release Date:
6/19/2008 8:00 AM

[Hari Sekhon]

Rainer Duffner wrote:

> Hi,
>
> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
> FreeBSD, Ubuntu and lately Solaris.
> We use these for a variety of reasons and each system does its job
> quite well.
>
> However, patch-management seems to be a weak spot in most cases.
> RedHat offers "RedHat Network", but it costs a lot of money (and they
> charge more if you want to put your servers in groups in the RHN - WTF?)
> FreeBSD offers the portaudit database - we should be able to hack
> together something with that.
> But what about CentOS? If you have an array of CentOS servers - how do
> you track which vulnerabilities each one has?
> Running yum update every night is no option.
>
> Does CentOS also maintain a vulnerability database along the lines of
> FreeBSD?
> How about Solaris?
> Ubuntu?
>
> How do you track vulnerabilities across your datacenter?
>
>
> Regards,
>
> Rainer
>

For CentOS: Nagios + check_yum (a plugin I wrote for Nagios to test for
updates on RedHat/CentOS servers). You will find it here

http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F2577.html;d=1

You may need to copy and paste that link as the funny links used on
nagiosexchange don't always come out well in mail clients.


For Ubuntu: Nagios + check_apt (from the standard Nagios plugins).

I have checks running every hour to watch for patches on my servers on
these distros.


If you ever rise to Gentoo, I wrote one for that too, you can find that
here in case you need it:

http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Detailed%2F1539.html;d=1


So much for expensive proprietary solutions. Nagios is truly excellent
open source.

-h

--
Hari Sekhon

[Eygene Ryabinkin-2]

Rainer, good day.

Thu, Jun 19, 2008 at 02:58:31PM +0200, Rainer Duffner wrote:
> But what about CentOS? If you have an array of CentOS servers - how do you
> track which vulnerabilities each one has?

Try Pakiti, http://pakiti.sourceforge.net/, it may be of some
interest to the users of RedHat-compatible systems and Ubuntu.
--
Eygene

[Josep L. Guallar-Esteve]

On Thursday 19 June 2008 08:58:31 Rainer Duffner wrote:

> Hi,
>
> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
> FreeBSD, Ubuntu and lately Solaris.
> We use these for a variety of reasons and each system does its job quite
> well.
>
> However, patch-management seems to be a weak spot in most cases.
> RedHat offers "RedHat Network", but it costs a lot of money (and they
> charge more if you want to put your servers in groups in the RHN - WTF?)
> FreeBSD offers the portaudit database - we should be able to hack
> together something with that.
> But what about CentOS? If you have an array of CentOS servers - how do
> you track which vulnerabilities each one has?
> Running yum update every night is no option.
>
> Does CentOS also maintain a vulnerability database along the lines of
> FreeBSD?
> How about Solaris?
> Ubuntu?
>
> How do you track vulnerabilities across your datacenter?
>
>
> Regards,
>
> Rainer

First, get subscribed to all your OS vendor security mailing lists.

Then, set up a series of scripts to do "Test updates" and send an email to the
sysadmin group with the result. A couple of lines of bash scripting plus an
easy cron entry and you are done.

Finally, if there is an update to be available, you can log into the system
and install the update.

If your servers are streamlined, not many updates will affect your server.

We do this in house for our Linux servers.


Regards,
Josep
--
Josep L. Guallar-Esteve - IT Department



This transmission is intended for the use of the entity or individual to which
or whom it is addressed.  The transmission or any documents accompanying the
transmission may contain confidential information.  If you are not the intended
recipient, you are hereby notified that any disclosure, copying, distribution,
or action taken in reliance on the contents of the transmission or the
documents is strictly prohibited. If you have received this confidential
transmission in error, please destroy it and any accompanying documents and
notify the sender immediately.  Thank you.

[Sylvain Robitaille]

On Thu, 19 Jun 2008, Rainer Duffner wrote:

> ..., patch-management seems to be a weak spot in most cases.
> RedHat offers "RedHat Network", but it costs a lot of money (and they
> charge more if you want to put your servers in groups in the RHN -
> WTF?)
> FreeBSD offers the portaudit database - we should be able to hack
> together something with that.
> But what about CentOS? If you have an array of CentOS servers - how do
> you track which vulnerabilities each one has?  Running yum update
> every night is no option.

One might argue that the process of selecting the OS distribution should
certainly involve consideration of patch management, release schedules,
and cost of subscription services from the vendor.

> Does CentOS also maintain a vulnerability database along the lines of
> FreeBSD?
> How about Solaris?
> Ubuntu?

Again, I think these are considerations that should be examined prior
to selecting the OS distribution.

It seems to me at the moment as though the model that is most suitable
to your situation is likely FreeBSD's, so you might want to be looking
at phasing out systems with other OS distributions.  In the (hopefully
small) number of cases where you must use a particular (non-FreeBSD)
OS distribution because of application software support requirements
(for example), you'll likely need to simply adapt the mechanisms that
you find work best.  These would be your "oddball" systems, but your
general environment should ultimately be easier to manage.

> How do you track vulnerabilities across your datacenter?

(for software installed as part of the OS distribution)
Primarily: http://www.therockgarden.ca/software/slackware/UPGRADE.sh
run daily from cron as an unprivileged user, and manually as root when
appropriate (see the script for details of how its behaviour differs
based on privilege).

In truth, the above script runs (unprivileged) on one system (my
workstation), and update packages are copied to each production system,
where the script is manually run as root, only when there's a package
that requires upgrading.  Ideally, I could simply NFS-export (read-only)
the directory where the script stores downloaded update packages, but
I haven't (yet?) done anything like that.  The process could certainly
be improved upon.

We have some oddball (mostly RHEL now, having phased out others) systems
that are handled by their own mechanisms.

I know this doesn't give you much help with your specific systems, but
I hope that it at least gives you food for thought, both during your
next phase of OS distribution selection, and to help resolve the problem
at hand.

--
----------------------------------------------------------------------
Sylvain Robitaille                              syl@...

Systems and Network analyst                       Concordia University
Instructional & Information Technology        Montreal, Quebec, Canada
----------------------------------------------------------------------

[Ram Prasad-7]

Does this help ?

http://www.redhat.com/spacewalk/

- Ram
On Jun 20, 2008, at 2:23 AM, <jacob@...> <jacob@...> wrote:

> Security plugin for YUM (which might also handle Redhat)
>
> http://wiki.linux.duke.edu/YumUtils/Plugins/Security?highlight=(Category
> Yum)
>
> I haven't tried it but we are just in the process of evaluating/moving
> to centos and it's on the todo list.
>
> With Debian I usually just used the "stable" tree for apt which only
> updates packages for security. It was never supposed to update the  
> major
> version number of a package (i.e. php-4 to php-5). There should be a  
> way
> to make Ubuntu do the same thing but I haven't used Ubuntu as a server
> platform yet.
>
> -----Original Message-----
> From: listbounce@... [mailto:listbounce@...
> ]
> On Behalf Of druid@...
> Sent: Thursday, June 19, 2008 1:09 PM
> To: Rainer Duffner
> Cc: focus-linux@...;
> focus-linux-return-3196@...
> Subject: Re: Vulnerability and Patch-Management in Linux (and other
> Unix)
>
> So, if you have the money you can use Opsware Server Automation System
> (SAS) which will patch and manage all of those OSes and more. Opsware
> was
> bought by HP so the product is now called HP Server Automation (HPSA).
>
> To be honest, this is a GREAT solution, but costs a lot. for medium to
> large enterprises totally worth it and actually kind of necassary, for
> small business, welcome to the wonderful world of scripting :P.
>
> http://en.wikipedia.org/wiki/Opsware
> https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto
> &cp=1-11-271-273^14711_4000_100__
>
> I know this will probably be out of your price range, but it is
> sometimes
> enlightening to see how large corporations handle this sort of thing.
>
> On Thu, 19 Jun 2008, Rainer Duffner wrote:
>
>> Hi,
>>
>> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
> FreeBSD,
>> Ubuntu and lately Solaris.
>> We use these for a variety of reasons and each system does its job
> quite
>> well.
>>
>> However, patch-management seems to be a weak spot in most cases.
>> RedHat offers "RedHat Network", but it costs a lot of money (and they
> charge
>> more if you want to put your servers in groups in the RHN - WTF?)
>> FreeBSD offers the portaudit database - we should be able to hack
> together
>> something with that.
>> But what about CentOS? If you have an array of CentOS servers - how  
>> do
> you
>> track which vulnerabilities each one has?
>> Running yum update every night is no option.
>>
>> Does CentOS also maintain a vulnerability database along the lines of
>> FreeBSD?
>> How about Solaris?
>> Ubuntu?
>>
>> How do you track vulnerabilities across your datacenter?
>>
>>
>> Regards,
>>
>> Rainer
>>
>>
>>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 8.0.100 / Virus Database: 270.4.0/1509 - Release Date:
> 6/19/2008 8:00 AM

[John Kunkel]

One word of caution with apt is if you use stable it will get major  
version updates when they move to a new stable project. With later  
installs of etch they have changed the default source.list to use etch  
instead of stable. This prevents any issues when project moves happen.  
I am sure Ubuntu will have something similar.

Hope that is helpful.

John Kunkel


On Jun 19, 2008, at 2:53 PM, <jacob@...> wrote:

> Security plugin for YUM (which might also handle Redhat)
>
> http://wiki.linux.duke.edu/YumUtils/Plugins/Security?highlight=(Category
> Yum)
>
> I haven't tried it but we are just in the process of evaluating/moving
> to centos and it's on the todo list.
>
> With Debian I usually just used the "stable" tree for apt which only
> updates packages for security. It was never supposed to update the  
> major
> version number of a package (i.e. php-4 to php-5). There should be a  
> way
> to make Ubuntu do the same thing but I haven't used Ubuntu as a server
> platform yet.
>
> -----Original Message-----
> From: listbounce@... [mailto:listbounce@...
> ]
> On Behalf Of druid@...
> Sent: Thursday, June 19, 2008 1:09 PM
> To: Rainer Duffner
> Cc: focus-linux@...;
> focus-linux-return-3196@...
> Subject: Re: Vulnerability and Patch-Management in Linux (and other
> Unix)
>
> So, if you have the money you can use Opsware Server Automation System
> (SAS) which will patch and manage all of those OSes and more. Opsware
> was
> bought by HP so the product is now called HP Server Automation (HPSA).
>
> To be honest, this is a GREAT solution, but costs a lot. for medium to
> large enterprises totally worth it and actually kind of necassary, for
> small business, welcome to the wonderful world of scripting :P.
>
> http://en.wikipedia.org/wiki/Opsware
> https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto
> &cp=1-11-271-273^14711_4000_100__
>
> I know this will probably be out of your price range, but it is
> sometimes
> enlightening to see how large corporations handle this sort of thing.
>
> On Thu, 19 Jun 2008, Rainer Duffner wrote:
>
>> Hi,
>>
>> we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
> FreeBSD,
>> Ubuntu and lately Solaris.
>> We use these for a variety of reasons and each system does its job
> quite
>> well.
>>
>> However, patch-management seems to be a weak spot in most cases.
>> RedHat offers "RedHat Network", but it costs a lot of money (and they
> charge
>> more if you want to put your servers in groups in the RHN - WTF?)
>> FreeBSD offers the portaudit database - we should be able to hack
> together
>> something with that.
>> But what about CentOS? If you have an array of CentOS servers - how  
>> do
> you
>> track which vulnerabilities each one has?
>> Running yum update every night is no option.
>>
>> Does CentOS also maintain a vulnerability database along the lines of
>> FreeBSD?
>> How about Solaris?
>> Ubuntu?
>>
>> How do you track vulnerabilities across your datacenter?
>>
>>
>> Regards,
>>
>> Rainer
>>
>>
>>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 8.0.100 / Virus Database: 270.4.0/1509 - Release Date:
> 6/19/2008 8:00 AM

[Rainer Duffner]

Radosław Antoniuk schrieb:
> Hi,
>
> For debian/ubuntu just a simple cure:
> cron-apt - automatic update of packages using apt-get
>
>
>  


Well, the point is: we don't want to have automatic updates.
I'd rather like to be able to answer questions like "Which of my
Linux-boxes
actually does have that stupid privilege escalation bug?"

We have to plan updates very carefully, as not to break
customer-applications (we do managed hosting).
In theory, a yum update shouldn't create a API/ABI breakage - but "In
theory, this shouldn't have happened" is a bad excuse to give to the
customer...

So, I'd like to have a tool at hand that gives me a good overview about
the "state of the datacenter", patch-wise.
Pakiti looks good - I must take a closer look and see how useful it is
in practice.


cheers,
Rainer

[Lee Fisher]

Perhaps AdvChk may be useful.

http://sourceforge.net/projects/advchk

[Jason Spears]

On Fri, Jun 20, 2008 at 12:43 PM, John Kunkel <jkunkel@...> wrote:
> One word of caution with apt is if you use stable it will get major version
> updates when they move to a new stable project. With later installs of etch
> they have changed the default source.list to use etch instead of stable.
> This prevents any issues when project moves happen. I am sure Ubuntu will
> have something similar.


You can always update your sources.list to replace 'stable' by sarge,
etch or whatever you want to track.

Jason

[jacob-47]

Try smpatch on solaris, I just remembered that command, it can be used
to report the missing patchs without attempting to apply them. It won't
help with anything manually installed or installed with blastwave or
from sunfreeware.com though, at least it didn't the last time I used it.

-----Original Message-----
From: listbounce@... [mailto:listbounce@...]
On Behalf Of Rainer Duffner
Sent: Thursday, June 19, 2008 5:59 AM
To: focus-linux@...
Subject: Vulnerability and Patch-Management in Linux (and other Unix)

Hi,

we've amassed a veritable "zoo" of Unix-versions: RHEL4+5, CentOS5,
FreeBSD, Ubuntu and lately Solaris.
We use these for a variety of reasons and each system does its job quite

well.

However, patch-management seems to be a weak spot in most cases.
RedHat offers "RedHat Network", but it costs a lot of money (and they
charge more if you want to put your servers in groups in the RHN - WTF?)
FreeBSD offers the portaudit database - we should be able to hack
together something with that.
But what about CentOS? If you have an array of CentOS servers - how do
you track which vulnerabilities each one has?
Running yum update every night is no option.

Does CentOS also maintain a vulnerability database along the lines of
FreeBSD?
How about Solaris?
Ubuntu?

How do you track vulnerabilities across your datacenter?


Regards,

Rainer

No virus found in this incoming message.
Checked by AVG.
Version: 8.0.100 / Virus Database: 270.4.0/1509 - Release Date:
6/19/2008 8:00 AM

[Kosala Atapattu-2]

>> For debian/ubuntu just a simple cure:
>> cron-apt - automatic update of packages using apt-get
>
> Well, the point is: we don't want to have automatic updates.
> I'd rather like to be able to answer questions like "Which of my Linux-boxes
> actually does have that stupid privilege escalation bug?"
>
> We have to plan updates very carefully, as not to break customer-applications (we do managed hosting).
> In theory, a yum update shouldn't create a API/ABI breakage - but "In theory, this shouldn't have happened" is a bad excuse to give to the customer...

Completely agree with you on this. And I would like to add something
related, I do not think it's a good approach to update every single
piece of software on a production system, just because there is a new
version. For example I would not update a NTP client if I'm not using
it withing any of my applications.

--
Kosala
--------------------------------------------
Disclaimer: Views expressed in this mail are my personal views and
they would not reflect views of the employer.
--------------------------------------------
blog.kosala.net
www.linux.lk/~kosala/
www.kosala.net