local vs. global counts for checksums

[Matus UHLAR - fantomas]

Hello,

since many phishing attacks targeted on our company in the near past, and
resulting spam outbreaks of which we are a victim for some time, I would
like to know if we can have some database of checksums that appeared locally
(at our servers) for a MANY times, where MANY would be different number than
global MANY.

The logical alternative is to run DCC servers only for our company (which
requires commercial version of DCC), and always query both servers with
public and private checksum databases.

Whle I don't have problems running commercial DCC, thiis would also require
double checking for checksums in both MTA and SpamAssassin, which I found a
bit hard to implement, unless some (commercial?) version implements it.

Any recommendations about this problem?

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Linux is like a teepee: no Windows, no Gates and an apache inside...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> From: Matus UHLAR - fantomas <uhlar@...>
> To: dcc@...

> since many phishing attacks targeted on our company in the near past, and
> resulting spam outbreaks of which we are a victim for some time, I would
> like to know if we can have some database of checksums that appeared locally
> (at our servers) for a MANY times, where MANY would be different number than
> global MANY.

Why would you need 2 different MANY numbers?  Why not share the
checksums of phishing attacks targeting your company with the world?
Some organizations apply DCC checks on their out-going email.  By
sharing phishing attacks targeting your company, you might stop
some of them at their soruce and possibly even alert the owners of
the source networks?

> The logical alternative is to run DCC servers only for our company (which
> requires commercial version of DCC), and always query both servers with
> public and private checksum databases.

Whether you need to buy a license for commercial use of DCC is
unrelated to whether you run your own DCC servers.  Some organizations
have commercial DCC licenses but do not run DCC servers.  Other
organizations run private DCC servers using the free DCC version.
You need a commercial license if you sell anti-spam appliances or
services, or if you do not share your checksums.


> Whle I don't have problems running commercial DCC, thiis would also require
> double checking for checksums in both MTA and SpamAssassin, which I found a
> bit hard to implement, unless some (commercial?) version implements it.
>
> Any recommendations about this problem?

The best way to use DCC is during the original SMTP transaction,
and so in the MTA.  I think it would be easy to configure Postfix
or sendmail to consult 2 sets of DCC servers.  With Postfix, use
two dccifd daemons as before-queue filters.  With sendmail, add two
Xdcc lines differing in DCC home directories.

If you must apply DCC checks after the SMTP transaction, I think
it would be straight forward to hack a copy of the SpamAssassin
DCC.pm to use a second set of parameters and so consult a second
dccifd daemon  The second module would be called something like DCC2.pm.
I'd probably write a sed recipe to generate DCC2.pm from DCC.pm from
apache.org or the misc directory in DCC source to ease handling updates.



Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Matus UHLAR - fantomas]

> > From: Matus UHLAR - fantomas <uhlar@...>
> > To: dcc@...
>
> > since many phishing attacks targeted on our company in the near past, and
> > resulting spam outbreaks of which we are a victim for some time, I would
> > like to know if we can have some database of checksums that appeared locally
> > (at our servers) for a MANY times, where MANY would be different number than
> > global MANY.

On 21.03.11 14:57, Vernon Schryver wrote:
> Why would you need 2 different MANY numbers?

because I found mail that appeared 1000 times on our servers much more
suspect than mail that appeared 1000 times in the whole world, while
only 5 times on our servers. The same can apply for slovakia etc...

Or is there I don't understand correctly about DCC?

> Why not share the
> checksums of phishing attacks targeting your company with the world?

Of course, but the difference above is what I'm interested in.

[...]

> Whether you need to buy a license for commercial use of DCC is
> unrelated to whether you run your own DCC servers.  Some organizations
> have commercial DCC licenses but do not run DCC servers.  Other
> organizations run private DCC servers using the free DCC version.
> You need a commercial license if you sell anti-spam appliances or
> services, or if you do not share your checksums.

that's it :)

> > Whle I don't have problems running commercial DCC, thiis would also require
> > double checking for checksums in both MTA and SpamAssassin, which I found a
> > bit hard to implement, unless some (commercial?) version implements it.
> >
> > Any recommendations about this problem?
>
> The best way to use DCC is during the original SMTP transaction,
> and so in the MTA.

Reporting in the MTA, checking in the spam filter is what I want to achieve
:)

> I think it would be easy to configure Postfix or sendmail to consult 2
> sets of DCC servers.  With Postfix, use two dccifd daemons as before-queue
> filters.  With sendmail, add two Xdcc lines differing in DCC home
> directories.
>
> If you must apply DCC checks after the SMTP transaction, I think
> it would be straight forward to hack a copy of the SpamAssassin
> DCC.pm to use a second set of parameters and so consult a second
> dccifd daemon  The second module would be called something like DCC2.pm.
> I'd probably write a sed recipe to generate DCC2.pm from DCC.pm from
> apache.org or the misc directory in DCC source to ease handling updates.

well, I'll goo this way if I won't find a better one.
--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Boost your system's speed by 500% - DEL C:\WINDOWS\*.*
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> From: Matus UHLAR - fantomas <uhlar@...>
> To: dcc@...

> because I found mail that appeared 1000 times on our servers much more
> suspect than mail that appeared 1000 times in the whole world, while
> only 5 times on our servers. The same can apply for slovakia etc...
>
> Or is there I don't understand correctly about DCC?

I think DCC detcts bulk mail but spam bulk mail that is also unsolicted.
I have always said that local whitelists are need to distinguish solicted
from unsolicited bulk mail.  However, most people disagree with me and
use DCC as if it detected spam.  That is because maintaining whitelists
for online order confirmations etc. is too much work for lusers.

A message that has been reported 1000 times to DCC servers is surely
bulk mail.


> > The best way to use DCC is during the original SMTP transaction,
> > and so in the MTA.
>
> Reporting in the MTA, checking in the spam filter is what I want to achieve
> :)

Checking spam in the spam filter after the end of SMTP transaction is
very popular but I think wrong.  It is wrong because false positives
(messages that are wrongly detected as spam) either disappear into
blackholes or are "bounced" in non-delivery reports (NDRs).
True positives (correctly detected spam) that is "bounced" often results
in "backscatter" to innocent third parties whose mailboxes have been
forged as the sender of of the spam.  Too much backscatter and you will
be blacklisted by enough of the Internet.
So bouncing spam is even worse than discarding it.

On the other hand, if you detect spam during the SMTP transaction reject
it with SMTP 5yz error codes, then the legitimate senders of false
positives are informed by their own MTA and know to try something else.
As a bonus, some lawful unsolicted bulk email advertisers honor persistent
SMTP rejections of their junk with an automatic "remove."

Unlike treating DCC as a spam filter instead of a bulk filter, this
is due to mail system operators instead of end users.  There is no
technical or administrative reason to delay spam filtering until after
the SMTP transaction.  Delaying does not save CPU cycles, because they
must be spent eventually.  You also do not need to delay filtering to
generate logs of rejected mail messages including message bodies as
the DCC client programs dccifd and dccm demonstrate.


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Matus UHLAR - fantomas]

> > From: Matus UHLAR - fantomas <uhlar@...>
> > To: dcc@...
>
> > because I found mail that appeared 1000 times on our servers much more
> > suspect than mail that appeared 1000 times in the whole world, while
> > only 5 times on our servers. The same can apply for slovakia etc...
> >
> > Or is there I don't understand correctly about DCC?

On 21.03.11 19:19, Vernon Schryver wrote:
> I think DCC detcts bulk mail but spam bulk mail that is also unsolicted.
> I have always said that local whitelists are need to distinguish solicted
> from unsolicited bulk mail.  However, most people disagree with me and
> use DCC as if it detected spam.  That is because maintaining whitelists
> for online order confirmations etc. is too much work for lusers.
>
> A message that has been reported 1000 times to DCC servers is surely
> bulk mail.

I see your point, and you can count me to those who think that maintaining
bunch of whitelists is a stuff that should be avoided whenever possible...

however I also think that there are different kinds/levels of bulkiness that
could have different scores and/or different ways to get handled.

> > > The best way to use DCC is during the original SMTP transaction,
> > > and so in the MTA.
> >
> > Reporting in the MTA, checking in the spam filter is what I want to achieve
> > :)
>
> Checking spam in the spam filter after the end of SMTP transaction is
> very popular but I think wrong.

Rejecting clear spam (SA score >10) while keeping the rest for later recheck
or delivering suspicious mail to spam folder is OK I think.
However since I don't plan to reject all bulk messages, I keep spamassassin
to work with the scores.

> Unlike treating DCC as a spam filter instead of a bulk filter, this
> is due to mail system operators instead of end users.  There is no
> technical or administrative reason to delay spam filtering until after
> the SMTP transaction.  Delaying does not save CPU cycles, because they
> must be spent eventually.  You also do not need to delay filtering to
> generate logs of rejected mail messages including message bodies as
> the DCC client programs dccifd and dccm demonstrate.

the same mail for multiple users can be scanned two times: first with global
set of rules at SMTP level, second time with per-user filters (and their
whitelists). First scan will report checksums to DCC, while the later will
only use them. While this doesn't space CPU cycles (at least for mail that
is not rejected in the first phase), it gives best results...

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Silvester Stallone: Father of the RISC concept.
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Matus UHLAR - fantomas]

On 21.03.11 14:57, Vernon Schryver wrote:
> You need a commercial license if you sell anti-spam appliances or
> services, or if you do not share your checksums.

so if I have two servers, one of which shares my checksums, I can run them
without commercial license?

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> From: Matus UHLAR - fantomas <uhlar@...>

> however I also think that there are different kinds/levels of bulkiness that
> could have different scores and/or different ways to get handled.

If you would say there are various kinds of mail that is usually bulk
and that has various probabilities of being unsolicited, I could agree.
But I think it is wrong to talk about different kinds of bulkiness
other than numbers of copies.  Precise language is important, because
sloppy language cause sloppy thinking, and sloppy thinking causes bad
results.  This week I had a discussion with a spammer who insisted
that his burst of around 100,000 identical messages was not "bulk mail"
because he claimed he wasn't selling anything.  



> Rejecting clear spam (SA score >10) while keeping the rest for later recheck
> or delivering suspicious mail to spam folder is OK I think.
> However since I don't plan to reject all bulk messages, I keep spamassassin
> to work with the scores.

SpamAssassin is like every other spam filter and imperfect.  If you
set the scoring so that SA can ever detect anything, than some
legitimate email will have a score >10 or whatever threshold you
choose.  In your configuration, such legitimate mail or false
positives will disappear into blackholes.

On the other hand, if you would do all SA scanning during the SMTP
transaction, you could reject instead of accept any mail that you
might eventually not deliver.  That prevents blackholes.  SpamAssassin
can be run in popular MTAs including sendmail and postfix so that
the SA tests can be completed before the end of the transaction and
so you could give 5yz response to any email not delivered.


> the same mail for multiple users can be scanned two times: first with global
> set of rules at SMTP level, second time with per-user filters (and their
> whitelists).

As I think dccifd+postfix and dccm+sendmail demonstrate, there is no
technical reason that absolutely prevents doing do both global and
per-user scanning in the MTA during the original SMTP transaction.
(You can deal with single response code to the DATA command by
temporarily rejecting second and later Rcpt_To mailboxes that have
whitelists or other settings that differ from the first Rcpt_To
value.)


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> > You need a commercial license if you sell anti-spam appliances or
> > services, or if you do not share your checksums.
>
> so if I have two servers, one of which shares my checksums, I can run them
> without commercial license?

If a server does not share its checksums, then it requires a
commercial license.

However, why would your internall server not share its checksums?
Why could it not flood its checksums to your other server while
refusing all incoming checksums with /var/dcc/flod lines like these:

/var/dcc/flod on external server at outside.domain.com:
  # connection to outside peers
   outside.example.com    5438 9876
   # connection to internal server
   inside.domain.com      4583 8967  all->reject

/var/dcc/flod on internal server at inside.domain.com:
   # connection to internal server
   outside.domain.com      8583 8967  -  all->reject


I wonder
   - if a single DCC server with two copies of dccifd with differing
       thresholds would be sufficient
   - if there will be enough email traffic to provide enough data for
       the inside DCC server to do any good


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Matus UHLAR - fantomas]

> > > You need a commercial license if you sell anti-spam appliances or
> > > services, or if you do not share your checksums.
> >
> > so if I have two servers, one of which shares my checksums, I can run them
> > without commercial license?

On 23.03.11 19:51, Vernon Schryver wrote:

> If a server does not share its checksums, then it requires a
> commercial license.
>
> However, why would your internall server not share its checksums?
> Why could it not flood its checksums to your other server while
> refusing all incoming checksums with /var/dcc/flod lines like these:
>
> /var/dcc/flod on external server at outside.domain.com:
>   # connection to outside peers
>    outside.example.com    5438 9876
>    # connection to internal server
>    inside.domain.com      4583 8967  all->reject
>
> /var/dcc/flod on internal server at inside.domain.com:
>    # connection to internal server
>    outside.domain.com      8583 8967  -  all->reject

> I wonder
>    - if a single DCC server with two copies of dccifd with differing
>        thresholds would be sufficient

I was thinking about such behaviour, but the main problem was always to
distinguish between checksums of local and global mail.

>    - if there will be enough email traffic to provide enough data for
>        the inside DCC server to do any good

still worth trying I guess :)

but, yes, I will think about all the stuff
--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
WinError #98652: Operation completed successfully.
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc