Please evaluate my understanding wrt access files

[RobertLopez]

I would like to confirm my understanding about access files.

Please let me know if any of this is not correct...

The man (5) access description describes a prototype file, where that
file could be a single file describing any host names, network
addresses, envelope senders or recipient addresses.

The file could also be a set of files all following the same format
rules.

Where such files might be
recipient_checks, helo_checks, sender_checks, client_checks, etc.


The usefulness of the content of an access file is dependent upon the
parameter that selects a routine that reads the file.

If check_client_access causes a read of the file it will only be
looking for IP addresses of a client server that sent the email or a
fully qualified domain name that successfully reverse maps to the IP
address of a client server that sent the email.

If check_sender_access causes a read of the file it will only be
looking for an email SMTP "MAIL FROM" address or a pattern which could
be a part that email address to the left of the "@" sign.

If check_helo_access causes a read of the file it will only be looking
for the HELO or EHLO hostname or any valid parent domain of that
hostname that is in the SMTP HELO.

The routines executed vi the parameters such as check_client_access,
check_sender_access, check_helo_access, etc. return the value the
check to the routine that called for the check where the calling
routine would be instigated by any of these parameters:

smtpd_client_restrictions
smtpd_helo_restrictions
smtpd_sender_restrictions
smtpd_recipient_restrictions
smtpd_data_restrictions

It is possible to have all the lookups done on a single
.../postfix/access.db file but that could mean the file gets confusing
so in practice multiple access files with names like client_access,
helo_access, sender_access, etc.

A single parameter such as check_client_access may be called multiple
times in a situation like this:

smtpd_sender_restrictions =
  check_sender_access hash:/etc/postfix/greylist
  check_sender_access hash:/etc/postfix/sender_access
  permit_mynetworks


However if the above causes a pattern to be found more than once then
only the last pattern match is used. (I think that is what "When the
same parameter is defined multiple times, only the last instance is
remembered." means.)

This is how I am putting this in practice on a new virtual server
where I hope to fix some problems on current production servers:

root@mg0x:/etc/postfix# postconf -d mail_version
mail_version = 2.5.5

I am using 2.5.5 because that is the latest from Ubuntu.


root@mg0x:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = yes
biff = no
bounce_size_limit = 1
config_directory = /etc/postfix
default_process_limit = 400
header_checks = regexp:/etc/postfix/header_checks
inet_interfaces = all
mailbox_size_limit = 0
masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org
max_use = 100
message_size_limit = 16777216
mydestination = $myhostname, $mydomain, localhost.localdomain,
cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu,
nmvc.org, mail.nmvc.org, mg0x.nmvc.org,  mg04.nmvc.org, mg05.nmvc.org,
mg06.nmvc,  nmvirtualcollege.org, mail.nmvirtualcollege.org,
mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org,
mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org,  nmln.net,
ideal-nm.org, ideal-nm.net,  idealnm.org, idealnm.net
myhostname = mg0x.cnm.edu
mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24,
172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8
[::ffff:127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
notify_classes = resource, software
readme_directory = no
recipient_delimiter = +
relay_domains = $mydestination
relayhost =
smtp_host_lookup = dns, native
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = cnm.edu
smtpd_client_restrictions = permit_mynetworks check_client_access
hash:/etc/postfix/access reject_rbl_client
zen.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client
dnsbl.njabl.org permit
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks reject_invalid_hostname
smtpd_recipient_restrictions = check_recipient_access
hash:/etc/postfix/overquota reject_non_fqdn_sender reject_unknown_sender_domain reject_non_fqdn_recipient reject_unknown_recipient_domain reject_unlisted_recipient permit_mynetworks reject_unauth_destination reject_unauth_pipeliningreject_invalid_helo_hostname
        reject_non_fqdn_helo_hostname reject_rbl_client zen.spamhaus.org
smtpd_sender_restrictions = check_sender_access
hash:/etc/postfix/greylist check_sender_access
hash:/etc/postfix/sender_access permit_mynetworks
        reject_unknown_sender_domain
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
virtual_alias_maps = hash:/etc/postfix/virtualaliases



--
Robert Lopez
Unix Systems Administrator
Central New Mexico Community College (CNM)
525 Buena Vista SE
Albuquerque, New Mexico 87106

[Noel Jones-2]

On 10/30/2009 12:55 PM, Robert Lopez wrote:

> I would like to confirm my understanding about access files.
>
> Please let me know if any of this is not correct...
>
> The man (5) access description describes a prototype file, where that
> file could be a single file describing any host names, network
> addresses, envelope senders or recipient addresses.
>
> The file could also be a set of files all following the same format
> rules.
>
> Where such files might be
> recipient_checks, helo_checks, sender_checks, client_checks, etc.
>
>
> The usefulness of the content of an access file is dependent upon the
> parameter that selects a routine that reads the file.
>
> If check_client_access causes a read of the file it will only be
> looking for IP addresses of a client server that sent the email or a
> fully qualified domain name that successfully reverse maps to the IP
> address of a client server that sent the email.
>
> If check_sender_access causes a read of the file it will only be
> looking for an email SMTP "MAIL FROM" address or a pattern which could
> be a part that email address to the left of the "@" sign.
>
> If check_helo_access causes a read of the file it will only be looking
> for the HELO or EHLO hostname or any valid parent domain of that
> hostname that is in the SMTP HELO.
>
> The routines executed vi the parameters such as check_client_access,
> check_sender_access, check_helo_access, etc. return the value the
> check to the routine that called for the check where the calling
> routine would be instigated by any of these parameters:
>
> smtpd_client_restrictions
> smtpd_helo_restrictions
> smtpd_sender_restrictions
> smtpd_recipient_restrictions
> smtpd_data_restrictions
>
> It is possible to have all the lookups done on a single
> .../postfix/access.db file but that could mean the file gets confusing
> so in practice multiple access files with names like client_access,
> helo_access, sender_access, etc.

so far so good...

>
> A single parameter such as check_client_access may be called multiple
> times in a situation like this:
>
> smtpd_sender_restrictions =
>   check_sender_access hash:/etc/postfix/greylist
>   check_sender_access hash:/etc/postfix/sender_access
>   permit_mynetworks
>
>
> However if the above causes a pattern to be found more than once then
> only the last pattern match is used. (I think that is what "When the
> same parameter is defined multiple times, only the last instance is
> remembered." means.)

No, the "last parameter is used" rule refers to when indexing
a single map -- when the map is indexed with postmap, a
duplicate key will overwrite the previous identical key.

With multiple maps -- or multiple restrictions of any type --
in smtpd_*_restrictions, the first match wins.

Postfix places no limit on how many maps you can use, but
there is system overhead with each map.  Rule of thumb is to
combine maps wherever possible -- don't use two
check_sender_access statements if you can do it with one.
The smart way to do this is use a Makefile to build a single
map from multiple similar input files.

>
> This is how I am putting this in practice on a new virtual server
> where I hope to fix some problems on current production servers:
>
> root@mg0x:/etc/postfix# postconf -d mail_version
> mail_version = 2.5.5
>
> I am using 2.5.5 because that is the latest from Ubuntu.
>
>
> root@mg0x:/etc/postfix# postconf -n
> alias_database = hash:/etc/aliases
> alias_maps = hash:/etc/aliases
> append_dot_mydomain = yes
> biff = no
> bounce_size_limit = 1
> config_directory = /etc/postfix
> default_process_limit = 400
> header_checks = regexp:/etc/postfix/header_checks
> inet_interfaces = all
> mailbox_size_limit = 0
> masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org
> max_use = 100
> message_size_limit = 16777216
> mydestination = $myhostname, $mydomain, localhost.localdomain,
> cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu,
> nmvc.org, mail.nmvc.org, mg0x.nmvc.org,  mg04.nmvc.org, mg05.nmvc.org,
> mg06.nmvc,  nmvirtualcollege.org, mail.nmvirtualcollege.org,
> mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org,
> mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org,  nmln.net,
> ideal-nm.org, ideal-nm.net,  idealnm.org, idealnm.net

Lots of domains in mydestination...  Are you sure these don't
belong in relay_domains instead?
http://www.postfix.org/BASIC_CONFIGURATION_README.html
http://www.postfix.org/STANDARD_CONFIGURATION_README.html
http://www.postfix.org/ADDRESS_CLASS_README.html

> myhostname = mg0x.cnm.edu
> mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24,
> 172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8
> [::ffff:127.0.0.0]/104 [::1]/128
> myorigin = /etc/mailname
> notify_classes = resource, software
> readme_directory = no
> recipient_delimiter = +
> relay_domains = $mydestination

relay_domains should be set explicitly, and generally should
not include $mydestination.  If there are no relay_domains, it
should be set empty.
http://www.postfix.org/ADDRESS_CLASS_README.html

> relayhost =
> smtp_host_lookup = dns, native
> smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
> smtpd_banner = cnm.edu

Should be "cnm.edu ESTMP", or better, just leave it at the
default.

> smtpd_client_restrictions = permit_mynetworks check_client_access
> hash:/etc/postfix/access reject_rbl_client
> zen.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client
> dnsbl.njabl.org permit

OK.  The final "permit" is a no-op and can be removed, but it
doesn't hurt anything.

> smtpd_helo_required = yes
> smtpd_helo_restrictions = permit_mynetworks reject_invalid_hostname
> smtpd_recipient_restrictions = check_recipient_access
> hash:/etc/postfix/overquota reject_non_fqdn_sender reject_unknown_sender_domain reject_non_fqdn_recipient reject_unknown_recipient_domain reject_unlisted_recipient permit_mynetworks reject_unauth_destination reject_unauth_pipeliningreject_invalid_helo_hostname
> reject_non_fqdn_helo_hostname reject_rbl_client zen.spamhaus.org

Caution!  lookup tables before reject_unauth_destination can
make you an open relay if something inintended gets put there.
  If you want to reject overquota recipients before
permit_mynetworks, do it in smtpd_sender_restrictions.
(Yes, you can use check_*_access in smtpd_**_restrictions,
with the default "smtpd_delay_reject = yes".)

> smtpd_sender_restrictions = check_sender_access
> hash:/etc/postfix/greylist check_sender_access
> hash:/etc/postfix/sender_access permit_mynetworks
> reject_unknown_sender_domain

Seems like permit_mynetworks should come before greylist or
other sender access checks.

> smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
> smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
> smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
> smtpd_use_tls = yes
> virtual_alias_maps = hash:/etc/postfix/virtualaliases

   -- Noel Jones

[RobertLopez]

On Fri, Oct 30, 2009 at 1:26 PM, Noel Jones <njones@...> wrote:

> On 10/30/2009 12:55 PM, Robert Lopez wrote:
>>
>> I would like to confirm my understanding about access files.
>>
>> Please let me know if any of this is not correct...
>>
>> The man (5) access description describes a prototype file, where that
>> file could be a single file describing any host names, network
>> addresses, envelope senders or recipient addresses.
>>
>> The file could also be a set of files all following the same format
>> rules.
>>
>> Where such files might be
>> recipient_checks, helo_checks, sender_checks, client_checks, etc.
>>
>>
>> The usefulness of the content of an access file is dependent upon the
>> parameter that selects a routine that reads the file.
>>
>> If check_client_access causes a read of the file it will only be
>> looking for IP addresses of a client server that sent the email or a
>> fully qualified domain name that successfully reverse maps to the IP
>> address of a client server that sent the email.
>>
>> If check_sender_access causes a read of the file it will only be
>> looking for an email SMTP "MAIL FROM" address or a pattern which could
>> be a part that email address to the left of the "@" sign.
>>
>> If check_helo_access causes a read of the file it will only be looking
>> for the HELO or EHLO hostname or any valid parent domain of that
>> hostname that is in the SMTP HELO.
>>
>> The routines executed vi the parameters such as check_client_access,
>> check_sender_access, check_helo_access, etc. return the value the
>> check to the routine that called for the check where the calling
>> routine would be instigated by any of these parameters:
>>
>> smtpd_client_restrictions
>> smtpd_helo_restrictions
>> smtpd_sender_restrictions
>> smtpd_recipient_restrictions
>> smtpd_data_restrictions
>>
>> It is possible to have all the lookups done on a single
>> .../postfix/access.db file but that could mean the file gets confusing
>> so in practice multiple access files with names like client_access,
>> helo_access, sender_access, etc.
>
> so far so good...
>
>
>>
>> A single parameter such as check_client_access may be called multiple
>> times in a situation like this:
>>
>> smtpd_sender_restrictions =
>>        check_sender_access hash:/etc/postfix/greylist
>>        check_sender_access hash:/etc/postfix/sender_access
>>        permit_mynetworks
>>
>>
>> However if the above causes a pattern to be found more than once then
>> only the last pattern match is used. (I think that is what "When the
>> same parameter is defined multiple times, only the last instance is
>> remembered." means.)
>
> No, the "last parameter is used" rule refers to when indexing a single map
> -- when the map is indexed with postmap, a duplicate key will overwrite the
> previous identical key.
>
> With multiple maps -- or multiple restrictions of any type -- in
> smtpd_*_restrictions, the first match wins.

Ah! Good. That now makes more sense.

>
> Postfix places no limit on how many maps you can use, but there is system
> overhead with each map.  Rule of thumb is to combine maps wherever possible
> -- don't use two check_sender_access statements if you can do it with one.
> The smart way to do this is use a Makefile to build a single map from
> multiple similar input files.

That is interesting. What is the advantage of that over directly
editing a single file?
I can see having unique names that pair with the parameters that cause
them to be read.
It is not clear to me what the benefit of multiple files is beyond
this association.

We do something similar with the virtualaliases table. There is a
table that has all
college employees who use an Exchange server, another that has all customers
(students) who use Sungard Luminis, and a third that has Mailman lists. So email
is delivered to one of those three systems based on that file. We
build that single file
from three separate files.

>
>>
>> This is how I am putting this in practice on a new virtual server
>> where I hope to fix some problems on current production servers:
>>
>> root@mg0x:/etc/postfix# postconf -d mail_version
>> mail_version = 2.5.5
>>
>> I am using 2.5.5 because that is the latest from Ubuntu.
>>
>>
>> root@mg0x:/etc/postfix# postconf -n
>> alias_database = hash:/etc/aliases
>> alias_maps = hash:/etc/aliases
>> append_dot_mydomain = yes
>> biff = no
>> bounce_size_limit = 1
>> config_directory = /etc/postfix
>> default_process_limit = 400
>> header_checks = regexp:/etc/postfix/header_checks
>> inet_interfaces = all
>> mailbox_size_limit = 0
>> masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org
>> max_use = 100
>> message_size_limit = 16777216
>> mydestination = $myhostname, $mydomain, localhost.localdomain,
>> cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu,
>> nmvc.org, mail.nmvc.org, mg0x.nmvc.org,  mg04.nmvc.org, mg05.nmvc.org,
>> mg06.nmvc,  nmvirtualcollege.org, mail.nmvirtualcollege.org,
>> mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org,
>> mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org,  nmln.net,
>> ideal-nm.org, ideal-nm.net,  idealnm.org, idealnm.net
>
> Lots of domains in mydestination...  Are you sure these don't belong in
> relay_domains instead?
> http://www.postfix.org/BASIC_CONFIGURATION_README.html
> http://www.postfix.org/STANDARD_CONFIGURATION_README.html
> http://www.postfix.org/ADDRESS_CLASS_README.html

No, I am not sure.

All email going students and employees are sent to either Sungard Luminis
servers or to Microsoft Exchange servers. There are Mailman lists that have
membership beyond this college. Members of some lists include state employees,
other college and university employees, state high school employees,
etc. So, for
example steering-board@... is a list and any mail sent to it
is then sent to
persons at many different other organizations.  The nmvirtualcollege is moving
toward a single service for students of many different colleges who take on-line
courses from a number of different colleges via a single web access point.

Does that mean they belong in relay_domains?

>
>> myhostname = mg0x.cnm.edu
>> mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24,
>> 172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8
>> [::ffff:127.0.0.0]/104 [::1]/128
>> myorigin = /etc/mailname
>> notify_classes = resource, software
>> readme_directory = no
>> recipient_delimiter = +
>> relay_domains = $mydestination
>
> relay_domains should be set explicitly, and generally should not include
> $mydestination.  If there are no relay_domains, it should be set empty.
> http://www.postfix.org/ADDRESS_CLASS_README.html

This may be another point where I am confused. I am thinking relay is when a
postfix server accepts email for user@... and then rewrites that address
to what is found in a table for the user where the email is then sent
to user@....
I may have to read that a few times to get it all straight.

>
>> relayhost =
>> smtp_host_lookup = dns, native
>> smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
>> smtpd_banner = cnm.edu
>
> Should be "cnm.edu ESTMP", or better, just leave it at the default.

Is that "cnm.edu ESMTP" or default?

>
>> smtpd_client_restrictions = permit_mynetworks   check_client_access
>> hash:/etc/postfix/access        reject_rbl_client
>> zen.spamhaus.org        reject_rbl_client bl.spamcop.net
>>  reject_rbl_client
>> dnsbl.njabl.org permit
>
> OK.  The final "permit" is a no-op and can be removed, but it doesn't hurt
> anything.
>
>> smtpd_helo_required = yes
>> smtpd_helo_restrictions = permit_mynetworks     reject_invalid_hostname
>> smtpd_recipient_restrictions = check_recipient_access
>> hash:/etc/postfix/overquota     reject_non_fqdn_sender
>>  reject_unknown_sender_domain    reject_non_fqdn_recipient
>> reject_unknown_recipient_domain reject_unlisted_recipient
>> permit_mynetworks       reject_unauth_destination
>> reject_unauth_pipeliningreject_invalid_helo_hostname
>>        reject_non_fqdn_helo_hostname   reject_rbl_client zen.spamhaus.org
>
> Caution!  lookup tables before reject_unauth_destination can make you an
> open relay if something inintended gets put there.  If you want to reject
> overquota recipients before permit_mynetworks, do it in
> smtpd_sender_restrictions.
> (Yes, you can use check_*_access in smtpd_**_restrictions, with the default
> "smtpd_delay_reject = yes".)

OK

>
>> smtpd_sender_restrictions = check_sender_access
>> hash:/etc/postfix/greylist      check_sender_access
>> hash:/etc/postfix/sender_access         permit_mynetworks
>>        reject_unknown_sender_domain
>
> Seems like permit_mynetworks should come before greylist or other sender
> access checks.

Seems like? :-)  This greylist program reads the tail of the mail log
and looks for
bursts of email in a short period of time. The program adds the bursting account
to the map which causes the email to be deferred.

>
>> smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
>> smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
>> smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
>> smtpd_use_tls = yes
>> virtual_alias_maps = hash:/etc/postfix/virtualaliases
>
>  -- Noel Jones
>

Thanks for the education assistance.

--
Robert Lopez
Unix Systems Administrator
Central New Mexico Community College (CNM)
525 Buena Vista SE
Albuquerque, New Mexico 87106

[Stan Hoeppner]

Robert Lopez put forth on 10/30/2009 6:57 PM:

> It is not clear to me what the benefit of multiple files is beyond
> this association.

Organization and ease of management for one.  For example:

smtpd_client_restrictions =
        check_recipient_access hash:/etc/postfix/access
        check_client_access hash:/etc/postfix/access
        pcre:/etc/postfix/check_client_fqdn.pcre
        hash:/etc/postfix/coolsavings.access
        hash:/etc/postfix/richk-1.access
        cidr:/etc/postfix/cidr_files/china.cidr
        cidr:/etc/postfix/cidr_files/korea.cidr
        cidr:/etc/postfix/cidr_files/russia.cidr
        cidr:/etc/postfix/cidr_files/ukraine.cidr
        cidr:/etc/postfix/cidr_files/malaysia.cidr
        cidr:/etc/postfix/cidr_files/belarus.cidr
        cidr:/etc/postfix/cidr_files/indonesia.cidr
        cidr:/etc/postfix/cidr_files/hongkong.cidr
        cidr:/etc/postfix/cidr_files/africa.cidr
        cidr:/etc/postfix/cidr_files/romania.cidr
        cidr:/etc/postfix/cidr_files/thailand.cidr
        cidr:/etc/postfix/cidr_files/poland.cidr
        cidr:/etc/postfix/cidr_files/spammer.cidr
        cidr:/etc/postfix/cidr_files/hurricane-electric.cidr
        cidr:/etc/postfix/cidr_files/richk-1.cidr
        pcre:/etc/postfix/access.pcre

My access file contains some whitelist email addresses, some whitelist
domains, some blacklist domains, and some whitelist and blacklist IP
addresses, so I do have some consolidation in the one file and I use it
in multiple restriction classes.  However, it's by far my smallest table
file.  Some of my cidr files are pretty large.  Note that I'm using the
IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp
connections.  Some of those files have thousands of entries.  Note also
that I'm using multiple table types, hash, cidr, and pcre.  It's better
to use multiple files in this kind of setup.

> This may be another point where I am confused. I am thinking relay is when a
> postfix server accepts email for user@... and then rewrites that address
> to what is found in a table for the user where the email is then sent
> to user@....
> I may have to read that a few times to get it all straight.

No, relay strictly means Postfix is not the final destination for a
given domain, thus Postfix relays the email to the appropriate server,
or rejects the email if Postfix is not configured to relay for a given
domain.  See:

http://www.postfix.org/postconf.5.html#relay_domains

> Is that "cnm.edu ESMTP" or default?

It's a simple banner message displayed to a remote host when it connects
to deliver mail to your MX.  You don't need to define it.  Postfix does
it for you.  Just delete that line from main.cf.  Humans never see it
anyway, unless they telnet to your port TCP 25 for testing or something.
 Here's an example of the Postfix default banner (mine):

220 greer.hardwarefreak.com ESMTP Postfix

Here's the banner on Wietse's server.  Wietse is the creator of Postfix.

220 spike.porcupine.org ESMTP Postfix (2.7-20091023)

IMHO, there's really no good reason to change the default banner.

--
Stan

[Noel Jones-2]

On 10/30/2009 9:05 PM, Stan Hoeppner wrote:

> Robert Lopez put forth on 10/30/2009 6:57 PM:
>
>> It is not clear to me what the benefit of multiple files is beyond
>> this association.
>
> Organization and ease of management for one.  For example:
>
> smtpd_client_restrictions =
>          check_recipient_access hash:/etc/postfix/access
>          check_client_access hash:/etc/postfix/access
>          pcre:/etc/postfix/check_client_fqdn.pcre
>          hash:/etc/postfix/coolsavings.access
>          hash:/etc/postfix/richk-1.access
>          cidr:/etc/postfix/cidr_files/china.cidr
>          cidr:/etc/postfix/cidr_files/korea.cidr
>          cidr:/etc/postfix/cidr_files/russia.cidr
>          cidr:/etc/postfix/cidr_files/ukraine.cidr
>          cidr:/etc/postfix/cidr_files/malaysia.cidr
>          cidr:/etc/postfix/cidr_files/belarus.cidr
>          cidr:/etc/postfix/cidr_files/indonesia.cidr
>          cidr:/etc/postfix/cidr_files/hongkong.cidr
>          cidr:/etc/postfix/cidr_files/africa.cidr
>          cidr:/etc/postfix/cidr_files/romania.cidr
>          cidr:/etc/postfix/cidr_files/thailand.cidr
>          cidr:/etc/postfix/cidr_files/poland.cidr
>          cidr:/etc/postfix/cidr_files/spammer.cidr
>          cidr:/etc/postfix/cidr_files/hurricane-electric.cidr
>          cidr:/etc/postfix/cidr_files/richk-1.cidr
>          pcre:/etc/postfix/access.pcre

It's poor form to omit the check_client_access in front of
each map name.

All these cidr maps are a great example of separate tables
that should be combined into a single table with a Makefile.

Maintain the separate input files from different sources for
easy management, then combine them into one table for postfix
to use.

>
> My access file contains some whitelist email addresses, some whitelist
> domains, some blacklist domains, and some whitelist and blacklist IP
> addresses, so I do have some consolidation in the one file and I use it
> in multiple restriction classes.However, it's by far my smallest table
> file.  Some of my cidr files are pretty large.  Note that I'm using the
> IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp
> connections.  Some of those files have thousands of entries.  Note also
> that I'm using multiple table types, hash, cidr, and pcre.  It's better
> to use multiple files in this kind of setup.

large cidr tables can use huge amounts of memory.  If you
really feel you need these, 1) consolidate them into one
table, and 2) use proxymap to open the table once per postfix
instance rather than once per smtpd process.

If the data is available in multiple forms, hash: or cdb:
tables scale much better than cidr, and a local DNSBL running
rbldnsd scales even better.
http://www.corpit.ru/mjt/rbldnsd.html

   -- Noel Jones

[Noel Jones-2]

On 10/30/2009 6:57 PM, Robert Lopez wrote:

>> Postfix places no limit on how many maps you can use, but there is system
>> overhead with each map.  Rule of thumb is to combine maps wherever possible
>> -- don't use two check_sender_access statements if you can do it with one.
>> The smart way to do this is use a Makefile to build a single map from
>> multiple similar input files.
>
> That is interesting. What is the advantage of that over directly
> editing a single file?
> I can see having unique names that pair with the parameters that cause
> them to be read.
> It is not clear to me what the benefit of multiple files is beyond
> this association.
>
> We do something similar with the virtualaliases table. There is a
> table that has all
> college employees who use an Exchange server, another that has all customers
> (students) who use Sungard Luminis, and a third that has Mailman lists. So email
> is delivered to one of those three systems based on that file. We
> build that single file
> from three separate files.

That's a good example of files that can be automated with a
Makefile.  Maintain the separate files for clear management
separation, then just type "make" to build a single postfix
file.  General example here:
http://www.postfix.org/DATABASE_README.html#safe_db

>>> message_size_limit = 16777216
>>> mydestination = $myhostname, $mydomain, localhost.localdomain,
>>> cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu,
>>> nmvc.org, mail.nmvc.org, mg0x.nmvc.org,  mg04.nmvc.org, mg05.nmvc.org,
>>> mg06.nmvc,  nmvirtualcollege.org, mail.nmvirtualcollege.org,
>>> mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org,
>>> mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org,  nmln.net,
>>> ideal-nm.org, ideal-nm.net,  idealnm.org, idealnm.net
>>
>> Lots of domains in mydestination...  Are you sure these don't belong in
>> relay_domains instead?
>> http://www.postfix.org/BASIC_CONFIGURATION_README.html
>> http://www.postfix.org/STANDARD_CONFIGURATION_README.html
>> http://www.postfix.org/ADDRESS_CLASS_README.html
>
> No, I am not sure.
>
> All email going students and employees are sent to either Sungard Luminis
> servers or to Microsoft Exchange servers.

At the most basic definition, relay_domains are domains that
are accepted by postfix and sent to the same address on
another box for final delivery.  Sounds as if these are all
relay_domains.  Valid recipients for relay_domains should be
listed in relay_recipient_maps, but it sounds as if you list
them in virtual_alias_maps.

Better to work with the system rather than against it.

>>> relay_domains = $mydestination
>>
>> relay_domains should be set explicitly, and generally should not include
>> $mydestination.  If there are no relay_domains, it should be set empty.
>> http://www.postfix.org/ADDRESS_CLASS_README.html
>
> This may be another point where I am confused. I am thinking relay is when a
> postfix server accepts email for user@... and then rewrites that address
> to what is found in a table for the user where the email is then sent
> to user@....

Postfix would call those virtual_alias_domains -- domains that
are accepted and rewritten to another domain for either local
or remote delivery.  Your domains are relay_domains.

>>> smtpd_banner = cnm.edu
>>
>> Should be "cnm.edu ESTMP", or better, just leave it at the default.
>
> Is that "cnm.edu ESMTP" or default?

The ESTMP is required to signal other mail servers that your
server accepts "enhanced" command syntax.

But there's usually no reason to change this setting from the
default "$myhostname ESTMP $mail_name", so just remove the
parameter from your main.cf.  You can't hide the fact you're
running postfix, so don't worry too much about the $mail_name
in there.

>>
>>> smtpd_sender_restrictions = check_sender_access
>>> hash:/etc/postfix/greylist      check_sender_access
>>> hash:/etc/postfix/sender_access         permit_mynetworks
>>>         reject_unknown_sender_domain
>>
>> Seems like permit_mynetworks should come before greylist or other sender
>> access checks.
>
> Seems like? :-)  This greylist program reads the tail of the mail log
> and looks for
> bursts of email in a short period of time. The program adds the bursting account
> to the map which causes the email to be deferred.

OK, you're using that as a outgoing quota control.  There are
policy servers that do similar and more, particularly policyd.
http://www.postfix.org/addon.html#policy

   -- Noel Jones

[Stan Hoeppner]

Noel Jones put forth on 10/30/2009 11:50 PM:

> On 10/30/2009 9:05 PM, Stan Hoeppner wrote:
>> Robert Lopez put forth on 10/30/2009 6:57 PM:
>>
>>> It is not clear to me what the benefit of multiple files is beyond
>>> this association.
>>
>> Organization and ease of management for one.  For example:
>>
>> smtpd_client_restrictions =
>>          check_recipient_access hash:/etc/postfix/access
>>          check_client_access hash:/etc/postfix/access
>>          pcre:/etc/postfix/check_client_fqdn.pcre
>>          hash:/etc/postfix/coolsavings.access
>>          hash:/etc/postfix/richk-1.access
>>          cidr:/etc/postfix/cidr_files/china.cidr
>>          cidr:/etc/postfix/cidr_files/korea.cidr
>>          cidr:/etc/postfix/cidr_files/russia.cidr
>>          cidr:/etc/postfix/cidr_files/ukraine.cidr
>>          cidr:/etc/postfix/cidr_files/malaysia.cidr
>>          cidr:/etc/postfix/cidr_files/belarus.cidr
>>          cidr:/etc/postfix/cidr_files/indonesia.cidr
>>          cidr:/etc/postfix/cidr_files/hongkong.cidr
>>          cidr:/etc/postfix/cidr_files/africa.cidr
>>          cidr:/etc/postfix/cidr_files/romania.cidr
>>          cidr:/etc/postfix/cidr_files/thailand.cidr
>>          cidr:/etc/postfix/cidr_files/poland.cidr
>>          cidr:/etc/postfix/cidr_files/spammer.cidr
>>          cidr:/etc/postfix/cidr_files/hurricane-electric.cidr
>>          cidr:/etc/postfix/cidr_files/richk-1.cidr
>>          pcre:/etc/postfix/access.pcre
>
> It's poor form to omit the check_client_access in front of each map name.

It may be poor form, but it's legal, gives the same result, and main.cf
doesn't look nearly as busy/cluttered by omitting it.  If/when it
becomes mandatory, I'll add it.  I hope it doesn't.

> All these cidr maps are a great example of separate tables that should
> be combined into a single table with a Makefile.

Interesting.  I may take a look into Makefile some time.  I was unaware
of it until today.  Just out of curiosity, how does putting it all in
one file make the resulting single file smaller than the sum of the
individual files' contents?  Does Makefile compress the contents in some
way, such as postmap indexes hash files?  I ask as the only overlapping
data in my CIDR files is the "REJECT" + "custom error" within each CIDR
file.

> Maintain the separate input files from different sources for easy
> management, then combine them into one table for postfix to use.

Again, why does a single large map file consume less memory than a bunch
of smaller map files, given there is no overlap but some of the custom
error message text?

>> My access file contains some whitelist email addresses, some whitelist
>> domains, some blacklist domains, and some whitelist and blacklist IP
>> addresses, so I do have some consolidation in the one file and I use it
>> in multiple restriction classes.However, it's by far my smallest table
>> file.  Some of my cidr files are pretty large.  Note that I'm using the
>> IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp
>> connections.  Some of those files have thousands of entries.  Note also
>> that I'm using multiple table types, hash, cidr, and pcre.  It's better
>> to use multiple files in this kind of setup.
>
> large cidr tables can use huge amounts of memory.  If you really feel
> you need these, 1) consolidate them into one table, and 2) use proxymap
> to open the table once per postfix instance rather than once per smtpd
> process.
>
> If the data is available in multiple forms, hash: or cdb: tables scale
> much better than cidr, and a local DNSBL running rbldnsd scales even
> better.
> http://www.corpit.ru/mjt/rbldnsd.html

This is moot in a low volume environment serving around 1000 connections
a day, with no concurrency, is it not?  My smptd's currently use 8MB
VIRT and 4MB RES, and IIRC the max I've ever seen were two smtpd
processes running simultaneously.  Now, should I start hitting 25 to 50K
connections per day, then I should probably look at optimizing my map
files a bit, no?

Thanks for the hints Noel.  I may need them down the road, although not
at the moment.  Though I am curious and may play around with Makefile
just to learn something.

--
Stan

[Noel Jones-2]

On 10/31/2009 12:32 AM, Stan Hoeppner wrote:

>> All these cidr maps are a great example of separate tables that should
>> be combined into a single table with a Makefile.
>
> Interesting.  I may take a look into Makefile some time.  I was unaware
> of it until today.  Just out of curiosity, how does putting it all in
> one file make the resulting single file smaller than the sum of the
> individual files' contents?  Does Makefile compress the contents in some
> way, such as postmap indexes hash files?  I ask as the only overlapping
> data in my CIDR files is the "REJECT" + "custom error" within each CIDR
> file.

Each lookup table requires overhead.  30 separate tables
requires considerably more overhead than one table.  The size
of the dataset doesn't change, it's the overhead that gets
smaller.  The more concurrent smtpd processes running, the
more it matters.

It probably doesn't matter in your environment, but posting
examples suitable for only a very small environment without
labeled as such doesn't help the wider user base.

And postmap doesn't compress files, but the index allows
postfix to find a particular key quickly -- lookup time and
system overhead for hash: and cdb: is essentially independent
of the size of the table, so it makes sense to use a single
table as big as necessary.  In particular, cdb: scales well up
to millions of entries, hash: tables easily to hundreds of
thousands.


   -- Noel Jones

[mouss-4]

Stan Hoeppner a écrit :
> [snip]
>
> Thanks for the hints Noel.  I may need them down the road, although not
> at the moment.  Though I am curious and may play around with Makefile
> just to learn something.
>

you can use a script if you prefer. the advantage of 'make' is that it
only re-generates files when needed (source change).

for your country maps and assuming gnu make, your Makefile would contain
something like this:

#---------------

#directpry where we store our cidr maps
CIDR_DIR=/etc/postfix/cidr_files

#file to generate
COUNTRIES_DEST=${CIDR_DIR}/countries

#"source" files, withput directory nor suffix.
COUNTRIES += china
COUNTRIES += korea
COUNTRIES += russia
...

#non portable commands. gnu make assumed.
CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES})
COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES})

${COUNTRIES_DEST}: ${COUNTRY_MAPS}
        cat $^  > $@

...
#-------------

this will catenate all your china.cidr, korea.cidr, ... into "countries"
which you can then use with
        check_client_access cidr:/etc/postfix/cidr_files/countries


PS. since you use a cidr specific directory, you can get rid of the
".cidr" suffix. you could then use (in main.cf):

cidr=cidr:/etc/postfix/cidr_files
smtpd_foo_restrictions =
        ...
        check_client_access ${cidr}/access_foo
        ...
        check_client_access ${cidr}/access_bar

[Stan Hoeppner]

Noel Jones put forth on 10/31/2009 1:12 AM:

> Each lookup table requires overhead.  30 separate tables requires
> considerably more overhead than one table.  The size of the dataset
> doesn't change, it's the overhead that gets smaller.  The more
> concurrent smtpd processes running, the more it matters.

I guess I was kinda looking for the technical definition of 'overhead'
here.  If the overhead per file is a straight percentage, then it
shouldn't matter if it's one big file or 30 smaller files.  What is the
actual memory footprint overhead that's being added by Posfix for each
map file?  And is the amount of overhead the same for all map types, or
different/variable depending on map type?  Is it something like 3 bytes
of overhead added for every 10 bytes in a map file, for example?  I can
understand that having everything in one map file, or few map files,
compared to many, can increase lookup performance due to indexing
efficiency.  What I don't understand is how having fewer larger map
files decreases memory footprint.

> It probably doesn't matter in your environment, but posting examples
> suitable for only a very small environment without labeled as such
> doesn't help the wider user base.

Agreed.  That's why I described my environment.  It's likely much
smaller scale than the average Postfix environment.

> And postmap doesn't compress files, but the index allows postfix to find
> a particular key quickly -- lookup time and system overhead for hash:
> and cdb: is essentially independent of the size of the table, so it
> makes sense to use a single table as big as necessary.  In particular,
> cdb: scales well up to millions of entries, hash: tables easily to
> hundreds of thousands.

Agreed.  Indexed files are always much faster to search than flat text
files (unless we're talking really tiny flat files).  And it's quicker
to search fewer indexes than more indexes.

This thread began with the premise being stated that many small map
files consume more memory in Postfix than fewer larger map files.
Indexes are typically proportional in size to the file being indexed,
unless the file contains many duplicate entries.  I freely admit I'm no
indexing or database expert.  However, that said, I don't understand how
map file consolidation decreases memory footprint.  Maybe Wieste can
chime in with a brief explanation.

Also, given that cidr files aren't postmap'd, is it safe to assume they
aren't indexed?  If they aren't indexed, how does putting the contents
in a single file speed up searching?  If they are indexed when Postfix
starts or reloads, then why wouldn't Postfix created a single index
_after_ reading in all the data from all the cidr files, creating one
internal program table?

--
Stan

[Stan Hoeppner]

mouss put forth on 10/31/2009 11:06 AM:

mouss, you rock.

> you can use a script if you prefer. the advantage of 'make' is that it
> only re-generates files when needed (source change).

The only likely changes would be adding another country.  In this case,
would I just add the file name to the "source" section below, and run
the make script?

> for your country maps and assuming gnu make, your Makefile would contain
> something like this:
>
> #---------------
>
> #directpry where we store our cidr maps
> CIDR_DIR=/etc/postfix/cidr_files
>
> #file to generate
> COUNTRIES_DEST=${CIDR_DIR}/countries
>
> #"source" files, withput directory nor suffix.
> COUNTRIES += china
> COUNTRIES += korea
> COUNTRIES += russia
> ...
>
> #non portable commands. gnu make assumed.
> CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES})
> COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES})
>
> ${COUNTRIES_DEST}: ${COUNTRY_MAPS}
> cat $^  > $@
>
> ...
> #-------------
>
> this will catenate all your china.cidr, korea.cidr, ... into "countries"
> which you can then use with
> check_client_access cidr:/etc/postfix/cidr_files/countries

Neat.  I'm using gnu make so this should all work.  Still don't
understand how this would affect memory footprint, but at least it would
clean up my main.cf a bit.

> PS. since you use a cidr specific directory, you can get rid of the
> ".cidr" suffix. you could then use (in main.cf):
>
> cidr=cidr:/etc/postfix/cidr_files
> smtpd_foo_restrictions =
> ...
> check_client_access ${cidr}/access_foo
> ...
> check_client_access ${cidr}/access_bar

Ahh, also sweet.  I'd seen folks using variables but I hadn't really
found a need yet to dig into it.  That and I was afraid of confusing
myself down the road. ;)  Just implemented this.  Works fine, main.cf
much cleaner looking, _and_ I'm now practicing "good form" with
"check_client_access" in front of each CIDR map. ;)

Lots of good ancillary info coming out of this thread.  Thanks guys.
Still wondering about the map file memory overhead details...

--
Stan

[mouss-4]

Stan Hoeppner a écrit :
> mouss put forth on 10/31/2009 11:06 AM:
>
> mouss, you rock.
>
>> you can use a script if you prefer. the advantage of 'make' is that it
>> only re-generates files when needed (source change).
>
> The only likely changes would be adding another country.

sometimes, you find a new IP that wasn't there, or an IP is assigned to
another country... rare, but may happen.

> In this case,
> would I just add the file name to the "source" section below, and run
> the make script?
>

yes, just add a
COUNTRIES += nowhereland
line and it should work.

>> [snip]
> Neat.  I'm using gnu make so this should all work.  Still don't
> understand how this would affect memory footprint, but at least it would
> clean up my main.cf a bit.
>

it is faster to search a single file, rather than doing 10 searches.
Think of an IP that is not listed in any of your country maps. with 10
files, you multiply the number of lookups by 10 compared to searching in
a single file.

of course latency isn't a problem in your setup. but beauty is enough
reason! see below.

>> PS. since you use a cidr specific directory, you can get rid of the
>> ".cidr" suffix. you could then use (in main.cf):
>>
>> cidr=cidr:/etc/postfix/cidr_files
>> smtpd_foo_restrictions =
>> ...
>> check_client_access ${cidr}/access_foo
>> ...
>> check_client_access ${cidr}/access_bar
>
> Ahh, also sweet.  I'd seen folks using variables but I hadn't really
> found a need yet to dig into it.  That and I was afraid of confusing
> myself down the road. ;)  Just implemented this.  Works fine, main.cf
> much cleaner looking,

yep. it avoids repetition and long lines.

maps_dir = /etc/postfix/maps
cdb = cdb:${maps_dir}/cdb
pcre = pcre:${maps_dir}/pcre
mysql = mysql:${maps_dir}/mysql
...

virtual_mailbox_maps =
        ${sql}/virtual_mailbox

virtual_alias_maps =
        ${sql}/virtual_alias

smtpd_foo_restrictions =
        ....
        # per recipient access rules
        check_recipient_access ${mysql}/access_recipient
        check_recipient_access ${pcre}/access_recipient
        ...
        # selectively reject mail from "generic rDNS" and invalid PTR:
        check_helo_access ${sql}/access_host
        check_helo_access ${pcre}/access_host
        check_reverse_client_hostname_access ${sql}/access_host
        check_reverse_client_hostname_access ${pcre}/access_host
        ...


PS. on the other hand, postconf -n doesn't show the custom variables.
but this shouldn't be a problem as long as you don't abuse variables.



>_and_ I'm now practicing "good form" with
> "check_client_access" in front of each CIDR map. ;)
>

yes. this will allow you to move/copy checks between different
restrictions. In particular, you can move checks to a restriction class
withouth having to wonder which restriction calls the class. (yes, I am
not very clear. but I hope you see what I mean ;-).


> Lots of good ancillary info coming out of this thread.  Thanks guys.
> Still wondering about the map file memory overhead details...
>
> --
> Stan

[Noel Jones-2]

On 10/31/2009 7:39 PM, Stan Hoeppner wrote:

> Noel Jones put forth on 10/31/2009 1:12 AM:
>
>> Each lookup table requires overhead.  30 separate tables requires
>> considerably more overhead than one table.  The size of the dataset
>> doesn't change, it's the overhead that gets smaller.  The more
>> concurrent smtpd processes running, the more it matters.
>
> I guess I was kinda looking for the technical definition of 'overhead'
> here.  If the overhead per file is a straight percentage, then it
> shouldn't matter if it's one big file or 30 smaller files.

"overhead" is pretty much independent of file size.  Each
extra lookup table is another open file, buffer space, another
library call (that might allocate its own buffer), and an
additional lookup.  In a tiny environment it's not enough to
make a difference -- probably not enough to even notice.

But one lookup always beats two lookups, one open file always
uses less resources than two open files.  Run enough smtpd
processes and it starts to make a difference; at some point it
can make a critical difference.  And it's always "prettier"
to do one thing one time vs. doing the same thing over and over.


> What is the
> actual memory footprint overhead that's being added by Posfix for each
> map file?

Measure it on your system.  Probably not a lot, but it adds up.

> And is the amount of overhead the same for all map types, or
> different/variable depending on map type?

Yes, different table types use different system libraries and
have different resource requirements.  But for each table type
it's a fairly constant value unaffected by table size (very
small tables might use a little less RAM under some conditions
-- but this really doesn't matter).

> Is it something like 3 bytes
> of overhead added for every 10 bytes in a map file, for example?

No, it's fixed.

> I can
> understand that having everything in one map file, or few map files,
> compared to many, can increase lookup performance due to indexing
> efficiency.  What I don't understand is how having fewer larger map
> files decreases memory footprint.

Fewer files = less overhead.  Size doesn't matter here.


> Also, given that cidr files aren't postmap'd, is it safe to assume they
> aren't indexed?  If they aren't indexed, how does putting the contents
> in a single file speed up searching?

One lookup beats two every time.

> If they are indexed when Postfix
> starts or reloads, then why wouldn't Postfix created a single index
> _after_ reading in all the data from all the cidr files, creating one
> internal program table?

How is postfix supposed to know you want all these tables
treated as one?  Anyway, postfix would still need to open the
files etc. so all you would save is the extra lookups.  It's
easy enough for you to control by building the tables the way
you want them used that postfix shouldn't be guessing.


  -- Noel Jones