XMail not seeding evenly

[Oliver Stöneberg]

There are a few cases in that XMail is distributing data randomly to
various destinations (e.g. relay domains or internal spool folders -
the code with "rand() % count"). It's not distributing the data
evenly in theses cases, so you end up with e.g. one server getting or
traffic or one spool folder being filled more than others. Especially
the first case has been problematic for some of our customers.

On a side note: all the different cases use basically the same logic,
so fixing it in one case would actually fix all of them.

A collegue did quite some testing on it and here are his findings:

When using load balancing with more than 2 relays, XMail is unevenly
distributing the load upon the configured relays.

Here is an example of the distribution to 3 mailservers. The data was
captured in a test running for 9 hours with approximately 25500
mails:

1. relay: 55.31%
2. relay: 22.31%
3. relay: 22.36%

The reason for this behaviour is the randomization / shuffling code
that is used for picking a mail relay to deliver an e-mail to.

Let's stick with the setup above using 3 mail relays. In this case
the randomization code will choose only one relay in the list of
available relays (floor(3 / 2)) and swap it with another relay, which
is also chosen at random and completely indepent of the first choice.
Thus, swapping the first with the first relay, the second with the
second, etc. is a possible outcome.

The easiest way to visualize the randomization in this case is the
following table:

        1     2     3
   1    x

   2          x     x

   3          x     x

The rows of this table represent the first chosen index of a relay,
the colums denote the index of the second choice (and thus the relay
that is swapped to the position of the first chosen relay). The 'x'
in the table is set, whenever the first element in the list will not
change (for example, swapping the second and the third relay).
As you can easily see in this table, in 5 out of 9 cases (55%) the
first relay is not swapped to another position and, consequently, 55%
of the e-mails get delivered to the first relay (which is almost
exactly the number we have
measured in our test environment).
_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Fri, 7 Aug 2009, Oliver Stöneberg wrote:

> There are a few cases in that XMail is distributing data randomly to
> various destinations (e.g. relay domains or internal spool folders -
> the code with "rand() % count"). It's not distributing the data
> evenly in theses cases, so you end up with e.g. one server getting or
> traffic or one spool folder being filled more than others. Especially
> the first case has been problematic for some of our customers.
>
> On a side note: all the different cases use basically the same logic,
> so fixing it in one case would actually fix all of them.
>
> A collegue did quite some testing on it and here are his findings:
>
> When using load balancing with more than 2 relays, XMail is unevenly
> distributing the load upon the configured relays.
>
> Here is an example of the distribution to 3 mailservers. The data was
> captured in a test running for 9 hours with approximately 25500
> mails:
>
> 1. relay: 55.31%
> 2. relay: 22.31%
> 3. relay: 22.36%
>
> The reason for this behaviour is the randomization / shuffling code
> that is used for picking a mail relay to deliver an e-mail to.
>
> Let's stick with the setup above using 3 mail relays. In this case
> the randomization code will choose only one relay in the list of
> available relays (floor(3 / 2)) and swap it with another relay, which
> is also chosen at random and completely indepent of the first choice.
> Thus, swapping the first with the first relay, the second with the
> second, etc. is a possible outcome.
>
> The easiest way to visualize the randomization in this case is the
> following table:
>
>         1     2     3
>    1    x
>
>    2          x     x
>
>    3          x     x
>
> The rows of this table represent the first chosen index of a relay,
> the colums denote the index of the second choice (and thus the relay
> that is swapped to the position of the first chosen relay). The 'x'
> in the table is set, whenever the first element in the list will not
> change (for example, swapping the second and the third relay).
> As you can easily see in this table, in 5 out of 9 cases (55%) the
> first relay is not swapped to another position and, consequently, 55%
> of the e-mails get delivered to the first relay (which is almost
> exactly the number we have
> measured in our test environment).

Thank you for reporting it. The problem, as I tried on my Linux box, is
the srand() done just before the randomization loop.
You can try to swap the position of the srand() in the code below, to see
how the distribution changes.
The problem is that across various Unix and Windows, the preservation of
the random seed changes WRT threads.
Removing the srand() and replacing it with a single one at program start,
works fine for systems using a shared (among threads) seed.
This is how it has been fixed ATM.



- Davide



#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>


static int StrStringsCount(char **s) {
        int i;

        for (i = 0; s[i] != NULL; i++);

        return i;
}

static int MscRandomizeStringsOrder(char **ppszStrings)
{
        int i, iCount = StrStringsCount(ppszStrings);

        // srand((unsigned int) time(NULL));

        for (i = 0; i < iCount - 1; i++) {
                int iChoice = rand() % (iCount - i);
                char *pszTmp = ppszStrings[i];

                ppszStrings[i] = ppszStrings[i + iChoice];
                ppszStrings[i + iChoice] = pszTmp;
        }

        return 0;
}


int main(int ac, char **av) {
        int i, j, count = 7, samples = 10000;
        int *vals;
        char **strs;

        srand((unsigned int) time(NULL));

        vals = (int *) malloc(count * sizeof(int));
        strs = (char **) malloc((count + 1) * sizeof(char *));

        for (i = 0; i < count; i++)
                vals[i] = 0;

        for (j = 0; j < samples; j++) {
                for (i = 0; i < count; i++) {
                        char *s;

                        asprintf(&s, "%d", i);
                        strs[i] = s;
                }

                MscRandomizeStringsOrder(strs);

                vals[atoi(strs[0])]++;

                for (i = 0; i < count; i++)
                        free(strs[i]);
        }

        for (i = 0; i < count; i++)
                printf("%d = %.2lf%%\n", i, 100.0 * (double) vals[i] / (double) samples);

        return 0;
}


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Hi All,

Some users are asking us about smtp tls to be used together with google.
What do we need to configure in Xmail to support smtp tls ?

obs: Xmail Win32 running on Windows 2000 Server SP4

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br
_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Hi All,
>
> Some users are asking us about smtp tls to be used together with google.
> What do we need to configure in Xmail to support smtp tls ?

Have you checked this?

http://www.xmailserver.org/Readme.html#ssl_configuration

And be sure EnableSMTP-TLS in not set to 0 in your server.tab file
(default, if missing, is 1).



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[David Lord-4]

On 19 Aug 2009 at 10:25, Edinilson - ATINET wrote:

> Hi All,
>
> Some users are asking us about smtp tls to be used together with google.
> What do we need to configure in Xmail to support smtp tls ?
>
> obs: Xmail Win32 running on Windows 2000 Server SP4
>

I found that although I can connect to my own server from a remote
blocklisted ip it's no use at all for getting email through to
other sites, ie blocked before authentication.

Not that I blame them. I route via my own server using smtps
on port 465.

My mobile broadand supplier suggests I contact recipient to add
me (+any of their users spreading virus or spam) to add their ip
block to their whitelist (and I have shares in this organisation).

Either way you need certificates etc, which took me a nights
session to setup (but several hours to get some tuits), and put
them in correct place as per xmail docs and have the required
lines in server.tab.

If you self certify you will get complaints, as from one of my
mates, that certificate isn't trusted, even though he had my
public key already and I'd told him to accept it.

On other users of my server I've done the add certificate bit
myself. Then they only use it when their own isp's mailserver
is down or blocklisted.

can of worms?

David

> Regards
>
> Edinilson
> ---------------------------------------------------------
> ATINET-Professional Web Hosting
> Tel Voz: (0xx11) 4412-0876
> http://www.atinet.com.br
> _______________________________________________
> xmail mailing list
> xmail@...
> http://xmailserver.org/mailman/listinfo/xmail

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Thanks Davide.

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Wednesday, August 19, 2009 1:39 PM
Subject: Re: [xmail] SMTP TLS


On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Hi All,
>
> Some users are asking us about smtp tls to be used together with google.
> What do we need to configure in Xmail to support smtp tls ?

Have you checked this?

http://www.xmailserver.org/Readme.html#ssl_configuration

And be sure EnableSMTP-TLS in not set to 0 in your server.tab file
(default, if missing, is 1).



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Davide, I checked:
http://www.xmailserver.org/Readme.html#ssl_configuration
and created server.cert and server.key

And in server.tab
"SMTP-TLS" "1"
"EnableSMTP-TLS" "1"

Using netstat -an I can see that port 465 was opened.

But, for some reason, we can´t authenticate any user using smtp tls.

Any tip?

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Wednesday, August 19, 2009 1:39 PM
Subject: Re: [xmail] SMTP TLS


On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Hi All,
>
> Some users are asking us about smtp tls to be used together with google.
> What do we need to configure in Xmail to support smtp tls ?

Have you checked this?

http://www.xmailserver.org/Readme.html#ssl_configuration

And be sure EnableSMTP-TLS in not set to 0 in your server.tab file
(default, if missing, is 1).



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail 

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Davide, I checked:
> http://www.xmailserver.org/Readme.html#ssl_configuration
> and created server.cert and server.key
>
> And in server.tab
> "SMTP-TLS" "1"
> "EnableSMTP-TLS" "1"
>
> Using netstat -an I can see that port 465 was opened.
>
> But, for some reason, we can´t authenticate any user using smtp tls.
>
> Any tip?

Port 465 is SMTPS, that's for pure SSL SMTP sessions.
By setting EnableSMTP-TLS you tell XMail to try a STARTTLS negotiation, if
the remote server allows it.
Did you setup the "certs" subdirectory, if you're using Windows (like
the SSLUseCertsDir suggests in the link I posted before)?



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Davide, I thing that I missing something.

in server.tab:
"SMTP-TLS" "1"
"EnableSMTP-TLS" "1"
"SSLUseCertsDir" "1"
"SSLAllowSelfSigned" "1"

I copied to windows\system32:
openssl.exe
libeay32.dll
ssleay32.dll
libeay32.lib
ssleay32.lib

And updated the files in mailroot/bin and mailroot/certs with those that are
in win32sll (from source package).

But, for some reason, it isn´t working yet.

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Wednesday, August 19, 2009 4:52 PM
Subject: Re: [xmail] SMTP TLS


On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Davide, I checked:
> http://www.xmailserver.org/Readme.html#ssl_configuration
> and created server.cert and server.key
>
> And in server.tab
> "SMTP-TLS" "1"
> "EnableSMTP-TLS" "1"
>
> Using netstat -an I can see that port 465 was opened.
>
> But, for some reason, we can´t authenticate any user using smtp tls.
>
> Any tip?

Port 465 is SMTPS, that's for pure SSL SMTP sessions.
By setting EnableSMTP-TLS you tell XMail to try a STARTTLS negotiation, if
the remote server allows it.
Did you setup the "certs" subdirectory, if you're using Windows (like
the SSLUseCertsDir suggests in the link I posted before)?



- Davide





_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail 

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Wed, 19 Aug 2009, David Lord wrote:

> On 19 Aug 2009 at 10:25, Edinilson - ATINET wrote:
>
> > Hi All,
> >
> > Some users are asking us about smtp tls to be used together with google.
> > What do we need to configure in Xmail to support smtp tls ?
> >
> > obs: Xmail Win32 running on Windows 2000 Server SP4
> >
>
> I found that although I can connect to my own server from a remote
> blocklisted ip it's no use at all for getting email through to
> other sites, ie blocked before authentication.
>
> Not that I blame them. I route via my own server using smtps
> on port 465.
>
> My mobile broadand supplier suggests I contact recipient to add
> me (+any of their users spreading virus or spam) to add their ip
> block to their whitelist (and I have shares in this organisation).
>
> Either way you need certificates etc, which took me a nights
> session to setup (but several hours to get some tuits), and put
> them in correct place as per xmail docs and have the required
> lines in server.tab.
>
> If you self certify you will get complaints, as from one of my
> mates, that certificate isn't trusted, even though he had my
> public key already and I'd told him to accept it.
>
> On other users of my server I've done the add certificate bit
> myself. Then they only use it when their own isp's mailserver
> is down or blocklisted.
>
> can of worms?

No shit! :)


- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Davide, I thing that I missing something.
>
> in server.tab:
> "SMTP-TLS" "1"
> "EnableSMTP-TLS" "1"
> "SSLUseCertsDir" "1"
> "SSLAllowSelfSigned" "1"
>
> I copied to windows\system32:
> openssl.exe
> libeay32.dll
> ssleay32.dll
> libeay32.lib
> ssleay32.lib
>
> And updated the files in mailroot/bin and mailroot/certs with those that are
> in win32sll (from source package).
>
> But, for some reason, it isn´t working yet.

Can you define "isn't working" a little bit further?
Check David Lord's email also, as for the difficulties of authenticate
with GOOG.


- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[David Lord-4]

On 19 Aug 2009 at 16:41, Edinilson - ATINET wrote:

> Davide, I checked:
> http://www.xmailserver.org/Readme.html#ssl_configuration
> and created server.cert and server.key
>
> And in server.tab
> "SMTP-TLS" "1"
> "EnableSMTP-TLS" "1"
>
> Using netstat -an I can see that port 465 was opened.
>
> But, for some reason, we can´t authenticate any user using smtp tls.
>
> Any tip?

I can't see you've got anything wrong.
Can you test locally.

Also the more competent isp I use for adsl has enabled
both smtps port 465 for ssl along with smmsp port 587
for smtpauth but that might also support tls (as I use
xmail from home I've not needed those facilities).

Last week by chance I tried out tls on port 25 from
notebook via mobile broadband, to send by xmail on
my server at home. That was delivered to one of my
accounts on another isp and ended up in spam folder.
Spam scores were due to mobile broadband ips I was
sending from being on several blocklists and their
dns was bad as well. At least using smtps or smmsp
I'd be sending from clean ips. I'd previously tried
smtps (then set as default) and those emails were
delivered without problem.


David

>
> Regards
>
> Edinilson
> ---------------------------------------------------------
> ATINET-Professional Web Hosting
> Tel Voz: (0xx11) 4412-0876
> http://www.atinet.com.br
>
>
> ----- Original Message -----
> From: "Davide Libenzi" <davidel@...>
> To: "XMail Users Mailing List" <xmail@...>
> Sent: Wednesday, August 19, 2009 1:39 PM
> Subject: Re: [xmail] SMTP TLS
>
>
> On Wed, 19 Aug 2009, Edinilson - ATINET wrote:
>
> > Hi All,
> >
> > Some users are asking us about smtp tls to be used together with google.
> > What do we need to configure in Xmail to support smtp tls ?
>
> Have you checked this?
>
> http://www.xmailserver.org/Readme.html#ssl_configuration
>
> And be sure EnableSMTP-TLS in not set to 0 in your server.tab file
> (default, if missing, is 1).
>
>
>
> - Davide
>
>
> _______________________________________________
> xmail mailing list
> xmail@...
> http://xmailserver.org/mailman/listinfo/xmail 
>
> _______________________________________________
> xmail mailing list
> xmail@...
> http://xmailserver.org/mailman/listinfo/xmail

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Davide, you are right. Looks like something is wrong with Google.
Using Outlook Express I can use TLS without problems.

Trying to use TLS via Google the following error is returned:
[Unspecified Error (SENT_SECOND_EHLO): Protocol error code(0) ]

Nothing is being logged in smtp* logs

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Wednesday, August 19, 2009 6:22 PM
Subject: Re: [xmail] SMTP TLS


On Wed, 19 Aug 2009, Edinilson - ATINET wrote:

> Davide, I thing that I missing something.
>
> in server.tab:
> "SMTP-TLS" "1"
> "EnableSMTP-TLS" "1"
> "SSLUseCertsDir" "1"
> "SSLAllowSelfSigned" "1"
>
> I copied to windows\system32:
> openssl.exe
> libeay32.dll
> ssleay32.dll
> libeay32.lib
> ssleay32.lib
>
> And updated the files in mailroot/bin and mailroot/certs with those that
> are
> in win32sll (from source package).
>
> But, for some reason, it isn´t working yet.

Can you define "isn't working" a little bit further?
Check David Lord's email also, as for the difficulties of authenticate
with GOOG.


- Davide




--------------------------------------------------------------------------------


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Thu, 20 Aug 2009, Edinilson - ATINET wrote:

> Davide, you are right. Looks like something is wrong with Google.
> Using Outlook Express I can use TLS without problems.
>
> Trying to use TLS via Google the following error is returned:
> [Unspecified Error (SENT_SECOND_EHLO): Protocol error code(0) ]
>
> Nothing is being logged in smtp* logs

When you say "sending with Google", what do you exactly mean?



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

Some users that have domains hosted here wants to use gmail for other
purposes.
But they wants to use yours accounts (hosted here) to send emails, but via
gmail (something like a relay).
Gmail have a configuration called: SEND MAIL FROM ANOTHER ADDRESS (
http://mail.google.com/support/bin/answer.py?hl=en&answer=22370&expand=sc1 )
where you can authenticate with an account from your domain and send email
(from gmail) using that account.

I could be wrong but looks like Google uses TLS in this point, to relay the
message.

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Thursday, August 20, 2009 2:38 PM
Subject: Re: [xmail] SMTP TLS


On Thu, 20 Aug 2009, Edinilson - ATINET wrote:

> Davide, you are right. Looks like something is wrong with Google.
> Using Outlook Express I can use TLS without problems.
>
> Trying to use TLS via Google the following error is returned:
> [Unspecified Error (SENT_SECOND_EHLO): Protocol error code(0) ]
>
> Nothing is being logged in smtp* logs

When you say "sending with Google", what do you exactly mean?



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Thu, 20 Aug 2009, Edinilson - ATINET wrote:

> Some users that have domains hosted here wants to use gmail for other
> purposes.
> But they wants to use yours accounts (hosted here) to send emails, but via
> gmail (something like a relay).
> Gmail have a configuration called: SEND MAIL FROM ANOTHER ADDRESS (
> http://mail.google.com/support/bin/answer.py?hl=en&answer=22370&expand=sc1 )
> where you can authenticate with an account from your domain and send email
> (from gmail) using that account.
>
> I could be wrong but looks like Google uses TLS in this point, to relay the
> message.

Unless GOOG requires that *your* certificate is *not* self-signed, by
refusing to talk to your server after the initial SSL negotiation.



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Edinilson - ATINET]

We have a ssl web server certificate from Thawte (www.thawte.com)
Could this kind of certificate be used with smtp TLS?

Regards

Edinilson
---------------------------------------------------------
ATINET-Professional Web Hosting
Tel Voz: (0xx11) 4412-0876
http://www.atinet.com.br


----- Original Message -----
From: "Davide Libenzi" <davidel@...>
To: "XMail Users Mailing List" <xmail@...>
Sent: Thursday, August 20, 2009 3:22 PM
Subject: Re: [xmail] SMTP TLS


On Thu, 20 Aug 2009, Edinilson - ATINET wrote:

> Some users that have domains hosted here wants to use gmail for other
> purposes.
> But they wants to use yours accounts (hosted here) to send emails, but via
> gmail (something like a relay).
> Gmail have a configuration called: SEND MAIL FROM ANOTHER ADDRESS (
> http://mail.google.com/support/bin/answer.py?hl=en&answer=22370&expand=sc1 
>  )
> where you can authenticate with an account from your domain and send email
> (from gmail) using that account.
>
> I could be wrong but looks like Google uses TLS in this point, to relay
> the
> message.

Unless GOOG requires that *your* certificate is *not* self-signed, by
refusing to talk to your server after the initial SSL negotiation.



- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail

[Davide Libenzi]

On Fri, 21 Aug 2009, Edinilson - ATINET wrote:

> We have a ssl web server certificate from Thawte (www.thawte.com)
> Could this kind of certificate be used with smtp TLS?

I honestly dunno. It depends on Google I guess.


- Davide


_______________________________________________
xmail mailing list
xmail@...
http://xmailserver.org/mailman/listinfo/xmail