Nabble - dbmail users

Re: Found [16] un-cached physmessages

2009-12-16T03:08:02Z

On Mittwoch, 16. Dezember 2009 Reindl Harald wrote:
> Nearly since migration to dbmail i get "Found [16] un-cached
> physmessages." At the begin there was 10 increased now to 16
>
That's a bug. Skip -b and ignore it. I wanted to do a fixup but didn't
have time so far. IIRC it's about dbmail-util -b inserting headers that
already exist and it breaks in the middle. See the thread about it on
march 17, 2009.

It doesn't affect anything, so it's not important. Some indices should
be cleaned up.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-16T03:00:30Z

On Mittwoch, 16. Dezember 2009 Daniel Mejia wrote:
> db.c,db_icheck_isheader(+1788): could not access messageblks table
> Failed. An error occured. Please check log.
> </code>
>
> what should i do?

Did you check the log, as the error message suggested?

we run this command daily:
dbmail-util -ubdsy

and this once a week:
dbmail-util -tpubdsy -r 168h ; dbmail-util -py

and for use those commands take about 30-40 seconds (the daily ones) and
20 minutes (the weekly stuff).

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Found [16] un-cached physmessages

2009-12-16T02:30:54Z

Hi

Nearly since migration to dbmail i get "Found [16] un-cached physmessages."
At the begin there was 10 increased now to 16
dbmail-util -c -t -u -b -p -d -y

2.2.13 does not fix this and i do not really know what this means
and how get it away
____________________

Repairing DBMAIL messageblocks integrity...
Ok. Found [0] unconnected messageblks.

Repairing DBMAIL physmessage integrity...
Ok. Found [0] unconnected physmessages.

Repairing DBMAIL message integrity...
Ok. Found [0] unconnected messages.

Repairing DBMAIL mailbox integrity...
Ok. Found [0] unconnected mailboxes.

Repairing DBMAIL for NULL messages...
Ok. Found [0] NULL messages.

Checking DBMAIL for NULL physmessages...
Ok. Found [0] physmessages without messageblocks.

Deleting messages with DELETE status...
Ok. [0] messages deleted.

Repairing DBMAIL for rfcsize field...
Ok. Found [0] missing rfcsize values.

Repairing DBMAIL for incorrect is_header flags...
Ok. Found [0] incorrect is_header flags.

Repairing DBMAIL for cached envelopes...
Ok. Found [0] missing envelope values.

Repairing DBMAIL for cached header values...
Ok. Found [16] un-cached physmessages.
................
Setting DELETE status for deleted messages...
Ok. [0] messages set for deletion.
Re-calculating used quota for all users...
Ok. Used quota updated for all users.

Vacuuming and optimizing database...
Ok. Database cleaned up.

Maintenance done. Errors were found and fixed.
We suggest running dbmail-util again to confirm that all errors were repaired.

--

Reindl Harald
the lounge interactive design GmbH
A-1060 Vienna | Hofmühlgasse 17
software-development / cms-solutions

phone: +43 (1) 595 3999 33
cellular: +43 (676) 40 221 40
icq: 154546673

mailto:h.reindl@...
http://www.thelounge.net/

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (268 bytes) Download Attachment

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-16T02:11:23Z

Daniel Mejia wrote:

>
> Paul J Stevens wrote:
>> So with 2000 users you have 2 to 3TB storage allocated already?
>>
>
> There is a 3TB of storage, with 800Gb used.
>
>
>
>> So don't run dbmail-util -ay on a 800GB database!
>> ...
>> That is the header check (-b). You can safely skip that one, until you
>> decide to rebuild the header cache tables.
>>
>
> Rebuild header cache tables = ALTER TABLE ?

You won't rebuild header caches, unless you want/need to change you
database's encoding, or have changed it in the past.

If you do, you'll want to drop/create all the header cache tables:
(dbmail_headername, dbmail_headervalue, dbmail_subjectfield,
dbmail_datefield, dbmail_referencesfield, dbmail_fromfield,
dbmail_tofield, dbmail_replytofield, dbmail_ccfield).

But the queries that check the consistency and completeness of the
headercaches may be long running, keeping a read-lock on those tables
along the way. So for that reason alone, you will want to avoid them if
at all possible.

Same goes for some of the other queries run by the maintenance code. If
it takes a long time to run them, but they don't normally find anything
to fix, consider running them less often, if ever.

The only queries you will probably want to run daily are -p and -d to
queue messages for deletion that are flagged as such. To actually
cleanout the messageblks involved in deleted messages you'll also want
to add -t.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-16T01:55:59Z

Paul J Stevens wrote:

So with 2000 users you have 2 to 3TB storage allocated already?

There is a 3TB of storage, with 800Gb used.

So don't run dbmail-util -ay on a 800GB database!
...
That is the header check (-b). You can safely skip that one, until you
decide to rebuild the header cache tables.

Rebuild header cache tables = ALTER TABLE ?

Run them all manually, and separately, to find out how long they take,
if they're needed at all, how they affect the database, etc...

Some of the maintenance options are probably not needed at all, only
need to be run once every few days or maybe even weeks.

Simply running -ay every night is ok for small to medium sized
installations. But with 800GB you're not in Kansas anymore.

at last someone gives an answer straight! thanx! will try exactly that.

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-16T01:13:18Z

Daniel Mejia wrote:
>
> Michael Monnerie-5 wrote:
>> Just put a max quota on your users accounts, then they *must* delete
>> some old e-mails.
>>
>
> We have a 1GB quota on all users, except for more or less, 20 users in the
> management level, with the biggest quota at 3GB.

So with 2000 users you have 2 to 3TB storage allocated already?

>> How many users do you have, and how much space do they use on average?
>>
>
> Currently, we have 1,879 users, but the number increases, slowly but surely,
> when the organization hire new staff (which happens all year round).
>
> I run the dbmail-util -ay command from a different server, whereas the mysql
> database server sits on a different server. This is what happens:

So don't run dbmail-util -ay on a 800GB database!

> <code>
> Repairing DBMAIL for incorrect is_header flags...
> Dec 16 13:13:04 mailadmin.mpob.g dbmail-util[16803]: Error:[sql]
> dbmysql.c,db_query(+290): [Lost connection to MySQL server during query]
> [SELECT MIN(messageblk_idnr),MAX(is_header) FROM dbmail_messageblks GROUP BY
> physmessage_id HAVING MAX(is_header)=0]

That is the header check (-b). You can safely skip that one, until you
decide to rebuild the header cache tables.

> Dec 16 13:13:04 mailadmin.mpob.g dbmail-util[16803]: Error:[db]
> db.c,db_icheck_isheader(+1788): could not access messageblks table
> Failed. An error occured. Please check log.
> </code>
>
> what should i do? from the mysql website:
> http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_connect_timeout,
> i'm kind of confused with all the *_timeout variables.
>
> I was thinking of increasing the number of seconds, but not sure which one.
> would that help?

You're barking up the wrong tree. There's a reason for those timeouts.
First find out which query is killing the connection (you did), next
find out which command triggers the query (-by).

In your case I would very carefully start to tune the dbmail-util runs.

Run them all manually, and separately, to find out how long they take,
if they're needed at all, how they affect the database, etc...

Some of the maintenance options are probably not needed at all, only
need to be run once every few days or maybe even weeks.

Simply running -ay every night is ok for small to medium sized
installations. But with 800GB you're not in Kansas anymore.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-16T00:33:15Z

Michael Monnerie-5 wrote:

Just put a max quota on your users accounts, then they *must* delete
some old e-mails.

We have a 1GB quota on all users, except for more or less, 20 users in the management level, with the biggest quota at 3GB.

How many users do you have, and how much space do they use on average?

Currently, we have 1,879 users, but the number increases, slowly but surely, when the organization hire new staff (which happens all year round).

I run the dbmail-util -ay command from a different server, whereas the mysql database server sits on a different server. This is what happens:

<code>
Repairing DBMAIL for incorrect is_header flags...
Dec 16 13:13:04 mailadmin.mpob.g dbmail-util[16803]: Error:[sql] dbmysql.c,db_query(+290): [Lost connection to MySQL server during query] [SELECT MIN(messageblk_idnr),MAX(is_header) FROM dbmail_messageblks GROUP BY physmessage_id HAVING MAX(is_header)=0]
Dec 16 13:13:04 mailadmin.mpob.g dbmail-util[16803]: Error:[db] db.c,db_icheck_isheader(+1788): could not access messageblks table
Failed. An error occured. Please check log.
</code>

what should i do? from the mysql website: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_connect_timeout, i'm kind of confused with all the *_timeout variables.

I was thinking of increasing the number of seconds, but not sure which one. would that help?

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-14T17:42:05Z

> still, the database size is increasing average 2GB per day. btw, do u happen
> to know the maximum size mysql can handle? with the rate this is going, its
> gonna be 2TB at the end of next year..

run the following query:

select userid,curmail_size from dbmail_users order by curmail_size desc
limit 10;

Now go tell these people to clean up their mailboxes, they're the space
hogs. You might be surprised how big some people's mailboxes are.

I personally also have a query that deletes all unread emails older than
6 months. Reason for this is people who never read their mailbox due to
it being forwarded elsewhere, but cc'ing locally:

echo "update dbmail_messages m, dbmail_mailboxes b, dbmail_physmessage p
set m.deleted_flag=1,m.status=2 where m.mailbox_idnr=b.mailbox_idnr and
p.id=m.physmessage_id and m.seen_flag = 0 and p.internal_date <
date_sub(now(), interval 180 day);" | mysql -u USER -pPASS dbname

Regards,
Josh.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-14T06:13:15Z

On Montag, 14. Dezember 2009 Daniel Mejia wrote:
> still, the database size is increasing average 2GB per day

Just put a max quota on your users accounts, then they *must* delete
some old e-mails. If you don't use quota, people will never delete, as
they just don't realize they need some space.

For POP accounts, I allow 100MB. IMAP gets 500MB per default. Those who
want more pay more. Simple as that.

How many users do you have, and how much space do they use on average?
Maybe you have some "top 5" users who need a lot of space, and the rest
is small anyway?

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-14T00:51:11Z

tabris wrote:

> Ok. Didn't know that. I know that it's supposed to handle LDAP dying,
> but haven't had much luck with that either.

The code only deals with temporary ldap disconnections. If a ldap
connection goes stale the code will try to reconnect every 2 seconds,
and will try this 5 times. So if the ldap server goes down for more than
10 seconds authentication will start to fail. Contrary to disrupted
database connections such failures will not propagate to the main server
process.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-14T00:20:33Z

Paul J Stevens wrote:

> tabris wrote:
>
>
>>> 1. shut down the mysql server,
>>> 2. change the mysql.cnf file, and
>>> 3. restart the mysql server back up
>>>
>> Well, shutdown dbmail-lmtp and dbmail-imap (and dbmail-pop too),
>> then restart mysqld.
>>
>
> Not really required since dbmail-2.2 will shutdown all child process
> handling incoming traffic when the database goes, and restart them when
> the connection is restored.
>
>
>

Ok. Didn't know that. I know that it's supposed to handle LDAP dying,

but haven't had much luck with that either.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (266 bytes) Download Attachment

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-14T00:16:18Z

tabris wrote:

>> 1. shut down the mysql server,
>> 2. change the mysql.cnf file, and
>> 3. restart the mysql server back up
>
> Well, shutdown dbmail-lmtp and dbmail-imap (and dbmail-pop too),
> then restart mysqld.

Not really required since dbmail-2.2 will shutdown all child process
handling incoming traffic when the database goes, and restart them when
the connection is restored.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T23:12:55Z

Daniel Mejia wrote:

Josh Marshall-2 wrote:

It only recovers empty space and re-orders (like a defrag). If your
usage is increasing, the best way is to remove emails. Maybe you need to
run a dbmail-util to remove the deleted emails (there might be a SQL you
can run manually)


i did a cronjob to take care of this, every 2 & 6 am daily. (dbmail-util
-ay)
the cronjob also runs a custom SQL that would clear the trash & spam folder
older than 30 days.

still, the database size is increasing average 2GB per day. btw, do u happen
to know the maximum size mysql can handle? with the rate this is going, its
gonna be 2TB at the end of next year..

would ANALYZE TABLE gives any performance speed increase etc?

I would consider a MySQL upgrade, I'm not sure how good MySQL 4.0 is
with dbmail.


Oops, typing error! its mysql 5.0.45. i guess that's what happen when u only
get 3 hours of sleep for the past week. xD

you could have a backup server, replicating away, then flip over to that for maintenance.
(not that nice really)
or perhaps just replicate the table in question, then shut down, stop replication, rename the table on the main system to a temp name, create a federated table on the main that points to the secondary
do the optimise table on the main, stop everything again, take out the federated table, restart replication, (so the main synchs up with the changes to the secondary) and your done.

Personally all that implies to me the table should be sharded.

shouldn't be that hard, split it based on the hash, hashes starting with a 1 go into table mimeparts_1 etc
you can easily split it based on that to 10,100 or 1000 tables.
1000 tables should give good enough performance for some pretty decent scaling ;-> especially if you spread them over the spindles.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T23:01:47Z

Josh Marshall-2 wrote:

It only recovers empty space and re-orders (like a defrag). If your
usage is increasing, the best way is to remove emails. Maybe you need to
run a dbmail-util to remove the deleted emails (there might be a SQL you
can run manually)

i did a cronjob to take care of this, every 2 & 6 am daily. (dbmail-util -ay)
the cronjob also runs a custom SQL that would clear the trash & spam folder older than 30 days.

still, the database size is increasing average 2GB per day. btw, do u happen to know the maximum size mysql can handle? with the rate this is going, its gonna be 2TB at the end of next year..

would ANALYZE TABLE gives any performance speed increase etc?

I would consider a MySQL upgrade, I'm not sure how good MySQL 4.0 is
with dbmail.

Oops, typing error! its mysql 5.0.45. i guess that's what happen when u only get 3 hours of sleep for the past week. xD

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T22:07:34Z

Daniel Mejia wrote:

> Josh Marshall-2 wrote:
>
>>
>>> 1. Is the dbmail_messageblks table cant be optimized?
>>>
>> It can be optimized, but there's probably no good reason to do it. I did
>> it once and it didn't increase the database speed it just took a long
>> time to do it.
>>
>>
>>
>
> Would the optimization retrieve some space? ie: reducing the table size?
>

It would, but it wouldn't matter if you're getting messages in at
1000/day. You'd delete 800 messages and get 1000 more... and that data
space would then get used by the new messages. If you want things to be
faster, instead try using 'ANALYZE TABLE' to update the indices. It's a
lot faster (and it doesn't rebuild the indices, just rebalance the index
tree and update the index cardinality).

>> innodb_log_file_size=192M
>> innodb_log_buffer_size=8M
>> innodb_file_per_table
>> [mysqldump]
>> quick
>> quote-names
>> max_allowed_packet = 128M
>> [mysql]
>> [isamchk]
>> key_buffer = 16M
>>
>>
>>
>
> ok, let say my database size is 760GB (and increasing), can i just:
> 1. shut down the mysql server,
> 2. change the mysql.cnf file, and
> 3. restart the mysql server back up
>

Well, shutdown dbmail-lmtp and dbmail-imap (and dbmail-pop too),
then restart mysqld.

> would that do the trick?
>
> p/s: my mail server hardware is similar to urs, running Fedora 10 with mysql
> ver 4.0.45.
> The number of incoming mails is larger than 20,000 daily, but most of them
> are blocked by the Barracuda spam firewall, reducing it to minimum 1000.
>

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (269 bytes) Download Attachment

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T22:05:37Z

> Would the optimization retrieve some space? ie: reducing the table
> size?

It only recovers empty space and re-orders (like a defrag). If your
usage is increasing, the best way is to remove emails. Maybe you need to
run a dbmail-util to remove the deleted emails (there might be a SQL you
can run manually)

> ok, let say my database size is 760GB (and increasing), can i just:
> 1. shut down the mysql server,
> 2. change the mysql.cnf file, and
> 3. restart the mysql server back up

My config file is for MySQL 5.0, so some parameters might be slightly
different. Otherwise, yes that should work fine. Take a backup though.

> p/s: my mail server hardware is similar to urs, running Fedora 10 with mysql
> ver 4.0.45.

I would consider a MySQL upgrade, I'm not sure how good MySQL 4.0 is
with dbmail.

> The number of incoming mails is larger than 20,000 daily, but most of them
> are blocked by the Barracuda spam firewall, reducing it to minimum 1000.

Indeed, we see 95% of emails blocked at our firewall, one day I turned
that off and averaged 10 incoming emails/second.

Regards,
Josh.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T21:56:05Z

Josh Marshall-2 wrote:

> 1. Is the dbmail_messageblks table cant be optimized?

It can be optimized, but there's probably no good reason to do it. I did
it once and it didn't increase the database speed it just took a long
time to do it.

Would the optimization retrieve some space? ie: reducing the table size?

innodb_log_file_size=192M
innodb_log_buffer_size=8M
innodb_file_per_table
[mysqldump]
quick
quote-names
max_allowed_packet = 128M
[mysql]
[isamchk]
key_buffer = 16M

ok, let say my database size is 760GB (and increasing), can i just:
1. shut down the mysql server,
2. change the mysql.cnf file, and
3. restart the mysql server back up

would that do the trick?

p/s: my mail server hardware is similar to urs, running Fedora 10 with mysql ver 4.0.45.
The number of incoming mails is larger than 20,000 daily, but most of them are blocked by the Barracuda spam firewall, reducing it to minimum 1000.

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T20:51:09Z

On Sun, 2009-12-13 at 20:14 -0800, Daniel Mejia wrote:

> 1. Is the dbmail_messageblks table cant be optimized?

It can be optimized, but there's probably no good reason to do it. I did
it once and it didn't increase the database speed it just took a long
time to do it.

> 2. What would u suggest the mysql.cnf config should be like for 2000+ users
> & at least 1000 incoming mails per day?

I've posted my mysql.cnf before. I had a few people pick it to pieces. I
implemented some of them. So it's still not perfect but it does the job
fine. This particular mysql.cnf is used on a Debian Etch server with a
single 3GHz Celeron CPU, 3Gb RAM and a pair of 160Gb hard disks in
software RAID-1. Serving 5000 mailboxes and between 25,000-30,000
emails/day:

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
skip-name-resolve
skip-name-resolve
bind-address = 0.0.0.0
key_buffer = 1024M
key_buffer_size = 16M
max_allowed_packet = 128M
table_cache = 2048
sort_buffer_size = 32M
read_buffer_size = 8M
read_rnd_buffer_size = 62M
myisam_sort_buffer_size = 64M
thread_concurrency = 4

thread_stack = 128K
thread_cache_size = 8
max_connections = 1800
query_cache_limit = 5M
query_cache_size = 512M
log_bin = /storage/binlog/mysql-bin.log
expire_logs_days = 5
max_binlog_size = 100M
skip-bdb
innodb_buffer_pool_size=100M
innodb_log_file_size=192M
innodb_log_buffer_size=8M
innodb_file_per_table
[mysqldump]
quick
quote-names
max_allowed_packet = 128M
[mysql]
[isamchk]
key_buffer = 16M

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: OPTIMIZE TABLE dbmail_messageblks in MySQL

2009-12-13T20:14:57Z

Paul J Stevens wrote:

kost@ropnet.ru wrote:
> -rw-rw---- 1 mysql4 mysql4 23995613184 Oct 20 11:47 dbmail_messageblks.ibd

he! are those innodb table spaces? Been reading about them, but haven't
used them yet..

> Table "dbmail_messageblks" has almost 24Gb size (we have about 45000
> emails each day!). So when "OPTIMIZE TABLE dbmail_messageblks" starts
> Mysql sets locking on this table and write operations delays while
> optimize process runnig.
> It has a very long time to parse 25Gb of data (MySQL creates temporary
> files of the same size to rebuild table) and all clients who fetch
> mail with pop3 or imap get time out error. In this state our MX server
> (that push mail in MySQL server through dbmail-smtp utility) increases
> number of processes in many times...
>
> If anybody now how to do optimize MySQL tables in such hard terms
> - please answer.

I'm open to suggestions and ideas here. I've been having this very same
problem.

1. Is the dbmail_messageblks table cant be optimized?
2. What would u suggest the mysql.cnf config should be like for 2000+ users & at least 1000 incoming mails per day?

I tried asking in another thread, but nobody seems to have the answer.... if anyone do have an answer, please help.. thanx ;)

Re: Optimizing Dbmail Database

2009-12-11T05:11:58Z

Am 11.12.2009 13:06, schrieb Tomas Kuliavas:

> Are you sure that syntax of your select query is correct?

That is not needed
It shows that you are more flexible using a database

In most cases a well configured database with enough memory
will be much faster because you have query-caches and
buffers which are more effective than any generic os-cache

> how complex select call you will make in order to cover all variations?
> flowed format, quoted-printable, headers and body that might have text in
> n different charsets.

how will you do that in a flat.file?

> SQL is not designed to decode MIME on the fly.

Why should anyone do that?
It's enough if you encode the search-string in every variation

>> The question is anyway: Does an IMAP SEARCH search in several variations
>> of "test"? What if it's base64 encoded?

and here is the benefit of a db-backend
imap himself has nothing to do with it
You can add a plaintext-field for text-only mimeparts,
decode them there and use this column for searching as sample

In this case you never touch the original mail-data AND can search
You can add the column long after receive the mails and build
the index for existing mails and play with many helper-columns/tables
whereever it makes sense for better performance

the point is that imap/mua has nothing to know about this internals

> Headers must be decoded, if charset is specified in search command. You
> are free to read all IMAP stardards if you want as long as you don't
> invent new SQL syntax in order to prove your point.

look above
imap-standards, encodings and sql-syntax are not the point
you can change the backend-structure in many ways to improve
things without thinking about frontend-protocols

and with rdbm-only backend you get much more benefits

BACKUPS:
* dbmail runs on a vmware-esxi
* a 1:1 copy runs on another hardware
* this is configured as replication-slave
* replication runs in a own mysql-instance
* every week replication is stopped and the data copied into the default-instance
* before copy the current data we stop both instances and make a "last-week"-copy
* so we have a backup-imap to go back until last sunday
* in worst-case we can switch back one week more
* the master is never stopped for this
* there is no work except read the weekly statusmail from cron

SCALEABLE:
* you want 10 mx-records to split the spam-filter-overhead?
* No problem, every mx has the same readonly-tables for valid addresses
* every mx can filter spam independet
* clean messages are going in the same database
* in fact postfix needs readonly you can use slaves to reduce db-load
* you could also do master-master-replication if needed to split write-load
* you can split cpu-load by using more machines with the same db (dbmail-imapd often uses more cpu as mysql)

FLEXIBLE:
* you want your own quota-notifies?: write a script in which anguage you want
* you want special guis for some things: no problem, write them
* you want combine the cinfiguration.infos with other services? do that!

The admin-benefit in our case is thwat we use two dbmail-servers this time and have a self-written webui for all
our standard-admin-jobs (ftp, www, dns, dhcp, domain-registry) with a clear permission-handling, it does not matter
how much dbmail-installations we use in the future: it needs 10 minutes to include all of them in the central panel
and give operators the right permissions

Additionally we are able to write scripts for often needed and/or complex actions in a few time and can combine the
setup-data from every service and machine because all other services are mysql-based (some of thmen by generating
standard-configs based on the databases).

Many overview-pages for non-technicals are based on this things because nobody has to write doumentations which
host/client was created and what happens in our company - only the reference host/service/client has to be choosed
manually, if we delete a host we can be sure it goes away in all overviews and from the billing

If there is a simple way to get a whole service in a database you should choose that, maybe you need more memorfy
or faster hardware but you are so much more flexible instead working with flat-files and configs

Reindl Harald
the lounge interactive design GmbH
A-1060 Vienna | Hofmühlgasse 17
software-development / cms-solutions

phone: +43 (1) 595 3999 33
cellular: +43 (676) 40 221 40
icq: 154546673

mailto:h.reindl@...
http://www.thelounge.net/

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (268 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T04:41:45Z

On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote:
> Are you sure that syntax of your select query is correct?

No, that was pseudo code to demonstrate you can search for variations
within one query.

> how complex select call you will make in order to cover all
> variations? flowed format, quoted-printable, headers and body that
> might have text in n different charsets.

Just exactly the same amount a server with flat files would have to.
There's no difference.

> SQL is not designed to decode MIME on the fly.

It all reduces to search for a certain byte combination. You just have
to encode your search string to all variations you need, and put all
those in a single SELECT. That was my point.

In case you have to decode the mail, you need to retrieve,decode,search,
and still this is the same work a flat file mailserver would do. All
this discussion is about the speed of searching, and I didn't see an
example where a flat file server could search faster than the DB so far.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T04:06:19Z

2009.12.11 13:14 Michael Monnerie rašė:

> On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote:
>> Emails are not raw text. There are at least two ways to write "test"
>> in email and if you go to 8bit text, number of same text variations
>> multiplies. SQL can't search emails stored in DB, because SQL does
>> not know about encodings, mime formats and character sets
>
> So where's the difference? You can
> SELECT * ... WHERE mailtext LIKE 'test'::utf8 OR mailtext LIKE
> 'test'::base64 etc.
> and a flat file server would do the same anyway. The e-mail is stored in
> original format, so it would also search for test in all encodings.

Are you sure that syntax of your select query is correct?

how complex select call you will make in order to cover all variations?
flowed format, quoted-printable, headers and body that might have text in
n different charsets.

SQL is not designed to decode MIME on the fly.

> The question is anyway: Does an IMAP SEARCH search in several variations
> of "test"? What if it's base64 encoded?

Headers must be decoded, if charset is specified in search command. You
are free to read all IMAP stardards if you want as long as you don't
invent new SQL syntax in order to prove your point.

Glad to see that Daniel got suggestions to his problem. Maybe size of
database can be reduced by moving some accounts to other server?

--
Tomas

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-11T03:14:00Z

On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote:
> Emails are not raw text. There are at least two ways to write "test"
> in email and if you go to 8bit text, number of same text variations
> multiplies. SQL can't search emails stored in DB, because SQL does
> not know about encodings, mime formats and character sets

So where's the difference? You can
SELECT * ... WHERE mailtext LIKE 'test'::utf8 OR mailtext LIKE
'test'::base64 etc.
and a flat file server would do the same anyway. The e-mail is stored in
original format, so it would also search for test in all encodings.

The question is anyway: Does an IMAP SEARCH search in several variations
of "test"? What if it's base64 encoded?

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T03:09:12Z

On Freitag, 11. Dezember 2009 Paul J Stevens wrote:
> Currently, a full body text search will do a full table scan of the
> mimeparts table and pull in all mimeparts part of the messages in the
> mailbox being searched. If we want to skip all non text/* mimeparts
> (as allowed by the imap rfc), we'd have to add some knowledge of the
> mimetype contained in the mimepart. Doing so would be trivial. And
> so would fixing the query be that does the search.
>
Sounds like a nice-to-have feature :-)
That would be a great reason to upgrade to 2.3.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T03:06:30Z

On Freitag, 11. Dezember 2009 Michael Monnerie wrote:
> LVM snapshots
>
Another thing to remember: You can only do a snapshot of a single
filesystem at a time. So if you have your DB and attachments in
different volumes, snapshots are not transactions anymore. Some people
may be happy to live with that, though.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T03:02:39Z

On Freitag, 11. Dezember 2009 Daniel Urstöger wrote:
> Well, one can also do that with a filesystem based storage, you
> just need something similar to the MySQL replication for flat
> files. DRDB for example.
>
DRBD puts a burden on the server all the time. For a secure replication
you need to wait until the I/O on the remote server is on disk too. Only
if you relax that, and allow buffered I/O to the remote, the impact is
negligible. But then you risk a munged DB in case your first machine
brutally crashes during high I/O, and suddenly you loose some parts of
your transactions which the DB does not expect. It's not nice, because
the DB claims everything went OK, while some data in some tables is
wrong...

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T02:57:44Z

On Freitag, 11. Dezember 2009 Josh Marshall wrote:
> I have found that since linux kernel 2.6 series, LVM snapshots have
> caused system lockups. I used it happily in the 2.4 series.

That's why LVM snapshots are not used in XenServer 5.x. They also said
it's unstable, especially under high load.

> Besides
> that, I did mention *impact-free*. Adding a snapshot and reading from
> a snapshot severely impacts the speed of the running system.

I totally agree with your arguing. Having all together is much easier to
administer. Once it's too slow, I'll throw in more hardware. It's
cheaper to throw in a new server than to have the extra burden with
redundancy, backup/restore, etc...

So far, I haven't seen a limit on dbmail, while we had limits with older
POP-only systems before, where users had the setting "leave mail on
server". The server had to copy the flat file all over again for each
user, I/O stalled...

BTW: we upgraded from PostgreSQL 8.1 to 8.3, which exactly *doubled* the
speed of our nightly backups and vacuum/cluster runs. So that was a nice
step which I can recommend to everybody. I wonder if 8.4 will bring
another improvement.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 6531 .network.your.ideas.
//
// Wir haben zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://willhaben.at/iad/realestate/object?adId=15306857

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (205 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-11T00:10:19Z

Michael Monnerie wrote:
> On Donnerstag, 10. Dezember 2009 Tomas Kuliavas wrote:
>> DBMail might find its niche in some setups, but large mailboxes are
>> not in that niche. 750 GB DB proves it. You can't do text search raw
>> email sources. There is no point of storing them in DB.
>
> And you believe doing a raw text search on a 750GB flat file mailserver
> would be fast?

Raw text searches are not your typical usage pattern. Doing so in a
truly high speed fashion is a principle goal for all imap
implementations. For dbmail, using an external full text indexes such as
solr/lucene would be the most logical (and scalable) solution.

>
> dbmail 2.3 is different in that it stores mimeparts separately. Maybe a
> full text search skips binary attachments there. Paul?

Currently, a full body text search will do a full table scan of the
mimeparts table and pull in all mimeparts part of the messages in the
mailbox being searched. If we want to skip all non text/* mimeparts (as
allowed by the imap rfc), we'd have to add some knowledge of the
mimetype contained in the mimepart. Doing so would be trivial. And so
would fixing the query be that does the search.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T22:51:25Z

2009.12.11 01:33 Michael Monnerie rašė:
> On Donnerstag, 10. Dezember 2009 Tomas Kuliavas wrote:
>> DBMail might find its niche in some setups, but large mailboxes are
>> not in that niche. 750 GB DB proves it. You can't do text search raw
>> email sources. There is no point of storing them in DB.
>
> And you believe doing a raw text search on a 750GB flat file mailserver
> would be fast?

Emails are not raw text. There are at least two ways to write "test" in
email and if you go to 8bit text, number of same text variations
multiplies. SQL can't search emails stored in DB, because SQL does not
know about encodings, mime formats and character sets

--
Tomas

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T17:58:09Z

> Which distro are you using? I havent had any snapshot related problems
> yet, but maybe that is related to how much data you have? The
> snapshots I create rately have more then 20GB of data.
> And I don't want to point out a better distro for you but I see this
> mailing list as quite a nice source of knowledge and of exchange ...
> So I just want to know to learn from :)

To be honest I haven't tried it for a long time. The snapshot problems I
had were back about a dozen point releases ago. I was using a vanilla
kernel compiled from source.

What I was trying to do was set up a netapp style snapshot system so
that it would have snapshots for the last 3 hours (hourly), the last day
(3-hourly) and the last week (daily). No this was not for mail, it was
for web shared space. Maybe I was trying to do too much? In the end I
just gave up, that was about 2 years ago.

As a side-note I've also tinkered with nilfs2 and it seems quite good
with its continuous snapshotting, however under heavy load I found it
started running away with continuous data writes. Maybe when it gets
closer to production status it'll be the way to go.

But we're digressing ;)

Josh.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T17:57:00Z

On Fri, 2009-12-11 at 02:01 +0100, Daniel Urstöger wrote:
> > Not quite. Having a separate slave database server to do the heavy
> > work
> > of backups has no impact on the master database during the backup
> > period. Therefore the master database is always at "normal speed".
>
> Well, one can also do that with a filesystem based storage, you just
> need something similar to the MySQL replication for flat files. DRDB
> for example.

I have also had DRBD in production, sharing in a similar way. However
when the DRBD reconnects it needs to scan through all the changes in the
master disk to find and copy across all the changed sectors. So there is
a performance hit when the copy completes. It depends on the number of
changed sectors to how much of a hit this will be. In most cases it
would be minor, so I am being picky here. One problem I did have with
DRBD (used 0.7 series) is that I would have the system lock me out if
only one side came up, so I was completely without service until the
timeout or I interfered with it to manually switch to the appropriate
master. They may have fixed this with the 0.8 series.

You could argue that with the mysql binary log there is a performance
hit when the copy completes and the slave reconnects, but if the binary
logs are on a separate disk spindle, the don't affect the performance of
the main database files.

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T17:00:50Z

> Not quite. Having a separate slave database server to do the heavy
> work
> of backups has no impact on the master database during the backup
> period. Therefore the master database is always at "normal speed".

Well, one can also do that with a filesystem based storage, you just
need something similar to the MySQL replication for flat files. DRDB
for example.
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T16:54:10Z

My message was not quite finished, sadly there isnt an App for that ;)

> I have found that since linux kernel 2.6 series, LVM snapshots have
> caused system lockups. I used it happily in the 2.4 series. Besides
> that, I did mention *impact-free*. Adding a snapshot and reading
> from a
> snapshot severely impacts the speed of the running system. Yes you
> get a
> clean backup, but the hard disk is being placed under a huge read
> strain, not to mention the extra COW load for every write to the
> filesystem.

Which distro are you using? I havent had any snapshot related problems
yet, but maybe that is related to how much data you have? The
snapshots I create rately have more then 20GB of data.
And I don't want to point out a better distro for you but I see this
mailing list as quite a nice source of knowledge and of exchange ...
So I just want to know to learn from :)

> I believe that dbmail on MySQL requires the use of InnoDB, which I
> believe (or has this recently changed?) does not support Full Text
> Index. Maybe using something like Sphinx as a bolt-on would be handy
> for
> doing IMAP searches. I generally sync all my emails to my desktop
> machine and do any searches on the local copies. Then searches don't
> impact the servers :)

It does require transactions, so for MySQL you are quite tied to
innodb, unless you want to try something more experimental.

It would be nice of customers to act like you, but usually they don't.
Though I really liked that Thunderbird 3 now even heavily suggest
syncing to local disk.

Sphinx for searches would be awesome! The new release is even
compatible to the MySQL client / libraries. So no API to fiddle with,
but still needs quite some work, I guess ...

>
>>
>>
_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Optimizing Dbmail Database

2009-12-10T16:52:13Z

Am 11.12.2009 01:38, schrieb "Daniel Urstöger" <daniel@...>":

> Yes, but after backing to the snapshot to some place one can remove it
> and the speed will be back to normal. So, running a db slave and using
> mysqldump for backups is not much different.

a) the slave can yun on one or more other physical machines
b) shut down the slave, run rsync, start the slave

There is no moment you have more load on the master

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

signature.asc (268 bytes) Download Attachment

Re: Optimizing Dbmail Database

2009-12-10T16:43:31Z

On Fri, 2009-12-11 at 01:38 +0100, Daniel Urstöger wrote:
> Yes, but after backing to the snapshot to some place one can remove
> it
> and the speed will be back to normal. So, running a db slave and
> using
> mysqldump for backups is not much different.

Not quite. Having a separate slave database server to do the heavy work
of backups has no impact on the master database during the backup
period. Therefore the master database is always at "normal speed".

_______________________________________________
DBmail mailing list
DBmail@...
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail