expire plugin + --exec-mail in 1.2.6 version

[Nikita Koshikov]

Hello list,

After upgrade to dovecot version 1.2.6 I have noticed some quota malfunctions(users become often overquota). Looking at logs closely I have found that quota value resets to general dovecot-config(
quota = maildir:Mailbox quota
quota_rule = *:storage=500M
quota_rule2 = Trash:storage=10%%
), after dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh command. The  problem that some users have their own quotas stored in ldap.

For example, user sr@... have maildirsize before running expire-tool(1G value from ldap):
# head -2 /data/mail/domain.com/sr/data/maildirsize
10485760000S
836423205 8285

Then I run expire-tool with mail_debug enabled:
# /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh
Info: Loading modules from directory: /usr/lib/dovecot/imap
Info: Module loaded: /usr/lib/dovecot/imap/lib10_quota_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib11_trash_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_autocreate_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_expire_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_virtual_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_zlib_plugin.so
Info: Quota root: name=Mailbox quota backend=maildir args=
Info: Quota rule: root=Mailbox quota mailbox=* bytes=524288000 messages=0
Info: Quota rule: root=Mailbox quota mailbox=Trash bytes=52428800 (10%) messages=0
Info: Quota warning: bytes=471859200 (90%) messages=0 command=/etc/dovecot/plugins/quota_warning.sh 90
Info: expire: pattern=Trash type=expunge secs=2592000
Info: expire: pattern=Spam type=expunge secs=2592000
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/koshikov.n
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/koshikov.n/data, index=, control=, inbox=/data/mail/domain.com/koshikov.n/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/koshikov.n/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: koshikov.n@.../Spam: seq=1 uid=1869: Expunge
Info: koshikov.n@.../Spam: seq=2 uid=1870: Expunge
Info: koshikov.n@.../Spam: seq=3 uid=1871: Expunge
Info: koshikov.n@.../Spam: seq=4 uid=1872: Expunge
Info: koshikov.n@.../Spam: seq=5 uid=1873: Expunge
Info: koshikov.n@.../Spam: seq=6 uid=1874: Expunge
Info: koshikov.n@.../Spam: seq=7 uid=1875: Expunge
Info: koshikov.n@.../Spam: seq=8 uid=1876: Expunge
Info: koshikov.n@.../Spam: seq=9 uid=1877: Expunge
Info: koshikov.n@.../Spam: seq=10 uid=1878: Expunge
Info: koshikov.n@.../Spam: seq=11 uid=1879: Expunge
Info: koshikov.n@.../Spam: seq=12 uid=1880: Expunge
Info: koshikov.n@.../Spam: timestamp 1256618680 (Tue Oct 27 06:44:40 2009) -> 1256716674 (Wed Oct 28 09:57:54 2009)
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/fsb
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/fsb/data, index=, control=, inbox=/data/mail/domain.com/fsb/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/fsb/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: fsb@.../Trash: seq=1 uid=578: Expunge
Info: fsb@.../Trash: seq=2 uid=579: Expunge
Info: fsb@.../Trash: seq=3 uid=580: Expunge
Info: fsb@.../Trash: seq=4 uid=581: Expunge
Info: fsb@.../Trash: seq=5 uid=582: Expunge
Info: fsb@.../Trash: seq=6 uid=583: Expunge
Info: fsb@.../Trash: seq=7 uid=584: Expunge
Info: fsb@.../Trash: seq=8 uid=585: Expunge
Info: fsb@.../Trash: seq=9 uid=586: Expunge
Info: fsb@.../Trash: seq=10 uid=587: Expunge
Info: fsb@.../Trash: seq=11 uid=588: Expunge
Info: fsb@.../Trash: seq=12 uid=589: Expunge
Info: fsb@.../Trash: seq=13 uid=590: Expunge
Info: fsb@.../Trash: timestamp 1256627284 (Tue Oct 27 09:08:04 2009) -> 1256732991 (Wed Oct 28 14:29:51 2009)
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/arsen
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/arsen/data, index=, control=, inbox=/data/mail/domain.com/arsen/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/arsen/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: arsen@.../Trash: seq=1 uid=14: Expunge
Info: arsen@.../Trash: seq=2 uid=15: Expunge
Info: arsen@.../Trash: seq=3 uid=16: Expunge
Info: arsen@.../Trash: seq=4 uid=17: Expunge
Info: arsen@.../Trash: seq=5 uid=18: Expunge
Info: arsen@.../Trash: seq=6 uid=19: Expunge
Info: arsen@.../Trash: seq=7 uid=20: Expunge
Info: arsen@.../Trash: seq=8 uid=21: Expunge
Info: arsen@.../Trash: seq=9 uid=22: Expunge
Info: arsen@.../Trash: seq=10 uid=23: Expunge
Info: arsen@.../Trash: seq=11 uid=24: Expunge
Info: arsen@.../Trash: timestamp 1256630371 (Tue Oct 27 09:59:31 2009) -> 1256733711 (Wed Oct 28 14:41:51 2009)
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/koshikov.n
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/koshikov.n/data, index=, control=, inbox=/data/mail/domain.com/koshikov.n/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/koshikov.n/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: koshikov.n@.../Trash: seq=1 uid=2561: Expunge
Info: koshikov.n@.../Trash: timestamp 1256709279 (Wed Oct 28 07:54:39 2009) -> 1256795118 (Thu Oct 29 07:45:18 2009)
Info: auth input: quota_rule=*:bytes=10485760000
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/sr
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/sr/data, index=, control=, inbox=/data/mail/domain.com/sr/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/sr/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: sr@.../Spam: seq=1 uid=405: Expunge
Info: sr@.../Spam: seq=2 uid=406: Expunge
Info: sr@.../Spam: seq=3 uid=407: Expunge
Info: sr@.../Spam: timestamp 1256710764 (Wed Oct 28 08:19:24 2009) -> 1256980000 (Sat Oct 31 11:06:40 2009)
Info: kolotilo.m@.../Trash: stop, expire time in future: Wed Oct 28 10:27:41 2009

And after expire-tool ends it's stuff, user sr@... have(512M from dovecot-config):
524288000SS
836423205 8285

As you can see - quota reset to default value.
In changes to version 1.2.6 I see
- dovecot --exec-mail was broken.

Is fixes to 1.2.6 touch userdb lookups from --exec-mail ? With 1.2.4 expire-tool works correct and I have no config changes to dovecot setup.

If dovecot -n output is needed, please let me know.

Thanks.

[Timo Sirainen]

On Wed, 2009-10-28 at 10:30 +0200, Nikita Koshikov wrote:
> The  problem that some users have their own quotas stored in ldap.

So the quota_rule should be returned by userdb lookup from ldap, right?

> Info: auth input: uid=8
> Info: auth input: gid=12
> Info: auth input: home=/data/mail/domain.com/koshikov.n
> Info: auth input: mail=maildir:~/data

This should have listed quota_rule.

> If dovecot -n output is needed, please let me know.

If the above doesn't help, then yeah, dovecot -n and dovecot-ldap.conf
contents.

[Nikita Koshikov]

On Thu, 29 Oct 2009 13:10:37 -0400
Timo Sirainen <tss@...> wrote:

> On Wed, 2009-10-28 at 10:30 +0200, Nikita Koshikov wrote:
> > The  problem that some users have their own quotas stored in ldap.
>
> So the quota_rule should be returned by userdb lookup from ldap, right?

Yes, and it's working fine with deliver and general imap process, it only failed(reset to general quota_rule) after expire-rool.sh script finished his duties.

>
> > Info: auth input: uid=8
> > Info: auth input: gid=12
> > Info: auth input: home=/data/mail/domain.com/koshikov.n
> > Info: auth input: mail=maildir:~/data
>
> This should have listed quota_rule.
>
The problematic user is sr, and it has quota_rule option(running with --test)

(Thu Oct 29 07:45:18 2009)
Info: auth input: quota_rule=*:bytes=10485760000
                        ^^^^^^^^^^^^^^^^^^^^^^^^^ returned from ldap
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/sr
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/sr/data, index=, control=, inbox=/data/mail/domain.com/sr/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/sr/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: sr@.../Spam: seq=1 uid=405: Expunge
Info: sr@.../Spam: seq=2 uid=406: Expunge
Info: sr@.../Spam: seq=3 uid=407: Expunge
Info: sr@.../Spam: timestamp 1256710764 (Wed Oct 28 08:19:24 2009) -> 1256980000 (Sat Oct 31 11:06:40 2009)

> > If dovecot -n output is needed, please let me know.
>
> If the above doesn't help, then yeah, dovecot -n and dovecot-ldap.conf
> contents.
>
dovecot -n:
# 1.2.6: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.26-gentoo-r4 i686 Gentoo Base System release 1.12.11.1
log_path: /var/log/dovecot/dovecot-error.log
info_log_path: /var/log/dovecot/dovecot.log
protocols: imaps managesieve
ssl_cert_file: /etc/ssl/dovecot/imaps.crt
ssl_key_file: /etc/ssl/dovecot/imaps.key
login_dir: /var/run/dovecot/login
login_executable(default): /usr/libexec/dovecot/imap-login
login_executable(imap): /usr/libexec/dovecot/imap-login
login_executable(managesieve): /usr/libexec/dovecot/managesieve-login
login_greeting: Server ready.
login_processes_count: 20
login_max_processes_count: 512
mail_max_userip_connections(default): 20
mail_max_userip_connections(imap): 20
mail_max_userip_connections(managesieve): 10
first_valid_uid: 8
last_valid_uid: 8
first_valid_gid: 12
last_valid_gid: 12
mail_drop_priv_before_exec: yes
mail_executable(default): /usr/libexec/dovecot/imap
mail_executable(imap): /usr/libexec/dovecot/imap
mail_executable(managesieve): /usr/libexec/dovecot/managesieve
mail_plugins(default): quota imap_quota trash expire zlib autocreate virtual
mail_plugins(imap): quota imap_quota trash expire zlib autocreate virtual
mail_plugins(managesieve):
mail_plugin_dir(default): /usr/lib/dovecot/imap
mail_plugin_dir(imap): /usr/lib/dovecot/imap
mail_plugin_dir(managesieve): /usr/lib/dovecot/managesieve
imap_client_workarounds(default): delay-newmail
imap_client_workarounds(imap): delay-newmail
imap_client_workarounds(managesieve):
namespace:
  type: private
  separator: /
  location: maildir:~/data
  inbox: yes
  list: yes
  subscriptions: yes
namespace:
  type: private
  separator: /
  prefix: Company/
  location: virtual:/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
  list: yes
lda:
  postmaster_address: postmaster@...
  hostname: mail.domain.com
  mail_plugins: quota trash expire sieve virtual
  quota_full_tempfail: yes
  sendmail_path: /usr/sbin/sendmail
  auth_socket_path: /var/run/dovecot/auth-master
  log_path: /var/log/dovecot/dovecot-deliver.log
  info_log_path: /var/log/dovecot/dovecot-deliver.log
  global_script_path: /etc/dovecot/sieve/default.sieve
  sieve_global_dir: /etc/dovecot/sieve
auth default:
  mechanisms: plain login
  default_realm: domain.com
  cache_size: 10240
  cache_negative_ttl: 0
  user: dovecot_auth
  master_user_separator: *
  worker_max_count: 50
  passdb:
    driver: passwd-file
    args: /etc/dovecot/passdb/master.pwd
    master: yes
  passdb:
    driver: passwd-file
    args: /etc/dovecot/passdb/users.pwd
  passdb:
    driver: ldap
    args: /etc/dovecot/dovecot-ldap.conf
  userdb:
    driver: prefetch
  userdb:
    driver: ldap
    args: /etc/dovecot/dovecot-userdb-ldap.conf
  userdb:
    driver: passwd-file
    args: /etc/dovecot/passdb/users.pwd
  socket:
    type: listen
    client:
      path: /var/run/dovecot/auth-client
      mode: 432
      user: mail
      group: dovecot_auth
    master:
      path: /var/run/dovecot/auth-master
      mode: 384
      user: mail
      group: mail
plugin:
  quota_warning: storage=90%% /etc/dovecot/plugins/quota_warning.sh 90
  quota: maildir:Mailbox quota
  quota_rule: *:storage=500M
  quota_rule2: Trash:storage=10%%
  trash: /etc/dovecot/plugins/dovecot-trash.conf
  expire: Trash 30 Spam 30
  expire_dict: proxy::expire
  autocreate: Drafts
  autocreate2: Sent
  autocreate3: Spam
  autocreate4: Trash
  autosubscribe: Drafts
  autosubscribe2: Sent
  autosubscribe3: Spam
  autosubscribe4: Trash
  sieve: ~/.dovecot.sieve
  sieve_dir: ~/sieve
  sieve_extensions: +notify
  sieve_before: /etc/dovecot/sieve/default.sieve
dict:
  expire: sqlite:/etc/dovecot/plugins/expire.conf

dovecot-ldap.conf:

uris = ldaps://off-dc-1 ldaps://off-dc-2
dn = imap@
dnpass = qweqwe
auth_bind = yes
ldap_version = 3
base = OU=Company,DC=company,DC=intranet
pass_attrs = =userdb_uid=8, =userdb_gid=12,=userdb_home=/data/mail/%d/%n, =userdb_mail=maildir:~/data, maxStorage=userdb_quota_rule=*:bytes=%$

pass_filter = (&(mail=%u)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

user_attrs = =uid=8, =gid=12,=home=/data/mail/%d/%n, =mail=maildir:~/data, maxStorage=quota_rule=*:bytes=%$

user_filter = (&(mail=%u)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

[Nikita Koshikov]

On Thu, 29 Oct 2009 13:10:37 -0400
Timo Sirainen <tss@...> wrote:

> On Wed, 2009-10-28 at 10:30 +0200, Nikita Koshikov wrote:
> > The  problem that some users have their own quotas stored in ldap.
>
> So the quota_rule should be returned by userdb lookup from ldap, right?
>
One more clean example.
For user koshikov.n@... I change quota in AD to 600M(default user quota is 512M,). After successful login to imap account, maildirsize is(first 2 lines):

629145600S
260658345 5484

So, quota value read from ldap database and applied to user's mailbox.
Then I run /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh --test:

Info: Loading modules from directory: /usr/lib/dovecot/imap
Info: Module loaded: /usr/lib/dovecot/imap/lib10_quota_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib11_trash_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_autocreate_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_expire_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_virtual_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_zlib_plugin.so
Info: Quota root: name=Mailbox quota backend=maildir args=
Info: Quota rule: root=Mailbox quota mailbox=* bytes=524288000 messages=0
Info: Quota rule: root=Mailbox quota mailbox=Trash bytes=52428800 (10%) messages=0
Info: Quota warning: bytes=471859200 (90%) messages=0 command=/etc/dovecot/plugins/quota_warning.sh 90
Info: expire: pattern=Trash type=expunge secs=2592000
Info: expire: pattern=Spam type=expunge secs=2592000
Info: auth input: quota_rule=*:bytes=629145600
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/koshikov.n
Info: auth input: mail=maildir:~/data
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/koshikov.n/data, index=, control=, inbox=/data/mail/domain.com/koshikov.n/data
Info: Namespace: type=private, prefix=Agromat/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/koshikov.n/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2
Info: koshikov.n@.../Spam: seq=1 uid=1933: Expunge
Info: koshikov.n@.../Spam: timestamp 1256902423 (Fri Oct 30 13:33:43 2009) -> 1256903611 (Fri Oct 30 13:53:31 2009)

Quota_rule are different from auth section and quota root\rule section. After expire-tool expunges e-mails, the maildirsize file become:

524288000S
260686038 5487

Which is default quota_rule value, not from ldap database.

[Timo Sirainen]

On Fri, 2009-10-30 at 14:03 +0200, Nikita Koshikov wrote:
> > > The  problem that some users have their own quotas stored in ldap.

See if the attached patch helps? I'm a bit afraid that it could also
break something. If it does, there's really no easy way to get this
fixed before v2.0.


[diff]

---

diff -r a0d72e0d88dd src/plugins/expire/expire-tool.c
--- a/src/plugins/expire/expire-tool.c Thu Oct 29 17:40:33 2009 -0400
+++ b/src/plugins/expire/expire-tool.c Fri Oct 30 13:15:45 2009 -0400
@@ -52,6 +52,7 @@
  /* user no longer exists */
  return 0;
  }
+ module_dir_init(modules);
 
  ctx->mail_user = mail_user_init(user);
  mail_user_set_home(ctx->mail_user, getenv("HOME"));
@@ -64,6 +65,7 @@
 {
  mail_user_unref(&ctx->mail_user);
  i_free_and_null(ctx->user);
+ module_dir_deinit(modules);
 }
 
 static int
@@ -265,8 +267,6 @@
  mail_storage_register_all();
  mailbox_list_register_all();
 
- module_dir_init(modules);
-
  expire_get_global_mail_ids();
 
  base_dir = getenv("BASE_DIR");

---

[Nikita Koshikov]

On Fri, 30 Oct 2009 13:16:47 -0400
Timo Sirainen <tss@...> wrote:

> On Fri, 2009-10-30 at 14:03 +0200, Nikita Koshikov wrote:
> > > > The  problem that some users have their own quotas stored in ldap.
>
> See if the attached patch helps? I'm a bit afraid that it could also
> break something. If it does, there's really no easy way to get this
> fixed before v2.0.
>

The patch helped with quotas, but it raises new error - "segmentation error"

Here details:
# /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh
Info: Loading modules from directory: /usr/lib/dovecot/imap
Info: Module loaded: /usr/lib/dovecot/imap/lib10_quota_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib11_trash_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_autocreate_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_expire_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_virtual_plugin.so
Info: Module loaded: /usr/lib/dovecot/imap/lib20_zlib_plugin.so

-->koshikov.n@... - have own quota in ldap, expire-tool is working fine:

Info: expire: pattern=Trash type=expunge secs=2592000
Info: expire: pattern=Spam type=expunge secs=2592000
Info: auth input: quota_rule=*:bytes=629145600
Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/koshikov.n
Info: auth input: mail=maildir:~/data
Info: Quota root: name=Mailbox quota backend=maildir args=
Info: Quota rule: root=Mailbox quota mailbox=* bytes=629145600 messages=0
Info: Quota rule: root=Mailbox quota mailbox=Trash bytes=62914560 (10%) messages=0
Info: Quota warning: bytes=566231040 (90%) messages=0 command=/etc/dovecot/plugins/quota_warning.sh 90
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/koshikov.n/data, index=, control=, inbox=/data/mail/domain.com/koshikov.n/data
Info: Namespace: type=private, prefix=Company/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/koshikov.n/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2

--> bunkina.l@... have default quota value, expire-tool is working fine:

Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/bunkina.l
Info: auth input: mail=maildir:~/data
Info: Quota root: name=Mailbox quota backend=maildir args=
Info: Quota rule: root=Mailbox quota mailbox=* bytes=524288000 messages=0
Info: Quota rule: root=Mailbox quota mailbox=Trash bytes=52428800 (10%) messages=0
Info: Quota warning: bytes=471859200 (90%) messages=0 command=/etc/dovecot/plugins/quota_warning.sh 90
Info: Namespace: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
Info: maildir: data=~/data
Info: maildir++: root=/data/mail/domain.com/bunkina.l/data, index=, control=, inbox=/data/mail/domain.com/bunkina.l/data
Info: Namespace: type=private, prefix=Company/, sep=/, inbox=no, hidden=no, list=yes, subscriptions=no
Info: virtual: data=/var/mail/virtual:INDEX=MEMORY:LAYOUT=maildir++
Info: maildir++: root=/var/mail/virtual, index=, control=, inbox=
Info: Namespace : Using permissions from /data/mail/domain.com/bunkina.l/data: mode=0700 gid=-1
Info: trash plugin: Added 'Trash' with priority 1
Info: trash plugin: Added 'Spam' with priority 2

--> podburtniy@... - have default quota, this account cause segmentation error, while running expire-tool:

Info: auth input: uid=8
Info: auth input: gid=12
Info: auth input: home=/data/mail/domain.com/podburtniy
Info: auth input: mail=maildir:~/data
Info: Quota root: name=Mailbox quota backend=maildir args=
Info: Quota rule: root=Mailbox quota mailbox=* bytes=524288000 messages=0
Info: Quota rule: root=Mailbox quota mailbox=Trash bytes=52428800 (10%) messages=0
Info: Quota warning: bytes=471859200 (90%) messages=0 command=/etc/dovecot/plugins/quota_warning.sh 90
Ошибка сегментирования

The only difference with mailbox podburtniy@... that it's ldap mail field is different from samaccountname:

podburtniy@...
mail: podburtniy@...
sAMAccountName: podburtn

koshikov.n@...
mail: koshikov.n@...
sAMAccountName: koshikov.n

But, I'm using only mail/maxstorage/useraccountcontrol attributes for mail staff.
Users login to their accounts using full e-mail address and AD password.

[Nikita Koshikov]

On Fri, 30 Oct 2009 13:16:47 -0400
Timo Sirainen <tss@...> wrote:

> On Fri, 2009-10-30 at 14:03 +0200, Nikita Koshikov wrote:
> > > > The  problem that some users have their own quotas stored in ldap.
>
> See if the attached patch helps? I'm a bit afraid that it could also
> break something. If it does, there's really no easy way to get this
> fixed before v2.0.
>

Note:
I rechecked expire-tool behavior with version 1.2.4 and it's working the same way as 1.2.6,
so my problem have more long history, than I thought.

[Nikita Koshikov]

>
> The patch helped with quotas, but it raises new error - "segmentation
> error"
>
>
After some more tests, I find out that crash happens, while trying to
proceed  the third user, and the user's quota and other settings not
important.

[Timo Sirainen]

On Sat, 2009-10-31 at 10:42 +0200, Nikita Koshikov wrote:
> The patch helped with quotas, but it raises new error - "segmentation error"

gdb backtrace would be helpful. But how to actually get it could be a
bit difficult, since I guess kernel thinks expire-tool is a setuid
program.

There are two ways I guess:

a) Apply attached diff1 and see if core dump gets written. Once you get
a core, use "gdb expire-tool core", "bt full".

b) If not, apply diff2 and start expire-tool. Then look up its pid and
run "gdb -p <pid>", "cont", wait for crash, "bt full".

[diff1]

---

diff -r 19aee456cec3 src/plugins/expire/auth-client.c
--- a/src/plugins/expire/auth-client.c Thu Nov 05 12:25:43 2009 -0500
+++ b/src/plugins/expire/auth-client.c Thu Nov 05 12:49:33 2009 -0500
@@ -64,6 +64,7 @@
  i_fatal("seteuid(%s) failed: %m", dec2str(reply->uid));
  current_uid = reply->uid;
  }
+ restrict_access_allow_coredumps(TRUE);
 
  fields = array_get(&reply->extra_fields, &count);
  for (i = 0; i < count; i++) {

---

[diff2]

---

diff -r 19aee456cec3 src/plugins/expire/expire-tool.c
--- a/src/plugins/expire/expire-tool.c Thu Nov 05 12:25:43 2009 -0500
+++ b/src/plugins/expire/expire-tool.c Thu Nov 05 12:50:09 2009 -0500
@@ -389,6 +389,7 @@
  struct ioloop *ioloop;
  bool test = FALSE;
 
+ sleep(10);
  lib_init();
  lib_signals_init();
  random_init();

---

[Simon Gao]

Hi,

I now find more information about the locking problem.

Users have multiple active imap connection from same machine for example:

13946 ?        S      0:00 imap [user1 192.168.1.11]
14491 ?        S      0:00 imap [user2 192.168.1.11]
28504 ?        S      0:00 imap [user1 192.168.1.11]
29040 ?        S      0:00 imap [user2 192.168.1.11]
29041 ?        S      0:00 imap [user2 192.168.1.11]

The /var/spool/mail is on NFS share.

Running strace on 13946 and 28504:

# strace -p 13946                    
Process 13946 attached - interrupt to quit      
gettimeofday({1257443764, 885542}, {0, 0}) = 0  
gettimeofday({1257443764, 885682}, NULL) = 0    
epoll_wait(6, {}, 4, 1079)              = 0      
gettimeofday({1257443765, 965259}, {0, 0}) = 0  
gettimeofday({1257443765, 965392}, NULL) = 0    
epoll_wait(6, {}, 4, 5000)              = 0      
gettimeofday({1257443770, 965380}, {0, 0}) = 0  
gettimeofday({1257443770, 965531}, NULL) = 0    
epoll_wait(6, {}, 4, 5000)              = 0      
gettimeofday({1257443775, 965409}, {0, 0}) = 0  
gettimeofday({1257443775, 965555}, NULL) = 0    
epoll_wait(6, {}, 4, 5000)              = 0      
gettimeofday({1257443780, 965498}, {0, 0}) = 0  
gettimeofday({1257443780, 965639}, NULL) = 0    
epoll_wait(6, {}, 4, 5000)              = 0      
gettimeofday({1257443785, 965779}, {0, 0}) = 0  
gettimeofday({1257443785, 965924}, NULL) = 0    
epoll_wait(6, {}, 4, 5000)              = 0      
gettimeofday({1257443790, 969831}, {0, 0}) = 0  
gettimeofday({1257443790, 969969}, NULL) = 0    
epoll_wait(6, {}, 4, 3111)              = 0      
gettimeofday({1257443794, 89867}, {0, 0}) = 0    
writev(1, [{"* OK Still here"..., 15}, {"\r\n"..., 2}], 2) = 17
gettimeofday({1257443794, 90356}, NULL) = 0                    
stat64("/var/mail/user1", {st_mode=S_IFREG|0600, st_size=8603552, ...}) = 0
gettimeofday({1257443794, 103769}, NULL) = 0                              
epoll_wait(6, {}, 4, 1866)              = 0                                
gettimeofday({1257443795, 969925}, {0, 0}) = 0                            
gettimeofday({1257443795, 970063}, NULL) = 0                              
epoll_wait(6, {}, 4, 4999)              = 0                                
gettimeofday({1257443800, 970025}, {0, 0}) = 0                            
gettimeofday({1257443800, 970156}, NULL) = 0                              
epoll_wait(6, {}, 4, 5000)              = 0                                
gettimeofday({1257443805, 970088}, {0, 0}) = 0                            
gettimeofday({1257443805, 970221}, NULL) = 0                              
epoll_wait(6, {}, 4, 5000)              = 0                                
gettimeofday({1257443810, 970121}, {0, 0}) = 0                            
gettimeofday({1257443810, 970261}, NULL) = 0                              
epoll_wait(6, {}, 4, 5000)              = 0


# strace -p 28504
Process 28504 attached - interrupt to quit
gettimeofday({1257443943, 506081}, {0, 0}) = 0
gettimeofday({1257443943, 506235}, NULL) = 0
epoll_wait(6, {}, 4, 22730)             = 0
gettimeofday({1257443966, 226716}, {0, 0}) = 0
gettimeofday({1257443966, 226855}, NULL) = 0
epoll_wait(6, {}, 4, 10)                = 0
gettimeofday({1257443966, 236612}, {0, 0}) = 0
stat64("/var/mail/user1", {st_mode=S_IFREG|0600, st_size=8603552, ...}) = 0
gettimeofday({1257443966, 259124}, NULL) = 0
epoll_wait(6,

Both imap connections try to access same /var/mail/user1 file. This caused
very slow performance.

Why are there multiple active imap connections? How to stop it?

Thanks,

Simon

           

[Nikita Koshikov]

On Thu, 05 Nov 2009 12:52:08 -0500
Timo Sirainen <tss@...> wrote:

> On Sat, 2009-10-31 at 10:42 +0200, Nikita Koshikov wrote:
> > The patch helped with quotas, but it raises new error - "segmentation error"
>
> gdb backtrace would be helpful. But how to actually get it could be a
> bit difficult, since I guess kernel thinks expire-tool is a setuid
> program.
>
> There are two ways I guess:
>
> a) Apply attached diff1 and see if core dump gets written. Once you get
> a core, use "gdb expire-tool core", "bt full".

This didn't create core file.

>
> b) If not, apply diff2 and start expire-tool. Then look up its pid and
> run "gdb -p <pid>", "cont", wait for crash, "bt full".

I don't understand why gdb says "no debugging symbols found" for some dovecot's executables, all binaries under /usr/libexec/dovecot have "not stripped"

GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
Attaching to process 26407
Reading symbols from /usr/libexec/dovecot/expire-tool...(no debugging symbols found)...done.
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libpthread.so.0...
(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f02ac0 (LWP 26407)]
Loaded symbols for /lib/libpthread.so.0
(no debugging symbols found)
0xffffe424 in __kernel_vsyscall ()
(gdb) cont
Continuing.
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7f02ac0 (LWP 26407)]
0xb7f88749 in ?? () from /lib/libc.so.6
(gdb) bt full
#0  0xb7f88749 in ?? () from /lib/libc.so.6
No symbol table info available.
Cannot access memory at address 0xbf690ff8
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/libexec/dovecot/expire-tool, process 26407

[Timo Sirainen]

On Fri, 2009-11-06 at 10:27 +0200, Nikita Koshikov wrote:
> I don't understand why gdb says "no debugging symbols found" for some
> dovecot's executables, all binaries under /usr/libexec/dovecot have
> "not stripped"
..
> (gdb) bt full
> #0  0xb7f88749 in ?? () from /lib/libc.so.6
> No symbol table info available.
> Cannot access memory at address 0xbf690ff8

Yeah, something in your system has broken gdb.. Makes the debugging
annoyingly difficult.

[Timo Sirainen]

On Thu, 2009-11-05 at 10:01 -0800, Simon Gao wrote:

> stat64("/var/mail/user1", {st_mode=S_IFREG|0600, st_size=8603552, ...}) = 0

These stats are fine.

> Both imap connections try to access same /var/mail/user1 file. This caused
> very slow performance.

Well, it usually shouldn't.. Although it depends on IMAP client I guess.

> Why are there multiple active imap connections? How to stop it?

Some clients do that. For example with Thunderbird you can configure how
many connections it creates.

As for your other email..:

> We just have another user reporting problem. There is no lock on
> /var/spool/mail/<user> file.
..
> fcntl64(8, F_SETLKW64, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0},
> 0xff9f4db4) = ? ERESTARTSYS (To be restarted)                                  

The straces show that the file is locked. Maybe you've just disabled
dotlocks? Or anyway read locks don't create the dotlock file so that
could be it also.

So if some processes are trying to do fcntl locks, you could find out
what process actually has the file locked and what it's doing.

Anyway, the easiest solution would be to just get rid of mbox and switch
to maildir. No more locking troubles.

[Nikita Koshikov]

On Fri, 06 Nov 2009 18:42:49 -0500
Timo Sirainen <tss@...> wrote:

> On Fri, 2009-11-06 at 10:27 +0200, Nikita Koshikov wrote:
> > I don't understand why gdb says "no debugging symbols found" for some
> > dovecot's executables, all binaries under /usr/libexec/dovecot have
> > "not stripped"
> ..
> > (gdb) bt full
> > #0  0xb7f88749 in ?? () from /lib/libc.so.6
> > No symbol table info available.
> > Cannot access memory at address 0xbf690ff8
>
> Yeah, something in your system has broken gdb.. Makes the debugging
> annoyingly difficult.
>

I have wrote simple wrapper that removes maildirsize  if the messages were expunged from user's mailbox and if user has non-default quota value. Is there any more convenient  way to force dovecot recalculate quota size ? I have overview IMAP-QOUTA rfc, but seemed that it lacks such imap command, maybe some internal dovecot command exist?

[Timo Sirainen]

On Mon, 2009-11-23 at 18:01 +0200, Nikita Koshikov wrote:

> On Fri, 06 Nov 2009 18:42:49 -0500
> Timo Sirainen <tss@...> wrote:
>
> > On Fri, 2009-11-06 at 10:27 +0200, Nikita Koshikov wrote:
> > > I don't understand why gdb says "no debugging symbols found" for some
> > > dovecot's executables, all binaries under /usr/libexec/dovecot have
> > > "not stripped"
> > ..
> > > (gdb) bt full
> > > #0  0xb7f88749 in ?? () from /lib/libc.so.6
> > > No symbol table info available.
> > > Cannot access memory at address 0xbf690ff8
> >
> > Yeah, something in your system has broken gdb.. Makes the debugging
> > annoyingly difficult.

You managed to get backtraces from the other bug. What did you do
differently? Maybe you can get backtrace from this bug now too?

> I have wrote simple wrapper that removes maildirsize  if the messages
> were expunged from user's mailbox and if user has non-default quota
> value. Is there any more convenient  way to force dovecot recalculate
> quota size ? I have overview IMAP-QOUTA rfc, but seemed that it lacks
> such imap command, maybe some internal dovecot command exist?

No, there's no other way.

[Nikita Koshikov]

On Mon, 23 Nov 2009 12:01:53 -0500
Timo Sirainen <tss@...> wrote:

> On Mon, 2009-11-23 at 18:01 +0200, Nikita Koshikov wrote:
> > On Fri, 06 Nov 2009 18:42:49 -0500
> > Timo Sirainen <tss@...> wrote:
> >
> > > On Fri, 2009-11-06 at 10:27 +0200, Nikita Koshikov wrote:
> > > > I don't understand why gdb says "no debugging symbols found" for some
> > > > dovecot's executables, all binaries under /usr/libexec/dovecot have
> > > > "not stripped"
> > > ..
> > > > (gdb) bt full
> > > > #0  0xb7f88749 in ?? () from /lib/libc.so.6
> > > > No symbol table info available.
> > > > Cannot access memory at address 0xbf690ff8
> > >
> > > Yeah, something in your system has broken gdb.. Makes the debugging
> > > annoyingly difficult.
>
> You managed to get backtraces from the other bug. What did you do
> differently? Maybe you can get backtrace from this bug now too?

I think gdb can't access memory due to gentoo kernel patches, so attaching to process is not working. Also, I can't change kernel on production server for testing purposes.
As I wrote above, core files not present in users's homedir, even with patch you provided.

>
> > I have wrote simple wrapper that removes maildirsize  if the messages
> > were expunged from user's mailbox and if user has non-default quota
> > value. Is there any more convenient  way to force dovecot recalculate
> > quota size ? I have overview IMAP-QOUTA rfc, but seemed that it lacks
> > such imap command, maybe some internal dovecot command exist?
>
> No, there's no other way.

Clear, thanks.