3.3.0 plans

[Justin Mason]

ok, what bugs really need to be fixed for 3.3.0?  feel free to set
Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
only real blocker:

5752    2008-02-04  nor     P1      Let's get rid of the default rules dir
and make sa-update mandatory

All the rest is just icing (and the odd non-blocker bug, but who's
counting ;).

Also, I would also like to get rid of the "rulesrc" external, and instead
just put its contents into each branch, separately.  I don't think the
idea of sharing rules in this way between branches has worked out; in my
opinion it's caused more trouble than help (unanticipated dependencies,
complexity, SVN external = horrible anyway).  Is anyone still attached to
this idea?

--j.

[Quanah Gibson-Mount-3]

--On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
wrote:

> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> only real blocker:

<http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
please. :)


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

[Justin Mason]

Quanah Gibson-Mount writes:
> --On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
> wrote:
>
> > ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> > Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> > only real blocker:
>
> <http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
> please. :)

unfortunately, there's no patch there, and it appears that BerkeleyDB
doesn't support upgrading of .db files anyway.  unlikely to get in,
given that!

--j.

[Warren Togami]

Justin Mason wrote:

> Quanah Gibson-Mount writes:
>> --On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
>> wrote:
>>
>>> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
>>> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
>>> only real blocker:
>> <http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
>> please. :)
>
> unfortunately, there's no patch there, and it appears that BerkeleyDB
> doesn't support upgrading of .db files anyway.  unlikely to get in,
> given that!
>
> --j.

How are the 3.3.0 plans coming along?

Warren

[Justin Mason]

Warren Togami writes:

> Justin Mason wrote:
> > Quanah Gibson-Mount writes:
> >> --On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
> >> wrote:
> >>
> >>> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> >>> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> >>> only real blocker:
> >> <http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
> >> please. :)
> >
> > unfortunately, there's no patch there, and it appears that BerkeleyDB
> > doesn't support upgrading of .db files anyway.  unlikely to get in,
> > given that!
>
> How are the 3.3.0 plans coming along?

Hmm. things have been pretty quiet -- I think we're all a bit distracted
by other, non-SpamAssassin-related stuff, unfortunately.

I've been meaning to try to push it along, the big work items are:

  - fixing the distribution process to work without rules in the dist
    tarball (since we'll be moving to a model where with distribute
    without rules and they're downloaded by the admin on install).

  - generating automated daily scores with Daryl.  this is kind of complex
    since it relies on having good rule-QA data coming in with the
    "reuse=yes" flag set, and I'm not sure what our current status is
    there.

I don't see anything else remaining in the 3.3.0 queue that are necessary
as much as those two.

--j.

[Warren Togami]

Justin Mason wrote:

> Warren Togami writes:
>> Justin Mason wrote:
>>> Quanah Gibson-Mount writes:
>>>> --On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
>>>> wrote:
>>>>
>>>>> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
>>>>> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
>>>>> only real blocker:
>>>> <http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
>>>> please. :)
>>> unfortunately, there's no patch there, and it appears that BerkeleyDB
>>> doesn't support upgrading of .db files anyway.  unlikely to get in,
>>> given that!
>> How are the 3.3.0 plans coming along?
>
> Hmm. things have been pretty quiet -- I think we're all a bit distracted
> by other, non-SpamAssassin-related stuff, unfortunately.
>
> I've been meaning to try to push it along, the big work items are:
>
>   - fixing the distribution process to work without rules in the dist
>     tarball (since we'll be moving to a model where with distribute
>     without rules and they're downloaded by the admin on install).
>
>   - generating automated daily scores with Daryl.  this is kind of complex
>     since it relies on having good rule-QA data coming in with the
>     "reuse=yes" flag set, and I'm not sure what our current status is
>     there.
>

Should 5899 be fixed before 3.3.0?  It seems to be a potential cause of
upgrade confusion.

Warren

[Justin Mason]

Warren Togami writes:

> Justin Mason wrote:
> > Warren Togami writes:
> >> Justin Mason wrote:
> >>> Quanah Gibson-Mount writes:
> >>>> --On Friday, March 14, 2008 3:36 PM +0000 Justin Mason <jm@...>
> >>>> wrote:
> >>>>
> >>>>> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> >>>>> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> >>>>> only real blocker:
> >>>> <http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4416> please please
> >>>> please. :)
> >>> unfortunately, there's no patch there, and it appears that BerkeleyDB
> >>> doesn't support upgrading of .db files anyway.  unlikely to get in,
> >>> given that!
> >> How are the 3.3.0 plans coming along?
> >
> > Hmm. things have been pretty quiet -- I think we're all a bit distracted
> > by other, non-SpamAssassin-related stuff, unfortunately.
> >
> > I've been meaning to try to push it along, the big work items are:
> >
> >   - fixing the distribution process to work without rules in the dist
> >     tarball (since we'll be moving to a model where with distribute
> >     without rules and they're downloaded by the admin on install).
> >
> >   - generating automated daily scores with Daryl.  this is kind of complex
> >     since it relies on having good rule-QA data coming in with the
> >     "reuse=yes" flag set, and I'm not sure what our current status is
> >     there.
> >
>
> Should 5899 be fixed before 3.3.0?  It seems to be a potential cause of
> upgrade confusion.

yes.

[Matthias Leisi]

>   - fixing the distribution process to work without rules in the dist
>     tarball (since we'll be moving to a model where with distribute
>     without rules and they're downloaded by the admin on install).

The longer I think about it, the less convinced I am that it is a good idea.

In environments requiring strict change control or that are otherwise
paranoid in terms of security, (direct) online updates are usually not
desired (or not possible, because eg the firewall concept forbids
mailservers from doing HTTP requests).

Thus actually *requiring* online update would make deployment in such
environments more complex (it would eg require repackaging *with* the
rules prior to deployment).

The same environments would also be rather reluctant to allow direct
online updates without at least some form of sanity check (maybe not only
lint, but also some messages being checked).

Although I'd like to see some clearer structure in how rules and rules
updates (of different sources) are handled, I suggest that some basic
ruleset ("rules du jour at the day of packing a release" or something
similar) be included with the tarball.

-- Matthias

[Justin Mason]

Matthias Leisi writes:

> >   - fixing the distribution process to work without rules in the dist
> >     tarball (since we'll be moving to a model where with distribute
> >     without rules and they're downloaded by the admin on install).
>
> The longer I think about it, the less convinced I am that it is a good idea.
>
> In environments requiring strict change control or that are otherwise
> paranoid in terms of security, (direct) online updates are usually not
> desired (or not possible, because eg the firewall concept forbids
> mailservers from doing HTTP requests).
>
> Thus actually *requiring* online update would make deployment in such
> environments more complex (it would eg require repackaging *with* the
> rules prior to deployment).
>
> The same environments would also be rather reluctant to allow direct
> online updates without at least some form of sanity check (maybe not only
> lint, but also some messages being checked).
>
> Although I'd like to see some clearer structure in how rules and rules
> updates (of different sources) are handled, I suggest that some basic
> ruleset ("rules du jour at the day of packing a release" or something
> similar) be included with the tarball.

hi Matthias --

if you check the bug (can't recall the number right now), there's plans to
make a tarball available of _just_ the rules, alongside the distro at
release time.  That way, that kind of users can dl the tarball and
install it using "sa-update --install SpamAssassin-rules-349584.tgz" (or
whatever).

Does that work?

--j.

[Matthias Leisi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Justin Mason schrieb:

| if you check the bug (can't recall the number right now), there's plans to
| make a tarball available of _just_ the rules, alongside the distro at
| release time.  That way, that kind of users can dl the tarball and
| install it using "sa-update --install SpamAssassin-rules-349584.tgz" (or
| whatever).

How would a typical distro handle this (or how are they thought to
handle it)? Pack the rules tarball at date of packaging? That should be
acceptable for most, as this would be basically the same as today (two
tarballs instead of one, but from the same source).

A rules package from the same source as the app package is definitely
acceptable. Would / should updated rules packages be made available
through the same channels as the initial one? (IMO, this *should* be
done - through distros, CPAN, ...).

- -- Matthias

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iD8DBQFIMc0nxbHw2nyi/okRAgIBAJ92gIAuBvsV51KucUMIrvgmpWEQPACeLHIQ
YlWoXASfBRA9/FhRM5cdOgk=
=ZjPM
-----END PGP SIGNATURE-----

[Justin Mason]

Matthias Leisi writes:

> Justin Mason schrieb:
>
> | if you check the bug (can't recall the number right now), there's plans to
> | make a tarball available of _just_ the rules, alongside the distro at
> | release time.  That way, that kind of users can dl the tarball and
> | install it using "sa-update --install SpamAssassin-rules-349584.tgz" (or
> | whatever).
>
> How would a typical distro handle this (or how are they thought to
> handle it)? Pack the rules tarball at date of packaging? That should be
> acceptable for most, as this would be basically the same as today (two
> tarballs instead of one, but from the same source).

Yes, that's what I was thinking.

> A rules package from the same source as the app package is definitely
> acceptable. Would / should updated rules packages be made available
> through the same channels as the initial one? (IMO, this *should* be
> done - through distros, CPAN, ...).

I'm fine with doing that ;)

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5752 is the bug.
could you add that recommendation there?

--j.

[Warren Togami]

Justin Mason wrote:
> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> only real blocker:
>
> 5752    2008-02-04  nor     P1      Let's get rid of the default rules dir
> and make sa-update mandatory
>
> All the rest is just icing (and the odd non-blocker bug, but who's
> counting ;).

It has been a while since this last update.  What is the status of 3.3.0
now?

Warren Togami
wtogami@...

[Justin Mason]

Warren Togami writes:

> Justin Mason wrote:
> > ok, what bugs really need to be fixed for 3.3.0?  feel free to set
> > Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
> > only real blocker:
> >
> > 5752    2008-02-04  nor     P1      Let's get rid of the default rules dir
> > and make sa-update mandatory
> >
> > All the rest is just icing (and the odd non-blocker bug, but who's
> > counting ;).
>
> It has been a while since this last update.  What is the status of 3.3.0
> now?

hi Warren --

There's been no real motion -- we've been infrequently bashing the odd bug
in the 3.3.0 list, but we have no concrete release schedule yet.  Sorry
about that...

--j.

[Warren Togami]

On 08/28/2008 06:47 AM, Justin Mason wrote:

> Warren Togami writes:
>> Justin Mason wrote:
>>> ok, what bugs really need to be fixed for 3.3.0?  feel free to set
>>> Priority on bugs on the 3.3.0 milestone.  in my opinion this is the
>>> only real blocker:
>>>
>>> 5752    2008-02-04  nor     P1      Let's get rid of the default rules dir
>>> and make sa-update mandatory
>>>
>>> All the rest is just icing (and the odd non-blocker bug, but who's
>>> counting ;).
>> It has been a while since this last update.  What is the status of 3.3.0
>> now?
>
> hi Warren --
>
> There's been no real motion -- we've been infrequently bashing the odd bug
> in the 3.3.0 list, but we have no concrete release schedule yet.  Sorry
> about that...
>
> --j.

How are things going now?

Warren Togami
wtogami@...

[Matt Kettler-3]

Warren Togami wrote:

> On 08/28/2008 06:47 AM, Justin Mason wrote:
>>
>> hi Warren --
>>
>> There's been no real motion -- we've been infrequently bashing the
>> odd bug
>> in the 3.3.0 list, but we have no concrete release schedule yet.  Sorry
>> about that...
>>
>> --j.
>
> How are things going now?
>
> Warren Togami
> wtogami@...
>

Well, a first hit would be looking at bugzilla


https://issues.apache.org/SpamAssassin/buglist.cgi?bug_file_loc=&bug_file_loc_type=allwordssubstr&bug_id=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bugidtype=include&chfieldfrom=&chfieldto=Now&chfieldvalue=&email1=&email2=&emailtype1=substring&emailtype2=substring&field-1-0-0=target_milestone&field-1-1-0=bug_status&field0-0-0=noop&keywords=&keywords_type=allwords&long_desc=&long_desc_type=allwordssubstr&query_format=advanced&remaction=&short_desc=&short_desc_type=allwordssubstr&status_whiteboard=&status_whiteboard_type=allwordssubstr&target_milestone=3.3.0&type-1-0-0=anyexact&type-1-1-0=anyexact&type0-0-0=noop&value-1-0-0=3.3.0&value-1-1-0=NEW%2CASSIGNED%2CREOPENED&value0-0-0=&order=bugs.priority&query_based_on=

In general we've been a little light on dev effort lately.. perhaps we
need to start rounding up for a 3.3.0 release.

[Justin Mason]

On Tue, Apr 14, 2009 at 01:48, Matt Kettler <mkettler_sa@...> wrote:

> Warren Togami wrote:
>> On 08/28/2008 06:47 AM, Justin Mason wrote:
>>>
>>> hi Warren --
>>>
>>> There's been no real motion -- we've been infrequently bashing the
>>> odd bug
>>> in the 3.3.0 list, but we have no concrete release schedule yet.  Sorry
>>> about that...
>>>
>>> --j.
>>
>> How are things going now?
>>
>> Warren Togami
>> wtogami@...
>>
> Well, a first hit would be looking at bugzilla
>
>
> https://issues.apache.org/SpamAssassin/buglist.cgi?bug_file_loc=&bug_file_loc_type=allwordssubstr&bug_id=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bugidtype=include&chfieldfrom=&chfieldto=Now&chfieldvalue=&email1=&email2=&emailtype1=substring&emailtype2=substring&field-1-0-0=target_milestone&field-1-1-0=bug_status&field0-0-0=noop&keywords=&keywords_type=allwords&long_desc=&long_desc_type=allwordssubstr&query_format=advanced&remaction=&short_desc=&short_desc_type=allwordssubstr&status_whiteboard=&status_whiteboard_type=allwordssubstr&target_milestone=3.3.0&type-1-0-0=anyexact&type-1-1-0=anyexact&type0-0-0=noop&value-1-0-0=3.3.0&value-1-1-0=NEW%2CASSIGNED%2CREOPENED&value0-0-0=&order=bugs.priority&query_based_on=
>
> In general we've been a little light on dev effort lately.. perhaps we
> need to start rounding up for a 3.3.0 release.

yeah, I think we should.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5752
'Let's get rid of the default rules dir and make sa-update mandatory'

is the only bug that IMO _needs_ to be done.  it'd be nice to get the
BerkeleyDB plugin in, too, though, and I'm sure there are a few
others.

we really need to get some tuits together ;)

--j.

[Mark Martinec]

On Tuesday 14 April 2009 10:50:23 Justin Mason wrote:

> > In general we've been a little light on dev effort lately.. perhaps we
> > need to start rounding up for a 3.3.0 release.
>
> yeah, I think we should.
>
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5752
> 'Let's get rid of the default rules dir and make sa-update mandatory'
>
> is the only bug that IMO _needs_ to be done.  it'd be nice to get the
> BerkeleyDB plugin in, too, though, and I'm sure there are a few
> others.
>
> we really need to get some tuits together ;)

I agree it's about time to get 3.3.0 wrapped up. There is some useful new
code there along with a couple of bug fixes, just sitting there. People are
reluctant to use a non-released version, even though I'd say it is just as
stable if not more than 3.2.5.

As new problems are being reported faster than the old ones are being
closed, it would be an illusion to wait for a majority of open tickets
to be closed before a release.

I would too like to see the BerkeleyDB bayes backend to be sorted out
before the release. I made a couple of minor attempts to deal with the
more obvious issues, but it would be nice to get  some help from Michael,
its author (or some other interested party), and some experience from
a production use. Bug 6046.

With DKIM ADSP rules I'm a bit stuck because 3.2.5 and 3.3 use the same
rules set (I'd like to add few adsp_override rules, drop now defunct
DKIM_POLICY_SIGNALL, DKIM_POLICY_SIGNSOME, DKIM_POLICY_TESTING;
and DKIM_VERIFIED now replaced by DKIM_VALID and DKIM_VALID_AU).
In principle the 'if version' could be used, but it gets unsightly.
If I remember right, we already heard suggestion (by JM?) to split again
the rules by version. Or maybe there could be a common subset, along
with a per-branch set.

I would like to get a dkim tests operational again. A few testing public keys
need to be published in a spamassassin.org zone (I know, Justin can push
them to DNS, I just need to provide them). This is the main item I need
to work on for 3.3.

Perhaps 3.3 is a good opportunity to make some Perl modules required
or phased out. One example that comes to mind is making Time::HiRes
a requirement - several SA modules are now jumping hoops and perform
suboptimally when they need to deal with integer seconds.

Another is perhaps to replace a Digest::SHA1 (which can only do a sha1)
with a Digest::SHA (which can do a sha1 as well as sha256).
The sha256 is required by a DKIM plugin, and Digest::SHA is upwards
compatible with Digest::SHA1. I can do some benchmarking if there
is a speed concern (sha1 is also used by Bayes).

Perhaps now the DomainKeys plugin could be removed, as its underlying
module is no longer supported, and its functionality is covered by a
DKIM plugin.

  Mark

[Justin Mason]

On Tue, Apr 14, 2009 at 12:21, Mark Martinec <Mark.Martinec+sa@...> wrote:

> On Tuesday 14 April 2009 10:50:23 Justin Mason wrote:
>> > In general we've been a little light on dev effort lately.. perhaps we
>> > need to start rounding up for a 3.3.0 release.
>>
>> yeah, I think we should.
>>
>> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5752
>> 'Let's get rid of the default rules dir and make sa-update mandatory'
>>
>> is the only bug that IMO _needs_ to be done.  it'd be nice to get the
>> BerkeleyDB plugin in, too, though, and I'm sure there are a few
>> others.
>>
>> we really need to get some tuits together ;)
>
> I agree it's about time to get 3.3.0 wrapped up. There is some useful new
> code there along with a couple of bug fixes, just sitting there. People are
> reluctant to use a non-released version, even though I'd say it is just as
> stable if not more than 3.2.5.
>
> As new problems are being reported faster than the old ones are being
> closed, it would be an illusion to wait for a majority of open tickets
> to be closed before a release.

Yes, agreed.  BTW, we generally use the Pri field in bugzilla to
determine importance of tickets to handle this issue. Here's a quick
guide to how to triage the 3.3.0 queue in our BZ:

- if a ticket has no patch, is a feature request or similar "I want a
pony" wishful thinking ;), and you think it's almost certainly not
going to be fixed for the release, move it to the "Future" milestone

- if a ticket *needs* to be fixed before release, set severity to "Blocker"

- if it should be fixed before release if possible, move Pri up to "1"

- if a ticket has a patch, or seems fixable for the release, set Pri
to somewhere between 2 and 4 based on your idea of its relative
importance

- if the ticket is a feature req with no suitable patch, set Pri to
"5", where it's probably not going to be fixed but is still "on the
radar".

You can then see the "queue" by searching against the 3.3.0 target milestone
and sorting by Pri.

I expect we'll release 3.3.0 with a lot of tickets in the Pri "3" to
"5" range. ;)


> I would too like to see the BerkeleyDB bayes backend to be sorted out
> before the release. I made a couple of minor attempts to deal with the
> more obvious issues, but it would be nice to get  some help from Michael,
> its author (or some other interested party), and some experience from
> a production use. Bug 6046.

is already in that list with Pri=3, which is about right.  I hope we
can get this working...

> With DKIM ADSP rules I'm a bit stuck because 3.2.5 and 3.3 use the same
> rules set (I'd like to add few adsp_override rules, drop now defunct
> DKIM_POLICY_SIGNALL, DKIM_POLICY_SIGNSOME, DKIM_POLICY_TESTING;
> and DKIM_VERIFIED now replaced by DKIM_VALID and DKIM_VALID_AU).
> In principle the 'if version' could be used, but it gets unsightly.
> If I remember right, we already heard suggestion (by JM?) to split again
> the rules by version. Or maybe there could be a common subset, along
> with a per-branch set.

I think we should split it again.  the attempts to use a common subset of rules
has caused a *lot* of integration/dependency problems IMO.  Here's what
I posted over a year ago(!):

'I would also like to get rid of the "rulesrc" external, and instead
just put its contents into each branch, separately.  I don't think the
idea of sharing rules in this way between branches has worked out; in my
opinion it's caused more trouble than help (unanticipated dependencies,
complexity, SVN external = horrible anyway).  Is anyone still attached to
this idea?'

there were no replies, so that sounds good ;)

> I would like to get a dkim tests operational again. A few testing public keys
> need to be published in a spamassassin.org zone (I know, Justin can push
> them to DNS, I just need to provide them). This is the main item I need
> to work on for 3.3.

is there a bug for that?  there probably should be...

> Perhaps 3.3 is a good opportunity to make some Perl modules required
> or phased out. One example that comes to mind is making Time::HiRes
> a requirement - several SA modules are now jumping hoops and perform
> suboptimally when they need to deal with integer seconds.

+1, that seems ok to me.  a lot of distros bundle Time::HiRes now.

> Another is perhaps to replace a Digest::SHA1 (which can only do a sha1)
> with a Digest::SHA (which can do a sha1 as well as sha256).
> The sha256 is required by a DKIM plugin, and Digest::SHA is upwards
> compatible with Digest::SHA1. I can do some benchmarking if there
> is a speed concern (sha1 is also used by Bayes).

I'd be more concerned about availability, esp whether or not it's
available in common distro packages (Ubuntu, Debian and Red Hat
particularly).

> Perhaps now the DomainKeys plugin could be removed, as its underlying
> module is no longer supported, and its functionality is covered by a
> DKIM plugin.

+1

--j.

[Michael Alan Dorman-3]

> I would too like to see the BerkeleyDB bayes backend to be sorted out
> before the release. I made a couple of minor attempts to deal with the
> more obvious issues, but it would be nice to get  some help from
> Michael, its author (or some other interested party), and some
> experience from a production use. Bug 6046.

Sorry, I've been swamped by work for the last couple of months.

I haven't had time to pull in your changes from a month or two ago,
which I want to look at, but I have done some work independently.

The solution to the consistency problems you were running into is, as
far as I can discern, to use transactions---which, to my eye, would
be perfectly fine: it slows things down a little bit, but it's still
faster than the full-on SQL back-ends.

Unfortunately, when I do that, I then have problems with BDB
complaining about lack of available transaction slots...even when
there's only ever been one or two transactions in play at one time.

(Oh, and you can only adjust the number of transaction slots using a
DB_CONFIG file, which is a whole other annoyance---that's just part of
the BDB API that isn't exposed in the perl module.)

Perhaps the combination of your changes and mine will make some forward
progress.  I'm going to be very busy for the next couple of weeks, but
maybe I can carve out some time to reconcile our changes and start up
some tests.

Mike.

[Quanah Gibson-Mount-3]

--On Tuesday, April 14, 2009 9:29 AM -0400 Michael Alan Dorman
<mdorman@...> wrote:

> Sorry, I've been swamped by work for the last couple of months.
>
> I haven't had time to pull in your changes from a month or two ago,
> which I want to look at, but I have done some work independently.
>
> The solution to the consistency problems you were running into is, as
> far as I can discern, to use transactions---which, to my eye, would
> be perfectly fine: it slows things down a little bit, but it's still
> faster than the full-on SQL back-ends.
>
> Unfortunately, when I do that, I then have problems with BDB
> complaining about lack of available transaction slots...even when
> there's only ever been one or two transactions in play at one time.
>
> (Oh, and you can only adjust the number of transaction slots using a
> DB_CONFIG file, which is a whole other annoyance---that's just part of
> the BDB API that isn't exposed in the perl module.)
>
> Perhaps the combination of your changes and mine will make some forward
> progress.  I'm going to be very busy for the next couple of weeks, but
> maybe I can carve out some time to reconcile our changes and start up
> some tests.

I don't know if it would be helpful or not, but you may wish to look at the
OpenLDAP back-bdb and back-hdb backends, which use BDB extensively, and are
highly performant.

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration