3.3.0 plans

[Michael Alan Dorman-3]

> I don't know if it would be helpful or not, but you may wish to look
> at the OpenLDAP back-bdb and back-hdb backends, which use BDB
> extensively, and are highly performant.

It's a good recommendation.

I also read a post just this morning from a linux kernel hacker who's
working with BDB, who fingered RMW as causing problems for him, so I
may check that as well.

Mike.

[Mark Martinec]

On Wednesday 15 April 2009 03:53:17 Michael Alan Dorman wrote:
> I also read a post just this morning from a linux kernel hacker who's
> working with BDB, who fingered RMW as causing problems for him, so I
> may check that as well.

Btw, I'm using bdb with DB_INIT_CDB | DB_INIT_MPOOL
(cursor locking with Concurrent Data Store) in amavisd,
and it is very reliable. I also keep signals disabled around
each short code section operating under a lock.
No secondary keys in use though.

  Mark

[Mark Martinec]

> > On Tuesday 14 April 2009 10:50:23 Justin Mason wrote:
> >> > In general we've been a little light on dev effort lately.. perhaps we
> >> > need to start rounding up for a 3.3.0 release.
> >>
> >> yeah, I think we should.

> > I agree it's about time to get 3.3.0 wrapped up. There is some useful new
> > code there along with a couple of bug fixes, just sitting there. People
> > are reluctant to use a non-released version, even though I'd say it is
> > just as stable if not more than 3.2.5.
> >
> > As new problems are being reported faster than the old ones are being
> > closed, it would be an illusion to wait for a majority of open tickets
> > to be closed before a release.

> I think we should split it again.  the attempts to use a common subset of
> rules has caused a *lot* of integration/dependency problems IMO.  Here's
> what I posted over a year ago(!):
>
> 'I would also like to get rid of the "rulesrc" external, and instead
> just put its contents into each branch, separately.  I don't think the
> idea of sharing rules in this way between branches has worked out; in my
> opinion it's caused more trouble than help (unanticipated dependencies,
> complexity, SVN external = horrible anyway).  Is anyone still attached to
> this idea?'
>
> there were no replies, so that sounds good ;)

Just needs to be done :)

> > I would like to get a dkim tests operational again. A few testing public
> > keys need to be published in a spamassassin.org zone (I know, Justin can
> > push them to DNS, I just need to provide them). This is the main item I
> > need to work on for 3.3.
>
> is there a bug for that?  there probably should be...

Bug 6100, now open.
DKIM keys are now ready, test mail samples need to be prepared.

> > Perhaps 3.3 is a good opportunity to make some Perl modules required
> > or phased out. One example that comes to mind is making Time::HiRes
> > a requirement - several SA modules are now jumping hoops and perform
> > suboptimally when they need to deal with integer seconds.
>
> +1, that seems ok to me.  a lot of distros bundle Time::HiRes now.

Bug 6095, now resolved.

> > Perhaps now the DomainKeys plugin could be removed, as its underlying
> > module is no longer supported, and its functionality is covered by a
> > DKIM plugin.
>
> +1

Bug 6098, now resolved.


  Mark

[Warren Togami]

On 04/14/2009 07:21 AM, Mark Martinec wrote:
>
> I agree it's about time to get 3.3.0 wrapped up. There is some useful new
> code there along with a couple of bug fixes, just sitting there. People are
> reluctant to use a non-released version, even though I'd say it is just as
> stable if not more than 3.2.5.
>

Might it be feasible to get 3.3.0 out sometime during the Fedora 12 dev
cycle?  It seems there is maybe 2-3 months.  A fresh spamassassin making
this release might possibly be particularly more important than other
releases for unnamed reasons...

March 2008 Justin Mason mentions only two major bugs necessary to be
fixed before 3.3.0.  Around that time 3.3.0 was considered to be
superior to 3.2.x.  Is it still considered better than 3.2.x?

Might it be a good idea to set a target date?  Time-based releases tend
to work well for projects.  Those stated deadlines tend to be stretched
a bit in reality, but at least goals were set and people work toward
those goals.

Warren Togami
wtogami@...

[Justin Mason]

On Thu, May 28, 2009 at 05:10, Warren Togami <wtogami@...> wrote:

> On 04/14/2009 07:21 AM, Mark Martinec wrote:
>>
>> I agree it's about time to get 3.3.0 wrapped up. There is some useful new
>> code there along with a couple of bug fixes, just sitting there. People
>> are
>> reluctant to use a non-released version, even though I'd say it is just as
>> stable if not more than 3.2.5.
>>
>
> Might it be feasible to get 3.3.0 out sometime during the Fedora 12 dev
> cycle?  It seems there is maybe 2-3 months.  A fresh spamassassin making
> this release might possibly be particularly more important than other
> releases for unnamed reasons...
>
> March 2008 Justin Mason mentions only two major bugs necessary to be fixed
> before 3.3.0.  Around that time 3.3.0 was considered to be superior to
> 3.2.x.  Is it still considered better than 3.2.x?
>
> Might it be a good idea to set a target date?  Time-based releases tend to
> work well for projects.  Those stated deadlines tend to be stretched a bit
> in reality, but at least goals were set and people work toward those goals.

It would be quite possible to do this, given a sufficiently-dedicated
release guy pushing
it.  It takes about 2 months to get through mass-checks, generate good scores,
etc.  Unfortunately, I'm out for this job, as child #2 is about to be born any
day now ;)

BTW in the short term, if ppl want to help 3.3.0 release plans, I suggest trying
out the SVN trunk code as "dogfood".  I've been doing this for the last couple
of years with very good results; I can't recall the last time I had to
restart my
spamd, for example, or spotted an FP...

--j.

[Warren Togami]

On 05/28/2009 03:48 AM, Justin Mason wrote:

> It would be quite possible to do this, given a sufficiently-dedicated
> release guy pushing
> it.  It takes about 2 months to get through mass-checks, generate good scores,
> etc.  Unfortunately, I'm out for this job, as child #2 is about to be born any
> day now ;)
>
> BTW in the short term, if ppl want to help 3.3.0 release plans, I suggest trying
> out the SVN trunk code as "dogfood".  I've been doing this for the last couple
> of years with very good results; I can't recall the last time I had to
> restart my
> spamd, for example, or spotted an FP...
>
> --j.

Is the release process and tools used documented?  Anyone else from the
core team might be able to do it?

Warren Togami
wtogami@...

[Justin Mason]

On Thu, May 28, 2009 at 19:35, Warren Togami <wtogami@...> wrote:

> On 05/28/2009 03:48 AM, Justin Mason wrote:
>>
>> It would be quite possible to do this, given a sufficiently-dedicated
>> release guy pushing
>> it.  It takes about 2 months to get through mass-checks, generate good
>> scores,
>> etc.  Unfortunately, I'm out for this job, as child #2 is about to be born
>> any
>> day now ;)
>>
>> BTW in the short term, if ppl want to help 3.3.0 release plans, I suggest
>> trying
>> out the SVN trunk code as "dogfood".  I've been doing this for the last
>> couple
>> of years with very good results; I can't recall the last time I had to
>> restart my
>> spamd, for example, or spotted an FP...
>>
>> --j.
>
> Is the release process and tools used documented?  Anyone else from the core
> team might be able to do it?

AFAIK, yep...

--j.

[Mark Martinec]

Sorry for my late response, and bringing the topic back to the list
(with some quotes omitted).

Warren Togami writes:
> Might it be feasible to get 3.3.0 out sometime during the Fedora 12 dev
> cycle?  It seems there is maybe 2-3 months.  A fresh spamassassin making
> this release might possibly be particularly more important than other
> releases for unnamed reasons...

Justin writes:
> It would be quite possible to do this, given a sufficiently-dedicated
> release guy pushing it.  It takes about 2 months to get through
> mass-checks, generate good scores, etc. Unfortunately, I'm out for this
> job, as child #2 is about to be born any day now ;)
>
> BTW in the short term, if ppl want to help 3.3.0 release plans, I suggest
> trying out the SVN trunk code as "dogfood".  I've been doing this for the
> last couple of years with very good results; I can't recall the last time
> I had to restart my spamd, for example, or spotted an FP...

I've also been running a fresh 3.3 in production at our site for more
than a year now, mostly because it provides bug fixes and some new
features or workarounds that I need. For example with a Perl upgrade
to 5.8.9 (and later to 5.10.0) on FreeBSD ports the old 3.2.5 just
couldn't survive any longer without crashing due to exceeded stack
size while compiling our set of rules (a perl bug).

I can confirm that 3.3 fares at least as well as 3.2.5, and in
many ways better. I do monitor for possible glitches, but there
just aren't any (new ones that is, that would not also be in 3.2.5).
I may also add that the more recent versions of amavisd-new are
happier with 3.3 than with 3.2.5, as this enables some nice SA
functionality (like SA timing breakdown reports, DKIM checking
of long mail messages).

The trouble with only a handful of sites running the new code is
that not all features are exercised, and not all setups tested.

Then there is a concern that some of the fixes for old bug reports
would still be nice to get into a 3.3 before a release, and some
nice features added/finished. Given the current progress trends
this is unfortunately just stretching time to a release, rising
a feeling that "it just isn't ripe yet". We might consider just
setting our goal/expectations closer to ground, and come out with
what we have.

> I'm pretty sure this won't be achievable in that timeframe if I'm
> driving the release process [...] while I'm busy running after babies ;)
> guys, I can totally provide advice/guidance as to how to go about
> this, if you feel up to it.
>
> basically this wiki page describes the basic idea of how to put
> together a new x.y.0 release:
> http://wiki.apache.org/spamassassin/ReleasePolicy
>
> the biggest task is working through this:
> http://wiki.apache.org/spamassassin/RescoreMassCheck

On my part I've spent less time lately on SpamAssassin then
I would like. Getting the amavis release out, preparing for
a conference, and then there will be a vacation time for me,
so I won't be of much help until the end of July.

I wonder if there's anything we could do without too much
trouble to let more people start using the 3.3 code from CVS.
I suppose there are still nightly tarball builds? (I haven't
checked). Perhaps just giving an encouraging hint every now
and then on a mailing list could be a good start.

  Mark

[Quanah Gibson-Mount-3]

--On Wednesday, June 24, 2009 5:28 PM +0200 Mark Martinec
<Mark.Martinec+sa@...> wrote:

> On my part I've spent less time lately on SpamAssassin then
> I would like. Getting the amavis release out, preparing for
> a conference, and then there will be a vacation time for me,
> so I won't be of much help until the end of July.
>
> I wonder if there's anything we could do without too much
> trouble to let more people start using the 3.3 code from CVS.
> I suppose there are still nightly tarball builds? (I haven't
> checked). Perhaps just giving an encouraging hint every now
> and then on a mailing list could be a good start.

Maybe a public alpha or beta release?  So folks know it isn't official, but
can start testing it out.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

[Theo Van Dinter]

fwiw, the process used to be:
- beta releases to get things stabilized
- use a beta release to do mass-check runs
- generate scores with mass-check data and submit to svn
- rc releases to get wider testing w/ scores
- release after rc appear to work



On Wed, Jun 24, 2009 at 2:11 PM, Quanah Gibson-Mount<quanah@...> wrote:

> --On Wednesday, June 24, 2009 5:28 PM +0200 Mark Martinec
> <Mark.Martinec+sa@...> wrote:
>
>
>
>> On my part I've spent less time lately on SpamAssassin then
>> I would like. Getting the amavis release out, preparing for
>> a conference, and then there will be a vacation time for me,
>> so I won't be of much help until the end of July.
>>
>> I wonder if there's anything we could do without too much
>> trouble to let more people start using the 3.3 code from CVS.
>> I suppose there are still nightly tarball builds? (I haven't
>> checked). Perhaps just giving an encouraging hint every now
>> and then on a mailing list could be a good start.
>
> Maybe a public alpha or beta release?  So folks know it isn't official, but
> can start testing it out.
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration
>

[Justin Mason]

On Wed, Jun 24, 2009 at 19:11, Quanah Gibson-Mount<quanah@...> wrote:

> --On Wednesday, June 24, 2009 5:28 PM +0200 Mark Martinec
> <Mark.Martinec+sa@...> wrote:
>
>> On my part I've spent less time lately on SpamAssassin then
>> I would like. Getting the amavis release out, preparing for
>> a conference, and then there will be a vacation time for me,
>> so I won't be of much help until the end of July.
>>
>> I wonder if there's anything we could do without too much
>> trouble to let more people start using the 3.3 code from CVS.
>> I suppose there are still nightly tarball builds? (I haven't
>> checked). Perhaps just giving an encouraging hint every now
>> and then on a mailing list could be a good start.
>
> Maybe a public alpha or beta release?  So folks know it isn't official, but
> can start testing it out.

A public alpha might be a good option.  The main stuff we need to do
after that would be procedural changes and score generation.

--j.

> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration
>
>

[Justin Mason]

On Wed, Jun 24, 2009 at 21:21, Theo Van Dinter<felicity@...> wrote:
> fwiw, the process used to be:
> - beta releases to get things stabilized
> - use a beta release to do mass-check runs
> - generate scores with mass-check data and submit to svn

I think we may be able to simplify that, now that Daryl's system is
generating scores weekly...

Also one hard part that we need to do is finish the "no rules in main
tarball" work item.

> - rc releases to get wider testing w/ scores
> - release after rc appear to work
>
>
>
> On Wed, Jun 24, 2009 at 2:11 PM, Quanah Gibson-Mount<quanah@...> wrote:
>> --On Wednesday, June 24, 2009 5:28 PM +0200 Mark Martinec
>> <Mark.Martinec+sa@...> wrote:
>>
>>
>>
>>> On my part I've spent less time lately on SpamAssassin then
>>> I would like. Getting the amavis release out, preparing for
>>> a conference, and then there will be a vacation time for me,
>>> so I won't be of much help until the end of July.
>>>
>>> I wonder if there's anything we could do without too much
>>> trouble to let more people start using the 3.3 code from CVS.
>>> I suppose there are still nightly tarball builds? (I haven't
>>> checked). Perhaps just giving an encouraging hint every now
>>> and then on a mailing list could be a good start.
>>
>> Maybe a public alpha or beta release?  So folks know it isn't official, but
>> can start testing it out.
>>
>> --Quanah
>>
>> --
>>
>> Quanah Gibson-Mount
>> Principal Software Engineer
>> Zimbra, Inc
>> --------------------
>> Zimbra ::  the leader in open source messaging and collaboration
>>
>
>

[Justin Mason]

On Wed, Jun 24, 2009 at 21:37, Justin Mason<jm@...> wrote:

> On Wed, Jun 24, 2009 at 21:21, Theo Van Dinter<felicity@...> wrote:
>> fwiw, the process used to be:
>> - beta releases to get things stabilized
>> - use a beta release to do mass-check runs
>> - generate scores with mass-check data and submit to svn
>
> I think we may be able to simplify that, now that Daryl's system is
> generating scores weekly...
>
> Also one hard part that we need to do is finish the "no rules in main
> tarball" work item.

which I've just done ;)

is there anything else that we should sort out before an alpha is viable?

--j.

>> - rc releases to get wider testing w/ scores
>> - release after rc appear to work
>>
>>
>>
>> On Wed, Jun 24, 2009 at 2:11 PM, Quanah Gibson-Mount<quanah@...> wrote:
>>> --On Wednesday, June 24, 2009 5:28 PM +0200 Mark Martinec
>>> <Mark.Martinec+sa@...> wrote:
>>>
>>>
>>>
>>>> On my part I've spent less time lately on SpamAssassin then
>>>> I would like. Getting the amavis release out, preparing for
>>>> a conference, and then there will be a vacation time for me,
>>>> so I won't be of much help until the end of July.
>>>>
>>>> I wonder if there's anything we could do without too much
>>>> trouble to let more people start using the 3.3 code from CVS.
>>>> I suppose there are still nightly tarball builds? (I haven't
>>>> checked). Perhaps just giving an encouraging hint every now
>>>> and then on a mailing list could be a good start.
>>>
>>> Maybe a public alpha or beta release?  So folks know it isn't official, but
>>> can start testing it out.
>>>
>>> --Quanah
>>>
>>> --
>>>
>>> Quanah Gibson-Mount
>>> Principal Software Engineer
>>> Zimbra, Inc
>>> --------------------
>>> Zimbra ::  the leader in open source messaging and collaboration
>>>
>>
>>
>

[Karsten Bräckelmann-2]

On Wed, 2009-06-24 at 23:00 +0100, Justin Mason wrote:
> is there anything else that we should sort out before an alpha is viable?

I believe re-thinking the minimum supported Perl version and related
stuff like screwing MakeMaker would be a *really* good target for a new
$minor release.

How much longer do we want to support Perl 5.6.x? There are comments in
bugzilla from years ago, that there's virtually no system without Perl
5.8.x for years (counting from the comment, not today).

I'd seriously prefer a "drop support for < Perl 5.8.1" in a separate
thread, though.


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

[Justin Mason]

2009/6/25 Karsten Bräckelmann <guenther@...>:

> On Wed, 2009-06-24 at 23:00 +0100, Justin Mason wrote:
>> is there anything else that we should sort out before an alpha is viable?
>
> I believe re-thinking the minimum supported Perl version and related
> stuff like screwing MakeMaker would be a *really* good target for a new
> $minor release.
>
> How much longer do we want to support Perl 5.6.x? There are comments in
> bugzilla from years ago, that there's virtually no system without Perl
> 5.8.x for years (counting from the comment, not today).
>
> I'd seriously prefer a "drop support for < Perl 5.8.1" in a separate
> thread, though.

I've done that -- and on the users@ list, since a lot of potential
commenters don't read dev@.  (I've also cc'd my blog to catch the
Planet Perl readers.)

my guess is it'll be doable.

--j.

[Justin Mason]

ok!  Here's what I see in bugzilla right now:

 ID   Sev   Pri   OS   Assignee   Status   Resolution   Summary
6077 maj P1 All dev@... NEW fix failing tests
6131 maj P1 All dev@... NEW Makefile.PL is
doing evil things to detect DESTDIR
6139 maj P1 All dev@... NEW remove
"rulesrc" external from SVN tree
5553 nor P1 All dev@... ASSI
                MIME_BASE64_TEXT does not handle charset properly
5871 enh P1 All dev@... REOP need docs for
new '--filter-retries' spamc switch

[so those are the 5 P1s.  these are all things that should be done
before a release, but none of them are blockers IMO.]

4949 nor P2 Free dev@... NEW spamd shutting
down when unable to fork new processes
5591 nor P2 Sola dev@... NEW make test
fails at spamd_hup test on Solaris 9, 10
6132 enh P2 All dev@... REOP FreeMail plugin

[would be nice to get done but again not blockers.]

everything else on the 3.3.0 milestone can be pushed off if nobody
gets the tuits to do them.
I think we could conceivably start the 3.3.0 release process soon....
paging Warren ;)

How's about I cut an alpha at the end of this week?

--j.


On Wed, Jun 24, 2009 at 23:00, Justin Mason<jm@...> wrote:

> On Wed, Jun 24, 2009 at 21:37, Justin Mason<jm@...> wrote:
>> On Wed, Jun 24, 2009 at 21:21, Theo Van Dinter<felicity@...> wrote:
>>> fwiw, the process used to be:
>>> - beta releases to get things stabilized
>>> - use a beta release to do mass-check runs
>>> - generate scores with mass-check data and submit to svn
>>
>> I think we may be able to simplify that, now that Daryl's system is
>> generating scores weekly...
>>
>> Also one hard part that we need to do is finish the "no rules in main
>> tarball" work item.
>
> which I've just done ;)
>
> is there anything else that we should sort out before an alpha is viable?
>
> --j.
>
>>> - rc releases to get wider testing w/ scores
>>> - release after rc appear to work

[Warren Togami]

On 06/29/2009 07:44 AM, Justin Mason wrote:
> How's about I cut an alpha at the end of this week?
>

Why end of the week if nothing on the list is blockers?

Warren

[Justin Mason]

On Mon, Jun 29, 2009 at 16:27, Warren Togami<wtogami@...> wrote:
> On 06/29/2009 07:44 AM, Justin Mason wrote:
>>
>> How's about I cut an alpha at the end of this week?
>>
>
> Why end of the week if nothing on the list is blockers?

ok ok.  good point ;)

Let's give it 3 days to garner some comments and possibly close out a
few of those P1s and P2s.  Wednesday evening...

--j.

[Mark Martinec]

Justin wrote:

> > Why end of the week if nothing on the list is blockers?
>
> ok ok.  good point ;)
>
> Let's give it 3 days to garner some comments and possibly close out a
> few of those P1s and P2s.  Wednesday evening...

Wednesday evening is good (or even Thursday),
I could announce its availability on Friday at a German mail server conference
http://www.heinlein-support.de/web/akademie/mailserver-konferenz-2009/

  Mark

[Justin Mason]

On Mon, Jun 29, 2009 at 22:24, Mark Martinec<Mark.Martinec+sa@...> wrote:

> Justin wrote:
>
>> > Why end of the week if nothing on the list is blockers?
>>
>> ok ok.  good point ;)
>>
>> Let's give it 3 days to garner some comments and possibly close out a
>> few of those P1s and P2s.  Wednesday evening...
>
> Wednesday evening is good (or even Thursday),
> I could announce its availability on Friday at a German mail server conference
> http://www.heinlein-support.de/web/akademie/mailserver-konferenz-2009/

great! we'll need to get 3 +1's from committers to release it though ;)  given
that you and I will be 2 of those, hopefully,  I don't think this will
be a problem.

--j.