calling all release branch managers

[Roland McGrath]

We haven't heard anything more about active collaboration lately.
Now we'll have the 2.11 release by next week.  So what's the plan?

Is Petr and/or Andreas the 2.10 release branch manager?
Is Andreas and/or Petr the 2.11 release branch manager?
Are there wiki pages about release branches and procedures?

Has anybody thought about what the bugzilla procedures ought to be?

If you ever want to be helping the glibc project behave more cohesively,
now is the time to be doing something about it.


Thanks,
Roland

[Joseph S. Myers]

On Wed, 28 Oct 2009, Roland McGrath wrote:

> Has anybody thought about what the bugzilla procedures ought to be?

A prerequisite for good bugzilla procedures for branches is such
procedures for mainline development.  Would it be useful for people to
identify bugs with patches that have gone unreviewed for some time (or
where submitters of patches are otherwise awaiting maintainer comment)?  
That is, if such bugs were identified would enough maintainers then review
the patches?

For example, bug 9894 has what seems a straightforward patch that has gone
unreviewed since February.

--
Joseph S. Myers
joseph@...

[Petr Baudis-2]

  Hi!

On Wed, Oct 28, 2009 at 06:31:11PM -0700, Roland McGrath wrote:

> We haven't heard anything more about active collaboration lately.
> Now we'll have the 2.11 release by next week.  So what's the plan?
>
> Is Petr and/or Andreas the 2.10 release branch manager?
> Is Andreas and/or Petr the 2.11 release branch manager?
> Are there wiki pages about release branches and procedures?
>
> Has anybody thought about what the bugzilla procedures ought to be?
>
> If you ever want to be helping the glibc project behave more cohesively,
> now is the time to be doing something about it.

  I'm sorry that I didn't take immediate action after your last mail, I
was quite busy lately, but I plan to get to write some wiki pages etc.
ASAP. I wonder who is currently the 2.10 release branch manager as well,
but I have always assumed Andreas to be one since he did the original
push. It finally turns out we will probably use 2.11 in one of our SUSE
products as well though, so I don't mind being 2.11 release manager and
Andreas staying 2.10 manager if he wants and is interested.

--
                                Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth

[Roland McGrath]

> A prerequisite for good bugzilla procedures for branches is such
> procedures for mainline development.

I disagree.  Certainly we would like to come to coherent procedures all
around.  But IMHO the place where the need for formality starts is with
the formal release branches.  Those are what users who are not themselves
libc hackers need to deal with, where there is concrete stable maintenance
going on that needs coordination, etc.

I certainly don't mean to stifle any discussion you would like to have.
I honestly don't think we'll ever achieve any useful formalities in
how we work on the trunk until we have the user-facing world of releases
and bug report management for release branches organized and functioning
as a daily practice.

> Would it be useful for people to identify bugs with patches that have
> gone unreviewed for some time (or where submitters of patches are
> otherwise awaiting maintainer comment)?  That is, if such bugs were
> identified would enough maintainers then review the patches?

If you think this might be useful then I'd suggest doing on the wiki.
It's a wiki precisely so that you can do first and ask later.

Frankly, this feels like a retread going where we have been before.
There is no shortage of complaints that trunk maintainers haven't paid
enough attention to some patch or other, nor of their repetition.  To me
it all still looks like simply complaining about maintainers not doing
more work.  I don't think that's going to get anyone anywhere.  

If you want things to go better and feel more organized, then do
something to help trunk maintainers feel some of the load of mayhem
taken off them, not add more.  The overall community has contributed so
little to getting the project organized that I cannot in good faith tell
any trunk maintainer that any new formalism reduces his work rather than
increases it.  

Perhaps what you have in mind will help in that way.  If it does, that's
great.  But experience makes me skeptical, if not downright pessimistic
(and I'm the nice one).  IMHO we'll have to achieve some working bug
procedures for release branches and experience them a bit in practice
before we can even decide whether and how we want to use bugzilla for
the trunk at all.


Thanks,
Roland

[Joseph S. Myers]

On Thu, 29 Oct 2009, Roland McGrath wrote:

> > A prerequisite for good bugzilla procedures for branches is such
> > procedures for mainline development.
>
> I disagree.  Certainly we would like to come to coherent procedures all
> around.  But IMHO the place where the need for formality starts is with
> the formal release branches.  Those are what users who are not themselves
> libc hackers need to deal with, where there is concrete stable maintenance
> going on that needs coordination, etc.

I propose the following bugzilla procedure:

Set a milestone on a bug if you think the fix for that bug is ready for a
given release branch and should go in it, or that the bug needs fixing for
that branch, and reopen if the bug was closed as fixed on trunk but the
fix should go in a previous branch.  Anyone can do this once for a bug but
should not restore a milestone if reset by the branch manager or reopen
if the branch manager decides against fixing on the branch.

Branch manager reviews bugs with the milestone set before branching and
decides which fixes should go in before the branchpoint (and determines as
necessary whether to delay branching or let the branch go without
everything wanted).  If there are persistent problems with getting things
fixed or reviewed in particular areas and so branches need to be made
without all the desired fixes, invite more people to become trunk
maintainers to fix and review patches in those areas.  By recording which
fixes were postponed past a branch we can get an idea of what areas need
more maintainers, and look at past contributors in those areas to find
suitable maintainer candidates.

Branch manager reviews bugs with the milestone set before subsequent point
releases from the branch (with the additional requirement there that they
are fixed on trunk first).

As you'll see, this applies both before and after the branchpoint, which I
think is the natural thing to do.  I have experience with this as one of
the release managers for GCC; Bugzilla is an important tool for monitoring
the state of things before branching and deciding when to branch.  
Release managers in GCC don't have any special rights over the trunk to
review or apply patches - only the power to delay or not to delay
branching - but the community still seems to work to get features and
fixes in at the right times.

--
Joseph S. Myers
joseph@...

[Roland McGrath]

(Trimmed CC's since everyone is on the list.)

I'll give my input, but don't give any special weight to my comments
on the details.  What I hope for is a consensus among the people who
will participate, and those people writing up things like wiki pages
for their proposals and/or agreements.  I personally hope to be a far
less active participant myself than others.

How does the reopen procedure address having multiple stable branches
alive at once?

Perhaps I'm worrying where I shouldn't, but I wonder if this way could
annoy trunk maintainers with bugzilla chatter about backport decisions and
administrivia they would rather not see in their mailboxes (assuming they
remained CC'd on the bugs because they were owners of them during the trunk
stage).  I guess if the procedure is rigorous about milestone settings,
then bugzilla mail can be filtered by X-Bugzilla-Milestone headers or
something like that.


Thanks,
Roland

[Joseph S. Myers]

On Thu, 29 Oct 2009, Roland McGrath wrote:

> How does the reopen procedure address having multiple stable branches
> alive at once?

Reopen if you think there's any branch that should have the fix that
doesn't.  If people prefer, the procedure could say "open a new bug" - I'm
just going by GCC practice where if we think something should be fixed on
a release branch (rather than just trunk), we leave it open until it is.

Another procedure we need is: open separate bugs for libc and ports, and
preferably separate bugs for different ports, rather than one bug with
both libc and ports patches, as different people will be dealing with
different areas and a maintainer should be able to close a bug once
they've dealt with the parts in their area.

> Perhaps I'm worrying where I shouldn't, but I wonder if this way could
> annoy trunk maintainers with bugzilla chatter about backport decisions and
> administrivia they would rather not see in their mailboxes (assuming they
> remained CC'd on the bugs because they were owners of them during the trunk
> stage).  I guess if the procedure is rigorous about milestone settings,
> then bugzilla mail can be filtered by X-Bugzilla-Milestone headers or
> something like that.

If we don't want this going to trunk maintainers once something is fixed
on trunk, then the procedure will need to be "open a new bug for the
branch" - and there will need to be a way for such new bugs not to CC the
trunk maintainers but to CC branch maintainers instead.

--
Joseph S. Myers
joseph@...

[Roland McGrath]

> Reopen if you think there's any branch that should have the fix that
> doesn't.  

How does that interact with milestone settings?  What (aside from the bug
comment trail) keeps track of which of various branches have merged the fix?

> If people prefer, the procedure could say "open a new bug" - I'm
> just going by GCC practice where if we think something should be fixed on
> a release branch (rather than just trunk), we leave it open until it is.

The practice you've suggested seems straightforward if in practice there is
only really one release branch that is active in getting fixes backported
at any one time.  I'm just wondering how well it works when that is the
other case.  I have experienced the "clone to a new bug and use bug
dependencies" method, and I'm ambivalent about its advantages and
disadvantages.

> Another procedure we need is: open separate bugs for libc and ports, and
> preferably separate bugs for different ports, rather than one bug with
> both libc and ports patches, as different people will be dealing with
> different areas and a maintainer should be able to close a bug once
> they've dealt with the parts in their area.

That sentence makes a lot of sense and keeps doing so when you
s/libc/trunk/ and s/ports/branches/.  For example, say Tom owns the bug on
the trunk, Dick owns the bug on the 2.72 branch, and Harry owns the bug on
the 3.14 branch.  To whom is the bug assigned?  How does each of them go
about doing their daily "things that are still my problem" bugzilla query?


Thanks,
Roland

[Joseph S. Myers]

On Thu, 29 Oct 2009, Roland McGrath wrote:

> > Reopen if you think there's any branch that should have the fix that
> > doesn't.  
>
> How does that interact with milestone settings?  What (aside from the bug
> comment trail) keeps track of which of various branches have merged the fix?

In GCC, milestone of closed bug = earliest release on earliest branch that
has the fix, milestone of open bug = next release on earliest branch for
which the fix is wanted.  Tracking statuses in many different trees is
something bug tracking systems in general don't seem very good at, but I
think the comment trail is good enough in practice.

> > Another procedure we need is: open separate bugs for libc and ports, and
> > preferably separate bugs for different ports, rather than one bug with
> > both libc and ports patches, as different people will be dealing with
> > different areas and a maintainer should be able to close a bug once
> > they've dealt with the parts in their area.
>
> That sentence makes a lot of sense and keeps doing so when you
> s/libc/trunk/ and s/ports/branches/.  For example, say Tom owns the bug on
> the trunk, Dick owns the bug on the 2.72 branch, and Harry owns the bug on
> the 3.14 branch.  To whom is the bug assigned?  How does each of them go
> about doing their daily "things that are still my problem" bugzilla query?

If you want different owners then multiple bugs makes sense - but I
generally presume that something should be fixed on trunk before 2.11 and
on 2.11 before 2.10, and with that principle you only need one active
owner at a time.

(There should also be some policy for when milestones are set at all.  
GCC bases it around regressions; for glibc I'd suggest something more
flexible related to whether a fix is available, how safe the fix appears,
when the fix is available and how important the bug seems to fix - ABI
breakage bugs would be important even without a fix ready, for example.)

--
Joseph S. Myers
joseph@...

[Roland McGrath]

> In GCC, milestone of closed bug = earliest release on earliest branch that
> has the fix, milestone of open bug = next release on earliest branch for
> which the fix is wanted.  Tracking statuses in many different trees is
> something bug tracking systems in general don't seem very good at, but I
> think the comment trail is good enough in practice.

Ok.  With bugzilla you can also use keywords and flags in ways that make
them usable for searching and filtering.  I think it merits some
attention to coming up with conventions that enable automation.  For me,
all should look toward the goal of each responsible person being able to
trivially peruse a priority list of things that require their attention,
and waste no time to discern and ignore things that do not.

> If you want different owners then multiple bugs makes sense - but I
> generally presume that something should be fixed on trunk before 2.11 and
> on 2.11 before 2.10, and with that principle you only need one active
> owner at a time.

That sounds reasonable enough to me.  My devil's advocate organ was
considering the case where 2.42 and 2.37 have active maintenance
(e.g. they are the system versions for heavily-used production systems
that pay maintainers for quick turnaround in fixes) while 2.39 is still
maintained but slowly (e.g. it is the system version for beloved arcane
hobby systems whose weekend hobby maintainers pay attention for a week
before their hobbyist convention twice a year).

> (There should also be some policy for when milestones are set at all.  
> GCC bases it around regressions; for glibc I'd suggest something more
> flexible related to whether a fix is available, how safe the fix appears,
> when the fix is available and how important the bug seems to fix - ABI
> breakage bugs would be important even without a fix ready, for example.)

Right.  We've discussed this a little in the context of branch managers'
discretion to backport or not.  Bits firmly established are things like any
behavior change (pure fixes included) in a branch only after the trunk has
committed to what the new behavior will be.  Other considerations like the
ones you mention we have mostly lumped as "good sense of a branch manager".
Of course, that means wise branch managers collaborate on documenting
guidelines for all to see, to whatever degree of codification is useful.


Thanks,
Roland

[Joseph S. Myers]

On Thu, 29 Oct 2009, Roland McGrath wrote:

> > (There should also be some policy for when milestones are set at all.  
> > GCC bases it around regressions; for glibc I'd suggest something more
> > flexible related to whether a fix is available, how safe the fix appears,
> > when the fix is available and how important the bug seems to fix - ABI
> > breakage bugs would be important even without a fix ready, for example.)
>
> Right.  We've discussed this a little in the context of branch managers'
> discretion to backport or not.  Bits firmly established are things like any
> behavior change (pure fixes included) in a branch only after the trunk has
> committed to what the new behavior will be.  Other considerations like the
> ones you mention we have mostly lumped as "good sense of a branch manager".
> Of course, that means wise branch managers collaborate on documenting
> guidelines for all to see, to whatever degree of codification is useful.

I'm thinking of the ones I mention as to some extent being guidance to
those who aren't branch managers on when to use milestones to flag
something for a branch manager's attention - "I think this ABI breakage
needs fixing before we branch" or "I think the fix for this bug, which is
on trunk, is safe and appropriate for this branch".  If the branch manager
disagrees, they then close the bug / remove the milestone.  If they think
the patch needs to be tested for another month on trunk before
backporting, they say so and leave the milestone or put it back to the
next point release.

--
Joseph S. Myers
joseph@...

[Carlos O'Donell-2]

On Thu, Oct 29, 2009 at 8:23 PM, Joseph S. Myers
<joseph@...> wrote:

> On Thu, 29 Oct 2009, Roland McGrath wrote:
>> Right.  We've discussed this a little in the context of branch managers'
>> discretion to backport or not.  Bits firmly established are things like any
>> behavior change (pure fixes included) in a branch only after the trunk has
>> committed to what the new behavior will be.  Other considerations like the
>> ones you mention we have mostly lumped as "good sense of a branch manager".
>> Of course, that means wise branch managers collaborate on documenting
>> guidelines for all to see, to whatever degree of codification is useful.
>
> I'm thinking of the ones I mention as to some extent being guidance to
> those who aren't branch managers on when to use milestones to flag
> something for a branch manager's attention - "I think this ABI breakage
> needs fixing before we branch" or "I think the fix for this bug, which is
> on trunk, is safe and appropriate for this branch".  If the branch manager
> disagrees, they then close the bug / remove the milestone.  If they think
> the patch needs to be tested for another month on trunk before
> backporting, they say so and leave the milestone or put it back to the
> next point release.

I reorganized the top level glibc wiki page to make it a little more coherent:
http://sourceware.org/glibc/wiki/HomePage

I've tried to document everything in this thread here:
http://sourceware.org/glibc/wiki/bugzilla/procedures

You can see the keyword list here to know which bugs are needed on
which release branches:
http://sources.redhat.com/bugzilla/describekeywords.cgi

I have also added the target milestone "2.11" for the upcoming release.

I am against creating new bugs to backport a fix to another branch.

I am for creating multiple bugs for each port, or target.

I am a big believer in getting the SNR as high as possible and
automating queries so maintainers don't have to waste time.

Cheers,
Carlos.

[Roland McGrath]

> I am for creating multiple bugs for each port, or target.

You mean when machine-specific work is required to fix the bug on each
machine?  Or what?

[Petr Baudis-2]

On Thu, Oct 29, 2009 at 09:05:08PM +0000, Joseph S. Myers wrote:

> On Thu, 29 Oct 2009, Roland McGrath wrote:
>
> > > A prerequisite for good bugzilla procedures for branches is such
> > > procedures for mainline development.
> >
> > I disagree.  Certainly we would like to come to coherent procedures all
> > around.  But IMHO the place where the need for formality starts is with
> > the formal release branches.  Those are what users who are not themselves
> > libc hackers need to deal with, where there is concrete stable maintenance
> > going on that needs coordination, etc.
>
> I propose the following bugzilla procedure:
>
> Set a milestone on a bug if you think the fix for that bug is ready for a
> given release branch and should go in it, or that the bug needs fixing for
> that branch, and reopen if the bug was closed as fixed on trunk but the
> fix should go in a previous branch.  Anyone can do this once for a bug but
> should not restore a milestone if reset by the branch manager or reopen
> if the branch manager decides against fixing on the branch.

This sounds fairly common-sense and sensible, I would expect that to be
the natural idea of anyone, but we can surely document this if noone is
against.  In case of multiple stable branches, why not just set the
milestone to the oldest branch, and the branch maintainer can reset it
to the next newer branch when he is done, if the maintainer of the newer
branch doesn't notice earlier, which is fairly likely given the rather
small amount of bugs that get opened.

> Branch manager reviews bugs with the milestone set before branching and
> decides which fixes should go in before the branchpoint (and determines as
> necessary whether to delay branching or let the branch go without
> everything wanted).  If there are persistent problems with getting things
> fixed or reviewed in particular areas and so branches need to be made
> without all the desired fixes, invite more people to become trunk
> maintainers to fix and review patches in those areas.  By recording which
> fixes were postponed past a branch we can get an idea of what areas need
> more maintainers, and look at past contributors in those areas to find
> suitable maintainer candidates.
>
> Branch manager reviews bugs with the milestone set before subsequent point
> releases from the branch (with the additional requirement there that they
> are fixed on trunk first).
>
> As you'll see, this applies both before and after the branchpoint, which I
> think is the natural thing to do.  I have experience with this as one of
> the release managers for GCC; Bugzilla is an important tool for monitoring
> the state of things before branching and deciding when to branch.  
> Release managers in GCC don't have any special rights over the trunk to
> review or apply patches - only the power to delay or not to delay
> branching - but the community still seems to work to get features and
> fixes in at the right times.

This would be a change from today in that branch manager would .

I have no idea how this would work in glibc, since it requires a lot of
cooperation between Ulrich and the branch manager; I don't know how this
would be done, and if Ulrich would be interested in any such cooperation
at all.

The policy you outline does seem as a overkill to me, though - I think
we are overdesigning for a case we know little about yet, it seems to me
to first try really maintaining stable branches for a release or two,
then we can decide if there would be major improvement in involving the
branch maintainer before tagging the major release; to get things
started, I think it's better to first have them just as stewards tagging
minor releases after the major one falls out from the trunk.

--
                                Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth

[Carlos O'Donell-2]

On Fri, Oct 30, 2009 at 2:51 AM, Roland McGrath <roland@...> wrote:
>> I am for creating multiple bugs for each port, or target.
>
> You mean when machine-specific work is required to fix the bug on each
> machine?  Or what?

Yes, when target-specific work or port-specific work is required to
fix the bug for each target.

For example when a new flag needs to be added to bits/fcntl.h we
should file a bug for each affected target.

Cheers,
Carlos.

[Andreas Schwab-4]

Petr Baudis <pasky@...> writes:

> It finally turns out we will probably use 2.11 in one of our SUSE
> products as well though, so I don't mind being 2.11 release manager and
> Andreas staying 2.10 manager if he wants and is interested.

I don't mind either way.

Andreas.

--
Andreas Schwab, schwab@...
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Petr Baudis-2]

On Wed, Oct 28, 2009 at 06:31:11PM -0700, Roland McGrath wrote:

> We haven't heard anything more about active collaboration lately.
> Now we'll have the 2.11 release by next week.  So what's the plan?
>
> Is Petr and/or Andreas the 2.10 release branch manager?
> Is Andreas and/or Petr the 2.11 release branch manager?
> Are there wiki pages about release branches and procedures?
>
> Has anybody thought about what the bugzilla procedures ought to be?
>
> If you ever want to be helping the glibc project behave more cohesively,
> now is the time to be doing something about it.

I have finally wrote up something on

        http://sourceware.org/glibc/wiki/Release

making myself the 2.11 release manager and Andreas the 2.10 release
manager based on the previous mails. I have also elaborated on the 2.11
release branch setup in a sub-page, and pre-filled some generic
information about 2.10 for Andreas to fill up.

Please everyone interested in release branches have a look to check if
this matches the spirit of what you have envisioned - I tried to base
the described policies on the general consensus from summer.

I have now requested commit permissions for glibc.git; for now, I expect
to just merge anything up to 2.11.99 commit in master to the branch,
unless Ulrich decides to make another release himself.

Andreas, I would like to push few further (conservative) cherry-picks
for 2.10 to release/2.10/master, would you be ok with that or do you
want all changes to go through your review? Do you have any plans about
2.10.2 release?

--
                                Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth

[Andreas Schwab-4]

Petr Baudis <pasky@...> writes:

> Andreas, I would like to push few further (conservative) cherry-picks
> for 2.10 to release/2.10/master, would you be ok with that or do you
> want all changes to go through your review?

If you're talking about the cherry-picks in your repo branch, they look
ok.

Andreas.

--
Andreas Schwab, schwab@...
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

[Petr Baudis-2]

On Tue, Nov 10, 2009 at 03:08:15PM +0100, Andreas Schwab wrote:
> Petr Baudis <pasky@...> writes:
>
> > Andreas, I would like to push few further (conservative) cherry-picks
> > for 2.10 to release/2.10/master, would you be ok with that or do you
> > want all changes to go through your review?
>
> If you're talking about the cherry-picks in your repo branch, they look
> ok.

Ok, I have pushed them on the branch.

Could you consider tagging this glibc-2.10.2? This version has been
extensively tested in openSUSE and is currently used in 11.2, it seems
to work well.

[With the exception that no fix for
http://sourceware.org/bugzilla/show_bug.cgi?id=10282 is in
release/2.10/master; a commit that probably fixes this bug has been
recently pushed onto master, though I didn't have time to verify it.]

--
                                Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth

[Andreas Schwab-4]

Petr Baudis <pasky@...> writes:

> Could you consider tagging this glibc-2.10.2? This version has been
> extensively tested in openSUSE and is currently used in 11.2, it seems
> to work well.

I'm considering to add 5b55d236 and 34df851b as well.  Other than that I
agree that it's a good time for a release.

Andreas.

--
Andreas Schwab, schwab@...
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."