[RFC] new -stable tag variant, Git workflow question

[Ingo Molnar]

FYI, today i committed a scheduler performance fix that has a number of
commit prerequisites for -stable integration. Those commits are not
marked -stable.

Previously, in similar situations, i solved it by email-forwarding the
prereq commits to stable@.... (or by waiting for your 'it does
not apply to -stable' email and figuring out the prereqs then.)

But we can move this into the Git commit space too, and minimize the
work for the -stable team, via a new -stable tag variant. So with this
new commit i started using a new tagging scheme in the commit itself:

 Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
 Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
 Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
 Cc: <stable@...> # .32.x
 LKML-Reference: <1257821402.5648.17.camel@...>
 Signed-off-by: Ingo Molnar <mingo@...>

The tag sequence has the meaning of:

  git cherry-pick a1f84a3
  git cherry-pick 1b9508f
  git cherry-pick fd21073
  git cherry-pick <this commit>

and i'm wondering whether this tagging scheme is fine with your -stable
scripting, etc.

A further question is, i can see using this tagging scheme in the future
in merge commits log messages too - will your scripts notice that too?

For example if there's a few commits left in tip:*/urgent branches
(tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
v2.6.32 is released, i will then merge them into tip:sched/core,
tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
a notification area to 'activate' them for -stable backporting via a
merge commit.

This is how such merge commits would look like:

  Merge branch 'core/urgent' into core/rcu

  Merge reason: Pick up urgent fixes that did not make it into .32.0

  Cc: <stable@...> # .32.x: 83f5b01: rcu: Fix long-grace-period race
  Signed-off-by: Ingo Molnar <mingo@...>

This is not so rare of a situation as it might seem - for the trees i
maintain it happens in almost every release cycle - i typically skip
urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
fixes - but they'd still be eligible for -stable.

I've attached the full commit below. The prereq commits are not uptream
yet, and they dont carry a -stable backporting tag as the -stable
relevance was not anticipated at that point yet. They will all be
upstream in the next merge window when Linus merges the relevant tree -
and then all these tags become visible to the -stable team's scripts.

What do you think about this new -stable tagging variant? To me it looks
quite intuitive, less error-prone and it is more informative as well.
Furthermore, it gives us some freedom to mark commits as backport
candidates later on. I kept them oneliners for the purpose of making
them all self-sufficient tags.

( Sidenote: i wouldnt go as far as to generate null Git commits to mark
  backports after the fact - this scheme is for a series of commits that
  get 'completed' - there's usually a final followup commit that can
  embedd this information. )

        Ingo

---------------------------->
From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001
From: Mike Galbraith <efault@...>
Date: Tue, 10 Nov 2009 03:50:02 +0100
Subject: [PATCH] sched: Fix and clean up rate-limit newidle code

Commit 1b9508f, "Rate-limit newidle" has been confirmed to fix
the netperf UDP loopback regression reported by Alex Shi.

This is a cleanup and a fix:

 - moved to a more out of the way spot

 - fix to ensure that balancing doesn't try to balance
   runqueues which haven't gone online yet, which can
   mess up CPU enumeration during boot.

Reported-by: Alex Shi <alex.shi@...>
Reported-by: Zhang, Yanmin <yanmin_zhang@...>
Signed-off-by: Mike Galbraith <efault@...>
Acked-by: Peter Zijlstra <a.p.zijlstra@...>
Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
Cc: <stable@...> # .32.x
LKML-Reference: <1257821402.5648.17.camel@...>
Signed-off-by: Ingo Molnar <mingo@...>
---
 kernel/sched.c |   28 +++++++++++++++-------------
 1 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 23e3535..ad37776 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -2354,17 +2354,6 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,
  if (rq != orig_rq)
  update_rq_clock(rq);
 
- if (rq->idle_stamp) {
- u64 delta = rq->clock - rq->idle_stamp;
- u64 max = 2*sysctl_sched_migration_cost;
-
- if (delta > max)
- rq->avg_idle = max;
- else
- update_avg(&rq->avg_idle, delta);
- rq->idle_stamp = 0;
- }
-
  WARN_ON(p->state != TASK_WAKING);
  cpu = task_cpu(p);
 
@@ -2421,6 +2410,17 @@ out_running:
 #ifdef CONFIG_SMP
  if (p->sched_class->task_wake_up)
  p->sched_class->task_wake_up(rq, p);
+
+ if (unlikely(rq->idle_stamp)) {
+ u64 delta = rq->clock - rq->idle_stamp;
+ u64 max = 2*sysctl_sched_migration_cost;
+
+ if (delta > max)
+ rq->avg_idle = max;
+ else
+ update_avg(&rq->avg_idle, delta);
+ rq->idle_stamp = 0;
+ }
 #endif
 out:
  task_rq_unlock(rq, &flags);
@@ -4098,7 +4098,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
  unsigned long flags;
  struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
 
- cpumask_setall(cpus);
+ cpumask_copy(cpus, cpu_online_mask);
 
  /*
  * When power savings policy is enabled for the parent domain, idle
@@ -4261,7 +4261,7 @@ load_balance_newidle(int this_cpu, struct rq *this_rq, struct sched_domain *sd)
  int all_pinned = 0;
  struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
 
- cpumask_setall(cpus);
+ cpumask_copy(cpus, cpu_online_mask);
 
  /*
  * When power savings policy is enabled for the parent domain, idle
@@ -9522,6 +9522,8 @@ void __init sched_init(void)
  rq->cpu = i;
  rq->online = 0;
  rq->migration_thread = NULL;
+ rq->idle_stamp = 0;
+ rq->avg_idle = 2*sysctl_sched_migration_cost;
  INIT_LIST_HEAD(&rq->migration_queue);
  rq_attach_root(rq, &def_root_domain);
 #endif

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Greg KH-2]

On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:

> FYI, today i committed a scheduler performance fix that has a number of
> commit prerequisites for -stable integration. Those commits are not
> marked -stable.
>
> Previously, in similar situations, i solved it by email-forwarding the
> prereq commits to stable@.... (or by waiting for your 'it does
> not apply to -stable' email and figuring out the prereqs then.)
>
> But we can move this into the Git commit space too, and minimize the
> work for the -stable team, via a new -stable tag variant. So with this
> new commit i started using a new tagging scheme in the commit itself:
>
>  Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
>  Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
>  Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
>  Cc: <stable@...> # .32.x
>  LKML-Reference: <1257821402.5648.17.camel@...>
>  Signed-off-by: Ingo Molnar <mingo@...>
>
> The tag sequence has the meaning of:
>
>   git cherry-pick a1f84a3
>   git cherry-pick 1b9508f
>   git cherry-pick fd21073
>   git cherry-pick <this commit>
>
> and i'm wondering whether this tagging scheme is fine with your -stable
> scripting, etc.

It would work just fine.

I only rely on one main script right now, one that runs from James's
directory on kernel.org that picks out the "Cc: <stable@...>"
lines and forwards the full commit message to stable@....

Then I have a simple script that I just pass the git commit id and
formats it properly for inclusion on the quilt tree for the stable
queue.  If you list the other git commit ids that are needed as a
prerequesite as you did above, that's trivial to also pick out.

So I think this is good for me and my workflow.

> A further question is, i can see using this tagging scheme in the future
> in merge commits log messages too - will your scripts notice that too?

Hm, I don't think we look at merges as there's nothing there to actually
commit.

> For example if there's a few commits left in tip:*/urgent branches
> (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> v2.6.32 is released, i will then merge them into tip:sched/core,
> tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> a notification area to 'activate' them for -stable backporting via a
> merge commit.
>
> This is how such merge commits would look like:
>
>   Merge branch 'core/urgent' into core/rcu
>
>   Merge reason: Pick up urgent fixes that did not make it into .32.0
>
>   Cc: <stable@...> # .32.x: 83f5b01: rcu: Fix long-grace-period race
>   Signed-off-by: Ingo Molnar <mingo@...>
>
> This is not so rare of a situation as it might seem - for the trees i
> maintain it happens in almost every release cycle - i typically skip
> urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> fixes - but they'd still be eligible for -stable.

Ok, that would be good and fine with me.

James, would your script pick this up, or does it need to also pay
attention to merge commits?

> I've attached the full commit below. The prereq commits are not uptream
> yet, and they dont carry a -stable backporting tag as the -stable
> relevance was not anticipated at that point yet. They will all be
> upstream in the next merge window when Linus merges the relevant tree -
> and then all these tags become visible to the -stable team's scripts.
>
> What do you think about this new -stable tagging variant? To me it looks
> quite intuitive, less error-prone and it is more informative as well.
> Furthermore, it gives us some freedom to mark commits as backport
> candidates later on. I kept them oneliners for the purpose of making
> them all self-sufficient tags.

I agree.

> ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
>   backports after the fact - this scheme is for a series of commits that
>   get 'completed' - there's usually a final followup commit that can
>   embedd this information. )

That's fine, don't worry about this.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Greg KH <gregkh@...> wrote:

> > I've attached the full commit below. The prereq commits are not
> > uptream yet, and they dont carry a -stable backporting tag as the
> > -stable relevance was not anticipated at that point yet. They will
> > all be upstream in the next merge window when Linus merges the
> > relevant tree - and then all these tags become visible to the
> > -stable team's scripts.
> >
> > What do you think about this new -stable tagging variant? To me it
> > looks quite intuitive, less error-prone and it is more informative
> > as well. Furthermore, it gives us some freedom to mark commits as
> > backport candidates later on. I kept them oneliners for the purpose
> > of making them all self-sufficient tags.
>
> I agree.

Ok - thanks for the confirmation - i've pushed out the first such
commit. (Let me know if there's any problem with it down the line - it
will be a few weeks, in the next merge window, until it truly
'activates' for -stable.)

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[James Bottomley-2]

On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:

> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> > FYI, today i committed a scheduler performance fix that has a number of
> > commit prerequisites for -stable integration. Those commits are not
> > marked -stable.
> >
> > Previously, in similar situations, i solved it by email-forwarding the
> > prereq commits to stable@.... (or by waiting for your 'it does
> > not apply to -stable' email and figuring out the prereqs then.)
> >
> > But we can move this into the Git commit space too, and minimize the
> > work for the -stable team, via a new -stable tag variant. So with this
> > new commit i started using a new tagging scheme in the commit itself:
> >
> >  Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
> >  Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
> >  Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
> >  Cc: <stable@...> # .32.x
> >  LKML-Reference: <1257821402.5648.17.camel@...>
> >  Signed-off-by: Ingo Molnar <mingo@...>
> >
> > The tag sequence has the meaning of:
> >
> >   git cherry-pick a1f84a3
> >   git cherry-pick 1b9508f
> >   git cherry-pick fd21073
> >   git cherry-pick <this commit>
> >
> > and i'm wondering whether this tagging scheme is fine with your -stable
> > scripting, etc.
>
> It would work just fine.
>
> I only rely on one main script right now, one that runs from James's
> directory on kernel.org that picks out the "Cc: <stable@...>"
> lines and forwards the full commit message to stable@....
>
> Then I have a simple script that I just pass the git commit id and
> formats it properly for inclusion on the quilt tree for the stable
> queue.  If you list the other git commit ids that are needed as a
> prerequesite as you did above, that's trivial to also pick out.
>
> So I think this is good for me and my workflow.

So I take this to mean that I don't alter my script and you pick out the
precursors with yours ...

> > A further question is, i can see using this tagging scheme in the future
> > in merge commits log messages too - will your scripts notice that too?
>
> Hm, I don't think we look at merges as there's nothing there to actually
> commit.
>
> > For example if there's a few commits left in tip:*/urgent branches
> > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > v2.6.32 is released, i will then merge them into tip:sched/core,
> > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > a notification area to 'activate' them for -stable backporting via a
> > merge commit.
> >
> > This is how such merge commits would look like:
> >
> >   Merge branch 'core/urgent' into core/rcu
> >
> >   Merge reason: Pick up urgent fixes that did not make it into .32.0
> >
> >   Cc: <stable@...> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> >   Signed-off-by: Ingo Molnar <mingo@...>
> >
> > This is not so rare of a situation as it might seem - for the trees i
> > maintain it happens in almost every release cycle - i typically skip
> > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > fixes - but they'd still be eligible for -stable.
>
> Ok, that would be good and fine with me.
>
> James, would your script pick this up, or does it need to also pay
> attention to merge commits?

No ... because merge commits should effectively be empty (and when
they're not you can't generate an applyable diff).  If I understand the
workflow, the desire is to have the whole branch sent to stable by
tagging the merge commit.  That's possible ... it's exactly the same
logic I use in the commit scripts for the SCSI tree, so it should be
possible to extract the logic.

By the looks of the above it's only a few commits, or is it the entire
branch?

> > I've attached the full commit below. The prereq commits are not uptream
> > yet, and they dont carry a -stable backporting tag as the -stable
> > relevance was not anticipated at that point yet. They will all be
> > upstream in the next merge window when Linus merges the relevant tree -
> > and then all these tags become visible to the -stable team's scripts.
> >
> > What do you think about this new -stable tagging variant? To me it looks
> > quite intuitive, less error-prone and it is more informative as well.
> > Furthermore, it gives us some freedom to mark commits as backport
> > candidates later on. I kept them oneliners for the purpose of making
> > them all self-sufficient tags.
>
> I agree.
>
> > ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
> >   backports after the fact - this scheme is for a series of commits that
> >   get 'completed' - there's usually a final followup commit that can
> >   embedd this information. )
>
> That's fine, don't worry about this.

The question is, how important is this?

One of the assumptions behind the current setup is that I assume
backports are independent (so the order of transmission doesn't matter
that much).  This isn't always true, but the exceptions tend to get
handled manually.  Part of what the above is requesting is an
implementation that starts to care about ordering.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[James Bottomley-2]

On Tue, 2009-11-10 at 05:20 +0100, Ingo Molnar wrote:

> * Greg KH <gregkh@...> wrote:
>
> > > I've attached the full commit below. The prereq commits are not
> > > uptream yet, and they dont carry a -stable backporting tag as the
> > > -stable relevance was not anticipated at that point yet. They will
> > > all be upstream in the next merge window when Linus merges the
> > > relevant tree - and then all these tags become visible to the
> > > -stable team's scripts.
> > >
> > > What do you think about this new -stable tagging variant? To me it
> > > looks quite intuitive, less error-prone and it is more informative
> > > as well. Furthermore, it gives us some freedom to mark commits as
> > > backport candidates later on. I kept them oneliners for the purpose
> > > of making them all self-sufficient tags.
> >
> > I agree.
>
> Ok - thanks for the confirmation - i've pushed out the first such
> commit. (Let me know if there's any problem with it down the line - it
> will be a few weeks, in the next merge window, until it truly
> 'activates' for -stable.)

Can you give me the commit id in your tree so I can run a few tests?

Thanks,

James


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Stefan Richter]

James Bottomley wrote:
[...]
>> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
>> > FYI, today i committed a scheduler performance fix that has a number of
>> > commit prerequisites for -stable integration. Those commits are not
>> > marked -stable.
[...]
>> > we can move this into the Git commit space too, and minimize the
>> > work for the -stable team, via a new -stable tag variant.
[...]
> The question is, how important is this?
>
> One of the assumptions behind the current setup is that I assume
> backports are independent (so the order of transmission doesn't matter
> that much).  This isn't always true, but the exceptions tend to get
> handled manually.  Part of what the above is requesting is an
> implementation that starts to care about ordering.

More importantly, isn't this against the character of the -stable kernel
branches as _safe and simple_ hotfix branches?

If a fix has a number of prerequisites which ar not -stable fixes
themselves, then it is more than a hint that this fix is not really well
suited for -stable.
--
Stefan Richter
-=====-==--= =-== -=-=-
http://arcgraph.de/sr/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* James Bottomley <James.Bottomley@...> wrote:

> On Tue, 2009-11-10 at 05:20 +0100, Ingo Molnar wrote:
> > * Greg KH <gregkh@...> wrote:
> >
> > > > I've attached the full commit below. The prereq commits are not
> > > > uptream yet, and they dont carry a -stable backporting tag as the
> > > > -stable relevance was not anticipated at that point yet. They will
> > > > all be upstream in the next merge window when Linus merges the
> > > > relevant tree - and then all these tags become visible to the
> > > > -stable team's scripts.
> > > >
> > > > What do you think about this new -stable tagging variant? To me it
> > > > looks quite intuitive, less error-prone and it is more informative
> > > > as well. Furthermore, it gives us some freedom to mark commits as
> > > > backport candidates later on. I kept them oneliners for the purpose
> > > > of making them all self-sufficient tags.
> > >
> > > I agree.
> >
> > Ok - thanks for the confirmation - i've pushed out the first such
> > commit. (Let me know if there's any problem with it down the line - it
> > will be a few weeks, in the next merge window, until it truly
> > 'activates' for -stable.)
>
> Can you give me the commit id in your tree so I can run a few tests?

Let me re-quote what you quoted above:

> > > > I've attached the full commit below. [...]

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Linus Torvalds-3]

On Tue, 10 Nov 2009, Ingo Molnar wrote:
>
> * James Bottomley <James.Bottomley@...> wrote:
> >
> > Can you give me the commit id in your tree so I can run a few tests?
>
> Let me re-quote what you quoted above:
>
> > > > > I've attached the full commit below. [...]

Let me re-quote James:

 "Can you give me the commit id in your tree so I can run a few tests?"

He wanted the tree. So that he can run tests on his machinery. His
machinery that does _not_ take some emailed commit description, but works
on git trees.

So Ingo, if you want to be snarky about quoting people, at least be snarky
with a reason.

                Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Linus Torvalds <torvalds@...> wrote:

>
>
> On Tue, 10 Nov 2009, Ingo Molnar wrote:
> >
> > * James Bottomley <James.Bottomley@...> wrote:
> > >
> > > Can you give me the commit id in your tree so I can run a few tests?
> >
> > Let me re-quote what you quoted above:
> >
> > > > > > I've attached the full commit below. [...]
>
> Let me re-quote James:
>
>  "Can you give me the commit id in your tree so I can run a few tests?"
>
> He wanted the tree. So that he can run tests on his machinery. His
> machinery that does _not_ take some emailed commit description, but
> works on git trees.
>
> So Ingo, if you want to be snarky about quoting people, at least be
> snarky with a reason.

The commit i quoted had the commit ID on its first line (i did that so
because i anticipated someone wanting to check my tree), but here it is
again plus the (well-known to James) URI of the -tip tree:

  > From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001

  git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git

Thanks,

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Chris Friesen]

On 11/10/2009 09:52 AM, Stefan Richter wrote:

> More importantly, isn't this against the character of the -stable kernel
> branches as _safe and simple_ hotfix branches?
>
> If a fix has a number of prerequisites which ar not -stable fixes
> themselves, then it is more than a hint that this fix is not really well
> suited for -stable.

Alternately, it's conceivable that the prerequisites were not
in-and-of-themselves candidates for -stable (maybe they didn't do
anything by themselves) but when combined with the final commit the
overall change is suitable for inclusion in -stable.

Chris
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Greg KH-2]

On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:

> On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> > > FYI, today i committed a scheduler performance fix that has a number of
> > > commit prerequisites for -stable integration. Those commits are not
> > > marked -stable.
> > >
> > > Previously, in similar situations, i solved it by email-forwarding the
> > > prereq commits to stable@.... (or by waiting for your 'it does
> > > not apply to -stable' email and figuring out the prereqs then.)
> > >
> > > But we can move this into the Git commit space too, and minimize the
> > > work for the -stable team, via a new -stable tag variant. So with this
> > > new commit i started using a new tagging scheme in the commit itself:
> > >
> > >  Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
> > >  Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
> > >  Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
> > >  Cc: <stable@...> # .32.x
> > >  LKML-Reference: <1257821402.5648.17.camel@...>
> > >  Signed-off-by: Ingo Molnar <mingo@...>
> > >
> > > The tag sequence has the meaning of:
> > >
> > >   git cherry-pick a1f84a3
> > >   git cherry-pick 1b9508f
> > >   git cherry-pick fd21073
> > >   git cherry-pick <this commit>
> > >
> > > and i'm wondering whether this tagging scheme is fine with your -stable
> > > scripting, etc.
> >
> > It would work just fine.
> >
> > I only rely on one main script right now, one that runs from James's
> > directory on kernel.org that picks out the "Cc: <stable@...>"
> > lines and forwards the full commit message to stable@....
> >
> > Then I have a simple script that I just pass the git commit id and
> > formats it properly for inclusion on the quilt tree for the stable
> > queue.  If you list the other git commit ids that are needed as a
> > prerequesite as you did above, that's trivial to also pick out.
> >
> > So I think this is good for me and my workflow.
>
> So I take this to mean that I don't alter my script and you pick out the
> precursors with yours ...

Exactly, it's easier for me that way, as I know the dependancy of what
needs to go before what.  And it's just so trivial to feed a git commit
id to my script :)

> > > A further question is, i can see using this tagging scheme in the future
> > > in merge commits log messages too - will your scripts notice that too?
> >
> > Hm, I don't think we look at merges as there's nothing there to actually
> > commit.
> >
> > > For example if there's a few commits left in tip:*/urgent branches
> > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > a notification area to 'activate' them for -stable backporting via a
> > > merge commit.
> > >
> > > This is how such merge commits would look like:
> > >
> > >   Merge branch 'core/urgent' into core/rcu
> > >
> > >   Merge reason: Pick up urgent fixes that did not make it into .32.0
> > >
> > >   Cc: <stable@...> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > >   Signed-off-by: Ingo Molnar <mingo@...>
> > >
> > > This is not so rare of a situation as it might seem - for the trees i
> > > maintain it happens in almost every release cycle - i typically skip
> > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > fixes - but they'd still be eligible for -stable.
> >
> > Ok, that would be good and fine with me.
> >
> > James, would your script pick this up, or does it need to also pay
> > attention to merge commits?
>
> No ... because merge commits should effectively be empty (and when
> they're not you can't generate an applyable diff).  If I understand the
> workflow, the desire is to have the whole branch sent to stable by
> tagging the merge commit.  That's possible ... it's exactly the same
> logic I use in the commit scripts for the SCSI tree, so it should be
> possible to extract the logic.
>
> By the looks of the above it's only a few commits, or is it the entire
> branch?

I'm thinking the commit would be the merge, right Ingo?  So it would
just be a single commit that has the marker in it.

> > > I've attached the full commit below. The prereq commits are not uptream
> > > yet, and they dont carry a -stable backporting tag as the -stable
> > > relevance was not anticipated at that point yet. They will all be
> > > upstream in the next merge window when Linus merges the relevant tree -
> > > and then all these tags become visible to the -stable team's scripts.
> > >
> > > What do you think about this new -stable tagging variant? To me it looks
> > > quite intuitive, less error-prone and it is more informative as well.
> > > Furthermore, it gives us some freedom to mark commits as backport
> > > candidates later on. I kept them oneliners for the purpose of making
> > > them all self-sufficient tags.
> >
> > I agree.
> >
> > > ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
> > >   backports after the fact - this scheme is for a series of commits that
> > >   get 'completed' - there's usually a final followup commit that can
> > >   embedd this information. )
> >
> > That's fine, don't worry about this.
>
> The question is, how important is this?
>
> One of the assumptions behind the current setup is that I assume
> backports are independent (so the order of transmission doesn't matter
> that much).  This isn't always true, but the exceptions tend to get
> handled manually.  Part of what the above is requesting is an
> implementation that starts to care about ordering.

No, I'll take care of the ordering myself.  Heck, I already have to do
that today with our current setup as we have dependant staging patches
right now.  I just want to make sure the merge commits will get picked
up and sent to me so I can then pick the correct git commit ids out of
them.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Greg KH-2]

On Tue, Nov 10, 2009 at 04:52:22PM +0100, Stefan Richter wrote:

> James Bottomley wrote:
> [...]
> >> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> >> > FYI, today i committed a scheduler performance fix that has a number of
> >> > commit prerequisites for -stable integration. Those commits are not
> >> > marked -stable.
> [...]
> >> > we can move this into the Git commit space too, and minimize the
> >> > work for the -stable team, via a new -stable tag variant.
> [...]
> > The question is, how important is this?
> >
> > One of the assumptions behind the current setup is that I assume
> > backports are independent (so the order of transmission doesn't matter
> > that much).  This isn't always true, but the exceptions tend to get
> > handled manually.  Part of what the above is requesting is an
> > implementation that starts to care about ordering.
>
> More importantly, isn't this against the character of the -stable kernel
> branches as _safe and simple_ hotfix branches?
>
> If a fix has a number of prerequisites which ar not -stable fixes
> themselves, then it is more than a hint that this fix is not really well
> suited for -stable.

Not true, we have been doing this kind of thing for quite some time now.
Sometimes it's just a simple "this patch cleans up the code, and the
second one fixes it in an obvious manner" type thing.  It is easier for
me and everyone else for us to apply 2 commits to the -stable tree,
instead of rewriting the second patch that actually does the fix and
hope I got it all correct in doing so.

It's also easier to review stuff, which is the most important thing.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Greg KH <gregkh@...> wrote:

> > By the looks of the above it's only a few commits, or is it the
> > entire branch?
>
> I'm thinking the commit would be the merge, right Ingo?  So it would
> just be a single commit that has the marker in it.

Correct.

This is really a special case, a small variation of the commit eae0c9d
-stable tagging scheme i outlined in the first mail.

When i merge */urgent branches into the for-linus branch in the merge
window, i cannot change the commits anymore (it would amount to a
rebase), but i have the opportunity to modify the merge commit message
itself. (which is typically a regular merge commit and does not carry
any -stable actionable change itself.)

I already annotate merge commits today - for example:

 | commit 43315956509ca6913764861ac7dec128b91eb1ec
 | Merge: 9bf4e7f 6beba7a
 | Author: Ingo Molnar <mingo@...>
 | Date:   Fri Oct 23 08:23:20 2009 +0200
 |
 |     Merge branch 'perf/core' into perf/probes
 |    
 |     Conflicts:
 |         tools/perf/Makefile
 |    
 |     Merge reason:
 |    
 |      - fix the conflict
 |      - pick up the pr_*() infrastructure to queue up dependent patch
 |    
 |     Signed-off-by: Ingo Molnar <mingo@...>

Note how i already put a SOB line into the merge commit - i treat every
merge as something that 'had to be done' so they are never arbitrary and
always carry real information.

So my idea was to potentially use the extended -stable notification
scheme in certain merge commits too. Here's a mockup merge commit log:

    Merge branch 'sched/urgent' into sched/core
     
    Conflicts:
        tools/perf/Makefile
     
    Merge reason:
     
     - resolve the conflict
     - queue up urgent fixes for the next merge window

    Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
    Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
    Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
    Signed-off-by: Ingo Molnar <mingo@...>

Note that the merge commit itself carries no action for -stable: there's
no "Cc: <stable@...>" line - only 'pointer' lines in the form
of:

    Cc: <stable@...> # .32.x: sha1: title

But ... if you or Linus dislikes this direction of tagging for some
reason i can still do the manual approach as well. It seemed useful to
me though and it would be a natural portion of my workflow.

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Greg KH <gregkh@...> wrote:

> > More importantly, isn't this against the character of the -stable
> > kernel branches as _safe and simple_ hotfix branches?
> >
> > If a fix has a number of prerequisites which ar not -stable fixes
> > themselves, then it is more than a hint that this fix is not really
> > well suited for -stable.
>
> Not true, we have been doing this kind of thing for quite some time
> now. Sometimes it's just a simple "this patch cleans up the code, and
> the second one fixes it in an obvious manner" type thing.  It is
> easier for me and everyone else for us to apply 2 commits to the
> -stable tree, instead of rewriting the second patch that actually does
> the fix and hope I got it all correct in doing so.
>
> It's also easier to review stuff, which is the most important thing.

Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were
-stable candidates anyway, in exactly the same order - the only
difference the new tagging scheme adds here is a more organized,
in-upsream-Git way of communicating it to you.

This is also easier and less error prone for me than using email: i can
do all the -stable tagging when i create a commit - or if i see that a
commit has prereqs and those should be in -stable too. In those
situations i check out the last stable kernel version, and cherry-pick
the prereqs and the target commit, to see that it cherry-picks without
conflicts.

But i cannot send you an email to stable@... just yet: as i
havent fully tested the last commit yet, and have not pushed it out yet.
The commit ID is not stable yet.

So without the in-commit tagging, i'd have to remember to send you an
email in an hour (or in a day - whenever testing is done) - and that is
error prone and easy to forget. The prereqs might be lost, etc. It's
better to do this all in one well-focused moment of time, gather the
information and mention it in the changelog.

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Junio C Hamano-4]

Ingo Molnar <mingo@...> writes:

> Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> - it just helps the existing practice some more. All these commits were
> -stable candidates anyway, in exactly the same order - the only
> difference the new tagging scheme adds here is a more organized,
> in-upsream-Git way of communicating it to you.

I am just a bystander, but if it were truly in-upstream-git way, wouldn't
you be forking a branch from the tagged target release (the latest of
2.6.32.X), and queuing only the changes meant for -stable to it, and
giving the name of the branch to git people and sending out patches from
that branch for e-mailed review and application?

There won't be any special tagging required, only a dedicated branch.

Or am I missing something?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Chris Friesen <cfriesen@...> wrote:

> On 11/10/2009 09:52 AM, Stefan Richter wrote:
>
> > More importantly, isn't this against the character of the -stable kernel
> > branches as _safe and simple_ hotfix branches?
> >
> > If a fix has a number of prerequisites which ar not -stable fixes
> > themselves, then it is more than a hint that this fix is not really well
> > suited for -stable.
>
> Alternately, it's conceivable that the prerequisites were not
> in-and-of-themselves candidates for -stable (maybe they didn't do
> anything by themselves) but when combined with the final commit the
> overall change is suitable for inclusion in -stable.

Yeah.

The way i do it as a maintainer is that when i add a new commit that i
realize as a -stable candidate, and i know that it has no semantic
prereqs (such as a new API, etc.), i git-cherry-pick it into stable in a
test branch. If that works fine i mark it -stable straight away.

If it conflicts, i figure out the prereqs, and look at those. If they
are too big, or too risky, i often decide not to mark a patch for
-stable backporting. (If it's not obvious to be in -stable then it
should not be in -stable, almost by definition - a _LOT_ of people are
using the stable kernels.)

If the prereqs look sane and are wanted for -stable, i end up with a
list of 2 or at most 3 commits that will cherry-pick cleanly. (rarely
more than that - the 4 commits here are really an exception - they are a
string of prereqs that are also fixes)

I send that list of commits to stable@....

( Sidenote: rarely does it make sense to port a conflicting commit to
  -stable. The risks of introducing some regression are just too high.
  Cherry-picking of commits, (while not entirely risk-free of course),
  is far more robust in practice. (conflicting backmerges do happen too
  occasionally, for high-profile bug fixes that justify the cost.) )

So this 'send the list to stable@...' step is simplified via
these tags:

    Cc: <stable@...> # .32.x: a1f84a3: sched: Check for an idle shared cache
    Cc: <stable@...> # .32.x: 1b9508f: sched: Rate-limit newidle
    Cc: <stable@...> # .32.x: fd21073: sched: Fix affinity logic
    Cc: <stable@...> # .32.x

I'd have done that via a plain email in any case - so this scheme does
not enable anything new - it just simplifies the process and makes it a
bit more robust.

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Greg KH-2]

On Tue, Nov 10, 2009 at 01:11:17PM -0800, Junio C Hamano wrote:

> Ingo Molnar <mingo@...> writes:
>
> > Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> > - it just helps the existing practice some more. All these commits were
> > -stable candidates anyway, in exactly the same order - the only
> > difference the new tagging scheme adds here is a more organized,
> > in-upsream-Git way of communicating it to you.
>
> I am just a bystander, but if it were truly in-upstream-git way, wouldn't
> you be forking a branch from the tagged target release (the latest of
> 2.6.32.X), and queuing only the changes meant for -stable to it, and
> giving the name of the branch to git people and sending out patches from
> that branch for e-mailed review and application?
>
> There won't be any special tagging required, only a dedicated branch.
>
> Or am I missing something?

Yes, these are patches going to Linus's tree, which would be 2.6.33 at
the time.  I need to know the ids to add them back to the older .32.y
tree.

hope this helps,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Ingo Molnar]

* Junio C Hamano <gitster@...> wrote:

> Ingo Molnar <mingo@...> writes:
>
> > Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> > - it just helps the existing practice some more. All these commits were
> > -stable candidates anyway, in exactly the same order - the only
> > difference the new tagging scheme adds here is a more organized,
> > in-upsream-Git way of communicating it to you.
>
> I am just a bystander, but if it were truly in-upstream-git way,
> wouldn't you be forking a branch from the tagged target release (the
> latest of 2.6.32.X), and queuing only the changes meant for -stable to
> it, and giving the name of the branch to git people and sending out
> patches from that branch for e-mailed review and application?
>
> There won't be any special tagging required, only a dedicated branch.
>
> Or am I missing something?

There's no Git flow towards -stable. It's either forwarded emails, or
tags in the upstream kernel. Also, _only_ commits that were pulled by
Linus are eligible for -stable.

So the pull requests all first go to Linus - then can any commit flow to
-stable.

But even if it was possible to send pull requests to Greg, marking
commits as -stable candidates is more natural in the commit log itself.

That informs people ('hey, that's a dangerous patch, dont mark it for
-stable!!' or 'hey, why isnt this commit tagged to stable??'), and it
also ensures it that only commits from Linus's tree flow towards
-stable.

        Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[James Bottomley-2]

On Tue, 2009-11-10 at 11:37 -0800, Greg KH wrote:

> On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> > On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > > > A further question is, i can see using this tagging scheme in the future
> > > > in merge commits log messages too - will your scripts notice that too?
> > >
> > > Hm, I don't think we look at merges as there's nothing there to actually
> > > commit.
> > >
> > > > For example if there's a few commits left in tip:*/urgent branches
> > > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > > a notification area to 'activate' them for -stable backporting via a
> > > > merge commit.
> > > >
> > > > This is how such merge commits would look like:
> > > >
> > > >   Merge branch 'core/urgent' into core/rcu
> > > >
> > > >   Merge reason: Pick up urgent fixes that did not make it into .32.0
> > > >
> > > >   Cc: <stable@...> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > >   Signed-off-by: Ingo Molnar <mingo@...>
> > > >
> > > > This is not so rare of a situation as it might seem - for the trees i
> > > > maintain it happens in almost every release cycle - i typically skip
> > > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > > fixes - but they'd still be eligible for -stable.
> > >
> > > Ok, that would be good and fine with me.
> > >
> > > James, would your script pick this up, or does it need to also pay
> > > attention to merge commits?
> >
> > No ... because merge commits should effectively be empty (and when
> > they're not you can't generate an applyable diff).  If I understand the
> > workflow, the desire is to have the whole branch sent to stable by
> > tagging the merge commit.  That's possible ... it's exactly the same
> > logic I use in the commit scripts for the SCSI tree, so it should be
> > possible to extract the logic.
> >
> > By the looks of the above it's only a few commits, or is it the entire
> > branch?
>
> I'm thinking the commit would be the merge, right Ingo?  So it would
> just be a single commit that has the marker in it.

OK, so I can make it send you this just by removing the --no-merge flag
from the git rev-list the script uses to sift through what changed
(which I've already done).

The slight problem is that further down, to generate the patch the
script uses git format-patch -k --stdout commit^..commit.  For a merge
commit, this will generate a patch equivalent to the entire branch that
was merged, even though the commit message will only pick out some of
these ... is this OK?

If not, I can look at using git show instead to generate the patches (it
will effectively generate null diffs for merge points with the stable
tag, which is closer to what you want).

Alternatively, if you pick up the commits from Linus' tree anyway, I
could just stop producing diffs, which will save email bandwidth and
then be automatically correct whether the commit is a merge or not.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Linus Torvalds-3]

On Wed, 11 Nov 2009, James Bottomley wrote:
>
> The slight problem is that further down, to generate the patch the
> script uses git format-patch -k --stdout commit^..commit.  For a merge
> commit, this will generate a patch equivalent to the entire branch that
> was merged, even though the commit message will only pick out some of
> these ... is this OK?

Don't do that.

Since you want to show a single commit, not a commit range, don't use
format-patch, use something like

        git show --pretty=email -M --cc --stat $commit

instead. That will not show the difference between a commit and its
first parent (which for a merge diff is more than one commit, and can be
absolutely _huge_), but will show just the named commit (which for a merge
commit will usually be an empty diff, but will show how conflicts were
resolved if they weren't just taken from one or the other side).

(Of course, maybe you'll want to change the exact flags in question. For
example, '--stat' for merge commits is often useful, but only if the merge
was done by an upper-level maintainer. Anybody who does reverse merges
will just get totally meaningless diffstats, since the diffstat is always
done against the first parent, so anybody who does back-merges will see
the stat of everthing _I_ have merged in the meantime, which is generally
not useful)

                Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/