lazy invoke projects
|
View:
New views
8 Messages
—
Rating Filter:
Alert me
|
 |
lazy invoke projects
by Ittay Dror
::
Rate this Message:
Hi,
The attached patch to buildr works for me
(invoking without arguments, with local task and with specific project
tasks). What do you think? Can this be included in Buildr?
The reason I'm requesting this is that I
have a buildfile with 150 projects and the current way Buildr works, by
invoking all projects takes too much time when I want to build just a
single project (and its dependencies of course). On developer windows
machine this "warmup" time is ~15 seconds and is increasing as more
modules are added. From my tests, this patch can lower it to just a few
seconds.
Ittay
-- -- Ittay Dror ittay.dror@... diff --git a/lib/buildr/core/application.rb b/lib/buildr/core/application.rb index 9898e3c..74cf7a3 100644 --- a/lib/buildr/core/application.rb +++ b/lib/buildr/core/application.rb @@ -404,7 +404,24 @@ module Buildr end end load_imports - Buildr.projects + #Buildr.projects + top_level_tasks.each do |task_name| + if task_name.index(':') + parts = [] + task_name.split(':').each do |part| + parts << part + begin + Buildr.project(parts.join(':')).invoke + rescue + nil + end + end + else + Buildr.projects.each do |project| + project.invoke + end + end + end end # Load/install all Gems specified in build.yaml file. diff --git a/lib/buildr/core/project.rb b/lib/buildr/core/project.rb index 959e863..d96a604 100644 --- a/lib/buildr/core/project.rb +++ b/lib/buildr/core/project.rb @@ -222,12 +222,12 @@ module Buildr @on_define.each { |callback| callback[project] } end if @on_define # Enhance the project using the definition block. - project.enhance { project.instance_eval &block } if block + project.enhance {|project| project.instance_eval &block } if block # Top-level project? Invoke the project definition. Sub-project? We don't invoke # the project definiton yet (allow project calls to establish order of evaluation), - # but must do so before the parent project's definition is done. - project.parent.enhance { project.invoke } if project.parent + # but must do so before the parent project's definition is done. + ### project.parent.enhance { project.invoke } if project.parent end end @@ -242,14 +242,18 @@ module Buildr @projects ||= {} name = args.first if options && options[:scope] - # We assume parent project is evaluated. - project = options[:scope].split(':').inject([[]]) { |scopes, scope| scopes << (scopes.last + [scope]) }. - map { |scope| @projects[(scope + [name]).join(':')] }. - select { |project| project }.last + scope = options[:scope].split(':').inject([[]]) { |scopes, scope| scopes << (scopes.last + [scope]) }. + reverse. # get all prefixes to name from longest to shortest + find do |scope| + name.split(':').inject([[]]) {|parts, part| parts << (parts.last + [part])}. + all? {|names| @projects[(scope + names).join(':')].tap {|project| project.invoke if project}} + end + project = @projects[(scope + [name]).join(':')] if scope end unless project # Parent project not evaluated. - name.split(':').tap { |parts| @projects[parts.first].invoke if parts.size > 1 } + name.split(':').inject([[]]) { |parts, part| parts << (parts.last + [part])}. + each {|scope| @projects[scope.join(':')].tap {|project| project.invoke if project}} project = @projects[name] end raise "No such project #{name}" unless project @@ -391,6 +395,11 @@ module Buildr # The parent project if this is a sub-project. attr_reader :parent + def invoke(*args) + return if @already_invoked + super + end + def initialize(*args) #:nodoc: super split = name.split(':') |
|
|
Re: lazy invoke projects
by Assaf Arkin
::
Rate this Message:
On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <ittay.dror@...> wrote:
> > The reason I'm requesting this is that I have a buildfile with 150 projects > and the current way Buildr works, by invoking all projects takes too much > time when I want to build just a single project (and its dependencies of > course). On developer windows machine this "warmup" time is ~15 seconds and > is increasing as more modules are added. From my tests, this patch can lower > it to just a few seconds. The reason I asked you to bring this back to the mailing list is the size issue. The projects I'm working on have, in the order of 1000~5000 files (not just source code), 5~25 projects, warmup around a second. Occasionally I would test things out against a larger corpus of code, ~50,000 files, I'm happy when that turns to go under 5 seconds. Only because I consider that at the top scale of what a single buildfile is expected to support. There are certain optimizations you can make that would benefits the smaller sizes (the 1000~5000 above), and would be acceptable on larger size (~50,000), but might be a drag on larger buildfiles. I never test with anything larger because I think the soft spot, the majority of uses fall in the smaller scale. Anything substantially larger you would want to (well, at least me) break up. So I'm wondering where do other people stand on this? What's the size range you expect buildfiles to be optimized for? Assaf > > Ittay > > -- > -- > Ittay Dror <ittay.dror@...> > |
|
|
Re: lazy invoke projects
by Ittay Dror-2
::
Rate this Message:
Assaf Arkin wrote: > On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <ittay.dror@...> wrote: > >> The reason I'm requesting this is that I have a buildfile with 150 projects >> and the current way Buildr works, by invoking all projects takes too much >> time when I want to build just a single project (and its dependencies of >> course). On developer windows machine this "warmup" time is ~15 seconds and >> is increasing as more modules are added. From my tests, this patch can lower >> it to just a few seconds. >> > > The reason I asked you to bring this back to the mailing list is the size issue. > > The projects I'm working on have, in the order of 1000~5000 files (not > just source code), 5~25 projects, warmup around a second. > Occasionally I would test things out against a larger corpus of code, > ~50,000 files, I'm happy when that turns to go under 5 seconds. Only > because I consider that at the top scale of what a single buildfile is > expected to support. > > also, in my case, buildr is used to build c++ modules, where unlike java, you need to run the compilation frequently. so even 5 seconds is annoying. > There are certain optimizations you can make that would benefits the > smaller sizes (the 1000~5000 above), and would be acceptable on larger > size (~50,000), but might be a drag on larger buildfiles. I never > test with anything larger because I think the soft spot, the majority > of uses fall in the smaller scale. Anything substantially larger you > would want to (well, at least me) break up. > majority of users in general, yes. but out of those users who will look into buildr? users with small scale projects, can probably manage with ant+ivy or maven. they will not need to learn a new language, there's a better community, more plugins, etc. so i think the sweet spot of buildr is in complex projects. i do want to break the buildfile, but unfortunately i can't, the projects are very interconnected. it is enough to have one dependency to make two otherwise separate groups be in one file. doesn't make the code complex anyway, i think the patch i sent is fairly simple. it took me a few hours to create, so why not incorporate it? ittay > So I'm wondering where do other people stand on this? What's the size > range you expect buildfiles to be optimized for? > > Assaf > > > >> Ittay >> >> -- >> -- >> Ittay Dror <ittay.dror@...> >> >> > > -- Ittay Dror <ittayd@...> Tikal <http://www.tikalk.com> Tikal Project <http://tikal.sourceforge.net> |
|
|
Re: lazy invoke projects
by Ittay Dror
::
Rate this Message:
ittay Ittay Dror wrote:
-- -- Ittay Dror ittay.dror@... >From f2604a0f035a3f8534e8b917219a8720206b76aa Mon Sep 17 00:00:00 2001 From: Ittay Dror <ittay.dror@...> Date: Wed, 5 Nov 2008 00:19:39 +0200 Subject: [PATCH] lazily invoking for more performance --- lib/buildr/core/application.rb | 19 ++++++++++++++++++- lib/buildr/core/project.rb | 34 +++++++++++++++++++++++++--------- 2 files changed, 43 insertions(+), 10 deletions(-) diff --git a/lib/buildr/core/application.rb b/lib/buildr/core/application.rb index 9898e3c..74cf7a3 100644 --- a/lib/buildr/core/application.rb +++ b/lib/buildr/core/application.rb @@ -404,7 +404,24 @@ module Buildr end end load_imports - Buildr.projects + #Buildr.projects + top_level_tasks.each do |task_name| + if task_name.index(':') + parts = [] + task_name.split(':').each do |part| + parts << part + begin + Buildr.project(parts.join(':')).invoke + rescue + nil + end + end + else + Buildr.projects.each do |project| + project.invoke + end + end + end end # Load/install all Gems specified in build.yaml file. diff --git a/lib/buildr/core/project.rb b/lib/buildr/core/project.rb index 959e863..c6cb3d2 100644 --- a/lib/buildr/core/project.rb +++ b/lib/buildr/core/project.rb @@ -222,12 +222,12 @@ module Buildr @on_define.each { |callback| callback[project] } end if @on_define # Enhance the project using the definition block. - project.enhance { project.instance_eval &block } if block + project.enhance {|project| project.instance_eval &block } if block # Top-level project? Invoke the project definition. Sub-project? We don't invoke # the project definiton yet (allow project calls to establish order of evaluation), - # but must do so before the parent project's definition is done. - project.parent.enhance { project.invoke } if project.parent + # but must do so before the parent project's definition is done. + ### project.parent.enhance { project.invoke } if project.parent end end @@ -242,14 +242,18 @@ module Buildr @projects ||= {} name = args.first if options && options[:scope] - # We assume parent project is evaluated. - project = options[:scope].split(':').inject([[]]) { |scopes, scope| scopes << (scopes.last + [scope]) }. - map { |scope| @projects[(scope + [name]).join(':')] }. - select { |project| project }.last + scope = options[:scope].split(':').inject([[]]) { |scopes, scope| scopes << (scopes.last + [scope]) }. + reverse. # get all prefixes to name from longest to shortest + find do |scope| + name.split(':').inject([[]]) {|parts, part| parts << (parts.last + [part])}. + all? {|names| @projects[(scope + names).join(':')].tap {|project| project.invoke if project}} + end + project = @projects[(scope + [name]).join(':')] if scope end unless project # Parent project not evaluated. - name.split(':').tap { |parts| @projects[parts.first].invoke if parts.size > 1 } + name.split(':').inject([[]]) { |parts, part| parts << (parts.last + [part])}. + each {|scope| @projects[scope.join(':')].tap {|project| project.invoke if project}} project = @projects[name] end raise "No such project #{name}" unless project @@ -391,6 +395,11 @@ module Buildr # The parent project if this is a sub-project. attr_reader :parent + def invoke(*args) + return if @already_invoked + super + end + def initialize(*args) #:nodoc: super split = name.split(':') @@ -537,9 +546,16 @@ module Buildr def recursive_task(*args, &block) task_name, arg_names, deps = Buildr.application.resolve_args(args) task = Buildr.options.parallel ? multitask(task_name) : task(task_name) - parent.task(task_name).enhance [task] if parent task.set_arg_names(arg_names) unless arg_names.empty? task.enhance Array(deps), &block + + + task.instance_variable_set :@project, self + def task.invoke_prerequisites(*args) + task_name = name.sub(@project.name + ':', '') + @prerequisites |= @project.projects.map {|project| project.task(task_name)} + super + end end # :call-seq: -- 1.6.0.3.640.g6331a |
|
|
Re: lazy invoke projects
by Assaf Arkin
::
Rate this Message:
On Wed, Nov 5, 2008 at 10:17 PM, Ittay Dror <ittayd@...> wrote:
> > > Assaf Arkin wrote: >> >> On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <ittay.dror@...> wrote: >> >>> >>> The reason I'm requesting this is that I have a buildfile with 150 >>> projects >>> and the current way Buildr works, by invoking all projects takes too much >>> time when I want to build just a single project (and its dependencies of >>> course). On developer windows machine this "warmup" time is ~15 seconds >>> and >>> is increasing as more modules are added. From my tests, this patch can >>> lower >>> it to just a few seconds. >>> >> >> The reason I asked you to bring this back to the mailing list is the size >> issue. >> >> The projects I'm working on have, in the order of 1000~5000 files (not >> just source code), 5~25 projects, warmup around a second. >> Occasionally I would test things out against a larger corpus of code, >> ~50,000 files, I'm happy when that turns to go under 5 seconds. Only >> because I consider that at the top scale of what a single buildfile is >> expected to support. >> >> > > did you test on windows machines? Yes. Give or take same performance. > also, in my case, buildr is used to build c++ modules, where unlike java, > you need to run the compilation frequently. so even 5 seconds is annoying. >> >> There are certain optimizations you can make that would benefits the >> smaller sizes (the 1000~5000 above), and would be acceptable on larger >> size (~50,000), but might be a drag on larger buildfiles. I never >> test with anything larger because I think the soft spot, the majority >> of uses fall in the smaller scale. Anything substantially larger you >> would want to (well, at least me) break up. >> > > majority of users in general, yes. but out of those users who will look into > buildr? users with small scale projects, can probably manage with ant+ivy or > maven. they will not need to learn a new language, there's a better > community, more plugins, etc. so i think the sweet spot of buildr is in > complex projects. That's why I'm asking the community, and I'm looking for data points about size not complexity. > i do want to break the buildfile, but unfortunately i can't, the projects > are very interconnected. it is enough to have one dependency to make two > otherwise separate groups be in one file. doesn't make the code complex > > anyway, i think the patch i sent is fairly simple. it took me a few hours to > create, so why not incorporate it? If the patch works for you, you are more than welcome to use it. I have to consider everything else that gets affected, beyond the needs of a single project. Assaf > > ittay >> >> So I'm wondering where do other people stand on this? What's the size >> range you expect buildfiles to be optimized for? >> >> Assaf >> >> >> >>> >>> Ittay >>> >>> -- >>> -- >>> Ittay Dror <ittay.dror@...> >>> >>> >> >> > > -- > Ittay Dror <ittayd@...> > Tikal <http://www.tikalk.com> > Tikal Project <http://tikal.sourceforge.net> > > |
|
|
Re: lazy invoke projects
by Ittay Dror-2
::
Rate this Message:
Assaf Arkin wrote: > If the patch works for you, you are more than welcome to use it. I > have to consider everything else that gets affected, beyond the needs > of a single project. > > can you please elaborate? Ittay > Assaf > > >> ittay >> >>> So I'm wondering where do other people stand on this? What's the size >>> range you expect buildfiles to be optimized for? >>> >>> Assaf >>> >>> >>> >>> >>>> Ittay >>>> >>>> -- >>>> -- >>>> Ittay Dror <ittay.dror@...> >>>> >>>> >>>> >>> >> -- >> Ittay Dror <ittayd@...> >> Tikal <http://www.tikalk.com> >> Tikal Project <http://tikal.sourceforge.net> >> >> >> > > -- Ittay Dror <ittayd@...> Tikal <http://www.tikalk.com> Tikal Project <http://tikal.sourceforge.net> |
|
|
Re: lazy invoke projects
by Assaf Arkin
::
Rate this Message:
On Thu, Nov 6, 2008 at 12:04 PM, Ittay Dror <ittayd@...> wrote:
> > > Assaf Arkin wrote: >> >> If the patch works for you, you are more than welcome to use it. I >> have to consider everything else that gets affected, beyond the needs >> of a single project. >> >> > > can you please elaborate? That would be the next step, if we decided that this path is worth pursuing. Assaf > > Ittay >> >> Assaf >> >> >>> >>> ittay >>> >>>> >>>> So I'm wondering where do other people stand on this? What's the size >>>> range you expect buildfiles to be optimized for? >>>> >>>> Assaf >>>> >>>> >>>> >>>> >>>>> >>>>> Ittay >>>>> >>>>> -- >>>>> -- >>>>> Ittay Dror <ittay.dror@...> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> -- >>> Ittay Dror <ittayd@...> >>> Tikal <http://www.tikalk.com> >>> Tikal Project <http://tikal.sourceforge.net> >>> >>> >>> >> >> > > -- > Ittay Dror <ittayd@...> > Tikal <http://www.tikalk.com> > Tikal Project <http://tikal.sourceforge.net> > > |
|
|
Re: lazy invoke projects
by lacton
::
Rate this Message:
I usually have about 10 projects and 1000 files.
Maybe this question would be better on buildr-users instead of -dev, to involve more of the community. Lacton On Wed, Nov 5, 2008 at 11:57 PM, Assaf Arkin <arkin@...> wrote: > On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <ittay.dror@...> wrote: >> >> The reason I'm requesting this is that I have a buildfile with 150 projects >> and the current way Buildr works, by invoking all projects takes too much >> time when I want to build just a single project (and its dependencies of >> course). On developer windows machine this "warmup" time is ~15 seconds and >> is increasing as more modules are added. From my tests, this patch can lower >> it to just a few seconds. > > The reason I asked you to bring this back to the mailing list is the size issue. > > The projects I'm working on have, in the order of 1000~5000 files (not > just source code), 5~25 projects, warmup around a second. > Occasionally I would test things out against a larger corpus of code, > ~50,000 files, I'm happy when that turns to go under 5 seconds. Only > because I consider that at the top scale of what a single buildfile is > expected to support. > > There are certain optimizations you can make that would benefits the > smaller sizes (the 1000~5000 above), and would be acceptable on larger > size (~50,000), but might be a drag on larger buildfiles. I never > test with anything larger because I think the soft spot, the majority > of uses fall in the smaller scale. Anything substantially larger you > would want to (well, at least me) break up. > > So I'm wondering where do other people stand on this? What's the size > range you expect buildfiles to be optimized for? > > Assaf > > >> >> Ittay >> >> -- >> -- >> Ittay Dror <ittay.dror@...> >> > |
| Free embeddable forum powered by Nabble | Forum Help |
