Improving our file operations support + archive handling

[hdockter]

One of the things I would like to start soon with, is to improve our  
file operations support (Delete, Move, archive handling)

I think Steve's copy stuff is leading the way. For each operation we  
want to have a task as well as a project method as well as a spec. So  
we will have a move/delete/copy/zip/jar/tar task/spec/method.  
Specially for the archive tasks this provides the nice feature that  
specs can be reused for multiple archive operations. For example in  
the Gradle build the different distributions would share a common spec  
(and then have additionally specific specs). The archive specs should  
extend from the copy specs so that copy operations and archive  
operations can share specs. Another thing that would be good to have  
are archives as input(from) values for copy/archives operations. That  
way we can use the copy task to unzip/tar/jar. For archives this would  
provide the current merge functionality.

Thoughts?

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org






---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Adam Murdoch-2]

Hans Dockter wrote:
> One of the things I would like to start soon with, is to improve our
> file operations support (Delete, Move, archive handling)
>
> I think Steve's copy stuff is leading the way.

+1

> For each operation we want to have a task as well as a project method
> as well as a spec. So we will have a move/delete/copy/zip/jar/tar
> task/spec/method. Specially for the archive tasks this provides the
> nice feature that specs can be reused for multiple archive operations.
> For example in the Gradle build the different distributions would
> share a common spec (and then have additionally specific specs). The
> archive specs should extend from the copy specs so that copy
> operations and archive operations can share specs.

I think most (or all?) of the stuff on ZipFileSet and TarFileSet could
live on CopySpec and be supported by both the copy action and archive
actions. For example, prefix (which is almost there in the into
property), filemode, dirmode, etc.

I would change the 'into' property to be a String path relative to the
destination directory/archive root, and allow into() to optionally take
a configure closure:

zip {
    into 'META-INF' {
        from 'src/metaInf'
        from '../shared/src/metaInfo' { include '**/staging/**' }
    }
}

And maybe (as, I think, Tom suggested a while back) support a builder style:

zip {
    bin(filemode: '755') {
        from 'src/toplevel/scripts'
    }
    libs {
        from configurations.distLibs
    }
}

> Another thing that would be good to have are archives as input(from)
> values for copy/archives operations. That way we can use the copy task
> to unzip/tar/jar. For archives this would provide the current merge
> functionality.
>

One question is how we distinguish between the case where you want to
copy the archive and the case where you want to copy the contents of the
archive.

One option is to have a method which takes an archive file and returns a
FileTree containing its contents:

copy {
    from 'some.zip' // copies the file
    from contentsOf('some.zip') // copies the contents of the file
}

This allows the expanding stuff to be reused elsewhere:

contentsOf('some.zip').matching { include 'org/gradle/**' }.visit {
println it.relativePath }


There's a few other things which might be useful for our file/archive
handling:

- Copy and archive actions preserve file permissions (as best they can)

- Allow an Ant pattern to be specified anywhere that accepts multiple
files as a shorthand:

files('lib/*.zip')
compileJava.source('src/*/java')
copy(from: 'src/*', into: ...)


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Tomek Kaczanowski-3]

Hi,

one comment - the idea is very nice, but let us not forget that all of
this can be achieved using ant, so I'd give this task a low priority
and concentrate on things that Gradle still lacks.

--
Tomek

2009/10/2 Adam Murdoch <a@...>:

>
>
> Hans Dockter wrote:
>>
>> One of the things I would like to start soon with, is to improve our file
>> operations support (Delete, Move, archive handling)
>>
>> I think Steve's copy stuff is leading the way.
>
> +1
>
>> For each operation we want to have a task as well as a project method as
>> well as a spec. So we will have a move/delete/copy/zip/jar/tar
>> task/spec/method. Specially for the archive tasks this provides the nice
>> feature that specs can be reused for multiple archive operations. For
>> example in the Gradle build the different distributions would share a common
>> spec (and then have additionally specific specs). The archive specs should
>> extend from the copy specs so that copy operations and archive operations
>> can share specs.
>
> I think most (or all?) of the stuff on ZipFileSet and TarFileSet could live
> on CopySpec and be supported by both the copy action and archive actions.
> For example, prefix (which is almost there in the into property), filemode,
> dirmode, etc.
>
> I would change the 'into' property to be a String path relative to the
> destination directory/archive root, and allow into() to optionally take a
> configure closure:
>
> zip {
>   into 'META-INF' {
>       from 'src/metaInf'
>       from '../shared/src/metaInfo' { include '**/staging/**' }
>   }
> }
>
> And maybe (as, I think, Tom suggested a while back) support a builder style:
>
> zip {
>   bin(filemode: '755') {
>       from 'src/toplevel/scripts'
>   }
>   libs {
>       from configurations.distLibs
>   }
> }
>
>> Another thing that would be good to have are archives as input(from)
>> values for copy/archives operations. That way we can use the copy task to
>> unzip/tar/jar. For archives this would provide the current merge
>> functionality.
>>
>
> One question is how we distinguish between the case where you want to copy
> the archive and the case where you want to copy the contents of the archive.
>
> One option is to have a method which takes an archive file and returns a
> FileTree containing its contents:
>
> copy {
>   from 'some.zip' // copies the file
>   from contentsOf('some.zip') // copies the contents of the file
> }
>
> This allows the expanding stuff to be reused elsewhere:
>
> contentsOf('some.zip').matching { include 'org/gradle/**' }.visit { println
> it.relativePath }
>
>
> There's a few other things which might be useful for our file/archive
> handling:
>
> - Copy and archive actions preserve file permissions (as best they can)
>
> - Allow an Ant pattern to be specified anywhere that accepts multiple files
> as a shorthand:
>
> files('lib/*.zip')
> compileJava.source('src/*/java')
> copy(from: 'src/*', into: ...)
>
>
> Adam
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

On Oct 2, 2009, at 6:38 AM, Tomek Kaczanowski wrote:

> Hi,
>
> one comment - the idea is very nice, but let us not forget that all of
> this can be achieved using ant, so I'd give this task a low priority
> and concentrate on things that Gradle still lacks.

This is a good point. On the other hand we want to reach 1.0 in the  
not too distant future and this is an important element of our API. It  
also has some urgency in that we have introduced new file system  
objects like FileTree which are used by the source sets. Using them  
now when creating an archive is pretty ugly for example if you want to  
store in a non root path in the archive.

What is your top list of missing features?

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Adam Murdoch-2]

Tomek Kaczanowski wrote:
> Hi,
>
> one comment - the idea is very nice, but let us not forget that all of
> this can be achieved using ant,

That might not be true for much longer. The key to these changes is that
they provide a declarative approach to defining the contents of a file
hierarchy, rather than the imperative approach offered by using Ant tasks.

That is, I can declare in one place the contents of my distribution, and
that same declaration can be reused in multiple places: to create a zip,
or tar, or exploded dist. Or later a self-extracting exe or a .dmg or an
os x app bundle, or whatever. Same with the contents of a war. I can
declare it once, and use it to create a war and an exploded war. The
jettyRun task can also make use of it, to configure the jetty web
context using the declaration of the war contents.

And because the API is declarative, it means Gradle can make use of it.
It will be integrated into the incremental builds of Gradle 0.9, so that
the archive will be rebuilt only if the set of input files changes. This
doesn't work with the Ant tasks. You get false positives when the
timestamp of an input file changes but the contents of the file hasn't.
And you get false negatives when the set of files change but the
timestamps haven't. And this means you almost always do a clean so that
you get reliable results. Using the declarative API + incremental builds
means you won't have to.

The API will also be integrated into the dependency auto-wiring of
Gradle 0.9. Because the API is declarative, Gradle knows what the input
files will be, and can figure out where they come from, to add in the
appropriate task dependencies. It won't be able to do that for the Ant
tasks.

To me, these changes are important. Almost every build customises the
archives it builds. Without these changes, these builds won't be able to
(easily) make use of some of the key 0.9 features.

>  so I'd give this task a low priority
> and concentrate on things that Gradle still lacks.
>
>  

This is an interesting point. We do lack any real input from the
community into prioritisation of the features for a release. We (the
Gradle devs) pretty much just choose the things we think are important,
but we really are just guessing. What could we do to improve this?

One option is to use the roadmap. Early in the release cycle, we would
update the roadmap with what we plan to do, and then invite the
community to comment on it.

Thoughts?


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[LightGuard]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On Mon, Oct 5, 2009 at 19:14, Adam Murdoch  wrote:

>
>
> Tomek Kaczanowski wrote:
>>
>> Hi,
>>
>> one comment - the idea is very nice, but let us not forget that all of
>> this can be achieved using ant,
>
> That might not be true for much longer. The key to these changes is that
> they provide a declarative approach to defining the contents of a file
> hierarchy, rather than the imperative approach offered by using Ant tasks.
>
> That is, I can declare in one place the contents of my distribution, and
> that same declaration can be reused in multiple places: to create a zip, or
> tar, or exploded dist. Or later a self-extracting exe or a .dmg or an os x
> app bundle, or whatever. Same with the contents of a war. I can declare it
> once, and use it to create a war and an exploded war. The jettyRun task can
> also make use of it, to configure the jetty web context using the
> declaration of the war contents.
>
> And because the API is declarative, it means Gradle can make use of it. It
> will be integrated into the incremental builds of Gradle 0.9, so that the
> archive will be rebuilt only if the set of input files changes. This doesn't
> work with the Ant tasks. You get false positives when the timestamp of an
> input file changes but the contents of the file hasn't. And you get false
> negatives when the set of files change but the timestamps haven't. And this
> means you almost always do a clean so that you get reliable results. Using
> the declarative API + incremental builds means you won't have to.
>
> The API will also be integrated into the dependency auto-wiring of Gradle
> 0.9. Because the API is declarative, Gradle knows what the input files will
> be, and can figure out where they come from, to add in the appropriate task
> dependencies. It won't be able to do that for the Ant tasks.
>
> To me, these changes are important. Almost every build customises the
> archives it builds. Without these changes, these builds won't be able to
> (easily) make use of some of the key 0.9 features.
>
>>  so I'd give this task a low priority
>> and concentrate on things that Gradle still lacks.
>>
>>
>
> This is an interesting point. We do lack any real input from the community
> into prioritisation of the features for a release. We (the Gradle devs)
> pretty much just choose the things we think are important, but we really are
> just guessing. What could we do to improve this?
>
> One option is to use the roadmap. Early in the release cycle, we would
> update the roadmap with what we plan to do, and then invite the community to
> comment on it.
>
> Thoughts?
>

I like the idea of using the road map.  I'd suggest making an
announcement to the mailing list and on the site.  Make sure people
are using the vote feature of JIRA.  Not really sure what else can
really be done.

>
> Adam
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>
>

- --
Jason Porter
Real Programmers think better when playing Adventure or Rogue.

PGP key id: 926CCFF5
PGP fingerprint: 64C2 C078 13A9 5B23 7738 F7E5 1046 C39B 926C CFF5
PGP key available at: keyserver.net, pgp.mit.edu


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.9)

iEYEAREIAAYFAkrLX4AACgkQEEbDm5Jsz/ULOgCgq1kik4hFLk4xfFk3aP4Q6Ppr
WzEAnRfmdH9Nw7gsN/cZFHL13YQMv0cw
=lGqB
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Tomek Kaczanowski-3]

Hi Adam,

> One option is to use the roadmap. Early in the release cycle, we would
> update the roadmap with what we plan to do, and then invite the community to
> comment on it.
People can already state their opinions using voting system on JIRA.
Judging by the vote numbers that I observe, not many people is using
this. Maybe you should encourage them more on mailing lists ?

--
Tomek

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Tomek Kaczanowski-3]

Hi Hans,

> What is your top list of missing features?
That is a good question. I haven't really thought about this.

Unfortunately JIRA is down, so I can't check issues right now. But I
remember some things:
  - update BND version - AFAIK Gradle uses some archaic version of
bnd, thus the bundles it creates are not perfect
  - make it possible for Gradle to do what other tools can, e.g.:
    - execute single tests
    - provide archetypes (to easily start projects using gradle)
    - provide easy way to integrate with other libraries (Cobertura, Aspectj)
  - make things that help people convert from other tools (e.g. there
was a ticket regarding XSLT transormation of pom.xml to print
dependencies in Gradle's format)

I think Gradle is quite mature, and offers a lot of super
functionality. What it lacks IMHO is a short way to achieve results. I
mean, not everyone will be willing to read the userguide. I expect
many people to just google for "Gradle Selenium" or "use Gradle with
AspecJ" and leave disappointed because there is no easy 3-lines code
snippet that they can use "just like that". That's why I'd put a lot
of effort into explaining how to do typical stuff with Gradle (yeah,
the cookbook is the place I'd love to see such explanations).

BTW. Sometimes on the mailing list you can find emails - especially
written by you and Adam (but also others) - that explains brilliantly
what are the features of Gradle and where its strength lies. For
example the recent one titled "Using modules to separate JDK
dependencies". I wonder if such things can be somehow included into
the userguide... They are worth "publishing", that is for sure !

--
Tomek

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email