« Return to Thread: Native test framework integration

Re: Native test framework integration

by Tom Eyckmans :: Rate this Message:

2009/6/3 Adam Murdoch <a@...>

Tom Eyckmans wrote:

Hi all,

just started a testing page http://docs.codehaus.org/display/GRADLE/Testing with my first thought on how to provide control over multiple VM forks to execute tests.

Your proposal looks like it is trying to solve 3 problems:

1. Defining a group of tests which need to run as a batch, with some settings about how to run those tests

2. Defining a workflow for these test groups, so that some are executed sequentially and some are executed in parallel

3. Defining composite test groups, each made up of other test groups

I think problem 1 is the only thing specific to testing. Problems 2 and 3 really apply to anything that Gradle does, and so would be better done as general-purpose facilities, rather than something fundamental to a test suite.

I agree I was also thinking in that direction.

For example, given that a task ends up being defined for each test suite, problem 2 could be solved by some facility to mark up which tasks should run in parallel. Or, a better option would be to use a constraint-based model and let Gradle figure out which tasks to run in parallel or sequentially, similar to what we do with task dependencies.

That would be cool but with the test task this really depends on whether the tests are written in a way that they can be executed in parallel. For example our integration tests, you could run these in parallel per integration test project, there is no way for Gradle to detect in some way that these can be executed in parallel. I think a constraint based model is really nice idea I'm not sure the serial / parallel blocks are sufficient for a general purpose facility. Also when a series of batch tests are executed, multiple test tasks are executed in serially or in parallel the test result xml files would be generated in the Gradle VM, how would we model this with serial / parallel in a generic way without troubling users with the inner workings of how test execution works? We might also allow naming the batch test groups so specific batch test groups are easily found in the test report.

We already have examples of problem 3: the 'libs' task builds all the JARs and WARs in the project, 'buildArtifacts' task builds the artifacts published by the project. We have solved these so far using task dependencies. We could do the same with grouping the execution of test suites. Or we could introduce some kind of domain object collection concept. I'm not so sure we need anything particularly comprehensive here for testing. In my experience, different test suites execute at very different points in a project's lifecycle (and usually in different builds), so there's little practical benefit in actually grouping them together.

So, I think you could trim the test suite proposal down to:

main(TestSuite) {
exclude '**/integtest/**'
useJUnit()
options.forkOptions.forkMode = ForkMode.Once
}

integTests(TestSuite) {
include '**/integtest/**'
useJUnit()
}

On the syntax: We're defining a thing with a type and a name here, just like tasks, or repositories (or source dirs etc). We have a syntax for defining these things, and we should use the same syntax for defining test suites as well.

I agree.

This could mean changing how we define tasks or repositories, or it could mean using the existing syntax for defining test suites.

I'll use the existing syntax.

Using the existing syntax, this means declaring test suites like

testsuites {
main {
... some options ...
}
}

or

testsuite main {
... some options ...
}

I think the test framework should be fundamental to the type of the test suite:

I agree

testsuites {
main(type: JUnit) { ... }
another(type: TestNg) { ... }
}

Where JUnit would be the default. Amongst other things, this typing means we can flatten the various options objects into the appropriate TestSuite subclass.

That is a really nice :)

Adam

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

« Return to Thread: Native test framework integration