Continuous Integration command

[Roberto Aguilar]

Hello developers,

I wanted to be able to run a project's test suite in the same manner
runserver restarts when a file is changed.  I did not find this
feature in Django so I wrote the following code:

http://www.djangosnippets.org/snippets/1763/

I think it would be useful to others and would like to propose to add
it into the core Django project.

Thanks,
-berto.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Russell Keith-Magee-2]

On Sun, Oct 18, 2009 at 12:59 AM, berto <roberto.c.aguilar@...> wrote:

>
> Hello developers,
>
> I wanted to be able to run a project's test suite in the same manner
> runserver restarts when a file is changed.  I did not find this
> feature in Django so I wrote the following code:
>
> http://www.djangosnippets.org/snippets/1763/
>
> I think it would be useful to others and would like to propose to add
> it into the core Django project.

This is an interesting proposal. I'm not completely convinced, though,
for a couple of reasons.

1) runserver is not especially robust to certain types of errors, such
as SyntaxError. I will admit that I haven't tried the snippet that you
referenced, but it looks like it will be prone to the same problem.
I'd like to fix this problem, but while this problem exists, adding a
continuous integration server that crashes every time the developer
makes a Syntax Error doesn't sound like an especially good idea.

2) I'm not necessarily convinced that a CI services should be running
on a developers box during development. Django's test suite takes a
long time to run - I've heard reports of test durations measured in
*hours* for some setups. Test durations like this aren't really
compatible with a "retest on every file change" CI approach. Given the
current state of Django's test framework, I'm more convinced that a CI
server based on checkins is more appropriate.

3) Most of my hesitation on (2) is driven by the fact that
setup/teardown is such an expensive operation. Most of the time spend
running the test suite isn't spent running tests - it's spent setting
up and tearing down tests. We've made some big improvements by using
transactions to speed up this process, but it's still slow. If you
want to convince me that per-file-modification CI is worthwhile,
you'll need to work out a way to dramatically reduce the
setup/teardown time for tests.

4) An alternate approach is to come up with a smarter way to run tests
so that the tests that are important to the file being modified
(whatever that means) are run more often, rather than just blanket
running the entire suite. This might mean reducing the CI process to a
Monte Carlo style sampling that eventually results in 100% coverage if
the file base remains stable for long enough.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Roberto Aguilar]

On Oct 17, 5:33 pm, Russell Keith-Magee <freakboy3...@...>
wrote:

> On Sun, Oct 18, 2009 at 12:59 AM, berto <roberto.c.agui...@...> wrote:
>
> > Hello developers,
>
> > I wanted to be able to run a project's test suite in the same manner
> > runserver restarts when a file is changed.  I did not find this
> > feature in Django so I wrote the following code:
>
> >http://www.djangosnippets.org/snippets/1763/
>
> > I think it would be useful to others and would like to propose to add
> > it into the core Django project.
>
> This is an interesting proposal. I'm not completely convinced, though,
> for a couple of reasons.
>
> 1) runserver is not especially robust to certain types of errors, such
> as SyntaxError. I will admit that I haven't tried the snippet that you
> referenced, but it looks like it will be prone to the same problem.
> I'd like to fix this problem, but while this problem exists, adding a
> continuous integration server that crashes every time the developer
> makes a Syntax Error doesn't sound like an especially good idea.

That's an excellent point.  My initial implementation did have the
same problem, but I have since fixed it as well as runserver.  The
changes are posted on github:

- autoreload patch adds checking for modifications on files in an
errored state
  http://github.com/rca/django/commit/ef96339f7e766700e8739a591b226b63b91cdcf7

- runserver patch adds files that have errors to the autoreload code
changed loop
  http://github.com/rca/django/commit/e1d0be02b001155a09b012f4867de1efd14454b4

- updated version of runtester using the functionality above:
  http://github.com/rca/django/commit/043e9723ae8fdb98745c2390f3ff92649bc106df
  - and a minor patch to it:
    http://github.com/rca/django/commit/c1d62dbb58b1113a864ed5b1d35dd0b60ec2c071

> 2) I'm not necessarily convinced that a CI services should be running
> on a developers box during development.

Ah, but when doing test-driven development, it's what one does.  My
current workflow is:

while True:
  write_tests()
  while are_tests_failing():
    write_code()
    run_tests()

> Django's test suite takes a
> long time to run - I've heard reports of test durations measured in
> *hours* for some setups. Test durations like this aren't really
> compatible with a "retest on every file change" CI approach. Given the
> current state of Django's test framework, I'm more convinced that a CI
> server based on checkins is more appropriate.

I can't really argue against your point that, if a test suite takes
hours, run it elsewhere.

But, just like the test command, the runtester command takes a list of
apps, or even a TestCase class within an app.  That way you are simply
running a subset of the tests on the app actively being worked on.
For example:

./manage.py runtester some_app.SubTest

With the above command, only the SubTest tests are run, which will
drastically reduce the time to run the tests.  Granted time to setup
and teardown of the test environment is still there.

> 3) Most of my hesitation on (2) is driven by the fact that
> setup/teardown is such an expensive operation. Most of the time spend
> running the test suite isn't spent running tests - it's spent setting
> up and tearing down tests. We've made some big improvements by using
> transactions to speed up this process, but it's still slow. If you
> want to convince me that per-file-modification CI is worthwhile,
> you'll need to work out a way to dramatically reduce the
> setup/teardown time for tests.

Much of this concern might be mitigated by listing the desired apps/
test cases to test when the command is run as described above.

> 4) An alternate approach is to come up with a smarter way to run tests
> so that the tests that are important to the file being modified
> (whatever that means) are run more often, rather than just blanket
> running the entire suite. This might mean reducing the CI process to a
> Monte Carlo style sampling that eventually results in 100% coverage if
> the file base remains stable for long enough.

Quite an interesting idea.  I can't afford to spend the time to really
hash this concept out though.  :\

Thanks for the reply,
-berto.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Russell Keith-Magee-2]

On Sun, Oct 18, 2009 at 6:14 PM, berto <roberto.c.aguilar@...> wrote:

>
> On Oct 17, 5:33 pm, Russell Keith-Magee <freakboy3...@...>
> wrote:
>> I'd like to fix this problem, but while this problem exists, adding a
>> continuous integration server that crashes every time the developer
>> makes a Syntax Error doesn't sound like an especially good idea.
>
> That's an excellent point.  My initial implementation did have the
> same problem, but I have since fixed it as well as runserver.  The
> changes are posted on github:
>
> - autoreload patch adds checking for modifications on files in an
> errored state
>  http://github.com/rca/django/commit/ef96339f7e766700e8739a591b226b63b91cdcf7
>
> - runserver patch adds files that have errors to the autoreload code
> changed loop
>  http://github.com/rca/django/commit/e1d0be02b001155a09b012f4867de1efd14454b4
>
> - updated version of runtester using the functionality above:
>  http://github.com/rca/django/commit/043e9723ae8fdb98745c2390f3ff92649bc106df
>  - and a minor patch to it:
>    http://github.com/rca/django/commit/c1d62dbb58b1113a864ed5b1d35dd0b60ec2c071

That's great - but if you leave it on github, it's going to get forgotten.

This issue is being tracked on ticket #9589. If you want to ensure
this contribution isn't forgotten, turn your github changeset into a
patch, and upload that patch to the ticket. You can also reference the
github changeset in the ticket comments, but not all the Django core
uses git.

> But, just like the test command, the runtester command takes a list of
> apps, or even a TestCase class within an app.  That way you are simply
> running a subset of the tests on the app actively being worked on.
> For example:
>
> ./manage.py runtester some_app.SubTest
>
> With the above command, only the SubTest tests are run, which will
> drastically reduce the time to run the tests.  Granted time to setup
> and teardown of the test environment is still there.

Well, sure, except for the fact that if you do this, you no longer
have continuous integration :-) The point of a test suite is that you
run *all* the tests, so that you can catch *all* the possible
regressions. Sure - if you're just doing a quick check of something,
you might just run a single test, but you need to run the full suite
to be sure.

That's why I suggest that CI is better suited as a checkin trigger. A
checkin is an indication that you think a body of work is complete.
Running the full test suite is then an appropriate validation step. A
single file save doesn't give you that sort of certainty that a body
of work is complete.

When it takes a while for a test suite to run (which will be true in
Django even if you're only running a small subset of the overall
suite), it could be easy for a test suite to start running on a
partially saved set of updates, which could result in confusion over
exactly which version of a which files were used to run a test.

kGiven the fact that it takes a long time to setup and teardown
Django's test suite, I feel that it is better to leave test execution
as a manually invoked process. That way the developer can be certain
of the exact file versions that will be used for a test run.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Thomas K. Adamcik-2]

On Sat, Oct 17, 2009 at 09:59:38AM -0700, berto wrote:
> I wanted to be able to run a project's test suite in the same manner
> runserver restarts when a file is changed.  I did not find this
> feature in Django so I wrote the following code:

Have you looked at http://github.com/lacostej/nosyd yet? This seems might be
of interest based on what you are looking for. As an added bonus it allready
supports Django :)

-Thomas Adamcik

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Roberto Aguilar]

On Oct 18, 7:24 am, Russell Keith-Magee <freakboy3...@...>
wrote:

> On Sun, Oct 18, 2009 at 6:14 PM, berto <roberto.c.agui...@...> wrote:
>
> > On Oct 17, 5:33 pm, Russell Keith-Magee <freakboy3...@...>
> > wrote:
> >> I'd like to fix this problem, but while this problem exists, adding a
> >> continuous integration server that crashes every time the developer
> >> makes a Syntax Error doesn't sound like an especially good idea.
>
> > That's an excellent point.  My initial implementation did have the
> > same problem, but I have since fixed it as well as runserver.  The
> > changes are posted on github:
>
> > - autoreload patch adds checking for modifications on files in an
> > errored state
> >  http://github.com/rca/django/commit/ef96339f7e766700e8739a591b226b63b...
>
> > - runserver patch adds files that have errors to the autoreload code
> > changed loop
> >  http://github.com/rca/django/commit/e1d0be02b001155a09b012f4867de1efd...
>
> > - updated version of runtester using the functionality above:
> >  http://github.com/rca/django/commit/043e9723ae8fdb98745c2390f3ff92649...
> >  - and a minor patch to it:
> >    http://github.com/rca/django/commit/c1d62dbb58b1113a864ed5b1d35dd0b60...
>
> That's great - but if you leave it on github, it's going to get forgotten.
>
> This issue is being tracked on ticket #9589. If you want to ensure
> this contribution isn't forgotten, turn your github changeset into a
> patch, and upload that patch to the ticket. You can also reference the
> github changeset in the ticket comments, but not all the Django core
> uses git.
>
> > But, just like the test command, the runtester command takes a list of
> > apps, or even a TestCase class within an app.  That way you are simply
> > running a subset of the tests on the app actively being worked on.
> > For example:
>
> > ./manage.py runtester some_app.SubTest
>
> > With the above command, only the SubTest tests are run, which will
> > drastically reduce the time to run the tests.  Granted time to setup
> > and teardown of the test environment is still there.
>
> Well, sure, except for the fact that if you do this, you no longer
> have continuous integration :-) The point of a test suite is that you
> run *all* the tests, so that you can catch *all* the possible
> regressions. Sure - if you're just doing a quick check of something,
> you might just run a single test, but you need to run the full suite
> to be sure.

Maybe I used the wrong name to describe the feature?  I was basically
looking for a way to run tests related to the work currently being
done.  I'm not suggesting that running the entire test suite is not
necessary, surely it is.

> That's why I suggest that CI is better suited as a checkin trigger. A
> checkin is an indication that you think a body of work is complete.
> Running the full test suite is then an appropriate validation step. A
> single file save doesn't give you that sort of certainty that a body
> of work is complete.

I think we're in agreement here.

> When it takes a while for a test suite to run (which will be true in
> Django even if you're only running a small subset of the overall
> suite), it could be easy for a test suite to start running on a
> partially saved set of updates, which could result in confusion over
> exactly which version of a which files were used to run a test.

Sounds like you're interpreting what I'm looking as a feature, a
bug.  :)  I want to write tests and see them fail continually until I
have written the code that makes them pass.

> kGiven the fact that it takes a long time to setup and teardown
> Django's test suite, I feel that it is better to leave test execution
> as a manually invoked process. That way the developer can be certain
> of the exact file versions that will be used for a test run.

Alright, well, thanks for your input.  I'll use the test runner
throughout the project I'm working on and see how useful it actually
is!

-berto.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Anssi Kaariainen-4]

On Oct 18, 8:08 pm, berto <roberto.c.agui...@...> wrote:
> Sounds like you're interpreting what I'm looking as a feature, a
> bug.  :)  I want to write tests and see them fail continually until I
> have written the code that makes them pass.

Is it possible to autorun only the failing tests? It would be very
useful (at least for me) if the following was supported:

1) run all the tests.
2) on file change, run tests that failed in step 1.
3) when no tests fail in step 2, run all the tests again. If failures,
start from step 1 again. If no failures, print "Completed".



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Dougal Matthews]

2009/10/18 Anssi Kaariainen <akaariai@...>

Is it possible to autorun only the failing tests? It would be very
useful (at least for me) if the following was supported

No. Not yet anyway...

It had been discussed recently; http://groups.google.com/group/django-developers/browse_thread/thread/825b6a1b676f24ed/c54aa0fd48ed7b8c?hl=en&lnk=gst&q=failed#c54aa0fd48ed7b8c

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[Dougal Matthews]

2009/10/18 Dougal Matthews <dougal85@...>

It had been discussed recently; http://groups.google.com/group/django-developers/browse_thread/thread/825b6a1b676f24ed/c54aa0fd48ed7b8c?hl=en&lnk=gst&q=failed#c54aa0fd48ed7b8c

Link fail. This one will work better;

http://groups.google.com/group/django-developers/browse_thread/thread/825b6a1b676f24ed/c54aa0fd48ed7b8c

Dougal

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

[gareth rushgrove-2]

On Oct 18, 6:08 pm, berto <roberto.c.agui...@...> wrote:

> On Oct 18, 7:24 am, Russell Keith-Magee <freakboy3...@...>
> wrote:
>
> > On Sun, Oct 18, 2009 at 6:14 PM, berto <roberto.c.agui...@...> wrote:
>
> > kGiven the fact that it takes a long time to setup and teardown
> > Django's test suite, I feel that it is better to leave test execution
> > as a manually invoked process. That way the developer can be certain
> > of the exact file versions that will be used for a test run.
>
> Alright, well, thanks for your input.  I'll use the test runner
> throughout the project I'm working on and see how useful it actually
> is!
>

For anyone else interested in seeing if this works in the real world,
me and Roberto have merged the runtester management command into my
django-test-extensions app.

http://github.com/garethr/django-test-extensions
http://pypi.python.org/pypi/django-test-extensions/0.7

Any other suggestions welcome.

Thanks

Gareth

> -berto.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers@...
To unsubscribe from this group, send email to django-developers+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---