incomplete vs broken

[Alan Baljeu]

I find I write tests that fail, and then I'm not able to make them pass yet because they need something else which doesn't exist or doesn't function yet.

When that happens, I suspend work on the current test, and develop the needed functionality through a new test.

This works.  The trouble is, I can't submit code because I have broken tests.  Soon I have huge reams of code to submit.

What I want to do is identify these tests as Incomplete, not Broken.  The app still works, because this is new stuff, and hasn't been integrated at the user level, and the old tests still work.

Source control branching is a good way to isolate incomplete code, and then we integrate when the feature set is complete.  Still, it would be nice in the working code set to clearly identify what's Broke vs. what's just waiting for more test development.  

Also I think it would be useful to write a test that simply says "TODO: xyz".  Then I will remember that I need to work on writing tests and code to do xyz.

Anybody do this?  Anybody do this with Google Test?

 Alan Baljeu


      __________________________________________________________________
The new Internet Explorer® 8 - Faster, safer, easier.  Optimized for Yahoo!  Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

[Dave Smith-22]

On Sat, Aug 29, 2009 at 11:01 AM, Alan Baljeu <alanbaljeu@...> wrote:

> I find I write tests that fail, and then I'm not able to make them pass yet
> because they need something else which doesn't exist or doesn't function
> yet.
>
> When that happens, I suspend work on the current test, and develop the
> needed functionality through a new test.
>
> This works.  The trouble is, I can't submit code because I have broken
> tests.  Soon I have huge reams of code to submit.


Submit in pieces, leaving your failing test checked out (or not checked in)
locally.

I often do this with a functional test that I write first as an "I'm done
with the happy path" indicator. Getting it to pass may force changes to
underlying substrates, so I write those tests, make those changes, and check
in everything except my new failing test and any changes at that level that
correspond.

A possible side-effect of this approach is submitted code that doesn't yet
correspond to a feature. A possible benefit of this approach is that you've
provided working code that indicates the direction you're going. Other
people can honor that direction (by not working at cross-purposes without
discussing it with you first), or just use the new code.

A common example of the latter is needing to extend test object builders to
enable testing a new feature. I can bundle up and submit the object builder
changes, along with tests for those, as soon as they're done.

Dave


[Non-text portions of this message have been removed]

[Michael Hill-10]

I will 'stack' up to three tests, but no more.  After that, I just say to
myself:  you bit off more than you could chew, and remove the bottom-most
element in the stack.  I used to comment it out.  Now I just remove it.  It
turns out that about half the time I was wrong about wanting exactly that
test, anyway.
Seeya,
Hill
(Read GeePawHill on TDD & other topics at http://anarchycreek.com .)


On Sat, Aug 29, 2009 at 2:01 PM, Alan Baljeu <alanbaljeu@...> wrote:

>
>
> I find I write tests that fail, and then I'm not able to make them pass yet
> because they need something else which doesn't exist or doesn't function
> yet.
>
> When that happens, I suspend work on the current test, and develop the
> needed functionality through a new test.
>
> This works. The trouble is, I can't submit code because I have broken
> tests. Soon I have huge reams of code to submit.
>
> What I want to do is identify these tests as Incomplete, not Broken. The
> app still works, because this is new stuff, and hasn't been integrated at
> the user level, and the old tests still work.
>
> Source control branching is a good way to isolate incomplete code, and then
> we integrate when the feature set is complete. Still, it would be nice in
> the working code set to clearly identify what's Broke vs. what's just
> waiting for more test development.
>
> Also I think it would be useful to write a test that simply says "TODO:
> xyz". Then I will remember that I need to work on writing tests and code to
> do xyz.
>
> Anybody do this? Anybody do this with Google Test?
>
> Alan Baljeu
>
> __________________________________________________________
> The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo!
> Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
>  
>

[Non-text portions of this message have been removed]

[Olof Bjarnason]

2009/8/29 Alan Baljeu <alanbaljeu@...>:

>
>
> I find I write tests that fail, and then I'm not able to make them pass yet
> because they need something else which doesn't exist or doesn't function
> yet.
>
> When that happens, I suspend work on the current test, and develop the
> needed functionality through a new test.
>
> This works. The trouble is, I can't submit code because I have broken tests.
> Soon I have huge reams of code to submit.
>
> What I want to do is identify these tests as Incomplete, not Broken. The app
> still works, because this is new stuff, and hasn't been integrated at the
> user level, and the old tests still work.
>
> Source control branching is a good way to isolate incomplete code, and then
> we integrate when the feature set is complete. Still, it would be nice in
> the working code set to clearly identify what's Broke vs. what's just
> waiting for more test development.
>
> Also I think it would be useful to write a test that simply says "TODO:
> xyz". Then I will remember that I need to work on writing tests and code to
> do xyz.
>
> Anybody do this? Anybody do this with Google Test?

Um not sure for C++ / Google Test.

In C#/NUnit I use the attribute [Ignore] to mark such tests.

>
> Alan Baljeu
>
> __________________________________________________________
> The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo!
> Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
>
>



--
twitter.com/olofb
olofb.wordpress.com
olofb.wordpress.com/tag/english

[George Dinwiddie]

Alan Baljeu wrote:
> Source control branching is a good way to isolate incomplete code,
> and then we integrate when the feature set is complete.  Still, it
> would be nice in the working code set to clearly identify what's
> Broke vs. what's just waiting for more test development.

See http://tech.groups.yahoo.com/group/extremeprogramming/message/151516 
for another opinion on this strategy.

> Also I think it would be useful to write a test that simply says
> "TODO: xyz".  Then I will remember that I need to work on writing
> tests and code to do xyz.
>
> Anybody do this?  Anybody do this with Google Test?

RSpec allows you to just put a "pending" step in a test.  Cucumber
allows you to to create your own, easily.  Either allows you to specify
an unimplemented scenario and treats it as pending.

I suspect you could add such capability to Google Test, but I've never
used it.

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------

[Keith Ray]

In Google Test for C++ [aka gTest], You can add the DISABLED_ prefix
to a test name to exclude it from execution.

If you need to disable all tests in a test case, you can either add
DISABLED_ to the front of the name of each test, or alternatively add
it to the front of the test case name.

[see also http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Temporarily_Disabling_Tests
]

On Sat, Aug 29, 2009 at 11:01 AM, Alan Baljeu<alanbaljeu@...> wrote:

>
>
> I find I write tests that fail, and then I'm not able to make them pass yet
> because they need something else which doesn't exist or doesn't function
> yet.
>
> When that happens, I suspend work on the current test, and develop the
> needed functionality through a new test.
>
> This works. The trouble is, I can't submit code because I have broken tests.
> Soon I have huge reams of code to submit.
>
> What I want to do is identify these tests as Incomplete, not Broken. The app
> still works, because this is new stuff, and hasn't been integrated at the
> user level, and the old tests still work.
>
> Source control branching is a good way to isolate incomplete code, and then
> we integrate when the feature set is complete. Still, it would be nice in
> the working code set to clearly identify what's Broke vs. what's just
> waiting for more test development.
>
> Also I think it would be useful to write a test that simply says "TODO:
> xyz". Then I will remember that I need to work on writing tests and code to
> do xyz.
>
> Anybody do this? Anybody do this with Google Test?
>
> Alan Baljeu
>
> __________________________________________________________
> The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo!
> Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
>
>

--
C. Keith Ray, IXP Coach, Industrial Logic, Inc.
http://industriallogic.com      866-540-8336 (toll free)
Groove with our Agile Greatest Hits: http://www.industriallogic.com/elearning/
http://agilesolutionspace.blogspot.com/

[Adam Sroka-3]

Good answers...

My reaction is:

a) You started with the wrong test. Delete it and write the right test.

b) You don't actually "need" it yet. Look for a simpler solution. Add
more when you actually need it.

c) You "need" something, but you don't know what it is yet. Use a
stub. In C you can just write the function header to a no-op. Later,
when you know what it should do, you will write a test for that.

On Sat, Aug 29, 2009 at 11:01 AM, Alan Baljeu<alanbaljeu@...> wrote:

>
>
> I find I write tests that fail, and then I'm not able to make them pass yet
> because they need something else which doesn't exist or doesn't function
> yet.
>
> When that happens, I suspend work on the current test, and develop the
> needed functionality through a new test.
>
> This works. The trouble is, I can't submit code because I have broken tests.
> Soon I have huge reams of code to submit.
>
> What I want to do is identify these tests as Incomplete, not Broken. The app
> still works, because this is new stuff, and hasn't been integrated at the
> user level, and the old tests still work.
>
> Source control branching is a good way to isolate incomplete code, and then
> we integrate when the feature set is complete. Still, it would be nice in
> the working code set to clearly identify what's Broke vs. what's just
> waiting for more test development.
>
> Also I think it would be useful to write a test that simply says "TODO:
> xyz". Then I will remember that I need to work on writing tests and code to
> do xyz.
>
> Anybody do this? Anybody do this with Google Test?
>
> Alan Baljeu
>
> __________________________________________________________
> The new Internet Explorer® 8 - Faster, safer, easier. Optimized for Yahoo!
> Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/
>
>

[Alan Baljeu]

Doesn't that lead to submitted code that won't even compile?

 




________________________________


Submit in pieces, leaving your failing test checked out (or not checked in)
locally.

Dave



   


      __________________________________________________________________
The new Internet Explorer® 8 - Faster, safer, easier.  Optimized for Yahoo!  Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/

[Non-text portions of this message have been removed]

[Dave Smith-22]

On Sun, Aug 30, 2009 at 4:09 PM, Alan Baljeu <alanbaljeu@...> wrote:

> Doesn't that lead to submitted code that won't even compile?


No. I don't check in smaller chunks unless all tests, except the new
functional test that I'm using as a target, pass.

Dave


[Non-text portions of this message have been removed]

[Adam Sroka-3]

I would once again humbly suggest that you might be starting with the
wrong test. If you are doing TDD you should be starting with a unit
test for the thing you are about to write, not with a functional test
that requires you to integrate pieces that don't yet exist. On the
other hand, some folks write customer/acceptance level tests that
aren't intended to pass until a feature is complete.

The distinction is important: programmer tests should be written with
the intent to immediately make them pass. That is TDD. Customer tests
can be allowed to fail until a feature is complete.

Customer tests, however, should be checked in even though they are
failing. Whether they pass or fail is an indication of progress not an
indication of brokenness.

On Sat, Aug 29, 2009 at 11:43 AM, Dave Smith<davewsmith@...> wrote:

>
>
> On Sat, Aug 29, 2009 at 11:01 AM, Alan Baljeu <alanbaljeu@...> wrote:
>
>> I find I write tests that fail, and then I'm not able to make them pass
>> yet
>> because they need something else which doesn't exist or doesn't function
>> yet.
>>
>> When that happens, I suspend work on the current test, and develop the
>> needed functionality through a new test.
>>
>> This works. The trouble is, I can't submit code because I have broken
>> tests. Soon I have huge reams of code to submit.
>
> Submit in pieces, leaving your failing test checked out (or not checked in)
> locally.
>
> I often do this with a functional test that I write first as an "I'm done
> with the happy path" indicator. Getting it to pass may force changes to
> underlying substrates, so I write those tests, make those changes, and check
> in everything except my new failing test and any changes at that level that
> correspond.
>
> A possible side-effect of this approach is submitted code that doesn't yet
> correspond to a feature. A possible benefit of this approach is that you've
> provided working code that indicates the direction you're going. Other
> people can honor that direction (by not working at cross-purposes without
> discussing it with you first), or just use the new code.
>
> A common example of the latter is needing to extend test object builders to
> enable testing a new feature. I can bundle up and submit the object builder
> changes, along with tests for those, as soon as they're done.
>
> Dave
>
> [Non-text portions of this message have been removed]
>
>

[Dave Smith-22]

On Sun, Aug 30, 2009 at 6:16 PM, Adam Sroka <adam.sroka@...> wrote:

> I would once again humbly suggest that you might be starting with the
> wrong test. If you are doing TDD you should be starting with a unit
> test for the thing you are about to write, not with a functional test
> that requires you to integrate pieces that don't yet exist.


Starting with a functional test as a statement of where you're heading is a
perfectly fine way to do TDD.

Starting too far down risks building well-testing bits that aren't in
support of the feature you're building.

Dave


[Non-text portions of this message have been removed]

[Donaldson, John (GEO)]

Alan,

I too suffer from this. I think it will take a lot of practice to avoid this completely (I am a long way from there).

My usual way of reacting to this is to comment out the failing test. Sometimes I forget this commented out code - but usually I find it again quickly.

I *do* try to do "the simplest thing" - such as "return 63" or whatever is appropriate. But still I occasionally end up with a cascade of tests which are all aimed at forcing the needed code into life.

Not much help - but sympathy anyway.

John D.

-----Original Message-----
From: testdrivendevelopment@... [mailto:testdrivendevelopment@...] On Behalf Of Alan Baljeu
Sent: 29 August 2009 20:01
To: testdrivendevelopment@...
Subject: [TDD] incomplete vs broken

I find I write tests that fail, and then I'm not able to make them pass yet because they need something else which doesn't exist or doesn't function yet.

When that happens, I suspend work on the current test, and develop the needed functionality through a new test.

This works.  The trouble is, I can't submit code because I have broken tests.  Soon I have huge reams of code to submit.

What I want to do is identify these tests as Incomplete, not Broken.  The app still works, because this is new stuff, and hasn't been integrated at the user level, and the old tests still work.

Source control branching is a good way to isolate incomplete code, and then we integrate when the feature set is complete.  Still, it would be nice in the working code set to clearly identify what's Broke vs. what's just waiting for more test development.  

Also I think it would be useful to write a test that simply says "TODO: xyz".  Then I will remember that I need to work on writing tests and code to do xyz.

Anybody do this?  Anybody do this with Google Test?

 Alan Baljeu


      __________________________________________________________________
The new Internet Explorer(r) 8 - Faster, safer, easier.  Optimized for Yahoo!  Get it Now for Free! at http://downloads.yahoo.com/ca/internetexplorer/


------------------------------------

Yahoo! Groups Links

[Kaleb Pederson-2]

On Sun, Aug 30, 2009 at 6:16 PM, Adam Sroka<adam.sroka@...> wrote:
> I would once again humbly suggest that you might be starting with the
> wrong test. If you are doing TDD you should be starting with a unit
> test for the thing you are about to write, not with a functional test
> that requires you to integrate pieces that don't yet exist. On the
> other hand, some folks write customer/acceptance level tests that
> aren't intended to pass until a feature is complete.

Although I'm not very proficient with mocks yet, this seems like the
perfect place for a mock.  The mock could emulate the behavior
necessary for the SUT to be tested and allow the test(s) to compile
and pass.

I'm curious if most TDD'ers use a top-down approach or a bottom-up
approach to software development?  The top-down approach seems to
benefit strongly from mocks, but the bottom-up approach seems to
naturally support TDD without such a requirement.

--Kaleb

[Olof Bjarnason]

2009/8/31 Kaleb Pederson <kaleb.pederson@...>:

>
>
> On Sun, Aug 30, 2009 at 6:16 PM, Adam Sroka<adam.sroka@...> wrote:
>> I would once again humbly suggest that you might be starting with the
>> wrong test. If you are doing TDD you should be starting with a unit
>> test for the thing you are about to write, not with a functional test
>> that requires you to integrate pieces that don't yet exist. On the
>> other hand, some folks write customer/acceptance level tests that
>> aren't intended to pass until a feature is complete.
>
> Although I'm not very proficient with mocks yet, this seems like the
> perfect place for a mock. The mock could emulate the behavior
> necessary for the SUT to be tested and allow the test(s) to compile
> and pass.
>
> I'm curious if most TDD'ers use a top-down approach or a bottom-up
> approach to software development? The top-down approach seems to
> benefit strongly from mocks, but the bottom-up approach seems to
> naturally support TDD without such a requirement.

I began with bottom-up TDD, but am trying to migrate to top-down.

The main reason for the switch was the "functionality in the
sky"-problem with bottom-up.

1. I built some classes "out of thin air"
2. When I were to integrate those into another, higher-level class,
some of the low-level classes were not needed at all.

The whole bottom-up/top-down discussion is interesting and I keep
returning to it; I've been through at least one or two switches even
before I started using TDD as a design tool.

But top-down has it hassles too; you start seeing duplication of
tests/examples at various levels of the system - something that I find
'smelly'.

>
> --Kaleb
>



--
twitter.com/olofb
olofb.wordpress.com
olofb.wordpress.com/tag/english

[George Dinwiddie]

Kaleb Pederson wrote:
> I'm curious if most TDD'ers use a top-down approach or a bottom-up
> approach to software development?  The top-down approach seems to
> benefit strongly from mocks, but the bottom-up approach seems to
> naturally support TDD without such a requirement.

Sorry to break your supposition, but I tend to work top-down without
mocks.  I use the "fake it 'til you make it" approach and add subsequent
tests to flesh out the functionality, dropping to tests on the lower
levels as appropriate.

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------

[Rob Park-3]

George, do you find the threads of functionality are small enough that you
come back up the stack quickly enough to check in?  Or what do you do while
the top tests are still failing, but you have some bottom tests passing (WRT
check in)?


On Mon, Aug 31, 2009 at 10:01 AM, George Dinwiddie
<lists@...>wrote:

>
>
> Kaleb Pederson wrote:
> > I'm curious if most TDD'ers use a top-down approach or a bottom-up
> > approach to software development? The top-down approach seems to
> > benefit strongly from mocks, but the bottom-up approach seems to
> > naturally support TDD without such a requirement.
>
> Sorry to break your supposition, but I tend to work top-down without
> mocks. I use the "fake it 'til you make it" approach and add subsequent
> tests to flesh out the functionality, dropping to tests on the lower
> levels as appropriate.
>
> - George
>
> --
> ----------------------------------------------------------
> * George Dinwiddie * http://blog.gdinwiddie.com
> Software Development http://www.idiacomputing.com
> Consultant and Coach http://www.agilemaryland.org
> ----------------------------------------------------------
>
>  
>

--
Rob
--
http://agileintention.blogspot.com
http://twitter.com/robpark


[Non-text portions of this message have been removed]

[Kaleb Pederson-2]

On Mon, Aug 31, 2009 at 9:01 AM, George
Dinwiddie<lists@...> wrote:

> Kaleb Pederson wrote:
>> I'm curious if most TDD'ers use a top-down approach or a bottom-up
>> approach to software development?  The top-down approach seems to
>> benefit strongly from mocks, but the bottom-up approach seems to
>> naturally support TDD without such a requirement.
>
> Sorry to break your supposition, but I tend to work top-down without
> mocks.  I use the "fake it 'til you make it" approach and add subsequent
> tests to flesh out the functionality, dropping to tests on the lower
> levels as appropriate.

Perhaps I'm committing the cardinal sin of being imprecise in my use
of the term mock, partially for lack of thought and partially for lack
of understanding.  I was using the standard definition for mock from
the OED:

"6.A.1. Preceding a noun: designating a person who or thing which
parodies, imitates, or deceptively resembles that which the noun
properly denotes; pretend, imitation, sham, counterfeit."

I gather from your statement that you create Fake objects used by the
SUT to make the tests pass?

Where do you add those subsequent tests?  Are they created as a
replacement for the fake that you dropped in?  Can you elaborate
and/or explain your views on the benefits of doing such?

I've been working mostly in C++ for the last couple of years and have
mostly avoided fakes, mocks, and stubs except where there was a really
obvious benefit.  However, I'm starting to find that I believe I could
benefit quite a bit from mocks (again, using the imprecise English
definition), so I'm probably going to start taking a look at some of
the C++ mocking frameworks and investigate better ways to structure
our project so it's not so painful to add in fakes, mocks, and stubs.

Thanks.

--Kaleb

[George Dinwiddie]

Rob Park wrote:
> George, do you find the threads of functionality are small enough that you
> come back up the stack quickly enough to check in?  Or what do you do while
> the top tests are still failing, but you have some bottom tests passing (WRT
> check in)?

I don't typically leave the top tests failing.  As a general strategy,
I'll typically start with the empty case.  Then a single value with a
hard-coded return.  Then a second value, which requires doing something.
  Then, if that 'doing something' represents a distinct concept,
refactor that to it's own home.  Then flesh out that 'doing something'
with more tests.  Then bounce back to the upper level to continue the story.

The order and nature of these things changes with the context and what
occurs to me at the time.

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------

[George Dinwiddie]

Kaleb Pederson wrote:
> I gather from your statement that you create Fake objects used by the
> SUT to make the tests pass?

I might, but more often I use hard-coded primitives as my fakes, or
incompletely implemented real objects as collaborators.

> Where do you add those subsequent tests?  Are they created as a
> replacement for the fake that you dropped in?  Can you elaborate
> and/or explain your views on the benefits of doing such?

I don't understand these questions.  How can a test replace a test double?

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------

[Kaleb Pederson-2]

On Mon, Aug 31, 2009 at 10:17 AM, George
Dinwiddie<lists@...> wrote:

> Kaleb Pederson wrote:
>> I gather from your statement that you create Fake objects used by the
>> SUT to make the tests pass?
>
> I might, but more often I use hard-coded primitives as my fakes, or
> incompletely implemented real objects as collaborators.
>
>> Where do you add those subsequent tests?  Are they created as a
>> replacement for the fake that you dropped in?  Can you elaborate
>> and/or explain your views on the benefits of doing such?
>
> I don't understand these questions.  How can a test replace a test double?

Sorry I was unclear.  I'm was trying to identify which behavior was
being refined.  Were the subsequently added tests for the fake object
that you used as a stand-in in the original test or were you writing
further tests for the SUT initially under test.

--Kaleb