Update over scm-local does not delete files removed from source dir

[ArneD]

Hello everybody,

my Continuum installation is accessing a file system directory using scm-local provider to get sources (BTW, the directory is on a ClearCase dynamic view). Updates are working fine as long as files are only changed or added. But when files are removed from the source directory, they still exist in the target directory. Especially after refactoring activities, this leads to build errors.

Seems like a bug to me... Or am I missing something?

Many thanks,
Arne

[Emmanuel Venisse]

scm-local was developped for tests and it's used in some case when a scm provider isn't available
like dynamic-view support for Clearcase.

It's a very simple provider that copy file from sources directory, but it can't know if some files
are deleted because it doesn't have some metadata that can inform it.

The only possibility you have for the moment is to remove the checkout directory and run a new
build. But the best way would be to add dynamic-view support in clearcase provider. Only clearcase
users will can add this features because we don't know clearcase and we don't have access to a
clearcase server.

Emmanuel

ArneD a écrit :

> Hello everybody,
>
> my Continuum installation is accessing a file system directory using
> scm-local provider to get sources (BTW, the directory is on a ClearCase
> dynamic view). Updates are working fine as long as files are only changed or
> added. But when files are removed from the source directory, they still
> exist in the target directory. Especially after refactoring activities, this
> leads to build errors.
>
> Seems like a bug to me... Or am I missing something?
>
> Many thanks,
> Arne
>
>
>
>
>

[ArneD]

Emmanuel Venisse wrote:

scm-local was developped for tests and it's used in some case when a scm provider isn't available
like dynamic-view support for Clearcase.

It's a very simple provider that copy file from sources directory, but it can't know if some files
are deleted because it doesn't have some metadata that can inform it.

The only possibility you have for the moment is to remove the checkout directory and run a new
build. But the best way would be to add dynamic-view support in clearcase provider. Only clearcase
users will can add this features because we don't know clearcase and we don't have access to a
clearcase server.

Hello Emmanuel,

thanks a lot for your answer. A ClearCase dynamic view looks like a normal filesystem, so I think scm-local in principle is a fine solution. I think there wouldn't be much what a dedicated dynamic-view support in the Clearcase SCM provider could add.

Couldn't the scm-local adapter consider all files that are in the checkout dir but not in the source dir as deleted? This should be easy to implement and do the job.

Best regards,
Arne

[Emmanuel Venisse]

ArneD a écrit :

>
> Emmanuel Venisse wrote:
>> scm-local was developped for tests and it's used in some case when a scm
>> provider isn't available
>> like dynamic-view support for Clearcase.
>>
>> It's a very simple provider that copy file from sources directory, but it
>> can't know if some files
>> are deleted because it doesn't have some metadata that can inform it.
>>
>> The only possibility you have for the moment is to remove the checkout
>> directory and run a new
>> build. But the best way would be to add dynamic-view support in clearcase
>> provider. Only clearcase
>> users will can add this features because we don't know clearcase and we
>> don't have access to a
>> clearcase server.
>>
>
> Hello Emmanuel,
>
> thanks a lot for your answer. A ClearCase dynamic view looks like a normal
> filesystem, so I think scm-local in principle is a fine solution. I think
> there wouldn't be much what a dedicated dynamic-view support in the
> Clearcase SCM provider could add.

Maybe it looks like a normal filesystem, but I think the clearcase server know if files are deleted
and update your local copy, right?

>
> Couldn't the scm-local adapter consider all files that are in the checkout
> dir but not in the source dir as deleted? This should be easy to implement
> and do the job.

We can't, because some users (or the build) add some files in the checkout directory like the target
directory and they don't want to remove them at each build.

Emmanuel

[ArneD]

Emmanuel Venisse wrote:

> thanks a lot for your answer. A ClearCase dynamic view looks like a normal
> filesystem, so I think scm-local in principle is a fine solution. I think
> there wouldn't be much what a dedicated dynamic-view support in the
> Clearcase SCM provider could add.

Maybe it looks like a normal filesystem, but I think the clearcase server know if files are deleted
and update your local copy, right?

ClearCase of course has the information. But the problem is that a ClearCase dynamic view resides on a virtual network share, e.g. \\view\some_view. You can map it to a drive letter under windows, e.g. Z:, but I don't see a way to define it in Continuum's working directory. So the files have to be copied from the dynamic view to the working directory - and that's what scm-local is doing.

>
> Couldn't the scm-local adapter consider all files that are in the checkout
> dir but not in the source dir as deleted? This should be easy to implement
> and do the job.

We can't, because some users (or the build) add some files in the checkout directory like the target
directory and they don't want to remove them at each build.

That's true. Maybe we could enhance scm-local to keep its own metadata? In particular,
scm-local could maintain a simple file, say .maven-scm-local, that contains as plain text the list of files in the source directory, as seen during the last checkout or update operation:

- During checkout, the file .maven-scm-local is created in the checkout base directory. Its just a plain text file containing the list of files that have been checked out.
- The update command looks for the file. If it is there, it compares the contents of that file to the current source directory contents (including subdirs). All files that are in .maven-scm-local but are no longer in the source dir, have been deleted in the source dir. The update command therefore removes them from the checkout dir.
- If for whatever reason .maven-scm-local is not there, the update command won't delete any files. That way, we're backwards compatible.
- After completing the update process, the update command rewrites the .maven-scm-local metadata file.
- Even the changelog command can interpret .maven-scm-local
- For add and checkin commands, I don't think that changes are needed.

What do you think?

Regards,
Arne

[Emmanuel Venisse]

Are you sure it isn't possible to checkout sources in a specific folder with cleartool?

Emmanuel

ArneD a écrit :

>
> Emmanuel Venisse wrote:
>>
>>> thanks a lot for your answer. A ClearCase dynamic view looks like a
>>> normal
>>> filesystem, so I think scm-local in principle is a fine solution. I think
>>> there wouldn't be much what a dedicated dynamic-view support in the
>>> Clearcase SCM provider could add.
>> Maybe it looks like a normal filesystem, but I think the clearcase server
>> know if files are deleted
>> and update your local copy, right?
>>
>
> ClearCase of course has the information. But the problem is that a ClearCase
> dynamic view resides on a virtual network share, e.g. \\view\some_view. You
> can map it to a drive letter under windows, e.g. Z:, but I don't see a way
> to define it in Continuum's working directory. So the files have to be
> copied from the dynamic view to the working directory - and that's what
> scm-local is doing.
>
>
>
>>> Couldn't the scm-local adapter consider all files that are in the
>>> checkout
>>> dir but not in the source dir as deleted? This should be easy to
>>> implement
>>> and do the job.
>> We can't, because some users (or the build) add some files in the checkout
>> directory like the target
>> directory and they don't want to remove them at each build.
>>
>
> That's true. Maybe we could enhance scm-local to keep its own metadata? In
> particular,
> scm-local could maintain a simple file, say .maven-scm-local, that contains
> as plain text the list of files in the source directory, as seen during the
> last checkout or update operation:
>
> - During checkout, the file .maven-scm-local is created in the checkout base
> directory. Its just a plain text file containing the list of files that have
> been checked out.
> - The update command looks for the file. If it is there, it compares the
> contents of that file to the current source directory contents (including
> subdirs). All files that are in .maven-scm-local but are no longer in the
> source dir, have been deleted in the source dir. The update command
> therefore removes them from the checkout dir.
> - If for whatever reason .maven-scm-local is not there, the update command
> won't delete any files. That way, we're backwards compatible.
> - After completing the update process, the update command rewrites the
> .maven-scm-local metadata file.
> - Even the changelog command can interpret .maven-scm-local
> - For add and checkin commands, I don't think that changes are needed.
>
> What do you think?
>
> Regards,
> Arne
>

[ArneD]

Emmanuel Venisse wrote:

Are you sure it isn't possible to checkout sources in a specific folder with cleartool?

It is possible with snapshot views. My ClearCase know-how is limited, but I am quite sure it is not possible with dynamic views.

Do you think the suggested enhancement for scm-local would be useful and make sense? Not only in combination with ClearCase dynamic views but for other usage as well, e.g. testing.

Regards,
Arne

> That's true. Maybe we could enhance scm-local to keep its own metadata? In
> particular,
> scm-local could maintain a simple file, say .maven-scm-local, that contains
> as plain text the list of files in the source directory, as seen during the
> last checkout or update operation:
>
> - During checkout, the file .maven-scm-local is created in the checkout base
> directory. Its just a plain text file containing the list of files that have
> been checked out.
> - The update command looks for the file. If it is there, it compares the
> contents of that file to the current source directory contents (including
> subdirs). All files that are in .maven-scm-local but are no longer in the
> source dir, have been deleted in the source dir. The update command
> therefore removes them from the checkout dir.
> - If for whatever reason .maven-scm-local is not there, the update command
> won't delete any files. That way, we're backwards compatible.
> - After completing the update process, the update command rewrites the
> .maven-scm-local metadata file.
> - Even the changelog command can interpret .maven-scm-local
> - For add and checkin commands, I don't think that changes are needed.
>
> What do you think?
>
> Regards,
> Arne
>

[Emmanuel Venisse]

ArneD a écrit :

>
> Emmanuel Venisse wrote:
>> Are you sure it isn't possible to checkout sources in a specific folder
>> with cleartool?
>>
>
> It is possible with snapshot views. My ClearCase know-how is limited, but I
> am quite sure it is not possible with dynamic views.
>
> Do you think the suggested enhancement for scm-local would be useful and
> make sense? Not only in combination with ClearCase dynamic views but for
> other usage as well, e.g. testing.

yes. Do you want to implement it?

>
> Regards,
> Arne
>
>
>
>>> That's true. Maybe we could enhance scm-local to keep its own metadata?
>>> In
>>> particular,
>>> scm-local could maintain a simple file, say .maven-scm-local, that
>>> contains
>>> as plain text the list of files in the source directory, as seen during
>>> the
>>> last checkout or update operation:
>>>
>>> - During checkout, the file .maven-scm-local is created in the checkout
>>> base
>>> directory. Its just a plain text file containing the list of files that
>>> have
>>> been checked out.
>>> - The update command looks for the file. If it is there, it compares the
>>> contents of that file to the current source directory contents (including
>>> subdirs). All files that are in .maven-scm-local but are no longer in the
>>> source dir, have been deleted in the source dir. The update command
>>> therefore removes them from the checkout dir.
>>> - If for whatever reason .maven-scm-local is not there, the update
>>> command
>>> won't delete any files. That way, we're backwards compatible.
>>> - After completing the update process, the update command rewrites the
>>> .maven-scm-local metadata file.
>>> - Even the changelog command can interpret .maven-scm-local
>>> - For add and checkin commands, I don't think that changes are needed.
>>>
>>> What do you think?
>>>
>>> Regards,
>>> Arne
>>>
>>
>>
>

[David Roussel-5]

> my Continuum installation is accessing a file system directory using
> scm-local provider to get sources (BTW, the directory is on a ClearCase
> dynamic view). Updates are working fine as long as files are only changed or
> added. But when files are removed from the source directory, they still
> exist in the target directory. Especially after refactoring activities, this
> leads to build errors.

Make sure you are runnning clean goal in maven.  When you do a clean it
should delete this info from the target directories.  And that will fix
your problem.

BTW, you'll speed up your build if you get maven to put the target
directories somewhere else, outside of the dynamic view, as dynamic
views can be slow.

David

[ArneD]

Emmanuel Venisse wrote:

> Do you think the suggested enhancement for scm-local would be useful and
> make sense? Not only in combination with ClearCase dynamic views but for
> other usage as well, e.g. testing.

yes. Do you want to implement it?

Yes, I can try to do so and then provide a patch.

Regards,
Arne

[Emmanuel Venisse]

ArneD a écrit :

>
> Emmanuel Venisse wrote:
>>
>>> Do you think the suggested enhancement for scm-local would be useful and
>>> make sense? Not only in combination with ClearCase dynamic views but for
>>> other usage as well, e.g. testing.
>> yes. Do you want to implement it?
>>
>>
>
> Yes, I can try to do so and then provide a patch.

Cool. Thanks.

Emmanuel

[ArneD]

Thanks for your answer, David.

continuum-3 wrote:

> my Continuum installation is accessing a file system directory using
> scm-local provider to get sources (BTW, the directory is on a ClearCase
> dynamic view). Updates are working fine as long as files are only changed or
> added. But when files are removed from the source directory, they still
> exist in the target directory. Especially after refactoring activities, this
> leads to build errors.

Make sure you are runnning clean goal in maven.  When you do a clean it
should delete this info from the target directories.  And that will fix
your problem.

This is what I am doing. The problem is: My project source dir resides on a dynamic view on drive Z:\VOB\some\where. My Continuum working directory is e.g. D:\continuum-work\99.

What Continuum does (with help of scm-local) is to copy all files from Z:\VOB\some\where to D:\continuum-work\99 before running the build. If someone removes an outdated class from ClearCase, it won't be on Z:\VOB\some\where any longer. As scm-local currently does not delete anything from the checkout directory, the outdated class will still be there at d:\continuum-work\99. The clean goal will only delete d:\continuum-work\99\target.

BTW, you'll speed up your build if you get maven to put the target
directories somewhere else, outside of the dynamic view, as dynamic
views can be slow.

Did you manage to let Continuum operate directly on a dynamic view, in my example on Z:\VOB\some\where, without copying to a working directory?

Probably you are only talking about Maven stand-alone usage. Then, I agree, it is no problem, as long as you are running the clean goal.

Regards,
Arne

[David Roussel-5]

> What Continuum does (with help of scm-local) is to copy all files from
> Z:\VOB\some\where to D:\continuum-work\99 before running the build. If
> someone removes an outdated class from ClearCase, it won't be on
> Z:\VOB\some\where any longer.

Is there a reason you can't use a snapshot view?  It deals with copying
the files to your C: drive for you, _and_ it deletes the old ones too.

> Did you manage to let Continuum operate directly on a dynamic view, in my
> example on Z:\VOB\some\where, without copying to a working directory?
>
> Probably you are only talking about Maven stand-alone usage. Then, I
> agree, it is no problem, as long as you are running the clean goal.

I've not tried.  I've done a manual maven build on a dynamic view and it
took 43 minutes, compared to 2 minutes on my snapshot view.  Bit that's
mainly because my clearcase server is in another country, and behind a
firewall.  Don't ask! I's a latency nightnare!

David

[ArneD]

Emmanuel Venisse wrote:

ArneD a écrit :
>
> Emmanuel Venisse wrote:
>>
>>> Do you think the suggested enhancement for scm-local would be useful and
>>> make sense? Not only in combination with ClearCase dynamic views but for
>>> other usage as well, e.g. testing.
>> yes. Do you want to implement it?
>>
>>
>
> Yes, I can try to do so and then provide a patch.

Cool. Thanks.

Hi Emmanuel,

I've filed issue http://jira.codehaus.org/browse/SCM-231 that includes a patch.

Could you please review it and, if it's okay, commit it to svn?

Many thanks
Arne