[68cat] More on scanning - interaction with VCS

[Tomas Pavek]

Hi all,

one more scanning-related improvement made it into 6.8 beta: optimized
interaction with VCS operations executed from the IDE. A typical example is a
long running update command that changes lots of files in the local working
copy. Before the fix the IDE might react on the changes and invoke scanning
many times during the update command, so rescanning incomplete data
needlessly, slowing down the update command itself, etc.

With the fix the IDE should do the check for changes and possible scanning
only once after the VCS command is completed.

To give it some more testing we'd like to ask you to help us. Don't need to do
anything special, just watch the VCS commands you perform and let us know if
you see any problems or unexpected behavior. The main points are:
1) Scanning should happen only once after the VCS commands is complete (unless
you change some files during the time in another way),
2) VCS commands that don't change files (e.g. commit) should not be followed
with scanning.

Note this applies to invoking a VCS command from the IDE (not externally on a
command line).

Let us know especially if you see things like:
* it's obviously scanning much more than what was updated, or
* it starts scanning after an non-modifying operation.

If you see problems please use the following switch to generate more logging
info and send us the messages.log file (or attach to a bug):
-J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE

Thanks,
Tomas

[Matthies, Niklas]

I wonder how this applies to dependent projects (or source roots), each
having their own CVS module. Ideally, if I have dependent projects
A<-B<-C,
and start CVS updates in this order (which are queued by NetBeans), the
following should happen:
1) Update A from CVS.
2) Once the update of A has completed, start rescanning A, but don't
   start rescanning the depending B (or C) yet, because an update of
   B has been started/is pending, and also because A is being
   rescanned.
3) In parallel to (2), update B.
4) Once (2) and (3) have completed, start rescanning B, but don't
   start rescanning the depending C yet, because an update of C has
   been started/is pending, and also because B is being rescanned.
5) In parallel to (4), update C.
6) Once (4) and (5) have completed, start rescanning C.

(Of course it gets more complicated/more things can be done in parallel
when the dependencies are not linear.)

Is the new logic designed to have an effect like that?

-- Niklas Matthies


On Mon, 26. Oct 2009 16:48, Tomas.Pavek@... wrote:
> Hi all,
>
> one more scanning-related improvement made it into 6.8 beta: optimized
> interaction with VCS operations executed from the IDE. A
> typical example is a
> long running update command that changes lots of files in the local
working

> copy. Before the fix the IDE might react on the changes and
> invoke scanning
> many times during the update command, so rescanning incomplete data
> needlessly, slowing down the update command itself, etc.
>
> With the fix the IDE should do the check for changes and
> possible scanning
> only once after the VCS command is completed.
>
> To give it some more testing we'd like to ask you to help us. Don't

need to do
> anything special, just watch the VCS commands you perform and
> let us know if
> you see any problems or unexpected behavior. The main points are:
> 1) Scanning should happen only once after the VCS commands is complete
(unless
> you change some files during the time in another way),
> 2) VCS commands that don't change files (e.g. commit) should
> not be followed
> with scanning.
>
> Note this applies to invoking a VCS command from the IDE (not
externally on a

> command line).
>
> Let us know especially if you see things like:
> * it's obviously scanning much more than what was updated, or
> * it starts scanning after an non-modifying operation.
>
> If you see problems please use the following switch to
> generate more logging
> info and send us the messages.log file (or attach to a bug):
> -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdat

er.level=FINE
>
> Thanks,
> Tomas

[Tomas Pavek]

No, it's actually very simple. The VCS operation has no notion of the projects
it runs on - it just tells the language support that it's going to update
selected folder and so the lang. support should not react on changes
underneath that folder until the VCS operation is finished.

   - Tomas


Matthies, Niklas wrote:

> I wonder how this applies to dependent projects (or source roots), each
> having their own CVS module. Ideally, if I have dependent projects
> A<-B<-C,
> and start CVS updates in this order (which are queued by NetBeans), the
> following should happen:
> 1) Update A from CVS.
> 2) Once the update of A has completed, start rescanning A, but don't
>    start rescanning the depending B (or C) yet, because an update of
>    B has been started/is pending, and also because A is being
>    rescanned.
> 3) In parallel to (2), update B.
> 4) Once (2) and (3) have completed, start rescanning B, but don't
>    start rescanning the depending C yet, because an update of C has
>    been started/is pending, and also because B is being rescanned.
> 5) In parallel to (4), update C.
> 6) Once (4) and (5) have completed, start rescanning C.
>
> (Of course it gets more complicated/more things can be done in parallel
> when the dependencies are not linear.)
>
> Is the new logic designed to have an effect like that?
>
> -- Niklas Matthies
>
>
> On Mon, 26. Oct 2009 16:48, Tomas.Pavek@... wrote:
>> Hi all,
>>
>> one more scanning-related improvement made it into 6.8 beta: optimized
>> interaction with VCS operations executed from the IDE. A
>> typical example is a
>> long running update command that changes lots of files in the local
> working
>> copy. Before the fix the IDE might react on the changes and
>> invoke scanning
>> many times during the update command, so rescanning incomplete data
>> needlessly, slowing down the update command itself, etc.
>>
>> With the fix the IDE should do the check for changes and
>> possible scanning
>> only once after the VCS command is completed.
>>
>> To give it some more testing we'd like to ask you to help us. Don't
> need to do
>> anything special, just watch the VCS commands you perform and
>> let us know if
>> you see any problems or unexpected behavior. The main points are:
>> 1) Scanning should happen only once after the VCS commands is complete
> (unless
>> you change some files during the time in another way),
>> 2) VCS commands that don't change files (e.g. commit) should
>> not be followed
>> with scanning.
>>
>> Note this applies to invoking a VCS command from the IDE (not
> externally on a
>> command line).
>>
>> Let us know especially if you see things like:
>> * it's obviously scanning much more than what was updated, or
>> * it starts scanning after an non-modifying operation.
>>
>> If you see problems please use the following switch to
>> generate more logging
>> info and send us the messages.log file (or attach to a bug):
>> -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdat
> er.level=FINE
>> Thanks,
>> Tomas

[Michel Graciano]

Well, the first tests works as expected. I will do some more tests soon and inform you about the progress.

Thanks for the fix.

On Mon, Oct 26, 2009 at 3:51 PM, Tomas Pavek <Tomas.Pavek@...> wrote:

No, it's actually very simple. The VCS operation has no notion of the projects it runs on - it just tells the language support that it's going to update selected folder and so the lang. support should not react on changes underneath that folder until the VCS operation is finished.

 - Tomas

Matthies, Niklas wrote:

I wonder how this applies to dependent projects (or source roots), each
having their own CVS module. Ideally, if I have dependent projects
A<-B<-C,
and start CVS updates in this order (which are queued by NetBeans), the
following should happen:
1) Update A from CVS.
2) Once the update of A has completed, start rescanning A, but don't
  start rescanning the depending B (or C) yet, because an update of
  B has been started/is pending, and also because A is being
  rescanned.
3) In parallel to (2), update B.
4) Once (2) and (3) have completed, start rescanning B, but don't
  start rescanning the depending C yet, because an update of C has
  been started/is pending, and also because B is being rescanned.
5) In parallel to (4), update C.
6) Once (4) and (5) have completed, start rescanning C.

(Of course it gets more complicated/more things can be done in parallel
when the dependencies are not linear.)

Is the new logic designed to have an effect like that?

-- Niklas Matthies


On Mon, 26. Oct 2009 16:48, Tomas.Pavek@... wrote:

Hi all,

one more scanning-related improvement made it into 6.8 beta: optimized
interaction with VCS operations executed from the IDE. A
typical example is a
long running update command that changes lots of files in the local

working

copy. Before the fix the IDE might react on the changes and
invoke scanning
many times during the update command, so rescanning incomplete data
needlessly, slowing down the update command itself, etc.

With the fix the IDE should do the check for changes and
possible scanning
only once after the VCS command is completed.

To give it some more testing we'd like to ask you to help us. Don't

need to do

anything special, just watch the VCS commands you perform and
let us know if
you see any problems or unexpected behavior. The main points are:
1) Scanning should happen only once after the VCS commands is complete

(unless

you change some files during the time in another way),
2) VCS commands that don't change files (e.g. commit) should
not be followed
with scanning.

Note this applies to invoking a VCS command from the IDE (not

externally on a

command line).

Let us know especially if you see things like:
* it's obviously scanning much more than what was updated, or
* it starts scanning after an non-modifying operation.

If you see problems please use the following switch to
generate more logging
info and send us the messages.log file (or attach to a bug):
-J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdat

er.level=FINE

Thanks,
Tomas

--
Michel Graciano
Summa Technologies do Brasil Ltda.
http://www.michelgraciano.com
https://genesis.dev.java.net/
http://translatedfiles.netbeans.org/index_pt_BR.html
https://copypastehistory.dev.java.net/

[Matthies, Niklas]

Ok, but what consequences does that have for the use case I describe?
What event triggers the rescanning of B (and C) after (or during?)
an update of A?
And would a concurrent update of B defer this rescanning of B?
...even if the update of B is queued but hasn't started yet?

I'm asking because this is my usual use case and I'd like to know
what the expected behavior is, so I know what too look out for.

-- Niklas Matthies


On Mon, 26. Oct 2009 18:52, Tomas.Pavek@... wrote:

> No, it's actually very simple. The VCS operation has no
> notion of the projects
> it runs on - it just tells the language support that it's
> going to update
> selected folder and so the lang. support should not react on changes
> underneath that folder until the VCS operation is finished.
>
>    - Tomas
>
>
> Matthies, Niklas wrote:
> > I wonder how this applies to dependent projects (or source roots),

each
> > having their own CVS module. Ideally, if I have dependent projects
A<-B<-C,
> > and start CVS updates in this order (which are queued by NetBeans),
the following should
> > happen:
> > 1) Update A from CVS.
> > 2) Once the update of A has completed, start rescanning A, but don't
> >    start rescanning the depending B (or C) yet, because an update of
> >    B has been started/is pending, and also because A is being
rescanned.
> > 3) In parallel to (2), update B.
> > 4) Once (2) and (3) have completed, start rescanning B, but don't
> >    start rescanning the depending C yet, because an update of C has
> >    been started/is pending, and also because B is being rescanned.
> > 5) In parallel to (4), update C.
> > 6) Once (4) and (5) have completed, start rescanning C.
> >
> > (Of course it gets more complicated/more things can be done in
parallel

> > when the dependencies are not linear.)
> >
> > Is the new logic designed to have an effect like that?
> >
> > -- Niklas Matthies
> >
> >
> > On Mon, 26. Oct 2009 16:48, Tomas.Pavek@... wrote:
> > > Hi all,
> > >
> > > one more scanning-related improvement made it into 6.8 beta:

optimized
> > > interaction with VCS operations executed from the IDE. A
> > > typical example is a
> > > long running update command that changes lots of files in the
local working
> > > copy. Before the fix the IDE might react on the changes and
> > > invoke scanning
> > > many times during the update command, so rescanning incomplete
data
> > > needlessly, slowing down the update command itself, etc.
> > >
> > > With the fix the IDE should do the check for changes and
> > > possible scanning
> > > only once after the VCS command is completed.
> > >
> > > To give it some more testing we'd like to ask you to help us.
Don't need to do
> > > anything special, just watch the VCS commands you perform and
> > > let us know if
> > > you see any problems or unexpected behavior. The main points are:
> > > 1) Scanning should happen only once after the VCS commands is
complete (unless
> > > you change some files during the time in another way),
> > > 2) VCS commands that don't change files (e.g. commit) should
> > > not be followed
> > > with scanning.
> > >
> > > Note this applies to invoking a VCS command from the IDE (not
externally on a command line).
> > >
> > > Let us know especially if you see things like:
> > > * it's obviously scanning much more than what was updated, or
> > > * it starts scanning after an non-modifying operation.
> > >
> > > If you see problems please use the following switch to
> > > generate more logging
> > > info and send us the messages.log file (or attach to a bug):
> > > -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdat
er.level=FINE Thanks,
> > > Tomas

[Tomas Pavek]

Just to make sure I understand: your projects A, B and C are each in a
different repository (but of the same type)?

I may need to confirm with guys who implemented it, but I think if B depends
on A, and update of A is finished with changes detected then also B is
rescanned because it may be affected. This rescan would be done regardless if
update of B is running at that moment (because it is not triggered by update
changes in B, but by already finished update of A).

I can see there is no efficient way to update your 3 projects, unless the
whole update operation can be done as one - I'm not sure if that is possible
over different repositories.

   - Tomas


Matthies, Niklas wrote:

> Ok, but what consequences does that have for the use case I describe?
> What event triggers the rescanning of B (and C) after (or during?)
> an update of A?
> And would a concurrent update of B defer this rescanning of B?
> ...even if the update of B is queued but hasn't started yet?
>
> I'm asking because this is my usual use case and I'd like to know
> what the expected behavior is, so I know what too look out for.
>
> -- Niklas Matthies
>
>
> On Mon, 26. Oct 2009 18:52, Tomas.Pavek@... wrote:
>> No, it's actually very simple. The VCS operation has no
>> notion of the projects
>> it runs on - it just tells the language support that it's
>> going to update
>> selected folder and so the lang. support should not react on changes
>> underneath that folder until the VCS operation is finished.
>>
>>    - Tomas
>>
>>
>> Matthies, Niklas wrote:
>>> I wonder how this applies to dependent projects (or source roots),
> each
>>> having their own CVS module. Ideally, if I have dependent projects
> A<-B<-C,
>>> and start CVS updates in this order (which are queued by NetBeans),
> the following should
>>> happen:
>>> 1) Update A from CVS.
>>> 2) Once the update of A has completed, start rescanning A, but don't
>>>    start rescanning the depending B (or C) yet, because an update of
>>>    B has been started/is pending, and also because A is being
> rescanned.
>>> 3) In parallel to (2), update B.
>>> 4) Once (2) and (3) have completed, start rescanning B, but don't
>>>    start rescanning the depending C yet, because an update of C has
>>>    been started/is pending, and also because B is being rescanned.
>>> 5) In parallel to (4), update C.
>>> 6) Once (4) and (5) have completed, start rescanning C.
>>>
>>> (Of course it gets more complicated/more things can be done in
> parallel
>>> when the dependencies are not linear.)
>>>
>>> Is the new logic designed to have an effect like that?
>>>
>>> -- Niklas Matthies
>>>
>>>
>>> On Mon, 26. Oct 2009 16:48, Tomas.Pavek@... wrote:
>>>> Hi all,
>>>>
>>>> one more scanning-related improvement made it into 6.8 beta:
> optimized
>>>> interaction with VCS operations executed from the IDE. A
>>>> typical example is a
>>>> long running update command that changes lots of files in the
> local working
>>>> copy. Before the fix the IDE might react on the changes and
>>>> invoke scanning
>>>> many times during the update command, so rescanning incomplete
> data
>>>> needlessly, slowing down the update command itself, etc.
>>>>
>>>> With the fix the IDE should do the check for changes and
>>>> possible scanning
>>>> only once after the VCS command is completed.
>>>>
>>>> To give it some more testing we'd like to ask you to help us.
> Don't need to do
>>>> anything special, just watch the VCS commands you perform and
>>>> let us know if
>>>> you see any problems or unexpected behavior. The main points are:
>>>> 1) Scanning should happen only once after the VCS commands is
> complete (unless
>>>> you change some files during the time in another way),
>>>> 2) VCS commands that don't change files (e.g. commit) should
>>>> not be followed
>>>> with scanning.
>>>>
>>>> Note this applies to invoking a VCS command from the IDE (not
> externally on a command line).
>>>> Let us know especially if you see things like:
>>>> * it's obviously scanning much more than what was updated, or
>>>> * it starts scanning after an non-modifying operation.
>>>>
>>>> If you see problems please use the following switch to
>>>> generate more logging
>>>> info and send us the messages.log file (or attach to a bug):
>>>> -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdat
> er.level=FINE Thanks,
>>>> Tomas

[Matthies, Niklas]

On Mon, 26. Oct 2009 19:26, Tomas.Pavek@... wrote:
> Just to make sure I understand: your projects A, B and C are
> each in a different repository (but of the same type)?

More or less. These are J2EE (EAR, EJB, WAR, app-client) and
standard Jar projects of different types. The source tree of
each project has its own CVS module, which may or may not reside
within the same CVS repository.

("Repository" and "module" in the CVS sense: "Repository" is the
entity defined by $CVSROOT, which can contain any number of
"modules", which in turn are what commit and update commands are
generally applied to.)

I should also say that the dependencies are established by
referencing the ${project}/dist output of the relevant projects
as libraries, which for some reason (but desirably so) causes
the language infrastructure not to use the Jars, but the project
sources for internal compilation. Which is the reason why CVS
updates are relevant for the depending projects.

> I may need to confirm with guys who implemented it, but I think
> if B depends on A, and update of A is finished with changes
> detected then also B is rescanned because it may be affected.
> This rescan would be done regardless if update of B is running
> at that moment (because it is not triggered by update changes
> in B, but by already finished update of A).

Yeah, but wouldn't the update of B that has been initiated tell
the language support to not rescan B just now?
Or does it only if the update of B is already in progress,
rather than queued?
Or not at all?

What is undesirable, of course, is for B to be rescanned twice,
once because A was updated and a second time because B was
updated. In particular, the first rescan is likely to cause
errors because B hasn't been updated yet, or only partially.
And this multiplies/escalates with further dependencies.

And, of course, a rescan of B should not start before the
update and (relevant parts of) the rescan of A has been
completed. I'm not sure what the previous and current NB logic
for that is.

> I can see there is no efficient way to update your 3 projects,
> unless the whole update operation can be done as one - I'm not
> sure if that is possible over different repositories.

It's actually possible to multi-select (Ctrl-click) several
folders across projects and initiate a CVS command for that
selection. Not sure how that is mapped internally though.
It's certainly still multiple commands on the CVS level.
My experience has been that NB doesn't always correctly
register the changes for such a "bulk" operation, so I tend
to avoid them.

In any case, treating it as one big operation would prevent the
rescan of A being started while B is still updating, which might
actually make the whole thing slower overall. It would be better
if the rescan of B triggered by the update of A would notice that
there is an update of B pending or running, and wait until that
is finished.

-- Niklas Matthies

[Tomas Pavek]

Matthies, Niklas wrote:
>> I may need to confirm with guys who implemented it, but I think
>> if B depends on A, and update of A is finished with changes
>> detected then also B is rescanned because it may be affected.
>> This rescan would be done regardless if update of B is running
>> at that moment (because it is not triggered by update changes
>> in B, but by already finished update of A).
>
> Yeah, but wouldn't the update of B that has been initiated tell
> the language support to not rescan B just now?

No, it does not work that way.

> Or does it only if the update of B is already in progress,
> rather than queued?
> Or not at all?

If VCS update of a project is in progress, it tells the language
infrastructure not to react on file changes within this project. So files
changed by the update will not provoke a new scan. However, if the scan of
this project is triggered by a change in another project (already finished VCS
update), it will run even though the project is just being updated and not
watched for changes. So you may get the dependent project scanned twice. The
only solution is to run update of both projects as one operation. That's at
least how it is implemented now. (If you select multiple projects and invoke
the update command on them, you get this.)

>
> In any case, treating it as one big operation would prevent the
> rescan of A being started while B is still updating, which might
> actually make the whole thing slower overall. It would be better
> if the rescan of B triggered by the update of A would notice that
> there is an update of B pending or running, and wait until that
> is finished.

A more advanced way you describe might be better, i.e. allow some part of
scanning run even during VCS update, but we are not there yet. For now the
improvement is that we eliminated multiple rescans of incomplete data during a
long VCS update.

   - Tomas