FF 3.5 and JS_GC Crashes

[Glenn Boysko]

Hello:

I have written an extension that analyzes the performance of our
company's multi-tier Web application. It inspects client-side behavior
(it has a builtin JavaScript profiler and analyzes HTTP resource
loading) as well as tracks backend delays (that contributes to end-to-
end performance).

The extension uses nsIWebProgressListener and jsdIDebuggerService APIs
extensively. It also features a C++ XPCOM (to monitor browser process
CPU usage) and an JavaScript XPCOM component.

We are finding many crashes associated with Firefox 3.5 that we are
not seeing with FF 3.0.x. The debug stacks seem to indicate failures
in JS_GC and other JavaScript related APIs.

Has anyone observed similar problems with the use of these APIs and FF
3.5?

Does anyone have any suggestions for diagnosing the source of problems
related to GC corruption? Or perhaps suggestions for coding that
prevents these types of problems?

Thanks,
Glenn
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Blake Kaplan]

In mozilla.dev.platform Glenn Boysko <gboysko@...> wrote:
> The extension uses nsIWebProgressListener and jsdIDebuggerService APIs
> extensively. It also features a C++ XPCOM (to monitor browser process
> CPU usage) and an JavaScript XPCOM component.

Hi Glenn,

Are you dealing with JavaScript values in your C++? If so, then you might be
running into what we call garbage collection hazards: the JS garbage collector
doesn't know about references to JS objects from the C++ stack or heap, so
unless you explicitly tell it about your references, it will collect objects
that you're holding (see
<https://developer.mozilla.org/en/SpiderMonkey_Garbage_Collection_Tips> for
more information about this).

It's also possible that you're running into bugs in either the engine (which
is susceptable to the same sorts of problem) or in the implementation of
jsdIDebuggerService... It's hard to tell exactly what's going on from here,
though.

There are a few debugging tricks you can use to track down these sorts of
problems, but they require a debug build.
--
Blake Kaplan
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Glenn Boysko]

On Jul 23, 4:20 pm, Blake Kaplan <mrb...@...> wrote:
> Are you dealing with JavaScript values in your C++? If so, then you might be
> running into what we call garbage collection hazards: the JS garbage collector
> doesn't know about references to JS objects from the C++ stack or heap, so
> unless you explicitly tell it about your references, it will collect objects
> that you're holding (see
> <https://developer.mozilla.org/en/SpiderMonkey_Garbage_Collection_Tips> for
> more information about this).

The C++ XPCOM component simply exposes a service which can return the
current (instantaneous) CPU for a given Win32 process. As such, I
don't believe there could be any JavaScript objects involved, right?
The IDL for the component is:

interface ICpuMonitor : nsISupports
{
  long GetPercentageCpuTime();
  attribute boolean enableMonitoring;
};

> It's also possible that you're running into bugs in either the engine (which
> is susceptable to the same sorts of problem) or in the implementation of
> jsdIDebuggerService... It's hard to tell exactly what's going on from here,
> though.
>
> There are a few debugging tricks you can use to track down these sorts of
> problems, but they require a debug build.

I see. I haven't used a debug build before. Do you archive a debug
build of the 3.5 GA release? If so, let me know where it is located
and the "debugging tricks" and I'll try to track down these crashes.

Thanks,
Glenn
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Jean-Marc Desperrier]

Blake Kaplan wrote:
> There are a few debugging tricks you can use to track down these sorts of
> problems, but they require a debug build.

Using the symbol/source server with a release version really won't do ?

_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[dbradley-3]

On Jul 23, 10:27 am, Glenn Boysko <gboy...@...> wrote:
> We are finding many crashes associated with Firefox 3.5 that we are
> not seeing with FF 3.0.x. The debug stacks seem to indicate failures
> in JS_GC and other JavaScript related APIs.

Are any of the calls to the JS component on a thread other than the
main thread?

David
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Glenn Boysko]

> > We are finding many crashes associated with Firefox 3.5 that we are
> > not seeing with FF 3.0.x. The debug stacks seem to indicate failures
> > in JS_GC and other JavaScript related APIs.
>
> Are any of the calls to the JS component on a thread other than the
> main thread?

I believe that all of the calls to the JS XPCOM component are on the
main thread. The JS XPCOM component creates a Timer and updates a
series of broadcast elements when the timer is handled. Does that mean
that the JS XPCOM component is being called (internally) on another
thread if it invoked from a nsITimer?

If it was being invoked from another thread (other than the main),
what should we do? Are there specially precautions I can take to
prevent GC crashes?

Thanks,
Glenn
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[dbradley-3]

On Jul 24, 11:45 am, Glenn Boysko <gboy...@...> wrote:

> > > We are finding many crashes associated with Firefox 3.5 that we are
> > > not seeing with FF 3.0.x. The debug stacks seem to indicate failures
> > > in JS_GC and other JavaScript related APIs.
>
> > Are any of the calls to the JS component on a thread other than the
> > main thread?
>
> I believe that all of the calls to the JS XPCOM component are on the
> main thread. The JS XPCOM component creates a Timer and updates a
> series of broadcast elements when the timer is handled. Does that mean
> that the JS XPCOM component is being called (internally) on another
> thread if it invoked from a nsITimer?
>
> If it was being invoked from another thread (other than the main),
> what should we do? Are there specially precautions I can take to
> prevent GC crashes?

Sounds like your fine. The timer fires on the main thread event queue.
I just wanted to make sure you were specifically creating a thread to
do some work and invoking some JS code on that. That causes lots of
problems, since GC is to only occur on the main thread.

David
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Glenn Boysko]

David:

> Sounds like your fine. The timer fires on the main thread event queue.
> I just wanted to make sure you were specifically creating a thread to
> do some work and invoking some JS code on that. That causes lots of
> problems, since GC is to only occur on the main thread.

Thanks for the reply. What about the callbacks from
jsdIDebuggerService methods (e.g., enumerateScripts)? Are they called
back on the main thread? Same goes for the nsIWebProgressListener
callbacks (e.g., onStateChange).

Can you get GC problems if you don't call QueryInterface in JS code?
I'm thinking specifically of the instances supplied in
nsIWebProgressListener callbacks (e.g., going from nsIRequest to
nsIHttpChannel).

Any suggestions are appreciated. Can a debug version of FF help to
identify the source of these crashes? Can I get debug versions of FF
without building them on my own machine?

Thanks,
Glenn
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[dbradley-3]

On Jul 27, 10:55 am, Glenn Boysko <gboy...@...> wrote:
> David:
>
> Thanks for the reply. What about the callbacks from
> jsdIDebuggerService methods (e.g., enumerateScripts)? Are they called
> back on the main thread? Same goes for the nsIWebProgressListener
> callbacks (e.g., onStateChange).

Pretty sure they don't. Most of the Mozilla code is event driven and
only a few things like Necko use threads and then it's very isolated.

> Can you get GC problems if you don't call QueryInterface in JS code?
> I'm thinking specifically of the instances supplied in
> nsIWebProgressListener callbacks (e.g., going from nsIRequest to
> nsIHttpChannel).

That wouldn't cause GC problems. If something was off there, XPConnect
would detect it and return an error.

> Any suggestions are appreciated. Can a debug version of FF help to
> identify the source of these crashes? Can I get debug versions of FF
> without building them on my own machine?

Yes, there is diagnostics and logging that you can turn on in a debug
build.  However I don't know of any distributions of debug builds. So
I think the only solution is to build it yourself.
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[John J Barton]

Glenn Boysko wrote:

> David:
>
>> Sounds like your fine. The timer fires on the main thread event queue.
>> I just wanted to make sure you were specifically creating a thread to
>> do some work and invoking some JS code on that. That causes lots of
>> problems, since GC is to only occur on the main thread.
>
> Thanks for the reply. What about the callbacks from
> jsdIDebuggerService methods (e.g., enumerateScripts)? Are they called
> back on the main thread? Same goes for the nsIWebProgressListener
> callbacks (e.g., onStateChange).

Yes, yes, yes.

>
> Can you get GC problems if you don't call QueryInterface in JS code?
> I'm thinking specifically of the instances supplied in
> nsIWebProgressListener callbacks (e.g., going from nsIRequest to
> nsIHttpChannel).

No. You can generally assume that the entire engine is single threaded,
anything that looks like another thread is based on yield. The
exceptions are small enough you have to work to find them.

>
> Any suggestions are appreciated. Can a debug version of FF help to
> identify the source of these crashes?

I don't think you should plan on using debugger on GC crashes, you will
waste a lot of time.  The crash will occur in GC but the bug is in your
code: the debugger will lead you astray.

jjb
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Blake Kaplan]

In mozilla.dev.platform Glenn Boysko <gboysko@...> wrote:
> I see. I haven't used a debug build before. Do you archive a debug
> build of the 3.5 GA release? If so, let me know where it is located
> and the "debugging tricks" and I'll try to track down these crashes.

Unfortunately not. You'd have to build your own. The problem with GC crashes
is that very often the code that caused the bug happened a *long* time ago,
and we only notice (and crash) at a much later date.
--
Blake Kaplan
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Glenn Boysko]

Thanks for all of your replies. Let me try a different question: what
is the most common way to get crashes due to JS_GC? What are the most
common pitfalls among new extension developers as they relate to
JavaScript GC problems?

You had mentioned that multiple threads can cause problems--how do you
spawn new threads in a Firefox extension?

We also have an observer service running in this extension (routing
messages). Are those all being raised on the main thread?

Thanks for any help you can provide as it relates to highlighting the
causes of JS_GC.
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger

[Glenn Boysko]

For anyone interested, I was able to resolve the crashing problems
that we were seeing in our extension. We are using the
jsdIDebuggerService interface (and, in particular, objects exposed
from this interface that extend nsIEphemeral). We had not been
checking the isValid property before accessing methods of the object
(e.g., jsdIScript.callCount).

After adding these checks, we are now running without crashing.
_______________________________________________
dev-apps-js-debugger mailing list
dev-apps-js-debugger@...
https://lists.mozilla.org/listinfo/dev-apps-js-debugger