Bug Report: Ecasound goes silent

[S. Massy-4]

Bug report:

Symptoms:
No more sound is heard, but the engine still seems to be running (clock
is moving, engine-status is "running", seems to obey transport
commands).

Temporary solution:
cs-disconnect/cs-connect restores sound.

Triggers:
- Transport commands: fw/rw/setpos, occasionally start/stop
- On copp-set, especially multiple ones.
  (In nama terms, mostly on solo and mute/unmute, which means multiple
  copp-set on eadb ops)
- More rarely, it happens for no obvious reason while playing back.

Remarks:
- The more complex a chainsetup is, the more likely it is to happen.
- Th shorter the JACK period (-p) is, the more likely it is to happen.
- Nama manages to mitigate this problem by introducing a sleep period
  between operations, though this does not entirely eliminate the
  problem.
  (e.g stop, sleep, fw, sleep, start)

Impact:
Usability is affected on complex chainsetups using the IAM or ECI.
Usability might also be affected on very large chainsetups in batch
mode. For a DAW like nama using ecasound as its backend, it means
rearming 1-5 times an hour on a modest project (5-10 tracks) while
mixing.

Software info:
- Ecasound 2.8.1 (also observed on 2.8.0)
- jackdmp version 1.9.7 (also observed on jackd1 0.118)
- JACK cmdline: "jackd -R -P65 -t1000 -dalsa -d hw:1,0 -r48000 -p128 -n3
                       -M -H -Xseq"
- kernel 2.6.33.7.2-rt30 SMP PREEMPT RT x86_64
- Debian Squeeze with some custom software

Case study:
I have put together a downloadable project for debugging purposes.
Thanks go to the packet-in band[1] for allowing me to use the material
from a song[2] I helped mix.
Link: http://wolfdream.ca/files/bugreport_smassy.tar.bz2
After untarring, you'll find in the bugreport_smassy directory:
- bugreport_smassy.ecs (The chainsetup)
- bugreport_smassy.log (A log of the bug being reproduced as described
  below)
- wav (A subdir with the audio material)
(NOTE: The audio material is flac compressed for convenience, so you
need to do:
$ cd wav;flac -d *flac;rm *flac;cd ..
...to make it ready for testing.)
(NOTE: The following ladspa plugins should be installed for the project
to work as expected: swh-plugins, tap-plugins,
invada-studio-plugins-ladspa, fil-plugins)

Reproducing the bug:
The simplest but by no means the only way to reproduce this bug is to
do (with jackd running):
$ ecasound -s bugreport_smassy.ecs -c
(In IAM)
> cs-connect
> setpos 10
> start
> fw 20

Please, let me know if I can be of any further assistance with this.

REFERENCES:
1. http://packet-in.org
2. http://packet-in.org/repo/RPM12/Marianas_Trench/attribution.txt

Cheers,
S.M.


--

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[Kai Vehmanen]

Hi,

> Symptoms:
> No more sound is heard, but the engine still seems to be running (clock
> is moving, engine-status is "running", seems to obey transport
> commands).

thanks for the excellent bugreport! I managed to reproduce this with your
test session, and am looking for the root cause. Upon initial look, this
would seem to be an issue with the JACK callbacks. I also got one crash
while ecasound called jack_port_get_latency_range(). That is suspicous as
the semantics of that call have changes a bit (since the related ecasound
code was written).

This looks serious enough that I've added it on top of my 2.8.2/2.9
release laundry list.

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[S. Massy-4]

On Thu, Mar 15, 2012 at 09:42:49PM +0200, Kai Vehmanen wrote:

> Hi,
>
> >Symptoms:
> >No more sound is heard, but the engine still seems to be running (clock
> >is moving, engine-status is "running", seems to obey transport
> >commands).
>
> thanks for the excellent bugreport! I managed to reproduce this with
> your test session, and am looking for the root cause. Upon initial
> look, this would seem to be an issue with the JACK callbacks. I also
> got one crash while ecasound called jack_port_get_latency_range().
> That is suspicous as the semantics of that call have changes a bit
> (since the related ecasound code was written).
>
> This looks serious enough that I've added it on top of my 2.8.2/2.9
> release laundry list.

Wonderful! Let me know if you ever need extra testing/debugging done.

Cheers,
S.M.

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[S. Massy-4]

On Thu, Mar 15, 2012 at 09:42:49PM +0200, Kai Vehmanen wrote:

> Hi,
>
> >Symptoms:
> >No more sound is heard, but the engine still seems to be running (clock
> >is moving, engine-status is "running", seems to obey transport
> >commands).
>
> thanks for the excellent bugreport! I managed to reproduce this with
> your test session, and am looking for the root cause. Upon initial
> look, this would seem to be an issue with the JACK callbacks. I also
> got one crash while ecasound called jack_port_get_latency_range().
> That is suspicous as the semantics of that call have changes a bit
> (since the related ecasound code was written).

Two more interesting facts:
- When the bug is triggered, the clock stops updating in real-time and
  instead seems to be refreshed every 4 or 5 seconds, jumping by that
  about that amount.
- Ocasionally, eeven jackd needs to be restarted in order for ecasound
  to work with jack again, suggesting some gremlins in the jack client
  code as you mention.

Cheers,
S.M.

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[Joel Roth-2]

On Tue, Apr 03, 2012 at 02:20:08PM -0400, S. Massy wrote:

> On Thu, Mar 15, 2012 at 09:42:49PM +0200, Kai Vehmanen wrote:
> > Hi,
> >
> > >Symptoms:
> > >No more sound is heard, but the engine still seems to be running (clock
> > >is moving, engine-status is "running", seems to obey transport
> > >commands).
> >
> > thanks for the excellent bugreport! I managed to reproduce this with
> > your test session, and am looking for the root cause. Upon initial
> > look, this would seem to be an issue with the JACK callbacks. I also
> > got one crash while ecasound called jack_port_get_latency_range().
> > That is suspicous as the semantics of that call have changes a bit
> > (since the related ecasound code was written).
> Two more interesting facts:
> - When the bug is triggered, the clock stops updating in real-time and
>   instead seems to be refreshed every 4 or 5 seconds, jumping by that
>   about that amount.

Seeking can cause problems with JACK because repositioning
in multiple files can take more time than a running JACK
transport allows to its clients.

Nama deals with this by stopping the transport (if running),
issuing a seek command, allowing a short time for Ecasound
to update file positions, then restarting the transport.

Except that you found several cases where Nama was using
Ecasound setpos rather than wrapping it with this
special logic.

Will be interested to see if your problems are reduced by
these fixes to Nama.

Regards,

Joel

> - Ocasionally, eeven jackd needs to be restarted in order for ecasound
>   to work with jack again, suggesting some gremlins in the jack client
>   code as you mention.
>
> Cheers,
> S.M.
--
Joel Roth

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[S. Massy-4]

On Tue, Apr 03, 2012 at 10:10:11AM -1000, Joel Roth wrote:

> On Tue, Apr 03, 2012 at 02:20:08PM -0400, S. Massy wrote:
> > On Thu, Mar 15, 2012 at 09:42:49PM +0200, Kai Vehmanen wrote:
> > > Hi,
> > >
> > > >Symptoms:
> > > >No more sound is heard, but the engine still seems to be running (clock
> > > >is moving, engine-status is "running", seems to obey transport
> > > >commands).
> > >
> > > thanks for the excellent bugreport! I managed to reproduce this with
> > > your test session, and am looking for the root cause. Upon initial
> > > look, this would seem to be an issue with the JACK callbacks. I also
> > > got one crash while ecasound called jack_port_get_latency_range().
> > > That is suspicous as the semantics of that call have changes a bit
> > > (since the related ecasound code was written).
> > Two more interesting facts:
> > - When the bug is triggered, the clock stops updating in real-time and
> >   instead seems to be refreshed every 4 or 5 seconds, jumping by that
> >   about that amount.
>
> Seeking can cause problems with JACK because repositioning
> in multiple files can take more time than a running JACK
> transport allows to its clients.
>
> Nama deals with this by stopping the transport (if running),
> issuing a seek command, allowing a short time for Ecasound
> to update file positions, then restarting the transport.
>
> Except that you found several cases where Nama was using
> Ecasound setpos rather than wrapping it with this
> special logic.
>
> Will be interested to see if your problems are reduced by
> these fixes to Nama.

That definitely contributes to making nama more stable by a significant
margin, but remember that the bug isn't merely triggered by seeking but
also by mute/solo, especially as the number of tracks/busses grows. It
could be ecasound needs to have the ability to skip a jack period if it
can't meet the deadline, or perhaps a mechanism to throttle the rate of
iam/eci commands it can process within a given timeframe.

Cheers,
S.M.

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[Joel Roth-2]

On Thu, Apr 05, 2012 at 11:56:55PM -0400, S. Massy wrote:
> ...remember that the bug isn't merely triggered by seeking but
> also by mute/solo, especially as the number of tracks/busses grows. It
> could be ecasound needs to have the ability to skip a jack period if it
> can't meet the deadline, or perhaps a mechanism to throttle the rate of
> iam/eci commands it can process within a given timeframe.

Perhaps Kai will have some ideas on the Ecasound side.
Certainly, it would help us to have a way of testing
how fast Ecasound can process commands when operating
under JACK.

For Nama, I will tweak the commands to make less demands
on Ecasound, for example, using fewer steps to fade
during mute/unmute.

Regards,

Joel
 
> Cheers,
> S.M.
--
Joel Roth

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[S. Massy-4]

On Sat, Mar 10, 2012 at 04:02:54PM -0500, S. Massy wrote:

> Bug report:
>
> Symptoms:
> No more sound is heard, but the engine still seems to be running (clock
> is moving, engine-status is "running", seems to obey transport
> commands).
>
> Temporary solution:
> cs-disconnect/cs-connect restores sound.
>

[...]
>
> Impact:
> Usability is affected on complex chainsetups using the IAM or ECI.
> Usability might also be affected on very large chainsetups in batch
> mode. For a DAW like nama using ecasound as its backend, it means
Not sure whether it's the same bug, but definitely jack support related:
if you run my demo setup in batch mode and then hit CTRL-C to terminate
ecasound, further instances will be silent when using jack. The only way
to get sound back is restarting jackd.

Cheers,
S.M.

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list

[Joel Roth-2]

On Sun, Apr 15, 2012 at 08:08:15PM -0400, S. Massy wrote:
> Not sure whether it's the same bug, but definitely jack support related:
> if you run my demo setup in batch mode and then hit CTRL-C to terminate
> ecasound, further instances will be silent when using jack. The only way
> to get sound back is restarting jackd.

You may like to 'pgrep ecasound' to see if any ecasound
processes are lying around after your CTRL-C.

--
Joel Roth

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecasound-list mailing list
Ecasound-list@...
https://lists.sourceforge.net/lists/listinfo/ecasound-list