[ruby-core:26361] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

View: New views
20 Messages — Rating Filter:   Alert me  
< Prev | 1 - 2 | Next >

[ruby-core:26361] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Feature #2294: [PATCH] ruby_bind_stack() to embed Ruby in coroutine
http://redmine.ruby-lang.org/issues/show/2294

Author: Suraj Kurapati
Status: Open, Priority: Normal
Category: core, Target version: 1.9.2

Hi,

I am attaching a "ruby_bind_stack.patch" patch file
that adds a ruby_bind_stack() function to the Ruby C API.

This function allows me to inform the GC about the stack
boundaries of the coroutine inside which Ruby is embedded:

  void ruby_bind_stack(void *lower, void *upper);

I am also attaching tarballs containing code examples that
embed Ruby inside two different coroutine environments:
UNIX System V contexts[1] and libpcl[2] coroutines.

Each tarball has an "output.log" file which contains the
result of running `script -c ./run.sh output.log` on my
machine:

Linux yantram 2.6.31-ARCH #1 SMP PREEMPT Tue Oct 13 13:36:23 CEST 2009 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux

The last section in "output.log" corresponds to Ruby @ SVN
trunk that is patched with the "ruby_bind_stack.patch"
patch file that is attached to this issue.

Thanks for your consideration.

[1]: http://www.gnu.org/s/libc/manual/html_node/System-V-contexts.html
[2]: http://www.xmailserver.org/libpcl.html

See also:
* http://redmine.ruby-lang.org/issues/show/2258
* http://redmine.ruby-lang.org/issues/show/2126


----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26376] Re: [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Nobuyoshi Nakada-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,

At Wed, 28 Oct 2009 02:03:01 +0900,
Suraj Kurapati wrote in [ruby-core:26361]:
> I am attaching a "ruby_bind_stack.patch" patch file
> that adds a ruby_bind_stack() function to the Ruby C API.

This patch does not work with multithreading at all.

--
Nobu Nakada


[ruby-core:26380] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack.patch added

Hi,

Nobu Nakada wrote:
> This patch does not work with multithreading at all.

Thank you for pointing out this problem.  I have updated
my patch accordingly and am reattaching it to this issue.

Here is my approach for solving this problem:
(Please correct me if I am wrong.)

Since Ruby 1.9 threads are native kernel threads, they
dynamically allocate and manage their own stacks.  So
the ruby_bind_stack() GC marking restriction must only
be applied to the main Ruby thread---which isn't really
a thread at all; it runs on the native C program stack.

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26384] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby-ucontext-thread.tgz added
File ruby-libpcl-thread.tgz added

Hi,

I am attaching two updated code examples which
test the multi-threading support of my updated
"ruby_bind_stack.patch" patch file.

One example uses UNIX System V contexts and the
other uses libpcl for embedding Ruby in coroutine.

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26387] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Roman Shterenzon.


I'm embedding a Ruby 1.9.1 in my app, and it would die with segmentation fault, and I was suspicious about the stack, as it's multithreaded. I applied your patch and it looks fine so far. I'm using the following code (and assume that stack grows upward):

static void pthread_get_stack(void **stack_begin, void **stack_end) {
        size_t stack_size;
#if defined(HAVE_STACKADDR_NP) && defined(HAVE_GET_STACKSIZE_NP) /* MacOS X */
        pthread_t t_id = pthread_self();
        *stack_begin = pthread_get_stackaddr_np(t_id);
        stack_size = pthread_get_stacksize_np(t_id);
#else /* Linux */
        pthread_attr_t attr;
        pthread_getattr_np(pthread_self(), &attr);
        pthread_attr_getstack(&attr, stack_begin, &stack_size);
#endif
        *stack_end = *stack_begin + stack_size;
}

----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26391] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.


Hi Roman,

I did not understand your code example.
Where do you call ruby_bind_stack() ?

Without that call, I don't see how my patch
can make any difference to your program.

Thanks.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26396] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Roman Shterenzon.


Sorry for the lack of explanation, I thought that it was implicit and apparent.
After I get thread's stack_begin and stack_end I'm calling your bind stack function, of course.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26398] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.


Roman, thanks for clarifying.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26418] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack.patch added

Hi,

I am attaching an updated "ruby_bind_stack.patch" file which adds:

* API documentation for the ruby_bind_stack() function in ruby.h
* an assertion to ensure that upper > lower inside ruby_bind_stack()

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26444] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack_after_refactoring.patch added

Hi,

To reduce your risk of applying (or even considering) this patch,
I moved the refactoring of *duplicated* machine stack calculation
code into a new "get_machine_stack_bounds.patch" file on this issue:

  http://redmine.ruby-lang.org/issues/show/2315

I am attaching a new "ruby_bind_stack_after_refactoring.patch" which
basically contains the result of "ruby_bind_stack.patch" *minus* the
changes in the "get_machine_stack_bounds.patch" mentioned above.

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26452] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack.patch added
File ruby_bind_stack_after_refactoring.patch added

Hi,

I'm attaching updated patches that contain better API documentation:

/*
 *  Binds the stack of Ruby's main thread to the region of memory that spans
 *  inclusively from the given lower boundary to the given upper boundary:
 *
 *    lower boundary <= stack pointer of Ruby's main thread <= upper boundary
 *
 *  These boundaries *do not* protect Ruby's main thread against stack
 *  overflow and they *do not* apply to non-main Ruby threads (whose stacks
 *  are dynamically allocated and managed by the native Operating System).
 */
void ruby_bind_stack(void *lower_boundary, void *upper_boundary);

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26454] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack.patch added
File ruby_bind_stack_after_refactoring.patch added

Hi,

I'm attaching updated patches that reduce the
runtime overhead of stack bound correction.

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26502] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.

File ruby_bind_stack_r25604.patch added

Hi,

Since my refactoring patch (from issue #2315) was accepted in r25604,
I am attaching a new "ruby_bind_stack_r25604.patch" file accordingly.

Thanks for your consideration.


----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26550] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.


Hi,

According to Matz's suggestion in [ruby-core:25139], I wrote
a detailed explanation of the problem this patch solves. I
hope this explanation is helpful.  Please do not hesitate
to ask for clarifications or to correct any misunderstandings.

Thanks for your thoughtful consideration.

== Introduction

The patch adds a ruby_bind_stack() function to the Ruby C API.
This function allows the person who is embedding Ruby to
tell the Ruby GC about the stack boundaries of the embedded
environment:

  void ruby_bind_stack(VALUE *lower_bound, VALUE *upper_bound);

In order to understand why this function is important, please
consider the following two modes of operation: normal & embedded.


== Normal operation: Ruby runs in a C program's main()

Initially, Ruby assumes that the stack of Ruby's main
thread exists in a high memory address range, like this:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary
               
  (low memory address)

As Ruby runs, the lower boundary is adjusted (by the
SET_STACK_END macro) to reflect the machine stack pointer:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary (not changed)
 
  0xc0ff1e80    Ruby's stack lower boundary
                (after update by SET_STACK_END)

  (low memory address)


== Embedded operation: Ruby runs inside a C coroutine

Initially, Ruby assumes that the stack of Ruby's main
thread exists in a high memory address range, like this:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary
               
  (low memory address)

However, the stack of the C coroutine (which runs Ruby)
exists at a low memory address range, because it is
statically allocated:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary

  0x086032a0    System V context's stack upper boundary
               
  0x082032a0    System V context's stack lower boundary
               
  (low memory address)

As Ruby runs, the lower boundary is adjusted (by the
SET_STACK_END macro) to reflect the machine stack pointer:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0x086032a0    System V context's stack upper boundary
 
  0x08601680    Ruby's stack lower boundary
                (after update by SET_STACK_END)
               
  0x082032a0    System V context's stack lower boundary
 
  (low memory address)


See the problem?  Ruby's stack and the C coroutine stack
do not agree.  They overlap!

This situation becomes worse (and causes a segfault) when
the Ruby GC runs: it marks VALUEs in the Ruby stack, which
currently contains all of the heap memory!  Somewhere in
the vast heap memory, it finds and dereferences a NULL value
and BOOM! a segfault occurs.  :-)

To solve this problem, the ruby_bind_stack() function corrects
Ruby's stack to reflect the stack boundaries of the C coroutine:

  (high memory address)
 
  0x086032a0    Ruby's stack upper boundary and also
                System V context's stack upper boundary
 
  0x08601680    Ruby's stack lower boundary
                (after update by SET_STACK_END)
               
  0x082032a0    System V context's stack lower boundary
 
  (low memory address)

Now, when the Ruby GC runs, it marks VALUEs in the correct
memory region.  It does not travel into heap memory and
cause a segfault.

That is all.  Thanks for reading!
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26661] Re: [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Roman Shterenzon-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,

I would like to say that without applying this patch, my ruby interpreter, embedded in a pthread, would cause a segmentation fault as soon as GC was invoked. I would like to see this applied to 1.9.1 as well as http://redmine.ruby-lang.org/issues/show/2279 . Without these, it's hardly possible to have ruby 1.9.1 embedded in a useful way.

Thanks,
--Roman


----- Original Message ----
From: Suraj Kurapati <redmine@...>
To: ruby-core@...
Sent: Thu, November 5, 2009 9:27:17 AM
Subject: [ruby-core:26550] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

Issue #2294 has been updated by Suraj Kurapati.


Hi,

According to Matz's suggestion in [ruby-core:25139], I wrote
a detailed explanation of the problem this patch solves. I
hope this explanation is helpful.  Please do not hesitate
to ask for clarifications or to correct any misunderstandings.

Thanks for your thoughtful consideration.

== Introduction

The patch adds a ruby_bind_stack() function to the Ruby C API.
This function allows the person who is embedding Ruby to
tell the Ruby GC about the stack boundaries of the embedded
environment:

  void ruby_bind_stack(VALUE *lower_bound, VALUE *upper_bound);

In order to understand why this function is important, please
consider the following two modes of operation: normal & embedded.


== Normal operation: Ruby runs in a C program's main()

Initially, Ruby assumes that the stack of Ruby's main
thread exists in a high memory address range, like this:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary
               
  (low memory address)

As Ruby runs, the lower boundary is adjusted (by the
SET_STACK_END macro) to reflect the machine stack pointer:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary (not changed)
 
  0xc0ff1e80    Ruby's stack lower boundary
                (after update by SET_STACK_END)

  (low memory address)


== Embedded operation: Ruby runs inside a C coroutine

Initially, Ruby assumes that the stack of Ruby's main
thread exists in a high memory address range, like this:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary
             
  (low memory address)

However, the stack of the C coroutine (which runs Ruby)
exists at a low memory address range, because it is
statically allocated:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0xbffff1f0    Ruby's stack lower boundary

  0x086032a0    System V context's stack upper boundary
               
  0x082032a0    System V context's stack lower boundary
               
  (low memory address)

As Ruby runs, the lower boundary is adjusted (by the
SET_STACK_END macro) to reflect the machine stack pointer:

  (high memory address)
 
  0xc1bff1f0    Ruby's stack upper boundary
 
  0x086032a0    System V context's stack upper boundary
 
  0x08601680    Ruby's stack lower boundary
                (after update by SET_STACK_END)
               
  0x082032a0    System V context's stack lower boundary
 
  (low memory address)


See the problem?  Ruby's stack and the C coroutine stack
do not agree.  They overlap!

This situation becomes worse (and causes a segfault) when
the Ruby GC runs: it marks VALUEs in the Ruby stack, which
currently contains all of the heap memory!  Somewhere in
the vast heap memory, it finds and dereferences a NULL value
and BOOM! a segfault occurs.  :-)

To solve this problem, the ruby_bind_stack() function corrects
Ruby's stack to reflect the stack boundaries of the C coroutine:

  (high memory address)
 
  0x086032a0    Ruby's stack upper boundary and also
                System V context's stack upper boundary
 
  0x08601680    Ruby's stack lower boundary
                (after update by SET_STACK_END)
               
  0x082032a0    System V context's stack lower boundary
 
  (low memory address)

Now, when the Ruby GC runs, it marks VALUEs in the correct
memory region.  It does not travel into heap memory and
cause a segfault.

That is all.  Thanks for reading!
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


     


[ruby-core:26797] [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Suraj Kurapati.


Hi,

Sorry to be impatient, but has there been any
further decision or consideration about this patch?

The only feedback I've received so far is that:

* An early version of this patch did not support
  multi-threading (thanks to Mr. Nobu).

* A later version of this patch worked for embedding
  Ruby 1.9 inside a pthread (thanks to Mr. Roman).

The silent suspense is "killing" me, so to speak. :-)

Thanks for your consideration.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26799] Re: [Feature #2294] [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Nobuyoshi Nakada-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,

At Wed, 18 Nov 2009 15:55:07 +0900,
Suraj Kurapati wrote in [ruby-core:26797]:
> Sorry to be impatient, but has there been any
> further decision or consideration about this patch?

Sorry to be late.

> The only feedback I've received so far is that:
>
> * An early version of this patch did not support
>   multi-threading (thanks to Mr. Nobu).
>
> * A later version of this patch worked for embedding
>   Ruby 1.9 inside a pthread (thanks to Mr. Roman).

Switching stack using setcontext() can't work on all platforms.
For instance, on NetBSD and older LinuxThread stack address is
tightly bound to thread, and can't be changed.  That is, your
strategy is not portable.

Why don't you simply use a thread instead?

--
Nobu Nakada


[ruby-core:26801] [Feature #2294](Closed) [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Nobuyoshi Nakada.

Status changed from Open to Closed
% Done changed from 0 to 100

This issue was solved with changeset r25842.
Suraj, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26802] [Feature #2294](Open) [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Nobuyoshi Nakada.

Status changed from Closed to Open

Sorry, mistaken.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org


[ruby-core:26803] [Feature #2294](Rejected) [PATCH] ruby_bind_stack() to embed Ruby in coroutine

by Usaku NAKAMURA :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Issue #2294 has been updated by Nobuyoshi Nakada.

Status changed from Open to Rejected

I don't think it's good idea to it as public API.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2294

----------------------------------------
http://redmine.ruby-lang.org

< Prev | 1 - 2 | Next >