« Return to Thread: jruby-memcache-client project started on GitHub

Re: jruby-memcache-client project started on GitHub

by Ikai Lan :: Rate this Message:

Re: [jruby-user] jruby-memcache-client project started on GitHub The threadsafety option only wraps the calls in a Mutex. There's no connection pooling, and all the connections use the same socket. In a high traffic application, the Ruby MemCache client ends up in a state where it blocks a connector thread but doesn't timeout, and the JRuby Timeout implementation should not be relied on to recover from IO blocking. Enabling Multithread support exacerbates this problem because all connectors threads become blocked waiting for the Mutex to be released. We sent the thread dumps to Sun, and Charlie and team put together a fix in 1.1.6. It seems to have helped a lot but we are still seeing issues.

The dump is below. Our application servers completely freeze up and we end up needing to restart them. This happens to us about once every two days per server. The Ruby MemCache client implementation works pretty well in an MRI environment because you can use Timeout as a fallback, and because it was never designed for multithreading.

Ikai

Thread "httpSSLWorkerThread-8080-2" thread-id 86 thread-stateBLOCKEDWaiting on lock: org.jruby.util.io.ChannelStream@1a230ce
         Owned by: httpSSLWorkerThread-8080-0 Id: 42     at: org.jruby.util.io.ChannelStream.feof(ChannelStream.java:486)
         at: org.jruby.util.io.OpenFile.checkWritable(OpenFile.java:128)
         at: org.jruby.RubyIO.write(RubyIO.java:1004)
         at: org.jruby.RubyIO$i_method_1_0$RUBYINVOKER$write.call(org/jruby/RubyIO$i_method_1_0$RUBYINVOKER$write.gen)
         at: org.jruby.runtime.CallSite$InlineCachingCallSite.call(CallSite.java:391)
         at: ruby.jit.ruby.opt.local.glassfish.domains.domain1.applications.j2ee_minus_modules.polls.WEB_minus_INF.gems.gems.activesupport_minus_2_dot_2_dot_2.lib.active_support.vendor.memcache_minus_client_minus_1_dot_5_dot_1.memcache.cache_get20982185_4606416.__file__(memcache.rb:585)
         at: org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:286)
         at: org.jruby.runtime.CallSite$InlineCachingCallSite.call(CallSite.java:442)
         at: ruby.jit.ruby.opt.local.glassfish.domains.domain1.applications.j2ee_minus_modules.polls.WEB_minus_INF.gems.gems.activesupport_minus_2_dot_2_dot_2.lib.active_support.vendor.memcache_minus_client_minus_1_dot_5_dot_1.memcache.get7349425_4606416.rescue_1$RUBY$__rescue__(memcache.rb:236)

On 12/22/08 5:24 PM, "Rich Manalang" <rich.manalang@...> wrote:

What's wrong with the standard memcache-client? We use it in production (since last year) with JRuby/Rails 2.2.2. If it has to do with the thread safety issues, we got around it by setting the threadsafety flag to true in memcache-client.

Rich

On Mon, Dec 22, 2008 at 2:34 PM, Ikai Lan <ilan@...> wrote:

It's at a state where it is functional. I'm still testing it in my application to see if I can find problems. I'm going to push my application to staging and bang on it.

Basically there's one thing might might be problematic: Marshal.dump and Marshal.load act weird. Some objects serialize using Marshal with a \222 character, which, after being stored on MemCache, on a subsequent get comes back as a weird ASCII character, and Marshal.load blows up. For this reason I'm using YAML serialization enough though YAML serialization is an order of magnitude slower. YAML deserialization is about 4 times slower.

On 12/18/08 3:43 PM, "Ikai Lan" <ilan@...> wrote:

If you've been following the thread about MemCache issues in production, you know that this has been bothering me.

Well, hopefully, no more! I've started a GitHub project to track my work here:

http://github.com/ikai/jruby-memcache-client/tree/master

Obviously, do not use this build. I'll let everyone know when the basic functionality is hammered down and usable.

Ikai

« Return to Thread: jruby-memcache-client project started on GitHub