Multi-Master Memcached Replication

[Kris Drebin]

Greetings everyone,

I'm trying to setup the following scenario.. I thought you all would probably have some great advise. Here's the idea:

I currently have:

1 Tomcat Web Server
1 Memcached Server
1 Postgresql database

In the event that any of these services fail, my web application does not work properly.

I would like to create the scenario where I have:

2 tomcat web servers
2 memcached servers
1 Postgresql database

So that the memcached servers are staying in sync with each other, in a multi-master fashion. I found the tool repcached, but it is single-master, single slave. This will not work for my application. Any ideas?

One more piece of information is that my Postgresql database does not currently support multi-master replication. I'm in the process of migrating to MySQL to achieve this, but that's a ways out.. so if you were thinking of that avenue, unfortunately it's not available quite yet.

Your ideas are appreciated!

Thanks,
Kris Drebin

[Brian Moon-2]

Kris Drebin wrote:
> So that the memcached servers are staying in sync with each other, in a
> multi-master fashion. I found the tool repcached, but it is
> single-master, single slave. This will not work for my application. Any
> ideas?

My first question is of course, why do you need multi-master memcached
servers?  Even the largest memcached installations don't duplicate data.
  I am just curious why you believe it to be the solution for you.

Second, you would have to build this into your application logic.  Brian
Aker's new library will support this, but it is not ready for production
use afaik.

--

Brian Moon
Senior Developer/Engineer
------------------------------
When you care enough to spend the very least.
http://dealnews.com/

[Brian Aker-6]

Hi!

On Apr 21, 2008, at 11:54 AM, Kris Drebin wrote:
> So that the memcached servers are staying in sync with each other,  
> in a multi-master fashion. I found the tool repcached, but it is  
> single-master, single slave. This will not work for my application.  
> Any ideas?

We are very close to pushing the replia code into libmemcached. The  
only issue for Postgres is that no one has taken the time to write  
UDF's like what have been written for MySQL.

Cheers,
        -Brian

BTW this is a hint for someone to write them...

--
_______________________________________________________
Brian "Krow" Aker, brian at tangent.org
Seattle, Washington
http://krow.net/                     <-- Me
http://tangent.org/                <-- Software
_______________________________________________________
You can't grep a dead tree.

[Dormando]

> I would like to create the scenario where I have:
>
> 2 tomcat web servers
> 2 memcached servers
> 1 Postgresql database
>
> So that the memcached servers are staying in sync with each other, in a
> multi-master fashion. I found the tool repcached, but it is
> single-master, single slave. This will not work for my application. Any
> ideas?

Be sure you're doing it right. The "normal" memcached pattern is to add
servers to a cloud. So if you lose one server your hit ratio will be cut
by 50% until that server comes back up or you add another one. If you
have three memcached's and you lose one, the cache hit ratio will drop
by 33%, and so on.

If you require master:master replication that will stunt your growth to
two servers max. Among other issues...

There's another replication option which Brian just mentioned, where you
may send multiple replicas of one cache item into the cloud. That
approach doesn't stunt growth, but halves effective cache size. Since
you're storing everything at least twice.

This is like RAID10'ing your drives. The cost becomes more obviously
painful as you add servers. If you had 100 memcached's and one died
you'd get a 1% hit to your hit ratio. If you're replicating you'd
probably need 200 memcached's to get the same memory capacity, but avoid
the 1% drop...

So you should weigh these options and do what makes the best sense for
your project.

-Dormando

[HIROSE, Masaaki]

hi!

repcached 2.0 supports master-master replication. 2.0's replication
cooperate with client library's server balancing feature.

            +--------+
            | client |
            +--------+
      key:foo    |    key:bar
         +-------+-------+
         |               |
         v               v
  +----------+       +----------+
  |  mcd-A   |       |  mcd-B   |
  |+--------+|repl   |+--------+|
  ||        ||------->|        ||
  ||key:foo ||       ||key:foo ||
  ||........||       ||........||
  ||        |<-------||        ||
  ||key:bar ||   repl||key:bar ||
  |+--------+|       |+--------+|
  +----------+       +----------+

 * client sets servers = [ 'mcd-A', 'mcd-B' ]
 * key:foo always access to server mcd-A
 * key:bar always access to server mcd-B
 * there is no conflict data replication between mcd-A and mcd-B
   because memory area is separated by client library's server
   balancing.

please try repcached-2.0!:
http://repcached.lab.klab.org/



in "Multi-Master Memcached Replication"
   <223a68c0804211154kab6d45bh812d8ce8bc52f68c@...>
at Mon, 21 Apr 2008 11:54:57 -0700,
   kris.drebin@... wrote:

> [1  <text/plain; ISO-8859-1 (7bit)>]
> Greetings everyone,
>
> I'm trying to setup the following scenario.. I thought you all would
> probably have some great advise. Here's the idea:
>
> I currently have:
>
> 1 Tomcat Web Server
> 1 Memcached Server
> 1 Postgresql database
>
> In the event that any of these services fail, my web application does not
> work properly.
>
> I would like to create the scenario where I have:
>
> 2 tomcat web servers
> 2 memcached servers
> 1 Postgresql database
>
> So that the memcached servers are staying in sync with each other, in a
> multi-master fashion. I found the tool repcached, but it is single-master,
> single slave. This will not work for my application. Any ideas?
>
> One more piece of information is that my Postgresql database does not
> currently support multi-master replication. I'm in the process of migrating
> to MySQL to achieve this, but that's a ways out.. so if you were thinking of
> that avenue, unfortunately it's not available quite yet.
>
> Your ideas are appreciated!
>
> Thanks,
> Kris Drebin
> [2  <text/html; ISO-8859-1 (7bit)>]
>

[Golan Zakai]

Kris Drebin wrote:

So that the memcached servers are staying in sync with each other, in a
multi-master fashion. I found the tool repcached, but it is single-master,
single slave. This will not work for my application. Any ideas?

Maybe this setup can help you:
http://golanzakai.blogspot.com/2008/11/memcached-replication-and-namespaces.html