Nabble - memcached

Failover mechanism for memcache server

2009-11-10T14:18:26Z

We have 2 memcached servers configured and use the Eniym client. When one of the server is down, it appears this server is added to the deadServers list (ServerPool.cs) and tries to resurrect the server every 10seconds (we have configured deadTimeOut to be 10seconds). Attempting to connect to the failed server causes a TCP timeout, the pages take a long time to load which results in bad user experience.

1) What is the standard way of resolving this issue? There are some posts about removing the server from the deadServers list. Is it okay to do this?

2) What is the recommended deadTimeOut setting (I understand by default it's 2 mins and we've changed it to 10seconds in our implementation)

3) Am I correct in my understanding that the cached data is not replicated across Server 1 and Server 2? If Server 1 is down, then it goes to the database to fetch these values (and it doesn't really check Server2)?

Any help is really appreciated.

(nginx) fcgi+PHP+memcache on RedHat: Class 'Memcache' not found...

2009-10-26T08:08:59Z

Hello

I have installed:
- libevent
- libmemcached (http://tangent.org/552/libmemcached.html)
- Done a PECL installation (pecl download memcached, phpize && ./configure && make)
- memcached

> cat /etc/php.d/memcached.ini
; Memcached default settings
extension=memcache.so

> ls -1 /usr/lib64/php/modules/memcache*
/usr/lib64/php/modules/memcached.so
/usr/lib64/php/modules/memcache.so

> memcached -h
memcached 1.2.6
...

> php -v
PHP 5.1.6 (cli) (built: Feb 26 2009 07:01:12)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies

/var/www/php_fcgi_memcache> cat memcache.php
<?php
$memcache = new Memcache;
print_r($memcache);
?>

/var/www/php_fcgi_memcache> php memcache.php
PHP Fatal error: Class 'Memcache' not found in /var/www/php_fcgi_memcache/memcache.php on line 2

Memcached is working fine (being use by Perl quite happily).

PHP/FCGI is working OK and phpinfo() shows:
memcached support enabled
Version 1.0.0
libmemcached version 0.34
Session support yes

Any pointers in the right direction would so very much appreciated.

sizing memcached

2009-09-20T15:01:01Z

Hi folks
We are trying to determine how to size memcached (how much memory should we allocate). The easiest way seemed to be:

. start a memcached
. keep writing items (that effectively do not expire) into the memcached till you start seeing evictions.

This would give us how many objects a given size store from which we can extrapolate.

When running such an experiment, I see the following stats after some time.

stats
STAT pid 5728
STAT uptime 2076
STAT time 1253483475
STAT version 1.2.6
STAT pointer_size 32
STAT curr_items 7497
STAT total_items 58725
STAT bytes 14178576
STAT curr_connections 7
STAT total_connections 8
STAT connection_structures 8
STAT cmd_get 0
STAT cmd_set 58725
STAT get_hits 0
STAT get_misses 0
STAT evictions 0
STAT bytes_read 110475920
STAT bytes_written 27168152
STAT limit_maxbytes 67108864
STAT threads 1

----
My question is why is curr_items so much less than total_items.? If the cache was bounced just before the experiment (which is the case here), should I consider total_items as the items stored in the cache? My sense from reading some other posts is that curr_items is not kept up to date due to efficiency reasons.

My guess in the above example is that this cache has total of 58725 items occupying 14178576 bytes.

Is this correct?

In general are there any articles on how to size memcached?

I saw some articles at http://code.google.com/p/memcached/wiki/HowTo but none seem to cover the right way of sizing memcached.

In our specific case we have just one type of object being stored in memcached - which means we may be able to store them more efficiently by following guidelines in articles dealing with storing "small" objects.

Thank you!

need help while using mecahced keys

2009-08-23T23:11:52Z

Hi All,

I am newbie to memcached. Currently i am working on PHP and i installed memcached on my system. I tired and succeed while create cache using set and get methods.

But i am confusing while assigning keys. Here is my requirement.

1) If user1 comes and fetch results from a table, fetched results are stored in cache using some key(say 'countryinfo')

then user2 comes and wants to fetch the same result from the database. Then how do we know the cache is already existed with the name 'countryinfo'.

I don't want to give the key name as static. I want it to be dynamic , coz i am running hundrads of queries in my project. It much complicate to maintain static keys for each query.

2) I am fetching different data from the same table using different where conditions. for example

SELECT * FROM tbl_users WHERE user_id = 40

SELECT username,gender FROM tbl_users WHERE user_id = 50
........
...........

In this case, How do i maintain cache

Thanks in advance

Re: PHP Warning: Memcache::pconnect(): Can't connect to 127.0.0.1:11211

2009-07-31T06:30:47Z

hi its such a bs not at all

kumar.c wrote:

Hi,

In my appliaction, I am getting below error.
*
PHP Warning: Memcache::pconnect(): Can't connect to 127.0.0.1:11211,
Connection refused (111) in /var/www/html/sample/memcache.class.php on line
26

Unable to connect the Memcache!!

*Please mail me the solution to fix above error.

Thanks in Advance,
kumar

Memcached data expiry on last access

2009-06-22T08:08:03Z

Hi,

Is anyone aware if you can choose to expire data from Memcached after a certain time period after the last update/access of the row?

Thanks,
Andy

Failover Mechanism with libmemcache

2009-06-17T01:54:37Z

Hi,

I am using repcached (memcache servers with support for replication)
servers and “libmemcached-based” PHP client.

The problem is that if the main repcached server goes for a toss, it
does not fallback to the other servers in the pool.

The fallback is happening with “zlib-based” PHP memcached client.

Is failover mechanism not there in the libmemcached based client ??
If there is any, how can I achieve this ?
I am currently using libmemcached-0.30 .

Any help would be greatly appreciated.

Thanks,
Anshul

Repcached failover mechnism with libmemcached based PHP client

2009-05-08T02:53:27Z

Hi,

I am using repcached (memcache servers with support for replication) servers and “libmemcached-based” PHP client.

The problem is that if the main repcached server goes for a toss, it does not fallback to the other servers in the pool.

The fallback is happening with “zlib-based” PHP memcached client.

Is failover mechanism not there in the libmemcached based client ?? If there is any, how can I achieve this ?

Re: Multi-Master Memcached Replication

2008-11-15T16:04:56Z

Kris Drebin wrote:

So that the memcached servers are staying in sync with each other, in a
multi-master fashion. I found the tool repcached, but it is single-master,
single slave. This will not work for my application. Any ideas?

Maybe this setup can help you:
http://golanzakai.blogspot.com/2008/11/memcached-replication-and-namespaces.html

Memcached Replication and Namespaces using MySQL Infrastructure.

2008-11-15T10:22:48Z

I would like to share with you guys a configuration method for memcached replication using mysql engine.

I wrote a blog entry describing it, I wish to hear feedback about the concept and if someone is already implemented such configuration on production environment.
http://golanzakai.blogspot.com/2008/11/memcached-replication-and-namespaces.html

Thanks
Golan Zakai

Windows client hangs when exceeding max connections

2008-11-10T06:26:42Z

We're running memcached 1.2.2 on linux, and using windows 2003 server clients. I noticed some of our memcache servers stopped responding to stats queries, and saw we have more client connection attempts than memcache was configured to handle. Using putty I could not get a stats query to respond from these servers, it would simply hang. Also a ruby script I wrote to get stats would not fail, it would just hang forever.

This has really bad side effects on our apps, where it can actually lock up request threads and cause server timeouts. Why does memcache not give back an error immediately if the connection is not available, or is this a sockets issue on windows? I've bumped up the connection limit on the server side, but I would like to know if there is something else I can do to ensure the client never gets hung, even if memcache is at its max connection limit. Would a memcache version upgrade fix this particular problem? Thanks.

Memcached + PHP Cli

2008-10-30T19:11:10Z

Memcached seems to work fine, but when I create a cron job to execute a php script using memcached, it complains about not findin the Memcache class. The cron job is simply a "php /var/www/schedule.php salt /var/www/schedule.log"

I tested that it was indeed the CLI by executing the same command in SSH and I get the same error.

How can I fix this?

MemCacheServer clustering

2008-10-06T04:10:46Z

Hi,

I am using memcached-1[1].2.6-win32-bin version of MemCache server.

Now, I have 2 instances of this server on different machine. I am using memcached-2[1].1.jar to make entry in memcache server through my java application.

When both servers are up, it makes entry in only one of the server.

Is it provide any configuration to make entry of Key-value in all memcache servers.

Waiting for your reply.
Thanking You,

Best Regards,

Jigna Joshi

Re: memcached 1.2.4 Beta for Windows

2008-09-14T10:22:19Z

Use this version of memcache for Windows. It's 1.2.5:
http://code.jellycan.com/memcached/

Works like a charm!

$memcache->set() failing, why?

2008-07-30T16:41:43Z

How can I determine why a call to set would fail? Are there any memcached logs or can I somehow peer into the process and read exceptions?

Oddly enough, I have some code here that fails to set *only* on certain keys...

For example, my script is as follows:

test_memcache('lb_top_selling_camera');
test_memcache('lb_top_selling_camera');
test_memcache('lb_top_selling_Printer');
test_memcache('lb_top_selling_camera');
test_memcache('lb_top_selling_camera');

And my output:

get lb_top_selling_camera
get lb_top_selling_camera
get lb_top_selling_printer
- set lb_top_selling_printer
- !!broken!!
get lb_top_selling_camera
get lb_top_selling_camera

The function (sloppy atm):

function test_memcache($memcache_key)
{
$g_memcache = memcache_init();

debug("get $memcache_key");

$html = $g_memcache->get($memcache_key);

if ($html === false)
{
$handle = @fopen('http://www.google.com', 'r');

if ($handle)
{
$html = '';

while (!feof($handle))
{
$html += fgets($handle);
}

debug(" - set $memcache_key");

$a = $g_memcache->set($memcache_key, $html, 0, 86400) or $error = true;

if ($error === true && is_admin())
{
//if ($a === false)print'false';
//debug(gettype($a));
debug(' - !!broken!!');
}

fclose($handle);
}
else
{
$html = array('Data not currently available.');
}
}
}

function memcache_init()
{
if (!class_exists('Memcache'))
{
nonfatal_error_alert('Memcache appears to be down!', false);
return false;
}

try
{
{
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or trigger_error('Memcache::Connect failed @ localhost:11211', FATAL);

return $memcache;
}
}
catch (Exception $e)
{
nonfatal_error_alert('Memcache appears to be down!', false);
return false;
}
}

function debug()
{
foreach(func_get_args() as $arg)
{
print '<xmp>';
if (is_array($arg) | is_object($arg)) print_r($arg); else print $arg;
print '</xmp>';
}
}

Strangely enough, when I set $g_memcache outside of the function and use global $g_memcache, the call for "lb_top_selling_Printer" actually breaks the connection, and the subsequent camera calls fail as well.

Edit: I'm running PHP Version 5.2.0-8+etch11, memcached 1.1.12, and Debian 4.0r4.

Re: make: *** No targets specified and no makefile found. Stop.

2008-07-03T14:16:42Z

Hmm, I don't know if your Makefile really got generated. There's a libevent error at the end of the configure output. I'd suggest using the --with-libevent flag with your configure.

./configure --with-libevent=/PATH/TO/LIBEVENT

rexzhen wrote:

the ./configure output looks fine. i only found Makefile.am Makefile.in there. Here is the output for the ./configure

[root@localhost memcached-1.2.2]# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for a BSD-compatible install... /usr/bin/install -c
checking for libevent directory... configure: error: libevent is required. You can get it from http://www.monkey.org/~provos/libevent/

If it's already installed, specify its path using --with-libevent=/dir/

Re: make: *** No targets specified and no makefile found. Stop.

2008-07-03T10:10:21Z

the ./configure output looks fine. i only found Makefile.am Makefile.in there. Here is the output for the ./configure

[root@localhost memcached-1.2.2]# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for a BSD-compatible install... /usr/bin/install -c
checking for libevent directory... configure: error: libevent is required. You can get it from http://www.monkey.org/~provos/libevent/

If it's already installed, specify its path using --with-libevent=/dir/

rexzhen wrote:

My system is CentOS5.2

Whatever I do configure, when I do a make, I get this:

make: *** No targets specified and no makefile found. Stop.

I download memcached-1.2.5.tar.gz
<http://www.danga.com/memcached/dist/memcached-1.2.5.tar.gz>

Re: make: *** No targets specified and no makefile found. Stop.

2008-07-02T14:53:03Z

Rex Zhen-3 wrote:

My system is CentOS5.2

Whatever I do configure, when I do a make, I get this:

make: *** No targets specified and no makefile found. Stop.

I download memcached-1.2.5.tar.gz
<http://www.danga.com/memcached/dist/memcached-1.2.5.tar.gz>

Sorry to ask a dumb question, but does your configure finish successfully? And does a Makefile exist after you run configure?

make: *** No targets specified and no makefile found. Stop.

2008-07-02T14:36:37Z

My system is CentOS5.2

Whatever I do configure, when I do a make, I get this:

make: *** No targets specified and no makefile found. Stop.

I download memcached-1.2.5.tar.gz

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T08:34:09Z

32-bit, I never managed to compile a 64-bit version, and since I don't need one myself I didn't spend a lot of effort on getting one. Feel free to use Brodie's working src archives here: http://code.jellycan.com/memcached/ to compile one yourself.

/Henrik Schröder

On Wed, Jul 2, 2008 at 2:00 PM, Josef Finsel <carpdeus@...> wrote:

Is that the 64 bit or 32 bit version?

Thanks

"If you see a whole thing - it seems that it's always beautiful. Planets, lives... But up close a world's all dirt and rocks. And day to day, life's a hard job, you get tired, you lose the pattern."
Ursula K. Le Guin

On Wed, Jul 2, 2008 at 7:55 AM, Henrik Schröder <skrolle@...> wrote:
I put up an archive at http://code.google.com/p/beitmemcached/downloads/list, grab it from there and test it out. Would love to get more people testing it so we know if the weird CPU usage is gone for good.

/Henrik Schröder

On Wed, Jul 2, 2008 at 1:33 PM, Stephen Johnston <stephen.johnston@...> wrote:

I would be very much interested in a Windows Binary that didn't eat up one of our cores.

-Stephen

On Wed, Jul 2, 2008 at 5:34 AM, Henrik Schröder <skrolle@...> wrote:

On Fri, Jun 6, 2008 at 12:27 AM, Henrik Schröder <skrolle@...> wrote:

On Thu, Jun 5, 2008 at 8:01 PM, Jeff Rodenburg <jeff.rodenburg@...> wrote:

Henrik - can you elaborate on what you've found with this? I'm not looking to resolve the issues, just trying to get a better picture of where the bodies are buried, and to convince an all-windows shop that it's OK to run a few linux instances to support certain application services.

On our current project, we run memcached on two servers that are also web servers, and on both machines the memcached process consumes exactly 25% CPU. The weird thing is that those two servers have different hardware. One is a two-processor dual core Xeon at 2,5GHz, and the other is a two-processor dual core Xeon at 1,6GHz. The first one runs Windows Server 2008, the other Windows Server 2003. But the memcached process on each takes up exactly 25% CPU all the time. I can also see on the stats that the second server gets more memcached traffic than the first one, so the second server is slower than the first and gets more traffic, but the CPU use is 25% on both servers.

Ok, thanks to Brodie Thiesfield who managed to produce working Visual Studio projects of Libevent 1.4.4 and Memcached 1.2.5, I've compiled my own version. I took his project, added the old memcached icon (These things are important! :) ), fixed a file version number, and compiled everything in my Visual Studio 2005 with whatever optimizations it can do, and finally got to deploy this version live.

It's been running for a day now, and so far it looks good, still at 0% CPU utilization so hopefully whatever problems the older windows versions of memcached had are gone. I'll let it run for a week, and if it's still behaving after that time, I'll try to make available our binary for those that are interested.

/Henrik Schröder

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "memcached" group.
To post to this group, send email to memcached@...
To unsubscribe from this group, send email to memcached-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/memcached?hl=en
-~----------~----~----~----~------~----~------~--~---

RE: Multi-instance on Win2K3?

2008-07-02T06:59:51Z

Yep, this will be a dedicated server for Memcached. It’s most likely going to be a virtual server, and I think the 2xCPU / 2xGB is a standard template for our Systems group, so it’s actually easier for them to deploy an instance this way. Couple clicks and it’s ready to go.

From: Henrik Schröder [mailto:skrolle@...]
Sent: Tuesday, July 01, 2008 5:07 PM
To: Matthew Drayer
Cc: memcached@...
Subject: Re: Multi-instance on Win2K3?

Hi Matthew,

Kinda derailing here, but are you only going to run memcached on that server? If so, you really do not need the CPU power, spend the money on RAM instead.

/Henrik Schröder

On Tue, Jul 1, 2008 at 10:30 PM, Matthew Drayer <mdrayer@...> wrote:

So, most likely our official deployment will be to a 64bit Linux machine which would initially have 2GB of RAM and two monster CPUs. If situation #2 isn't an issue, would it be best to run a single instance of Memcached, or split the RAM into two instances?

From: Stephen Johnston [mailto:stephen.johnston@...]
Sent: Tuesday, July 01, 2008 4:26 PM

To: Matthew Drayer
Cc: memcached@...
Subject: Re: Multi-instance on Win2K3?

I think that CPU is rarely why people do this. From what I've seen and read there are a few common cases:

1. You have 384mb on one machine and 128mb on another available. You make 4 instances so their eviction pattern is similar and the client can treat them as identical, and your expected behavior for them will be similar, and write across them equally without a 384mb <-> 128mb pair of server causing wierd imbalances. The clients that I have seen don't take cache size into account when considering which instance to use.

2. You have a situation where you store items with no delete time (they live for ever), but you have limited memory. your no delete time items are expensive to recreate. You also have alot of less expensive items to recreate that may lead to your expensive ones being evicted. You use one instance sized for the items that live forever and another for the ongoing "evictable" items.

I'm sure others have some use cases, but those are the two I've seen mentioned commonly.

-Stephen

On Tue, Jul 1, 2008 at 4:17 PM, Matthew Drayer <mdrayer@...> wrote:

Probably not at such a low level, no J but, this was more for a proof-of-concept to show my team how it might work. I assume we'll only distribute out if we find we're pushing the limits of RAM or CPU utilization.

Matt

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T05:00:05Z

Is that the 64 bit or 32 bit version?

Thanks

"If you see a whole thing - it seems that it's always beautiful. Planets, lives... But up close a world's all dirt and rocks. And day to day, life's a hard job, you get tired, you lose the pattern."
Ursula K. Le Guin

On Wed, Jul 2, 2008 at 7:55 AM, Henrik Schröder <skrolle@...> wrote:

I put up an archive at http://code.google.com/p/beitmemcached/downloads/list, grab it from there and test it out. Would love to get more people testing it so we know if the weird CPU usage is gone for good.

/Henrik Schröder

On Wed, Jul 2, 2008 at 1:33 PM, Stephen Johnston <stephen.johnston@...> wrote:

I would be very much interested in a Windows Binary that didn't eat up one of our cores.

-Stephen

On Wed, Jul 2, 2008 at 5:34 AM, Henrik Schröder <skrolle@...> wrote:

On Fri, Jun 6, 2008 at 12:27 AM, Henrik Schröder <skrolle@...> wrote:

On Thu, Jun 5, 2008 at 8:01 PM, Jeff Rodenburg <jeff.rodenburg@...> wrote:

Henrik - can you elaborate on what you've found with this? I'm not looking to resolve the issues, just trying to get a better picture of where the bodies are buried, and to convince an all-windows shop that it's OK to run a few linux instances to support certain application services.

On our current project, we run memcached on two servers that are also web servers, and on both machines the memcached process consumes exactly 25% CPU. The weird thing is that those two servers have different hardware. One is a two-processor dual core Xeon at 2,5GHz, and the other is a two-processor dual core Xeon at 1,6GHz. The first one runs Windows Server 2008, the other Windows Server 2003. But the memcached process on each takes up exactly 25% CPU all the time. I can also see on the stats that the second server gets more memcached traffic than the first one, so the second server is slower than the first and gets more traffic, but the CPU use is 25% on both servers.

Ok, thanks to Brodie Thiesfield who managed to produce working Visual Studio projects of Libevent 1.4.4 and Memcached 1.2.5, I've compiled my own version. I took his project, added the old memcached icon (These things are important! :) ), fixed a file version number, and compiled everything in my Visual Studio 2005 with whatever optimizations it can do, and finally got to deploy this version live.

It's been running for a day now, and so far it looks good, still at 0% CPU utilization so hopefully whatever problems the older windows versions of memcached had are gone. I'll let it run for a week, and if it's still behaving after that time, I'll try to make available our binary for those that are interested.

/Henrik Schröder

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "memcached" group.
To post to this group, send email to memcached@...
To unsubscribe from this group, send email to memcached-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/memcached?hl=en
-~----------~----~----~----~------~----~------~--~---

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T04:55:37Z

I put up an archive at http://code.google.com/p/beitmemcached/downloads/list, grab it from there and test it out. Would love to get more people testing it so we know if the weird CPU usage is gone for good.

/Henrik Schröder

On Wed, Jul 2, 2008 at 1:33 PM, Stephen Johnston <stephen.johnston@...> wrote:

I would be very much interested in a Windows Binary that didn't eat up one of our cores.

-Stephen

On Wed, Jul 2, 2008 at 5:34 AM, Henrik Schröder <skrolle@...> wrote:

On Fri, Jun 6, 2008 at 12:27 AM, Henrik Schröder <skrolle@...> wrote:

On Thu, Jun 5, 2008 at 8:01 PM, Jeff Rodenburg <jeff.rodenburg@...> wrote:

Henrik - can you elaborate on what you've found with this? I'm not looking to resolve the issues, just trying to get a better picture of where the bodies are buried, and to convince an all-windows shop that it's OK to run a few linux instances to support certain application services.

On our current project, we run memcached on two servers that are also web servers, and on both machines the memcached process consumes exactly 25% CPU. The weird thing is that those two servers have different hardware. One is a two-processor dual core Xeon at 2,5GHz, and the other is a two-processor dual core Xeon at 1,6GHz. The first one runs Windows Server 2008, the other Windows Server 2003. But the memcached process on each takes up exactly 25% CPU all the time. I can also see on the stats that the second server gets more memcached traffic than the first one, so the second server is slower than the first and gets more traffic, but the CPU use is 25% on both servers.

Ok, thanks to Brodie Thiesfield who managed to produce working Visual Studio projects of Libevent 1.4.4 and Memcached 1.2.5, I've compiled my own version. I took his project, added the old memcached icon (These things are important! :) ), fixed a file version number, and compiled everything in my Visual Studio 2005 with whatever optimizations it can do, and finally got to deploy this version live.

It's been running for a day now, and so far it looks good, still at 0% CPU utilization so hopefully whatever problems the older windows versions of memcached had are gone. I'll let it run for a week, and if it's still behaving after that time, I'll try to make available our binary for those that are interested.

/Henrik Schröder

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T04:50:39Z

On Wed, Jul 2, 2008 at 1:46 PM, Dean Harding <dean.harding@...> wrote:

We use a version of 1.2.4 that was ported to win32 that we subsequently
modified to compile in 64-bit mode that I could share if people are
interested.

Which version of libevent did you compile that one against?

/Henrik Schröder

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T04:46:40Z

Stephen Johnston wrote:
> I would be very much interested in a Windows Binary that didn't eat up
> one of our cores.

We use a version of 1.2.4 that was ported to win32 that we subsequently
modified to compile in 64-bit mode that I could share if people are
interested.

Shouldn't take much effort for me to update it to 1.2.5 or whatever the
latest is, either (once we got it working, we didn't really bother with
updating it :p)

Dean.

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T04:33:20Z

I would be very much interested in a Windows Binary that didn't eat up one of our cores.

-Stephen

On Wed, Jul 2, 2008 at 5:34 AM, Henrik Schröder <skrolle@...> wrote:

On Fri, Jun 6, 2008 at 12:27 AM, Henrik Schröder <skrolle@...> wrote:

On Thu, Jun 5, 2008 at 8:01 PM, Jeff Rodenburg <jeff.rodenburg@...> wrote:

Henrik - can you elaborate on what you've found with this? I'm not looking to resolve the issues, just trying to get a better picture of where the bodies are buried, and to convince an all-windows shop that it's OK to run a few linux instances to support certain application services.

On our current project, we run memcached on two servers that are also web servers, and on both machines the memcached process consumes exactly 25% CPU. The weird thing is that those two servers have different hardware. One is a two-processor dual core Xeon at 2,5GHz, and the other is a two-processor dual core Xeon at 1,6GHz. The first one runs Windows Server 2008, the other Windows Server 2003. But the memcached process on each takes up exactly 25% CPU all the time. I can also see on the stats that the second server gets more memcached traffic than the first one, so the second server is slower than the first and gets more traffic, but the CPU use is 25% on both servers.

Ok, thanks to Brodie Thiesfield who managed to produce working Visual Studio projects of Libevent 1.4.4 and Memcached 1.2.5, I've compiled my own version. I took his project, added the old memcached icon (These things are important! :) ), fixed a file version number, and compiled everything in my Visual Studio 2005 with whatever optimizations it can do, and finally got to deploy this version live.

It's been running for a day now, and so far it looks good, still at 0% CPU utilization so hopefully whatever problems the older windows versions of memcached had are gone. I'll let it run for a week, and if it's still behaving after that time, I'll try to make available our binary for those that are interested.

/Henrik Schröder

Re: Announce: New feature-complete version of the BeIT Memcached Client released.

2008-07-02T02:34:57Z

On Fri, Jun 6, 2008 at 12:27 AM, Henrik Schröder <skrolle@...> wrote:

On Thu, Jun 5, 2008 at 8:01 PM, Jeff Rodenburg <jeff.rodenburg@...> wrote:

Henrik - can you elaborate on what you've found with this? I'm not looking to resolve the issues, just trying to get a better picture of where the bodies are buried, and to convince an all-windows shop that it's OK to run a few linux instances to support certain application services.

On our current project, we run memcached on two servers that are also web servers, and on both machines the memcached process consumes exactly 25% CPU. The weird thing is that those two servers have different hardware. One is a two-processor dual core Xeon at 2,5GHz, and the other is a two-processor dual core Xeon at 1,6GHz. The first one runs Windows Server 2008, the other Windows Server 2003. But the memcached process on each takes up exactly 25% CPU all the time. I can also see on the stats that the second server gets more memcached traffic than the first one, so the second server is slower than the first and gets more traffic, but the CPU use is 25% on both servers.

Ok, thanks to Brodie Thiesfield who managed to produce working Visual Studio projects of Libevent 1.4.4 and Memcached 1.2.5, I've compiled my own version. I took his project, added the old memcached icon (These things are important! :) ), fixed a file version number, and compiled everything in my Visual Studio 2005 with whatever optimizations it can do, and finally got to deploy this version live.

It's been running for a day now, and so far it looks good, still at 0% CPU utilization so hopefully whatever problems the older windows versions of memcached had are gone. I'll let it run for a week, and if it's still behaving after that time, I'll try to make available our binary for those that are interested.

/Henrik Schröder

MemcacheDB 1.1.0 beta is released, new BerkeleyDB 4.7 support

2008-07-02T00:18:48Z

*MemcacheDB* is a distributed key-value storage system designed for
persistent. It is NOT a cache solution, but a persistent storage
engine for fast and reliable key-value based object storage and
retrieval. It conforms to memcache protocol(not completed, see below),
so any memcached client can have connectivity with it. Memcachedb uses
Berkeley DB as a storing backend, so lots of features including
transaction and replication are supported.

ChangeLog since 1.0.3 version:

2008-07-03 Version 1.1.0-beta released

2008-06-26 Steve Chu <stvchu@...>
* unix socket connectivity now has be well tested.

2008-06-13 Steve Chu <stvchu@...>
* some code refactoring to improve a bit performance

2008-06-11 Steve Chu <stvchu@...>
* now '-b' option is a tuning way, not a limit. item size that smaller than
'-b' value use fast freelist alloc, otherwise, use system 'malloc' instead.
Many thanks to Davies Liu <davies.liu@...> for patches and ideas.

2008-06-03 Steve Chu <stvchu@...>
* new support to BerkeleyDB 4.7
- new version is BerkeleyDB 4.7 only, due to the changed bdb api
- 'rep_set_request' now use time, not the number of messages

* Bugfix: 'pkget' and "multiple get" that return lots of items may
cause memory overflow so the daemon crashes. Thanks to Davies Liu
<davies.liu@...>

* flooded verbose messages of bdb and replication now move to '-vv' option

2008-04-13 Steve Chu <stvchu@...>
* Changes:
- allow limitation to 'pkget' and 'pvget' command,
use 'pkget/pvget <prefix> [limit]' to get limited matched items.
- 'memcache.py' is also patched for this feature.

2008-04-03 Steve Chu <stvchu@...>
* New features:
- add '-T' option to set database type, 'btree' or 'hash'
- add '-E' option to enable second database, so we can use 'pvget'
- new command 'pkget' that you can get items with a key prefix
- new command 'pvget' that you can get items with a value prefix
* Changes:
- add support to memcache.py for 'pkget' and 'pvget' command, see
'tools/memcache.py'

for more info, please visit: http://memcachedb.org

Regards

--
Steve Chu
http://stvchu.org

Re: Help? seg-fault...

2008-07-01T14:20:27Z

On Jul 1, 2008, at 13:35, Michael Wieher wrote:

> It is a guarantee that "code" is never more than 8 characters,

That's not a guarantee. snprintf is a guarantee.

> file
> ranges from 2-digits to 3, max, and location wouldn't ever be more
> than 10 characters. So, I'm looking at a string of max 21 chars, I
> made it 32.
>
> These inputs come from controlled static sources, not user input.
> However I did recompile w/asserts and they passed w/o issues.

I can't say for sure what's going on in your code without seeing it
as a whole, but these kinds of things are smells. Any place you
assume something, just stick an assertion in the code to both document
and enforce it. Makes it a lot harder to write bugs.

>> You seem to be missing symbols from the valgrind output. Perhaps
>> you should
>> compile the debugging symbols in.
>
> I know. It's a bit of a strange situation, in that I'm actually
> compiling a shared-library.so file that is imported into and run by
> Python, but the compilation itself of the C++ library is done with -g
> ... so I'm at a bit of a loss as how to get a different angle on it,
> without basically writing a wrapper-driver to compile & test the
> sucker at the C/C++ level, before then wrapping it into Python.

Yeah, definitely test the parts in the most simple way possible.
You've got a really simple error somewhere. The less code you have to
look at, the easier it'll be to find.

See also: The Underhaded C Contest. :)

--
Dustin Sallings

Re: Multi-instance on Win2K3?

2008-07-01T14:07:04Z

Hi Matthew,

Kinda derailing here, but are you only going to run memcached on that server? If so, you really do not need the CPU power, spend the money on RAM instead.

/Henrik Schröder

On Tue, Jul 1, 2008 at 10:30 PM, Matthew Drayer <mdrayer@...> wrote:

So, most likely our official deployment will be to a 64bit Linux machine which would initially have 2GB of RAM and two monster CPUs. If situation #2 isn't an issue, would it be best to run a single instance of Memcached, or split the RAM into two instances?

From: Stephen Johnston [mailto:stephen.johnston@...]
Sent: Tuesday, July 01, 2008 4:26 PM

To: Matthew Drayer
Cc: memcached@...
Subject: Re: Multi-instance on Win2K3?

I think that CPU is rarely why people do this. From what I've seen and read there are a few common cases:

1. You have 384mb on one machine and 128mb on another available. You make 4 instances so their eviction pattern is similar and the client can treat them as identical, and your expected behavior for them will be similar, and write across them equally without a 384mb <-> 128mb pair of server causing wierd imbalances. The clients that I have seen don't take cache size into account when considering which instance to use.

2. You have a situation where you store items with no delete time (they live for ever), but you have limited memory. your no delete time items are expensive to recreate. You also have alot of less expensive items to recreate that may lead to your expensive ones being evicted. You use one instance sized for the items that live forever and another for the ongoing "evictable" items.

I'm sure others have some use cases, but those are the two I've seen mentioned commonly.

-Stephen

On Tue, Jul 1, 2008 at 4:17 PM, Matthew Drayer <mdrayer@...> wrote:

Probably not at such a low level, no J but, this was more for a proof-of-concept to show my team how it might work. I assume we'll only distribute out if we find we're pushing the limits of RAM or CPU utilization.

Matt

Re: Multi-instance on Win2K3?

2008-07-01T13:43:57Z

2008/7/1 Stephen Johnston <stephen.johnston@...>:
> I think that CPU is rarely why people do this. From what I've seen and read
> there are a few common cases:

Another case is if memcached crashes or must be restarted (for example
due to some critical bugfix). Sometimes it is very painful to restart
one instance with few GBs of cached data, it is better to have a
couple if instances with less data that can be restarted one at a
time. But this depends on what is cached and how painful it is when
the cache is lost.

--
Janusz Dziemidowicz

Re: Help? seg-fault...

2008-07-01T13:35:16Z

>
> sprintf(qstring,"%s%d%lu",code,cur->file,cur->location);
>
> sprintf is generally not recommended. It doesn't seem terribly hard to get
> it to exceed your stack buffer here.

It is a guarantee that "code" is never more than 8 characters, file
ranges from 2-digits to 3, max, and location wouldn't ever be more
than 10 characters. So, I'm looking at a string of max 21 chars, I
made it 32.
These inputs come from controlled static sources, not user input.
However I did recompile w/asserts and they passed w/o issues.

> You seem to be missing symbols from the valgrind output. Perhaps you should
> compile the debugging symbols in.

I know. It's a bit of a strange situation, in that I'm actually
compiling a shared-library.so file that is imported into and run by
Python, but the compilation itself of the C++ library is done with -g
... so I'm at a bit of a loss as how to get a different angle on it,
without basically writing a wrapper-driver to compile & test the
sucker at the C/C++ level, before then wrapping it into Python.

thanks though...

RE: Multi-instance on Win2K3?

2008-07-01T13:30:26Z

So, most likely our official deployment will be to a 64bit Linux machine which would initially have 2GB of RAM and two monster CPUs. If situation #2 isn’t an issue, would it be best to run a single instance of Memcached, or split the RAM into two instances?

From: Stephen Johnston [mailto:stephen.johnston@...]
Sent: Tuesday, July 01, 2008 4:26 PM
To: Matthew Drayer
Cc: memcached@...
Subject: Re: Multi-instance on Win2K3?

I think that CPU is rarely why people do this. From what I've seen and read there are a few common cases:

I'm sure others have some use cases, but those are the two I've seen mentioned commonly.

-Stephen

On Tue, Jul 1, 2008 at 4:17 PM, Matthew Drayer <mdrayer@...> wrote:

Matt

Re: Multi-instance on Win2K3?

2008-07-01T13:25:44Z

I think that CPU is rarely why people do this. From what I've seen and read there are a few common cases:

I'm sure others have some use cases, but those are the two I've seen mentioned commonly.

-Stephen

On Tue, Jul 1, 2008 at 4:17 PM, Matthew Drayer <mdrayer@...> wrote:

Probably not at such a low level, no J but, this was more for a proof-of-concept to show my team how it might work. I assume we'll only distribute out if we find we're pushing the limits of RAM or CPU utilization.

Matt

Re: Help? seg-fault...

2008-07-01T13:24:14Z

On Jul 1, 2008, at 13:03, Michael Wieher wrote:

(a few spaces would make this easier to read)

sprintf(qstring,"%s%d%lu",code,cur->file,cur->location);

sprintf is generally not recommended. It doesn't seem terribly hard to get it to exceed your stack buffer here.

It would be valuable at least to check the return value to ensure it is strictly less than the size of your buffer.

An assert(strlen(qstring) < sizeof(qstring)) would also help with that.

char * tempbuff = (char*)calloc(sizeof(char),cur->size);

assert(tempbuff);

You seem to be missing symbols from the valgrind output. Perhaps you should compile the debugging symbols in.

Dustin Sallings