Durable queues & bindings

[Levi Greenspan]

Dear list members,

I have a question w.r.t. clustering. Durable exchanges seem to survive
individual node crashes, but neither do queues and bindings if I am not
mistaken. The clustering guide [1] states:

"An exception to this [data & state replication] are message queues,
which currently only reside on the node that created them, though they
are visible and reachable from all nodes. Future releases of RabbitMQ
will introduce migration and replication of message queues."

Even though bindings are not mentioned I wanted to ask:

1. Is there an approximate time schedule, when to expect queue
replication in a cluster?

2. Assuming queues are replicated in a cluster, will I also get durable
bindings?

Many thanks for your replies!

Cheers,
Levi

-------------------------
[1] http://www.rabbitmq.com/clustering.html


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock-2]

Levi,

Levi Greenspan wrote:
> Dear list members,
>
> I have a question w.r.t. clustering. Durable exchanges seem to survive
> individual node crashes, but neither do queues and bindings if I am not
> mistaken.

The durable queues residing on the crashed node and their associated
bindings do indeed disappear - but only for the duration of the outage;
if/when the node comes back up the queues and bindings re-appear.

> 1. Is there an approximate time schedule, when to expect queue
> replication in a cluster?

Queue replication, or, more precisely, the ability of queues to stay
alive in the presence of individual node failures, is fairly high on our
priority list. There are a few other items we want/need to get out of
the way first though, so I doubt we'll get round to it for another few
months. As always, sponsors are welcome.

> 2. Assuming queues are replicated in a cluster, will I also get durable
> bindings?

As noted above, bindings are already durable. Moreover, they are already
replicated.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Levi Greenspan]

On Wed, 2009-04-08 at 22:15 +0100, Matthias Radestock wrote:
> The durable queues residing on the crashed node and their associated
> bindings do indeed disappear - but only for the duration of the outage;
> if/when the node comes back up the queues and bindings re-appear.

Oops. My fault. Need to to further investigation why I got the wrong
impression.

Cheers,
Levi



_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Levi Greenspan]

On Wed, 2009-04-08 at 23:43 +0200, Levi Greenspan wrote:
> On Wed, 2009-04-08 at 22:15 +0100, Matthias Radestock wrote:
> > The durable queues residing on the crashed node and their associated
> > bindings do indeed disappear - but only for the duration of the outage;
> > if/when the node comes back up the queues and bindings re-appear.
>
> Oops. My fault. Need to to further investigation why I got the wrong
> impression.

I upgraded to version 1.5.4 and now I see the queues and bindings after
the node is running again. However in 1.5.3 it didn't work for me.

Anyway I noticed another difference between 1.5.3 and 1.5.4 - the call
to com.rabbitmq.client.Connection.getKnownHosts() in that Java client
API returns the list of all running node addresses in 1.5.3, but only
one host address in 1.5.4 - a bug?

Thanks,
Levi


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock-2]

Levi,

Levi Greenspan wrote:
> I upgraded to version 1.5.4 and now I see the queues and bindings after
> the node is running again. However in 1.5.3 it didn't work for me.

I can't think of anything that has changed between 1.5.3 and 1.5.4 that
would cause this, except that startup times for when there are lots of
durable exchanges, queues or bindings have improved significantly. The
recreation of these entities is now also happening incrementally. So
it's possible that previously rabbitmqctl would not list any queues for
you if it was called right after you started the broker - though it
should have done so eventually - whereas now it does.

> Anyway I noticed another difference between 1.5.3 and 1.5.4 - the call
> to com.rabbitmq.client.Connection.getKnownHosts() in that Java client
> API returns the list of all running node addresses in 1.5.3, but only
> one host address in 1.5.4 - a bug?

Nothing has changed in that area of the code. Are you sure that all your
nodes are running and listening?


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Levi Greenspan]

Matthias,

On Sat, 2009-04-11 at 08:46 +0100, Matthias Radestock wrote:

> Levi,
>
> Levi Greenspan wrote:
> > I upgraded to version 1.5.4 and now I see the queues and bindings after
> > the node is running again. However in 1.5.3 it didn't work for me.
>
> I can't think of anything that has changed between 1.5.3 and 1.5.4 that
> would cause this, except that startup times for when there are lots of
> durable exchanges, queues or bindings have improved significantly. The
> recreation of these entities is now also happening incrementally. So
> it's possible that previously rabbitmqctl would not list any queues for
> you if it was called right after you started the broker - though it
> should have done so eventually - whereas now it does.

I have attached a small Java test client and a transcript which shows
the different behavior between version 1.5.3 and 1.5.4. Maybe you can
reproduce it.

> > Anyway I noticed another difference between 1.5.3 and 1.5.4 - the call
> > to com.rabbitmq.client.Connection.getKnownHosts() in that Java client
> > API returns the list of all running node addresses in 1.5.3, but only
> > one host address in 1.5.4 - a bug?
>
> Nothing has changed in that area of the code. Are you sure that all your
> nodes are running and listening?

Yes, I am. The test client prints the known hosts and for 1.5.3 the
reult is "localhost:5672", whereas for 1.5.4 it is "localhost:5673" and
"localhost:5672". Again I hope you can reproduce it with the attached
client. I am using Ubuntu 8.10 on x86 and I compiled erlang using the
offically released source, not the Ubuntu packages.

$ erl -version
Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 5.6.5

Many thanks again,
Levi


[Test.java]

---

import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Address;
import com.rabbitmq.client.QueueingConsumer;
import com.rabbitmq.client.ShutdownListener;
import com.rabbitmq.client.ReturnListener;
import com.rabbitmq.client.Channel ;
import com.rabbitmq.client.ConnectionParameters;
import com.rabbitmq.client.ConnectionFactory;


public class Test {
    public static void main(String[] args) throws Exception {
        final ConnectionParameters params = new ConnectionParameters();
        params.setUsername("guest");
        params.setPassword("guest");
        final ConnectionFactory fact = new ConnectionFactory(params);
        final Connection conn = fact.newConnection("localhost", 5672);
        final Channel channel = conn.createChannel();

        channel.exchangeDeclare("my-exchange", "direct", true);
        channel.queueDeclare("my-queue", true);
        channel.queueBind("my-queue", "my-exchange", "my-key");

        final Address[] hosts = conn.getKnownHosts();
        for (int i = 0; i < hosts.length; ++i)
            System.out.println(hosts[i]);
    }
}

---

user@host ~ $ cd rabbitmq-server-1.5.3/
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmq-multi start_all 2
Starting all nodes...
Starting node rabbit@localhost...
RabbitMQ 1.5.3 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http:/www.rabbitmq.com/

Logging to "/opt/logs/rabbitmq/rabbit.log"
SASL logging to "/opt/logs/rabbitmq/rabbit-sasl.log"

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done

broker running
OK
Starting node rabbit_1@localhost...
RabbitMQ 1.5.3 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http:/www.rabbitmq.com/

Logging to "/opt/logs/rabbitmq/rabbit_1.log"
SASL logging to "/opt/logs/rabbitmq/rabbit_1-sasl.log"

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done

broker running
OK
done.
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
...done.


    >>> Now I executed the test client in another shell. <<<


user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
        my-queue my-queue []
my-exchange my-queue my-key []
...done.
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmqctl stop
Stopping and halting node rabbit@localhost ...
Erlang has closed
...done.
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmq-server -detached
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
...done.
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
...done.
user@host ~/rabbitmq-server-1.5.3 $ scripts/rabbitmq-multi stop_all
Stopping all nodes...
Stopping node rabbit_1@localhost
Erlang has closed
....OK
Stopping node rabbit@localhost
.OK
done.
user@host ~/rabbitmq-server-1.5.3 $ cd ../rabbitmq-server-1.5.4/


    >>> Deleted mnesia files to start from scratch. <<<


user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmq-multi start_all 2
Starting all nodes...
Starting node rabbit@localhost...
RabbitMQ 1.5.4 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http:/www.rabbitmq.com/

Logging to "/opt/logs/rabbitmq/rabbit.log"
SASL logging to "/opt/logs/rabbitmq/rabbit-sasl.log"

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done

broker running
OK
Starting node rabbit_1@localhost...
RabbitMQ 1.5.4 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http:/www.rabbitmq.com/

Logging to "/opt/logs/rabbitmq/rabbit_1.log"
SASL logging to "/opt/logs/rabbitmq/rabbit_1-sasl.log"

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done

broker running
OK
done.
user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
...done.


    >>> Now I executed the test client in another shell. <<<


user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
        my-queue my-queue []
my-exchange my-queue my-key []
...done.
user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmqctl stop
Stopping and halting node rabbit@localhost ...
...done.
Erlang has closed
user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmq-server -detached
user@host ~/rabbitmq-server-1.5.4 $ scripts/rabbitmqctl list_bindings
Listing bindings ...
        my-queue my-queue []
my-exchange my-queue my-key []
...done.
user@host ~/rabbitmq-server-1.5.4 $

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock-2]

Levi,

Levi Greenspan wrote:

> On Sat, 2009-04-11 at 08:46 +0100, Matthias Radestock wrote:
>> Levi,
>>
>> Levi Greenspan wrote:
>>> I upgraded to version 1.5.4 and now I see the queues and bindings after
>>> the node is running again. However in 1.5.3 it didn't work for me.
>> I can't think of anything that has changed between 1.5.3 and 1.5.4 that
>> would cause this, except that startup times for when there are lots of
>> durable exchanges, queues or bindings have improved significantly. The
>> recreation of these entities is now also happening incrementally. So
>> it's possible that previously rabbitmqctl would not list any queues for
>> you if it was called right after you started the broker - though it
>> should have done so eventually - whereas now it does.
>
>
> I have attached a small Java test client and a transcript which shows
> the different behavior between version 1.5.3 and 1.5.4. Maybe you can
> reproduce it.

What user did you run these as? Looks like 'user', based on the
transcript. But usually only the 'rabbitmq' user can write to
/var/log/rabbitmq etc, which would cause the script invocations to fail.

Did you make any adjustments to the permissions, or set some env vars to
adjust the paths, or tweak the scripts?

Also, did you try running these tests when no clustering has been
configured and you have just a single node?

>>> Anyway I noticed another difference between 1.5.3 and 1.5.4 - the call
>>> to com.rabbitmq.client.Connection.getKnownHosts() in that Java client
>>> API returns the list of all running node addresses in 1.5.3, but only
>>> one host address in 1.5.4 - a bug?
>> Nothing has changed in that area of the code. Are you sure that all your
>> nodes are running and listening?
>
> Yes, I am. The test client prints the known hosts and for 1.5.3 the
> reult is "localhost:5672", whereas for 1.5.4 it is "localhost:5673" and
> "localhost:5672". Again I hope you can reproduce it with the attached
> client. I am using Ubuntu 8.10 on x86 and I compiled erlang using the
> offically released source, not the Ubuntu packages.

So is it 1.5.3 that returns both hosts (which is what you said
originally) or 1.5.4?

And how did you configure the clustering?


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Levi Greenspan]

On Sun, 2009-04-12 at 00:32 +0100, Matthias Radestock wrote:
> What user did you run these as? Looks like 'user', based on the
> transcript. But usually only the 'rabbitmq' user can write to
> /var/log/rabbitmq etc, which would cause the script invocations to fail.
>
> Did you make any adjustments to the permissions, or set some env vars to
> adjust the paths, or tweak the scripts?

I have the following settings in /etc/rabbitmq/rabbitmq.conf

RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
RABBITMQ_LOG_BASE=/opt/logs/rabbitmq
RABBITMQ_MNESIA_BASE=/opt/otp/mnesia
RABBITMQ_PIDS_FILE=/opt/otp/pids

and in /etc/rabbitmq/rabbitmq_cluster.conf:

[rabbit@localhost].

"user" has read & write permissions to these directories.

> Also, did you try running these tests when no clustering has been
> configured and you have just a single node?

I just did and with only one node everything is fine with both versions.

> >>> Anyway I noticed another difference between 1.5.3 and 1.5.4 - the call
> >>> to com.rabbitmq.client.Connection.getKnownHosts() in that Java client
> >>> API returns the list of all running node addresses in 1.5.3, but only
> >>> one host address in 1.5.4 - a bug?
> >> Nothing has changed in that area of the code. Are you sure that all your
> >> nodes are running and listening?
> >
> > Yes, I am. The test client prints the known hosts and for 1.5.3 the
> > reult is "localhost:5672", whereas for 1.5.4 it is "localhost:5673" and
> > "localhost:5672". Again I hope you can reproduce it with the attached
> > client. I am using Ubuntu 8.10 on x86 and I compiled erlang using the
> > offically released source, not the Ubuntu packages.
>
> So is it 1.5.3 that returns both hosts (which is what you said
> originally) or 1.5.4?

Sorry. Screwed versions. 1.5.3 returns both and 1.5.4 only one.

>
> And how did you configure the clustering?

Following http://www.rabbitmq.com/clustering.html#single-machine I just
put "[rabbit@localhost]." into /etc/rabbitmq_cluster.conf

Thanks,
Levi


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock-2]

Levi,

Levi Greenspan wrote:

> I have the following settings in /etc/rabbitmq/rabbitmq.conf
>
> RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
> RABBITMQ_LOG_BASE=/opt/logs/rabbitmq
> RABBITMQ_MNESIA_BASE=/opt/otp/mnesia
> RABBITMQ_PIDS_FILE=/opt/otp/pids
>
> and in /etc/rabbitmq/rabbitmq_cluster.conf:
>
> [rabbit@localhost].

The default location of the cluster config file is
/etc/rabbitmq/cluster.config. The docs say
/etc/rabbitmq/rabbitmq_cluster.config, which is wrong.

Did you change that setting to point to /etc/rabbitmq/rabbitmq_cluster.conf?

If not that would explain why getKnownHosts returns one node only,
though it doesn't explain why it would only do that for 1.5.4 and not
1.5.3. Perhaps your config changed?

To check whether the clustering is set up correctly, run "rabbitmqctl
status". That should show both nodes as running.

>>> I upgraded to version 1.5.4 and now I see the queues and bindings after
>>> > > the node is running again. However in 1.5.3 it didn't work for me.
> [...]
> I have attached a small Java test client and a transcript which shows
> the different behavior between version 1.5.3 and 1.5.4. Maybe you can
> reproduce it.

I can indeed reproduce your findings: bindings to durable queues
residing on nodes in a cluster are not restored when the queues' nodes
restart. Thanks for reporting this; I have filed a bug.

This happens for me in both 1.5.3 and 1.5.4; the reason that it works
for you in 1.5.4 is probably the same reason that getKnownHosts returns
just one node, namely that the clustering was not set up correctly when
you ran that test.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matthias Radestock-2]

Levi,

Matthias Radestock wrote:

> The default location of the cluster config file is
> /etc/rabbitmq/cluster.config. The docs say
> /etc/rabbitmq/rabbitmq_cluster.config, which is wrong.
>
> Did you change that setting to point to
> /etc/rabbitmq/rabbitmq_cluster.conf?
>
> If not that would explain why getKnownHosts returns one node only,
> though it doesn't explain why it would only do that for 1.5.4 and not
> 1.5.3. Perhaps your config changed?

Mystery solved. The default location of the cluster config is *supposed*
to be /etc/rabbitmq/rabbitmq_cluster.config; and indeed that's what it
was until we inadvertently changed it in 1.5.4.

Oops. Bug filed and fixed on the bug20593 branch. Until that fix has
been qa'ed and released, please either correct the setting in the
rabbitmq-server script, set the RABBITMQ_CLUSTER_CONFIG_FILE env var, or
create a symlink from the broken location to the correct one.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Levi Greenspan]

On Tue, 2009-04-14 at 17:14 +0100, Matthias Radestock wrote:
> Mystery solved. The default location of the cluster config is *supposed*
> to be /etc/rabbitmq/rabbitmq_cluster.config; and indeed that's what it
> was until we inadvertently changed it in 1.5.4.
>
> Oops. Bug filed and fixed on the bug20593 branch. Until that fix has
> been qa'ed and released, please either correct the setting in the
> rabbitmq-server script, set the RABBITMQ_CLUSTER_CONFIG_FILE env var, or
> create a symlink from the broken location to the correct one.

Hi Matthias,

Many thanks! I created the symlink and everything works fine.

Cheers,
Levi



_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@...
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss