JDBC Master Slave Question

[colonelx]

Hi,

I have 2 brokers setup in a JDBC Master Slave Configuration.

Each broker is on a different machine. BoxA and BoxB
The database is also on a different machine BoxC. (mysql 5.1)

The failover is working fine when I manually kill any instance of ActiveMQ on BoxA or BoxB. (Ctrl-C)

However, can anyone tell me what will happen if the if the Master instance looses its network connection ?
Will its lock on the database get dropped immediately ? or will it still hold a (now invalid) lock on the database ?
I ask this, because if this happens , there could possibly be a condition where the slave won't ever be able to get a lock on the database because the master has lock which did not get correctly released and manual intervention would be needed in order to fix up the problem.

I know that I could test this by plugging in and out the physical network connections, but I'm not really in a position to do this on any of the above boxes.

Thanks.

[Johan Stuyts-2]

> I know that I could test this by plugging in and out the physical network
> connections, but I'm not really in a position to do this on any of the  
> above
> boxes.

If you are on Windows you can use TCPView to kill connections without  
having to physically disconnect the network:
<http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx>

I'm sure there are similar tools for other platforms.

Regards,

Johan Stuyts

[Gary Tully]

You have got to ask the JDBC driver or database documentation. I imagine
there is a lock expiry option that can be tweaked.

2009/6/29 colonelx <zac@...>

>
> Hi,
>
> I have 2 brokers setup in a JDBC Master Slave Configuration.
>
> Each broker is on a different machine. BoxA and BoxB
> The database is also on a different machine BoxC. (mysql 5.1)
>
> The failover is working fine when I manually kill any instance of ActiveMQ
> on BoxA or BoxB. (Ctrl-C)
>
> However, can anyone tell me what will happen if the if the Master instance
> looses its network connection ?
> Will its lock on the database get dropped immediately ? or will it still
> hold a (now invalid) lock on the database ?
> I ask this, because if this happens , there could possibly be a condition
> where the slave won't ever be able to get a lock on the database because
> the
> master has lock which did not get correctly released and manual
> intervention
> would be needed in order to fix up the problem.
>
> I know that I could test this by plugging in and out the physical network
> connections, but I'm not really in a position to do this on any of the
> above
> boxes.
>
> Thanks.
>
> --
> View this message in context:
> http://www.nabble.com/JDBC-Master-Slave-Question-tp24251198p24251198.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

--
http://blog.garytully.com

Open Source Integration
http://fusesource.com

[colonelx]

I got to try out the scenario I described above (ie) physically remove the network cable from the Master.

Basically the database did not realize that the client had disconnected abruptly,
and there was still a lock on the database table, (which was now an invalid stale lock), and would
never get released.

After some further digging I realised that there was an issue raised about this,
and also explains the exact behaviour I experienced.

https://issues.apache.org/activemq/browse/AMQ-1958

I know that there's nothing that can be done really at the ActiveMQ level, its more of a DB issue really.
I did look into maybe setting the tcp_keepalive to a low value which might force the (invalid)connection
to close quicker at a TCP level, but I didn't want to mess around with stuff like that at such a low level.

Any lock options at the DB level were specific to the amount of time you wait until you get a lock,
not the amount of time you can hold onto it.

In the end I wrote a DB specific script which basically kills any sessions for a particular IP, which in
turn releases the lock and the slave can kick in.

If anyone knows of a different (perhaps better) way of implementing this please let us know.

Gary Tully wrote:

You have got to ask the JDBC driver or database documentation. I imagine
there is a lock expiry option that can be tweaked.

2009/6/29 colonelx <[e-mail]>

>
> Hi,
>
> I have 2 brokers setup in a JDBC Master Slave Configuration.
>
> Each broker is on a different machine. BoxA and BoxB
> The database is also on a different machine BoxC. (mysql 5.1)
>
> The failover is working fine when I manually kill any instance of ActiveMQ
> on BoxA or BoxB. (Ctrl-C)
>
> However, can anyone tell me what will happen if the if the Master instance
> looses its network connection ?
> Will its lock on the database get dropped immediately ? or will it still
> hold a (now invalid) lock on the database ?
> I ask this, because if this happens , there could possibly be a condition
> where the slave won't ever be able to get a lock on the database because
> the
> master has lock which did not get correctly released and manual
> intervention
> would be needed in order to fix up the problem.
>
> I know that I could test this by plugging in and out the physical network
> connections, but I'm not really in a position to do this on any of the
> above
> boxes.
>
> Thanks.
>
> --
> View this message in context:
> http://www.nabble.com/JDBC-Master-Slave-Question-tp24251198p24251198.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


--
http://blog.garytully.com

Open Source Integration
http://fusesource.com