Re: Active-active crash report

Ok, I got it again.
3 nodes. Stop slave, restart app(client), bring slave back online, stop master, restart app, bring master back online, BOOM!
So here are logs from both nodes, the one which I shutdown (dev-2) and the one which failed unexpectedly (dev-3)

dev-2 was the master node and when I stopped it, it brought dev-3 node down with it.
Did everything started because of this?
> Reason:     reached_max_restart_intensity

Vadim

dev-3 log :
===============================================
=INFO REPORT==== 26-Apr-2012::16:50:38 ===
Management agent started.

=INFO REPORT==== 26-Apr-2012::16:50:38 ===
Management plugin started. Port: 55672, path: /

=INFO REPORT==== 26-Apr-2012::16:51:07 ===
    application: rabbitmq_management
    exited: shutdown
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
stopped TCP Listener on 0.0.0.0:5672
===============================================

dev-3-sasl.log
===============================================
=SUPERVISOR REPORT==== 26-Apr-2012::16:51:07 ===
     Supervisor: {local,rabbit_mgmt_sup}
     Context:    child_terminated
     Reason:     shutdown
     Offender:   [{pid,<0.396.0>},
                  {name,mirroring},
                  {mfa,
                      {mirrored_supervisor,start_internal,
                          [rabbit_mgmt_sup,
                           [{rabbit_mgmt_db,
                                {rabbit_mgmt_db,start_link,[]},
                                permanent,4294967295,worker,
                                [rabbit_mgmt_db]}]]}},
                  {restart_type,permanent},
                  {shutdown,4294967295},
                  {child_type,worker}]


=SUPERVISOR REPORT==== 26-Apr-2012::16:51:07 ===
     Supervisor: {local,rabbit_mgmt_sup}
     Context:    shutdown
     Reason:     reached_max_restart_intensity
     Offender:   [{pid,<0.396.0>},
                  {name,mirroring},
                  {mfa,
                      {mirrored_supervisor,start_internal,
                          [rabbit_mgmt_sup,
                           [{rabbit_mgmt_db,
                                {rabbit_mgmt_db,start_link,[]},
                                permanent,4294967295,worker,
                                [rabbit_mgmt_db]}]]}},
                  {restart_type,permanent},
                  {shutdown,4294967295},
                  {child_type,worker}]

===============================================

dev-2 log
===============================================
=INFO REPORT==== 26-Apr-2012::16:51:07 ===
Stopping Rabbit

=INFO REPORT==== 26-Apr-2012::16:51:07 ===
    application: rabbitmq_management_visualiser
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:07 ===
    application: rabbitmq_management
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:07 ===
    application: rabbitmq_management_agent
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:07 ===
stopped TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
rabbit on node 'rabbit@rabbitmq-dev-3' down

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'Advertiser-ONYX' in vhost '/'): Slave <'rabbit@rabbitmq-dev-2'.1.265.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.290.0>

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'Search-JROSE6420' in vhost '/'): Slave <'rabbit@rabbitmq-dev-2'.1.276.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.328.0>

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'Advertiser-DHOHMEYERGX280' in vhost '/'): Slave <'rabbit@rabbitmq-dev-2'.1.268.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.288.0>

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'CustomerAdmin-DHOHMEYERGX280' in vhost '/'): Slave <'rabbit@rabbitmq-dev-2'.1.270.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.294.0>

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'Tracking-JROSE6420' in vhost '/'): Slave <'rabbit@rabbitmq-dev-2'.1.250.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.247.0>

=ERROR REPORT==== 26-Apr-2012::16:51:08 ===
** Generic server rabbit_mirror_queue_slave_sup terminating
** Last message in was {'EXIT',<0.127.0>,shutdown}
** When Server state == {state,
                         {local,rabbit_mirror_queue_slave_sup},
                         simple_one_for_one_terminate,
                         [{child,undefined,rabbit_mirror_queue_slave,
                           {rabbit_mirror_queue_slave,start_link,[]},
                           temporary,4294967295,worker,
                           [rabbit_mirror_queue_slave]}],
                         {dict,11,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],
                            [[<0.265.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,<<"Advertiser-ONYX">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.289.0>,
                               [<3065.334.0>],
                               all}]],
                            [],
                            [[undefined,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"Advertiser-heartbeat-DHOHMEYERGX280">>},
                               false,true,<3065.24765.0>,
                               [{<<"x-ha-policy">>,longstr,<<"all">>}],
                               <3065.24784.0>,
                               [<3067.12370.2>],
                               all}],
                             [<0.6730.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"Advertiser-heartbeat-DHOHMEYERGX280">>},
                               false,true,<3065.14349.1>,
                               [{<<"x-ha-policy">>,longstr,<<"all">>}],
                               <3065.14370.1>,[],all}]],
                            [[<0.268.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"Advertiser-DHOHMEYERGX280">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.287.0>,
                               [<3065.332.0>],
                               all}]],
                            [],
                            [[<0.270.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"CustomerAdmin-DHOHMEYERGX280">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.295.0>,
                               [<3065.338.0>],
                               all}]],
                            [],
                            [[<0.272.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"Sales-DHOHMEYERGX280">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.317.0>,
                               [<3065.356.0>],
                               all}]],
                            [[<0.250.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"Tracking-JROSE6420">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.247.0>,
                               [<3065.249.0>],
                               all}]],
                            [[<0.274.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"RelatedLinks-ROGERMOORE">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.331.0>,
                               [<3065.370.0>],
                               all}]],
                            [[<0.252.0>,
                              {amqqueue,
                               {resource,<<"ActivityLogging">>,queue,
                                <<"vsw.activitylogging.conversion.pendingconversion">>},
                               true,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>}],
                               <3067.253.0>,
                               [<3065.257.0>],
                               all}]],
                            [[<0.276.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,<<"Search-JROSE6420">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.26259.1>,
                               [<3065.4161.0>],
                               all}]],
                            [[<0.261.0>,
                              {amqqueue,
                               {resource,<<"/">>,queue,
                                <<"RelatedLinks-ONYX">>},
                               false,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>},
                                {<<"x-message-ttl">>,signedint,1200000}],
                               <3067.277.0>,
                               [<3065.322.0>],
                               all}]],
                            []}}},
                         10,10,[],rabbit_mirror_queue_slave_sup,[]}
** Reason for termination ==
** {badarg,[{erlang,unlink,[undefined]},
            {supervisor2,'-terminate_simple_children/3-fun-0-',4},
            {dict,fold_bucket,3},
            {dict,fold_seg,4},
            {dict,fold_segs,4},
            {supervisor2,terminate_simple_children,3},
            {supervisor2,terminate,2},
            {gen_server,terminate,6}]}

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Mirrored-queue (queue 'vsw.activitylogging.conversion.pendingconversion' in vhost 'ActivityLogging'): Slave <'rabbit@rabbitmq-dev-2'.1.252.0> saw deaths of mirrors <'rabbit@rabbitmq-dev-3'.2.253.0>

=ERROR REPORT==== 26-Apr-2012::16:51:08 ===
** Generic server <0.265.0> terminating
** Last message in was {'EXIT',<0.249.0>,
                           {badarg,
                               [{erlang,unlink,[undefined]},
                                {supervisor2,
                                    '-terminate_simple_children/3-fun-0-',4},
                                {dict,fold_bucket,3},
                                {dict,fold_seg,4},
                                {dict,fold_segs,4},
                                {supervisor2,terminate_simple_children,3},
                                {supervisor2,terminate,2},
                                {gen_server,terminate,6}]}}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"/">>,queue,<<"Advertiser-ONYX">>},
                          false,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>},
                           {<<"x-message-ttl">>,signedint,1200000}],
                          <3067.289.0>,
                          [<3065.334.0>],
                          all},
                         <0.267.0>,<3065.334.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          undefined,
                          {0,nil},
                          {qistate,
                           "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/queues/1ON03RTBZAE64B67H41L4ICDE",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,262144,
                           #Fun<rabbit_variable_queue.2.91259159>,
                           {0,nil}},
                          {undefined,
                           {client_msstate,msg_store_transient,
                            <<228,99,153,84,177,114,138,215,76,30,0,48,128,77,
                              68,245>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,368709,
                             "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient",
                            <0.239.0>,372806,364585,376903,381000}},
                          false,0,#Fun<rabbit_mirror_queue_slave.3.29941254>,
                          0,0,infinity,0,0,0,0,0,
                          {rates,
                           {{1335,483942,985021},0},
                           {{1335,483942,985021},0},
                           0.0,0.0,
                           {1335,484240,226498}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1335,483942,985021},0},
                           {{1335,483942,985021},0},
                           0.0,0.0,
                           {1335,484240,226498}}},
                         undefined,#Ref<0.0.9.158766>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         true}
** Reason for termination ==
** {badarg,[{erlang,unlink,[undefined]},
            {supervisor2,'-terminate_simple_children/3-fun-0-',4},
            {dict,fold_bucket,3},
            {dict,fold_seg,4},
            {dict,fold_segs,4},
            {supervisor2,terminate_simple_children,3},
            {supervisor2,terminate,2},
            {gen_server,terminate,6}]}

=ERROR REPORT==== 26-Apr-2012::16:51:08 ===
** Generic server <0.6730.0> terminating
** Last message in was {'EXIT',<0.249.0>,
                           {badarg,
                               [{erlang,unlink,[undefined]},
                                {supervisor2,
                                    '-terminate_simple_children/3-fun-0-',4},
                                {dict,fold_bucket,3},
                                {dict,fold_seg,4},
                                {dict,fold_segs,4},
                                {supervisor2,terminate_simple_children,3},
                                {supervisor2,terminate,2},
                                {gen_server,terminate,6}]}}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"/">>,queue,
                           <<"Advertiser-heartbeat-DHOHMEYERGX280">>},
                          false,true,<3065.14349.1>,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <3065.14370.1>,[],all},
                         <0.6731.0>,<3065.14370.1>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          undefined,
                          {0,nil},
                          {qistate,
                           "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/queues/5R0GE9UEB8S0EMSJORBUKHBTE",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,262144,
                           #Fun<rabbit_variable_queue.2.91259159>,
                           {0,nil}},
                          {undefined,
                           {client_msstate,msg_store_transient,
                            <<182,86,2,135,69,87,14,108,163,188,15,159,127,119,
                              55,241>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,368709,
                             "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient",
                            <0.239.0>,372806,364585,376903,381000}},
                          false,0,#Fun<rabbit_mirror_queue_slave.3.29941254>,
                          0,0,infinity,0,0,0,0,0,
                          {rates,
                           {{1335,484239,805553},0},
                           {{1335,484239,805553},0},
                           0.0,0.0,
                           {1335,484255,991021}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1335,484239,805553},0},
                           {{1335,484239,805553},0},
                           0.0,0.0,
                           {1335,484255,991021}}},
                         undefined,#Ref<0.0.9.158602>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         true}
** Reason for termination ==
** {badarg,[{erlang,unlink,[undefined]},
            {supervisor2,'-terminate_simple_children/3-fun-0-',4},
            {dict,fold_bucket,3},
            {dict,fold_seg,4},
            {dict,fold_segs,4},
            {supervisor2,terminate_simple_children,3},
            {supervisor2,terminate,2},
            {gen_server,terminate,6}]}

=ERROR REPORT==== 26-Apr-2012::16:51:08 ===
** Generic server <0.252.0> terminating
** Last message in was {'EXIT',<0.249.0>,shutdown}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"ActivityLogging">>,queue,
                           <<"vsw.activitylogging.conversion.pendingconversion">>},
                          true,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <3067.253.0>,
                          [<3065.257.0>],
                          all},
                         <0.253.0>,<3065.257.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          undefined,
                          {0,nil},
                          {qistate,
                           "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/queues/9KDJJKR11HTT40SVOPWG0AWDM",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,262144,
                           #Fun<rabbit_variable_queue.2.91259159>,
                           {0,nil}},
                          {{client_msstate,msg_store_persistent,
                            <<58,135,182,192,228,241,29,217,255,38,4,53,94,213,
                              95,102>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,389194,
                             "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_persistent"},
                            rabbit_msg_store_ets_index,
                            "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_persistent",
                            <0.244.0>,393291,385097,397388,401485},
                           {client_msstate,msg_store_transient,
                            <<63,170,201,248,36,125,208,20,133,142,139,71,219,
                              91,212,33>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,368709,
                             "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-dev-2/msg_store_transient",
                            <0.239.0>,372806,364585,376903,381000}},
                          true,0,#Fun<rabbit_mirror_queue_slave.3.29941254>,0,
                          0,infinity,0,0,0,0,0,
                          {rates,
                           {{1335,483943,551021},0},
                           {{1335,483943,551021},0},
                           0.0,0.0,
                           {1335,484240,51118}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1335,483943,551021},0},
                           {{1335,483943,551021},0},
                           0.0,0.0,
                           {1335,484240,51118}}},
                         undefined,#Ref<0.0.9.158618>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         true}
** Reason for termination ==
** {shutdown,
       {gen_server2,call,
           [msg_store_persistent,
            {client_terminate,
                <<58,135,182,192,228,241,29,217,255,38,4,53,94,213,95,102>>},
            infinity]}}
** In 'terminate' callback with reason ==
** shutdown

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
    application: rabbit
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
    application: os_mon
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
    application: mnesia
    exited: stopped
    type: permanent

=INFO REPORT==== 26-Apr-2012::16:51:08 ===
Halting Erlang VM

===============================================

dev-2-sasl.log
===============================================
=CRASH REPORT==== 26-Apr-2012::16:51:08 ===
  crasher:
    initial call: supervisor2:init/1
    pid: <0.249.0>
    registered_name: rabbit_mirror_queue_slave_sup
    exception exit: {badarg,
                        [{erlang,unlink,[undefined]},
                         {supervisor2,'-terminate_simple_children/3-fun-0-',4},
                         {dict,fold_bucket,3},
                         {dict,fold_seg,4},
                         {dict,fold_segs,4},
                         {supervisor2,terminate_simple_children,3},
                         {supervisor2,terminate,2},
                         {gen_server,terminate,6}]}
      in function  gen_server:terminate/6
    ancestors: [rabbit_sup,<0.126.0>]
    messages: [{'DOWN',#Ref<0.0.9.159703>,process,
                          {undefined,'rabbit@rabbitmq-dev-2'},
                          noproc}]
    links: [<0.265.0>,<0.6730.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 5064
  neighbours:

=SUPERVISOR REPORT==== 26-Apr-2012::16:51:08 ===
     Supervisor: {local,rabbit_sup}
     Context:    shutdown_error
     Reason:     {badarg,
                     [{erlang,unlink,[undefined]},
                      {supervisor2,'-terminate_simple_children/3-fun-0-',4},
                      {dict,fold_bucket,3},
                      {dict,fold_seg,4},
                      {dict,fold_segs,4},
                      {supervisor2,terminate_simple_children,3},
                      {supervisor2,terminate,2},
                      {gen_server,terminate,6}]}
     Offender:   [{pid,<0.249.0>},
                  {name,rabbit_mirror_queue_slave_sup},
                  {mfargs,{rabbit_mirror_queue_slave_sup,start_link,[]}},
                  {restart_type,transient},
                  {shutdown,infinity},
                  {child_type,supervisor}]


=CRASH REPORT==== 26-Apr-2012::16:51:08 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.265.0>
    registered_name: []
    exception exit: {badarg,
                        [{erlang,unlink,[undefined]},
                         {supervisor2,'-terminate_simple_children/3-fun-0-',4},
                         {dict,fold_bucket,3},
                         {dict,fold_seg,4},
                         {dict,fold_segs,4},
                         {supervisor2,terminate_simple_children,3},
                         {supervisor2,terminate,2},
                         {gen_server,terminate,6}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.126.0>]
    messages: [{'EXIT',<0.267.0>,normal},
                  {'DOWN',#Ref<0.0.3.44831>,process,<3065.334.0>,normal}]
    links: []
    dictionary: [{{xtype_to_module,direct},rabbit_exchange_type_direct},
                  {{xtype_to_module,topic},rabbit_exchange_type_topic},
                  {guid,{{3831732564,2977073879,1277034544,2152547573},0}}]
    trap_exit: true
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 4916
  neighbours:

=CRASH REPORT==== 26-Apr-2012::16:51:08 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.6730.0>
    registered_name: []
    exception exit: {badarg,
                        [{erlang,unlink,[undefined]},
                         {supervisor2,'-terminate_simple_children/3-fun-0-',4},
                         {dict,fold_bucket,3},
                         {dict,fold_seg,4},
                         {dict,fold_segs,4},
                         {supervisor2,terminate_simple_children,3},
                         {supervisor2,terminate,2},
                         {gen_server,terminate,6}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.126.0>]
    messages: [{'EXIT',<0.6731.0>,normal},
                  {'DOWN',#Ref<0.0.4.191216>,process,<3065.14370.1>,normal}]
    links: []
    dictionary: [{credit_blocked,[]},
                  {{xtype_to_module,direct},rabbit_exchange_type_direct},
                  {{xtype_to_module,fanout},rabbit_exchange_type_fanout},
                  {guid,{{3059090055,1163333228,2747010975,2138519537},0}}]
    trap_exit: true
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 8290
  neighbours:
===============================================