[Linux-HA] pingd, quorum, split-brain... should I give up?
Riccardo Perni
riccardo.perni at aslromab.it
Mon Oct 22 15:30:03 MDT 2007
Hi Dejan,
thank you for your reply
Dejan Muhamedagic <dejanmm at fastmail.fm> ha scritto:
> Hi,
>
> On Mon, Oct 22, 2007 at 02:45:42PM +0200, Riccardo Perni wrote:
>> Hello to all,
>> it is several days that I'm trying to set-up a split-site cluster but
>> with scarce results.
>>
>> Since the two cluster nodes will be several Km away I cannot set up a
>> reliable communication media between them, so I have to run heartbeat on the
>> main Ethernet; I hoped that using pingd and an external ping site could
>> help me to solve the potential conflict that will showup, but probably I'm
>> not smart enough to solve this problem... Can someone help me?
>>
>> Actually I've set up a test using virtual machines
>> Only one resource is running (a virtual IP using ocf:IPaddr) and only one
>> constraint copied from linux-ha.org pingd FAQ.
>> All seems to work right and if I broke the network connectivity of one of
>> the nodes the resource is runned by the node with woking network; but both
>> nodes get the "dc" status and when the connectivity is restored I have a
>> split-brain condition with both nodes running the resource.
>
> Right, because you can't prevent split-brain this way. What you
> may prevent though is running a resource on the node which lost
> connectivity. What does exactly happen once the connectivity's
> restored? Can you post logs for that?
Yes, I understand this, just I'd like to have the cluster resync
itself upon reconnecting, BTW I have attache the log of both nodes..
>
>> Is it possible
>> to handle this situation?
>
> You may try quorumd. See
>
> http://www.linux-ha.org/QuorumServerGuide
I'm going to look at it, but is'n it another SPOF?
>
> Thanks,
>
> Dejan
>
>> Thankyou
>> Riccardo
>>
>>
>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA at lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
>
--
Riccardo Perni
Unità Operativa Informatica Aziendale
ASL Roma-B
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
-------------- next part --------------
Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe02 is online
Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr): Started clusterpaghe01
Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: notice: NoRoleChange: Leave resource resource_Virtual_IP (clusterpaghe01)
Oct 22 23:07:46 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-114.bz2
Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 4: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 4 status: te_complete - <null>
Oct 22 23:07:46 clusterpaghe01 cib: [3669]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: d2a2afe10026f9c560848a27a594b2c7)
Oct 22 23:07:46 clusterpaghe01 cib: [3670]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: d2a2afe10026f9c560848a27a594b2c7)
************ Network disconnected on this node
Oct 22 23:16:38 clusterpaghe01 cib: [3313]: info: cib_stats: Processed 48 operations (1875.00us average, 0% utilization) in the last 10min
Oct 22 23:17:36 clusterpaghe01 heartbeat: [3142]: WARN: node clusterpaghe02: is dead
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe02 now has status [dead]
Oct 22 23:17:36 clusterpaghe01 heartbeat: [3142]: info: Link clusterpaghe02:eth0 dead.
Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 38): 0.79.1583 (ok)
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1583
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: WARN: match_down_event: No match for shutdown action on 84dcf392-c5d9-44c1-a923-e436ed5af73e
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: extract_event: Stonith/shutdown of 84dcf392-c5d9-44c1-a923-e436ed5af73e not matched
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: quorum plugin: majority
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: quorum plugin: twonodes
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 23:17:36 clusterpaghe01 haclient: on_event:evt:cib_changed
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: info: Break tie for 2 nodes cluster
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 23:17:36 clusterpaghe01 haclient: on_event:evt:cib_changed
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: no mbr_track info
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: no mbr_track info
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: LOST: clusterpaghe02
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: CURRENT: clusterpaghe01 [nodeid=0, born=3]
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: LOST: clusterpaghe02 [nodeid=1, born=2]
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 39): 0.79.1583 (ok)
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1583
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 5: 0 actions in 0 synapses
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1583" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 5: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 5 status: te_complete - <null>
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr): Started clusterpaghe01
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: NoRoleChange: Leave resource resource_Virtual_IP (clusterpaghe01)
Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-115.bz2
Oct 22 23:17:36 clusterpaghe01 cib: [3730]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 289e20d9cda935df1cf765f1c1d9883a)
Oct 22 23:17:36 clusterpaghe01 cib: [3731]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: f86aff211891e078596ce3e46df9bc0d)
Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: WARN: node 10.44.4.1: is dead
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: WARN: node 10.44.4.4: is dead
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.1:10.44.4.1 dead.
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.4:10.44.4.4 dead.
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.1 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: WARN: get_uuid: Could not calculate UUID for 10.44.4.1
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.1 is dead
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.4 now has status [dead]
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: WARN: get_uuid: Could not calculate UUID for 10.44.4.4
Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.4 is dead
Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: flush message from clusterpaghe01
Oct 22 23:17:38 clusterpaghe01 haclient: on_event:evt:cib_changed
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: Sent update 11: pingd=0
Oct 22 23:17:38 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3316, call:11): 0.79.1583 -> 0.79.1584 (ok)
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_modify): 0.79.1583 -> 0.79.1584
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: extract_event: Aborting on transient_attributes changes for 8b658843-7f87-4a86-a398-e996f92fa12b
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
Oct 22 23:17:38 clusterpaghe01 cib: [3732]: info: write_cib_contents: Wrote version 0.79.1584 of the CIB to disk (digest: d888f0742bc020138182a5012caf070e)
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1584" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr): Started clusterpaghe01
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: WARN: native_color: Resource resource_Virtual_IP cannot run anywhere
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: StopRsc: clusterpaghe01 Stop resource_Virtual_IP
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 6: 1 actions in 1 synapses
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_stop_0 on clusterpaghe01
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_stop_0 key=3:6:b9de1520-d6ef-4f22-a2b5-2fb6ccbba5e0)
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/heartbeat/pengine/pe-warn-264.bz2
Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.
Oct 22 23:17:38 clusterpaghe01 lrmd: [3314]: info: RA output: (resource_Virtual_IP:stop:stderr) SIOCDELRT: No such process
Oct 22 23:17:38 clusterpaghe01 IPaddr[3733]: [3742]: INFO: /sbin/ifconfig eth0:0 10.44.4.28 down
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: process_lrm_event: LRM operation resource_Virtual_IP_stop_0 (call=4, rc=0) complete
Oct 22 23:17:38 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3317, call:44): 0.79.1584 -> 0.79.1585 (ok)
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1584 -> 0.79.1585
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: match_graph_event: Action resource_Virtual_IP_stop_0 (3) confirmed on 8b658843-7f87-4a86-a398-e996f92fa12b
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 6: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 6 status: te_complete - <null>
Oct 22 23:17:38 clusterpaghe01 haclient: on_event:evt:cib_changed
Oct 22 23:17:38 clusterpaghe01 cib: [3745]: info: write_cib_contents: Wrote version 0.79.1585 of the CIB to disk (digest: 268fa49f9fc7c5dba32c5c0e0d412491)
*********************************network reconnected
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: CRIT: Cluster node clusterpaghe02 returning after partition.
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe02 now has status [up]
Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe02 now has status [active]
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [up]
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Deadtime value may be too small.
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [active]
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: See FAQ for information on tuning deadtime.
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: URL: http://linux-ha.org/FAQ#heavy_load
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link clusterpaghe02:eth0 up.
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node clusterpaghe02: interval 172210 ms
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node clusterpaghe02: status active
Oct 22 23:19:58 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 45): 0.79.1585 (ok)
Oct 22 23:19:58 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1585 -> 0.79.1585
Oct 22 23:19:58 clusterpaghe01 haclient: on_event:evt:cib_changed
Oct 22 23:19:58 clusterpaghe01 cib: [3746]: info: write_cib_contents: Wrote version 0.79.1585 of the CIB to disk (digest: ad9f85a2de6b590ea08a167777d9e78c)
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: all clients are now paused
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=937
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug:
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node 10.44.4.4: interval 172210 ms
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node 10.44.4.4: status ping
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.4:10.44.4.4 up.
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.1:10.44.4.1 up.
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node 10.44.4.1: interval 172210 ms
Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node 10.44.4.1: status ping
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [ping]
Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.4 now has status [ping]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.4 is ping
Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.1 now has status [ping]
Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.1 is ping
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.4 now has status [up]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [up]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.1 now has status [up]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [up]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 2 active ping nodes
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [ping]
Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 2 active ping nodes
Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=938
Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug:
Oct 22 23:19:59 clusterpaghe01 attrd: [3316]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=939
Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug:
Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=940
Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: expecting from clusterpaghe02
Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: it's ackseq=0
Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug:
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=941
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: expecting from clusterpaghe02
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: it's ackseq=0
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug:
Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: info: all clients are now resumed
Oct 22 23:20:02 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: flush message from clusterpaghe01
Oct 22 23:20:02 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: Sent update 13: pingd=200000
Oct 22 23:20:02 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3316, call:13): 0.79.1585 -> 0.79.1586 (ok)
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_modify): 0.79.1585 -> 0.79.1586
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: extract_event: Aborting on transient_attributes changes for 8b658843-7f87-4a86-a398-e996f92fa12b
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1586" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr): Stopped
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: StartRsc: clusterpaghe01 Start resource_Virtual_IP
Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 7: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-116.bz2
Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 7: 1 actions in 1 synapses
Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_start_0 on clusterpaghe01
Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_start_0 key=3:7:b9de1520-d6ef-4f22-a2b5-2fb6ccbba5e0)
Oct 22 23:20:03 clusterpaghe01 lrmd: [3314]: info: RA output: (resource_Virtual_IP:start:stderr) Rewrote octal netmask as: 24
Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3756]: INFO: Using calculated nic for 10.44.4.28: eth0
Oct 22 23:20:03 clusterpaghe01 cib: [3747]: info: write_cib_contents: Wrote version 0.79.1586 of the CIB to disk (digest: 59654cc89cf183c1576a9b8e988470a5)
Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3761]: INFO: Using calculated netmask for 10.44.4.28: 255.255.255.0
Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3782]: INFO: eval /sbin/ifconfig eth0:0 10.44.4.28 netmask 255.255.252.0 broadcast 10.44.7.255
Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3787]: DEBUG: Sending Gratuitous Arp for 10.44.4.28 on eth0:0 [eth0]
Oct 22 23:20:03 clusterpaghe01 haclient: on_event: from message queue: evt:cib_changed
Oct 22 23:20:04 clusterpaghe01 crmd: [3317]: info: process_lrm_event: LRM operation resource_Virtual_IP_start_0 (call=5, rc=0) complete
Oct 22 23:20:04 clusterpaghe01 lrmd: [3314]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms) before being called (GSource: 0x805af90)
Oct 22 23:20:04 clusterpaghe01 lrmd: [3314]: info: G_SIG_dispatch: started at 1718148810 should have started at 1718148710
Oct 22 23:20:04 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 23:20:04 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3317, call:48): 0.79.1586 -> 0.79.1587 (ok)
Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1586 -> 0.79.1587
Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: match_graph_event: Action resource_Virtual_IP_start_0 (3) confirmed on 8b658843-7f87-4a86-a398-e996f92fa12b
Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 7: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 7 status: te_complete - <null>
Oct 22 23:20:04 clusterpaghe01 cib: [3801]: info: write_cib_contents: Wrote version 0.79.1587 of the CIB to disk (digest: fd42bd74a0057307571b0f8814b63e7f)
Oct 22 23:20:04 clusterpaghe01 haclient: on_event: from message queue: evt:cib_changed
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: no mbr_track info
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: no mbr_track info
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: CURRENT: clusterpaghe02 [nodeid=1, born=1]
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: CURRENT: clusterpaghe01 [nodeid=0, born=2]
Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: NEW: clusterpaghe02 [nodeid=1, born=1]
clusterpaghe01:~ #
-------------- next part --------------
Oct 22 22:31:22 clusterpaghe02 gconfd (root-3430): Resolved address "xml:readonly:/etc/opt/gnome/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: do_node_walk: Requesting the list of configured nodes
Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: send_update: 2 active ping nodes
Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: main: Starting pingd
Oct 22 22:31:25 clusterpaghe02 gconfd (root-3430): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe02
Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 4: pingd=200000
Oct 22 22:31:26 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3343, call:4): 0.79.1582 -> 0.79.1583 (ok)
Oct 22 22:31:26 clusterpaghe02 cib: [3523]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: ad8fb22dacf5d202e98caecce65473fd)
**********************network disconnected on the other node
Oct 22 22:41:11 clusterpaghe02 cib: [3340]: info: cib_stats: Processed 25 operations (2400.00us average, 0% utilization) in the last 10min
Oct 22 22:41:15 clusterpaghe02 heartbeat: [3190]: WARN: node clusterpaghe01: is dead
Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
Oct 22 22:41:15 clusterpaghe02 heartbeat: [3190]: info: Link clusterpaghe01:eth0 dead.
Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe01 now has status [dead]
Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 16): 0.79.1583 (ok)
Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum lost after event=NOT PRIMARY (id=2)
Oct 22 22:41:15 clusterpaghe02 haclient: on_event:evt:cib_changed
Oct 22 22:41:15 clusterpaghe02 cib: [3633]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 323e31c76c14e627275db776cf056917)
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: WARN: check_dead_member: Our DC node (clusterpaghe01) left the cluster
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: CURRENT: clusterpaghe02 [nodeid=1, born=3]
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: LOST: clusterpaghe01 [nodeid=0, born=1]
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to <null> (<null>)
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
Oct 22 22:41:28 clusterpaghe02 haclient: on_event:evt:cib_changed
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: LOST: clusterpaghe01
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 17): 0.79.1583 (ok)
Oct 22 22:41:28 clusterpaghe02 cib: [3634]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 027540dbd4bf36ce353c1462a021d33d)
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_election_count_vote: Updated voted hash for clusterpaghe02 to vote
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_election_count_vote: Election ignore: our vote (clusterpaghe02)
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: start_subsystem: Starting sub-system "tengine"
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: start_subsystem: Starting sub-system "pengine"
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_dc_takeover: Taking over DC status for this partition
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_process_readwrite: We are now in R/W mode
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to <null> (<null>)
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Oct 22 22:41:28 clusterpaghe02 pengine: [3636]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Oct 22 22:41:28 clusterpaghe02 pengine: [3636]: info: pe_init: Starting pengine
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: G_main_add_TriggerHandler: Added signal manual handler
Oct 22 22:41:28 clusterpaghe02 cib: [3637]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 027540dbd4bf36ce353c1462a021d33d)
Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: te_init: Registering TE UUID: a43171f3-ea7a-4759-ad3b-554f3ccad59f
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: set_graph_functions: Setting custom graph functions
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: te_init: Starting tengine
Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to clusterpaghe02 (1.0.8)
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_local_callback: Sending full refresh
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: sync_our_cib: Syncing CIB to all peers
Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: update_attrd: Connecting to attrd...
Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:23): 0.79.1583 -> 0.79.1584 (ok)
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1584
Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:24): 0.79.1584 -> 0.80.1585 (ok)
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_bump): 0.79.1584 -> 0.80.1585
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: update_abort_priority: Abort priority upgraded to 1000000
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: update_abort_priority: 'DC Takeover' abort superceeded
Oct 22 22:41:29 clusterpaghe02 cib: [3638]: info: write_cib_contents: Wrote version 0.80.1585 of the CIB to disk (digest: 6c47a6cb394c3c8eb965cd3ab7752d71)
Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe02
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to clusterpaghe02 (1.0.8)
Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 6: pingd=200000
Oct 22 22:41:29 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
Oct 22 22:41:29 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
Oct 22 22:41:29 clusterpaghe02 cib: [3639]: info: write_cib_contents: Wrote version 0.80.1585 of the CIB to disk (digest: 6c47a6cb394c3c8eb965cd3ab7752d71)
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_dc_join_ack: join-1: Updating node state to member for clusterpaghe02)
Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:26): 0.80.1585 -> 0.80.1586 (ok)
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1585 -> 0.80.1586
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: process_graph_event: Action resource_Virtual_IP_monitor_0 initiated by a different transitioner
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" ccm_transition="3" dc_uuid="84dcf392-c5d9-44c1-a923-e436ed5af73e" epoch="80" num_updates="1586"/>
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Oct 22 22:41:29 clusterpaghe02 cib: [3640]: info: write_cib_contents: Wrote version 0.80.1586 of the CIB to disk (digest: 7eb6b9a59f21650623d53050a1f31e3b)
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: determine_online_status: Node clusterpaghe02 is online
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr): Stopped
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: StartRsc: clusterpaghe02 Start resource_Virtual_IP
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_start_0 key=3:0:a43171f3-ea7a-4759-ad3b-554f3ccad59f)
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: unpack_graph: Unpacked transition 0: 1 actions in 1 synapses
Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_start_0 on clusterpaghe02
Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-91.bz2
Oct 22 22:41:29 clusterpaghe02 lrmd: [3341]: info: RA output: (resource_Virtual_IP:start:stderr) Rewrote octal netmask as: 24
Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3650]: INFO: Using calculated nic for 10.44.4.28: eth0
Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3655]: INFO: Using calculated netmask for 10.44.4.28: 255.255.255.0
Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3676]: INFO: eval /sbin/ifconfig eth0:0 10.44.4.28 netmask 255.255.252.0 broadcast 10.44.7.255
Oct 22 22:41:30 clusterpaghe02 IPaddr[3641]: [3681]: DEBUG: Sending Gratuitous Arp for 10.44.4.28 on eth0:0 [eth0]
Oct 22 22:41:30 clusterpaghe02 crmd: [3344]: info: process_lrm_event: LRM operation resource_Virtual_IP_start_0 (call=3, rc=0) complete
Oct 22 22:41:30 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
Oct 22 22:41:30 clusterpaghe02 kernel: send_arp uses obsolete (PF_INET,SOCK_PACKET)
Oct 22 22:41:30 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:29): 0.80.1586 -> 0.80.1587 (ok)
Oct 22 22:41:30 clusterpaghe02 haclient: on_event:evt:cib_changed
Oct 22 22:41:30 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1586 -> 0.80.1587
Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: match_graph_event: Action resource_Virtual_IP_start_0 (3) confirmed on 84dcf392-c5d9-44c1-a923-e436ed5af73e
Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: run_graph: Transition 0: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: notify_crmd: Transition 0 status: te_complete - <null>
Oct 22 22:41:30 clusterpaghe02 kernel: NET: Registered protocol family 17
Oct 22 22:41:30 clusterpaghe02 cib: [3705]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: e8ea66f562d7aeb0a0eaeff3d0525b6a)
************Network reconnected
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: CRIT: Cluster node clusterpaghe01 returning after partition.
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: WARN: Deadtime value may be too small.
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: See FAQ for information on tuning deadtime.
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: URL: http://linux-ha.org/FAQ#heavy_load
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: Link clusterpaghe01:eth0 up.
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: WARN: Late heartbeat: Node clusterpaghe01: interval 172120 ms
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: Status update for node clusterpaghe01: status active
Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe01 now has status [up]
Oct 22 22:43:38 clusterpaghe02 crmd: [3344]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe01 now has status [active]
Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [up]
Oct 22 22:43:38 clusterpaghe02 haclient: on_event:evt:cib_changed
Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [active]
Oct 22 22:43:38 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 30): 0.80.1587 (ok)
Oct 22 22:43:38 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1587 -> 0.80.1587
Oct 22 22:43:38 clusterpaghe02 cib: [3706]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: 1ec8a6bf75fd6006aba055083aa85b4e)
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: all clients are now paused
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=863
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=864
Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=865
Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=866
Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=867
Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:42 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe01
Oct 22 22:43:42 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 8: pingd=200000
Oct 22 22:43:42 clusterpaghe02 cib: [3707]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: b197b4491faf083594aac256c6cb11ed)
Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=868
Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:42 clusterpaghe02 cib: [3340]: WARN: cib_peer_callback: Discarding cib_apply_diff message (3b3) from clusterpaghe01: not in our membership
Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=869
Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug:
Oct 22 22:43:44 clusterpaghe02 heartbeat: [3190]: info: all clients are now resumed
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: WARN: cib_peer_callback: Discarding cib_apply_diff message (3b5) from clusterpaghe01: not in our membership
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=1)
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=1, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3
Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: CURRENT: clusterpaghe02 [nodeid=1, born=1]
Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=2, member_quorum_votes=200
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: CURRENT: clusterpaghe02 [nodeid=1, born=1]
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: CURRENT: clusterpaghe01 [nodeid=0, born=2]
Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW: clusterpaghe01 [nodeid=0, born=2]
clusterpaghe02:~ #
More information about the Linux-HA
mailing list