[Linux-HA] pingd, quorum, split-brain... should I give up?

Dejan Muhamedagic dejanmm at fastmail.fm
Tue Oct 23 08:59:25 MDT 2007


Hi,

On Mon, Oct 22, 2007 at 11:30:03PM +0200, Riccardo Perni wrote:
> Hi Dejan,
> thank you for your reply
> 
> 
> Dejan Muhamedagic <dejanmm at fastmail.fm> ha scritto:
> 
> >Hi,
> >
> >On Mon, Oct 22, 2007 at 02:45:42PM +0200, Riccardo Perni wrote:
> >>Hello to all,
> >>it is several days that I'm trying	to set-up a split-site cluster but
> >>with scarce results.
> >>
> >>Since the two cluster nodes will be several Km away I cannot set up a
> >>reliable communication media between them, so I have to run heartbeat on 
> >>the
> >>main Ethernet;  I hoped that using pingd and an external ping site could
> >>help me to solve the potential conflict that will showup, but probably I'm
> >>not smart enough to solve this problem... Can someone help me?
> >>
> >>Actually I've set up a test using virtual machines
> >>Only one resource is running (a virtual IP using ocf:IPaddr) and only one
> >>constraint copied from linux-ha.org pingd FAQ.
> >>All seems to work right and if I broke the network connectivity of one of
> >>the nodes the resource is runned by the node with woking network; but both
> >>nodes get the "dc" status and when the connectivity is restored I have a
> >>split-brain condition with both nodes running the resource.
> >
> >Right, because you can't prevent split-brain this way. What you
> >may prevent though is running a resource on the node which lost
> >connectivity. What does exactly happen once the connectivity's
> >restored? Can you post logs for that?
> 
> Yes, I understand this, just I'd like to have the cluster resync  
> itself upon reconnecting, BTW I have attache the log of both nodes..

Unfortunately, they end before the interesting part.

Thanks,

Dejan

> >
> >>Is it possible
> >>to handle this situation?
> >
> >You may try quorumd. See
> >
> >http://www.linux-ha.org/QuorumServerGuide
> 
> I'm going to look at it, but is'n it another SPOF?
> 
> >
> >Thanks,
> >
> >Dejan
> >
> >>Thankyou
> >>Riccardo
> >>
> >>
> >>
> >>
> >>_______________________________________________
> >>Linux-HA mailing list
> >>Linux-HA at lists.linux-ha.org
> >>http://lists.linux-ha.org/mailman/listinfo/linux-ha
> >>See also: http://linux-ha.org/ReportingProblems
> >_______________________________________________
> >Linux-HA mailing list
> >Linux-HA at lists.linux-ha.org
> >http://lists.linux-ha.org/mailman/listinfo/linux-ha
> >See also: http://linux-ha.org/ReportingProblems
> >
> >
> 
> -- 
> Riccardo Perni
> Unità Operativa Informatica Aziendale
> ASL Roma-B
> 
> 
> 
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
> 

> Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe02 is online
> Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr):        Started clusterpaghe01
> Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: notice: NoRoleChange: Leave resource resource_Virtual_IP        (clusterpaghe01)
> Oct 22 23:07:46 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
> Oct 22 23:07:46 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-114.bz2
> Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 4: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
> Oct 22 23:07:46 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 4 status: te_complete - <null>
> Oct 22 23:07:46 clusterpaghe01 cib: [3669]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: d2a2afe10026f9c560848a27a594b2c7)
> Oct 22 23:07:46 clusterpaghe01 cib: [3670]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: d2a2afe10026f9c560848a27a594b2c7)
> 
> 
> 
> 
> ************ Network disconnected on this node
> 
> Oct 22 23:16:38 clusterpaghe01 cib: [3313]: info: cib_stats: Processed 48 operations (1875.00us average, 0% utilization) in the last 10min
> Oct 22 23:17:36 clusterpaghe01 heartbeat: [3142]: WARN: node clusterpaghe02: is dead
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe02 now has status [dead]
> Oct 22 23:17:36 clusterpaghe01 heartbeat: [3142]: info: Link clusterpaghe02:eth0 dead.
> Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
> Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
> Oct 22 23:17:36 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [dead]
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 38): 0.79.1583 (ok)
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1583
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: WARN: match_down_event: No match for shutdown action on 84dcf392-c5d9-44c1-a923-e436ed5af73e
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: extract_event: Stonith/shutdown of 84dcf392-c5d9-44c1-a923-e436ed5af73e not matched
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: quorum plugin: majority
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: quorum plugin: twonodes
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 23:17:36 clusterpaghe01 haclient: on_event:evt:cib_changed
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 23:17:36 clusterpaghe01 ccm: [3312]: info: Break tie for 2 nodes cluster
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 23:17:36 clusterpaghe01 haclient: on_event:evt:cib_changed
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: no mbr_track info
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: no mbr_track info
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: LOST: clusterpaghe02
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail:    CURRENT: clusterpaghe01 [nodeid=0, born=3]
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: ccm_event_detail:    LOST:    clusterpaghe02 [nodeid=1, born=2]
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Oct 22 23:17:36 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 39): 0.79.1583 (ok)
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1583
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 5: 0 actions in 0 synapses
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1583" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
> Oct 22 23:17:36 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 5: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
> Oct 22 23:17:36 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 5 status: te_complete - <null>
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr):        Started clusterpaghe01
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: notice: NoRoleChange: Leave resource resource_Virtual_IP        (clusterpaghe01)
> Oct 22 23:17:36 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-115.bz2
> Oct 22 23:17:36 clusterpaghe01 cib: [3730]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 289e20d9cda935df1cf765f1c1d9883a)
> Oct 22 23:17:36 clusterpaghe01 cib: [3731]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: f86aff211891e078596ce3e46df9bc0d)
> Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: WARN: node 10.44.4.1: is dead
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: WARN: node 10.44.4.4: is dead
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
> Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.1:10.44.4.1 dead.
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.4:10.44.4.4 dead.
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 pingd: [3320]: info: send_update: 0 active ping nodes
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.1 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: WARN: get_uuid: Could not calculate UUID for 10.44.4.1
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.1 is dead
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.4 now has status [dead]
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: WARN: get_uuid: Could not calculate UUID for 10.44.4.4
> Oct 22 23:17:37 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.4 is dead
> Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
> Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: flush message from clusterpaghe01
> Oct 22 23:17:38 clusterpaghe01 haclient: on_event:evt:cib_changed
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:38 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: Sent update 11: pingd=0
> Oct 22 23:17:38 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3316, call:11): 0.79.1583 -> 0.79.1584 (ok)
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_modify): 0.79.1583 -> 0.79.1584
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: extract_event: Aborting on transient_attributes changes for 8b658843-7f87-4a86-a398-e996f92fa12b
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
> Oct 22 23:17:38 clusterpaghe01 cib: [3732]: info: write_cib_contents: Wrote version 0.79.1584 of the CIB to disk (digest: d888f0742bc020138182a5012caf070e)
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1584" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr):        Started clusterpaghe01
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: WARN: native_color: Resource resource_Virtual_IP cannot run anywhere
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: notice: StopRsc:   clusterpaghe01       Stop resource_Virtual_IP
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 6: 1 actions in 1 synapses
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_stop_0 on clusterpaghe01
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_stop_0 key=3:6:b9de1520-d6ef-4f22-a2b5-2fb6ccbba5e0)
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/heartbeat/pengine/pe-warn-264.bz2
> Oct 22 23:17:38 clusterpaghe01 pengine: [3398]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
> Oct 22 23:17:38 clusterpaghe01 lrmd: [3314]: info: RA output: (resource_Virtual_IP:stop:stderr) SIOCDELRT: No such process
> Oct 22 23:17:38 clusterpaghe01 IPaddr[3733]: [3742]: INFO: /sbin/ifconfig eth0:0 10.44.4.28 down
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: process_lrm_event: LRM operation resource_Virtual_IP_stop_0 (call=4, rc=0) complete
> Oct 22 23:17:38 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3317, call:44): 0.79.1584 -> 0.79.1585 (ok)
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1584 -> 0.79.1585
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: match_graph_event: Action resource_Virtual_IP_stop_0 (3) confirmed on 8b658843-7f87-4a86-a398-e996f92fa12b
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 6: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
> Oct 22 23:17:38 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:17:38 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 6 status: te_complete - <null>
> Oct 22 23:17:38 clusterpaghe01 haclient: on_event:evt:cib_changed
> Oct 22 23:17:38 clusterpaghe01 cib: [3745]: info: write_cib_contents: Wrote version 0.79.1585 of the CIB to disk (digest: 268fa49f9fc7c5dba32c5c0e0d412491)
> 
> 
> 
> 
> *********************************network reconnected
> 
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: CRIT: Cluster node clusterpaghe02 returning after partition.
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe02 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe02 now has status [active]
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Deadtime value may be too small.
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe02 now has status [active]
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: See FAQ for information on tuning deadtime.
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: URL: http://linux-ha.org/FAQ#heavy_load
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link clusterpaghe02:eth0 up.
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node clusterpaghe02: interval 172210 ms
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node clusterpaghe02: status active
> Oct 22 23:19:58 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Local-only Change (client:3317, call: 45): 0.79.1585 (ok)
> Oct 22 23:19:58 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1585 -> 0.79.1585
> Oct 22 23:19:58 clusterpaghe01 haclient: on_event:evt:cib_changed
> Oct 22 23:19:58 clusterpaghe01 cib: [3746]: info: write_cib_contents: Wrote version 0.79.1585 of the CIB to disk (digest: ad9f85a2de6b590ea08a167777d9e78c)
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: all clients are now paused
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=937
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: debug:
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node 10.44.4.4: interval 172210 ms
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node 10.44.4.4: status ping
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.4:10.44.4.4 up.
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Link 10.44.4.1:10.44.4.1 up.
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: WARN: Late heartbeat: Node 10.44.4.1: interval 172210 ms
> Oct 22 23:19:58 clusterpaghe01 heartbeat: [3142]: info: Status update for node 10.44.4.1: status ping
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [ping]
> Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.4 now has status [ping]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
> Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.4 is ping
> Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: notice: crmd_ha_status_callback: Status update: Node 10.44.4.1 now has status [ping]
> Oct 22 23:19:58 clusterpaghe01 crmd: [3317]: info: crmd_ha_status_callback: Ping node 10.44.4.1 is ping
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.4 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.4 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 1 active ping nodes
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_lstatus_callback: Status update: Ping node 10.44.4.1 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [up]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 2 active ping nodes
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: notice: pingd_nstatus_callback: Status update: Ping node 10.44.4.1 now has status [ping]
> Oct 22 23:19:58 clusterpaghe01 pingd: [3320]: info: send_update: 2 active ping nodes
> Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
> Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=938
> Oct 22 23:19:59 clusterpaghe01 heartbeat: [3142]: debug:
> Oct 22 23:19:59 clusterpaghe01 attrd: [3316]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
> Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
> Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=939
> Oct 22 23:20:00 clusterpaghe01 heartbeat: [3142]: debug:
> Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
> Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=940
> Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: expecting from clusterpaghe02
> Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug: it's ackseq=0
> Oct 22 23:20:01 clusterpaghe01 heartbeat: [3142]: debug:
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: hist->ackseq =836
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: hist->lowseq =835, hist->hiseq=941
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: expecting from clusterpaghe02
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug: it's ackseq=0
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: debug:
> Oct 22 23:20:02 clusterpaghe01 heartbeat: [3142]: info: all clients are now resumed
> Oct 22 23:20:02 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: flush message from clusterpaghe01
> Oct 22 23:20:02 clusterpaghe01 attrd: [3316]: info: attrd_ha_callback: Sent update 13: pingd=200000
> Oct 22 23:20:02 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3316, call:13): 0.79.1585 -> 0.79.1586 (ok)
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_modify): 0.79.1585 -> 0.79.1586
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: extract_event: Aborting on transient_attributes changes for 8b658843-7f87-4a86-a398-e996f92fa12b
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: update_abort_priority: Abort priority upgraded to 1000000
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: te_update_diff: Aborting on transient_attributes deletions
> Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" epoch="79" num_updates="1586" cib-last-written="Mon Oct 22 23:05:22 2007" ccm_transition="3" dc_uuid="8b658843-7f87-4a86-a398-e996f92fa12b"/>
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: determine_online_status: Node clusterpaghe01 is online
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr):        Stopped
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: notice: StartRsc:  clusterpaghe01       Start resource_Virtual_IP
> Oct 22 23:20:02 clusterpaghe01 pengine: [3398]: info: process_pe_message: Transition 7: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-116.bz2
> Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: unpack_graph: Unpacked transition 7: 1 actions in 1 synapses
> Oct 22 23:20:02 clusterpaghe01 tengine: [3397]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_start_0 on clusterpaghe01
> Oct 22 23:20:02 clusterpaghe01 crmd: [3317]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_start_0 key=3:7:b9de1520-d6ef-4f22-a2b5-2fb6ccbba5e0)
> Oct 22 23:20:03 clusterpaghe01 lrmd: [3314]: info: RA output: (resource_Virtual_IP:start:stderr) Rewrote octal netmask as: 24
> Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3756]: INFO: Using calculated nic for 10.44.4.28: eth0
> Oct 22 23:20:03 clusterpaghe01 cib: [3747]: info: write_cib_contents: Wrote version 0.79.1586 of the CIB to disk (digest: 59654cc89cf183c1576a9b8e988470a5)
> Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3761]: INFO: Using calculated netmask for 10.44.4.28: 255.255.255.0
> Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3782]: INFO: eval /sbin/ifconfig eth0:0 10.44.4.28 netmask 255.255.252.0 broadcast 10.44.7.255
> Oct 22 23:20:03 clusterpaghe01 IPaddr[3748]: [3787]: DEBUG: Sending Gratuitous Arp for 10.44.4.28 on eth0:0 [eth0]
> Oct 22 23:20:03 clusterpaghe01 haclient: on_event: from message queue: evt:cib_changed
> Oct 22 23:20:04 clusterpaghe01 crmd: [3317]: info: process_lrm_event: LRM operation resource_Virtual_IP_start_0 (call=5, rc=0) complete
> Oct 22 23:20:04 clusterpaghe01 lrmd: [3314]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms) before being called (GSource: 0x805af90)
> Oct 22 23:20:04 clusterpaghe01 lrmd: [3314]: info: G_SIG_dispatch: started at 1718148810 should have started at 1718148710
> Oct 22 23:20:04 clusterpaghe01 crmd: [3317]: info: do_state_transition: clusterpaghe01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 23:20:04 clusterpaghe01 cib: [3313]: info: cib_diff_notify: Update (client: 3317, call:48): 0.79.1586 -> 0.79.1587 (ok)
> Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: te_update_diff: Processing diff (cib_update): 0.79.1586 -> 0.79.1587
> Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: match_graph_event: Action resource_Virtual_IP_start_0 (3) confirmed on 8b658843-7f87-4a86-a398-e996f92fa12b
> Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: run_graph: Transition 7: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
> Oct 22 23:20:04 clusterpaghe01 tengine: [3397]: info: notify_crmd: Transition 7 status: te_complete - <null>
> Oct 22 23:20:04 clusterpaghe01 cib: [3801]: info: write_cib_contents: Wrote version 0.79.1587 of the CIB to disk (digest: fd42bd74a0057307571b0f8814b63e7f)
> Oct 22 23:20:04 clusterpaghe01 haclient: on_event: from message queue: evt:cib_changed
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: no mbr_track info
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: no mbr_track info
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
> Oct 22 23:20:06 clusterpaghe01 cib: [3313]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail:    CURRENT: clusterpaghe02 [nodeid=1, born=1]
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail:    CURRENT: clusterpaghe01 [nodeid=0, born=2]
> Oct 22 23:20:06 clusterpaghe01 crmd: [3317]: info: ccm_event_detail:    NEW:     clusterpaghe02 [nodeid=1, born=1]
> 
> clusterpaghe01:~ #
> 

> Oct 22 22:31:22 clusterpaghe02 gconfd (root-3430): Resolved address "xml:readonly:/etc/opt/gnome/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
> Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: do_node_walk: Requesting the list of configured nodes
> Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: send_update: 2 active ping nodes
> Oct 22 22:31:24 clusterpaghe02 pingd: [3347]: info: main: Starting pingd
> Oct 22 22:31:25 clusterpaghe02 gconfd (root-3430): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
> Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
> Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe02
> Oct 22 22:31:25 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 4: pingd=200000
> Oct 22 22:31:26 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3343, call:4): 0.79.1582 -> 0.79.1583 (ok)
> Oct 22 22:31:26 clusterpaghe02 cib: [3523]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: ad8fb22dacf5d202e98caecce65473fd)
> 
> 
> 
> 
> 
> 
> **********************network disconnected on the other node
> 
> Oct 22 22:41:11 clusterpaghe02 cib: [3340]: info: cib_stats: Processed 25 operations (2400.00us average, 0% utilization) in the last 10min
> Oct 22 22:41:15 clusterpaghe02 heartbeat: [3190]: WARN: node clusterpaghe01: is dead
> Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
> Oct 22 22:41:15 clusterpaghe02 heartbeat: [3190]: info: Link clusterpaghe01:eth0 dead.
> Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
> Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
> Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
> Oct 22 22:41:15 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [dead]
> Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe01 now has status [dead]
> Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
> Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
> Oct 22 22:41:15 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 16): 0.79.1583 (ok)
> Oct 22 22:41:15 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum lost after event=NOT PRIMARY (id=2)
> Oct 22 22:41:15 clusterpaghe02 haclient: on_event:evt:cib_changed
> Oct 22 22:41:15 clusterpaghe02 cib: [3633]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 323e31c76c14e627275db776cf056917)
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: WARN: check_dead_member: Our DC node (clusterpaghe01) left the cluster
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    CURRENT: clusterpaghe02 [nodeid=1, born=3]
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    LOST:    clusterpaghe01 [nodeid=0, born=1]
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to <null> (<null>)
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:41:28 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
> Oct 22 22:41:28 clusterpaghe02 haclient: on_event:evt:cib_changed
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: LOST: clusterpaghe01
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 17): 0.79.1583 (ok)
> Oct 22 22:41:28 clusterpaghe02 cib: [3634]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 027540dbd4bf36ce353c1462a021d33d)
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_election_count_vote: Updated voted hash for clusterpaghe02 to vote
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_election_count_vote: Election ignore: our vote (clusterpaghe02)
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: start_subsystem: Starting sub-system "tengine"
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: start_subsystem: Starting sub-system "pengine"
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_dc_takeover: Taking over DC status for this partition
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_process_readwrite: We are now in R/W mode
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to <null> (<null>)
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
> Oct 22 22:41:28 clusterpaghe02 pengine: [3636]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Oct 22 22:41:28 clusterpaghe02 pengine: [3636]: info: pe_init: Starting pengine
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: G_main_add_TriggerHandler: Added signal manual handler
> Oct 22 22:41:28 clusterpaghe02 cib: [3637]: info: write_cib_contents: Wrote version 0.79.1583 of the CIB to disk (digest: 027540dbd4bf36ce353c1462a021d33d)
> Oct 22 22:41:28 clusterpaghe02 cib: [3340]: info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: te_init: Registering TE UUID: a43171f3-ea7a-4759-ad3b-554f3ccad59f
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: set_graph_functions: Setting custom graph functions
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
> Oct 22 22:41:28 clusterpaghe02 tengine: [3635]: info: te_init: Starting tengine
> Oct 22 22:41:28 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to clusterpaghe02 (1.0.8)
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_local_callback: Sending full refresh
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
> Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: sync_our_cib: Syncing CIB to all peers
> Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_timer_callback: Sending flush op to all hosts for: pingd
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: update_attrd: Connecting to attrd...
> Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:23): 0.79.1583 -> 0.79.1584 (ok)
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.79.1583 -> 0.79.1584
> Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:24): 0.79.1584 -> 0.80.1585 (ok)
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_bump): 0.79.1584 -> 0.80.1585
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: update_abort_priority: Abort priority upgraded to 1000000
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: update_abort_priority: 'DC Takeover' abort superceeded
> Oct 22 22:41:29 clusterpaghe02 cib: [3638]: info: write_cib_contents: Wrote version 0.80.1585 of the CIB to disk (digest: 6c47a6cb394c3c8eb965cd3ab7752d71)
> Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe02
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: update_dc: Set DC to clusterpaghe02 (1.0.8)
> Oct 22 22:41:29 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 6: pingd=200000
> Oct 22 22:41:29 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
> Oct 22 22:41:29 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
> Oct 22 22:41:29 clusterpaghe02 cib: [3639]: info: write_cib_contents: Wrote version 0.80.1585 of the CIB to disk (digest: 6c47a6cb394c3c8eb965cd3ab7752d71)
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_dc_join_ack: join-1: Updating node state to member for clusterpaghe02)
> Oct 22 22:41:29 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:26): 0.80.1585 -> 0.80.1586 (ok)
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1585 -> 0.80.1586
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: process_graph_event: Action resource_Virtual_IP_monitor_0 initiated by a different transitioner
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" ignore_dtd="false" num_peers="2" cib_feature_revision="1.3" generated="true" ccm_transition="3" dc_uuid="84dcf392-c5d9-44c1-a923-e436ed5af73e" epoch="80" num_updates="1586"/>
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
> Oct 22 22:41:29 clusterpaghe02 cib: [3640]: info: write_cib_contents: Wrote version 0.80.1586 of the CIB to disk (digest: 7eb6b9a59f21650623d53050a1f31e3b)
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: determine_online_status: Node clusterpaghe02 is online
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: native_print: resource_Virtual_IP (heartbeat::ocf:IPaddr):        Stopped
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: notice: StartRsc:  clusterpaghe02       Start resource_Virtual_IP
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 22:41:29 clusterpaghe02 crmd: [3344]: info: do_lrm_rsc_op: Performing op=resource_Virtual_IP_start_0 key=3:0:a43171f3-ea7a-4759-ad3b-554f3ccad59f)
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: unpack_graph: Unpacked transition 0: 1 actions in 1 synapses
> Oct 22 22:41:29 clusterpaghe02 tengine: [3635]: info: send_rsc_command: Initiating action 3: resource_Virtual_IP_start_0 on clusterpaghe02
> Oct 22 22:41:29 clusterpaghe02 pengine: [3636]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-91.bz2
> Oct 22 22:41:29 clusterpaghe02 lrmd: [3341]: info: RA output: (resource_Virtual_IP:start:stderr) Rewrote octal netmask as: 24
> Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3650]: INFO: Using calculated nic for 10.44.4.28: eth0
> Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3655]: INFO: Using calculated netmask for 10.44.4.28: 255.255.255.0
> Oct 22 22:41:29 clusterpaghe02 IPaddr[3641]: [3676]: INFO: eval /sbin/ifconfig eth0:0 10.44.4.28 netmask 255.255.252.0 broadcast 10.44.7.255
> Oct 22 22:41:30 clusterpaghe02 IPaddr[3641]: [3681]: DEBUG: Sending Gratuitous Arp for 10.44.4.28 on eth0:0 [eth0]
> Oct 22 22:41:30 clusterpaghe02 crmd: [3344]: info: process_lrm_event: LRM operation resource_Virtual_IP_start_0 (call=3, rc=0) complete
> Oct 22 22:41:30 clusterpaghe02 haclient: on_event: from message queue: evt:cib_changed
> Oct 22 22:41:30 clusterpaghe02 kernel: send_arp uses obsolete (PF_INET,SOCK_PACKET)
> Oct 22 22:41:30 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Update (client: 3344, call:29): 0.80.1586 -> 0.80.1587 (ok)
> Oct 22 22:41:30 clusterpaghe02 haclient: on_event:evt:cib_changed
> Oct 22 22:41:30 clusterpaghe02 crmd: [3344]: info: do_state_transition: clusterpaghe02: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
> Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1586 -> 0.80.1587
> Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: match_graph_event: Action resource_Virtual_IP_start_0 (3) confirmed on 84dcf392-c5d9-44c1-a923-e436ed5af73e
> Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: run_graph: Transition 0: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
> Oct 22 22:41:30 clusterpaghe02 tengine: [3635]: info: notify_crmd: Transition 0 status: te_complete - <null>
> Oct 22 22:41:30 clusterpaghe02 kernel: NET: Registered protocol family 17
> Oct 22 22:41:30 clusterpaghe02 cib: [3705]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: e8ea66f562d7aeb0a0eaeff3d0525b6a)
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ************Network reconnected
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: CRIT: Cluster node clusterpaghe01 returning after partition.
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: WARN: Deadtime value may be too small.
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: See FAQ for information on tuning deadtime.
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: URL: http://linux-ha.org/FAQ#heavy_load
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: Link clusterpaghe01:eth0 up.
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: WARN: Late heartbeat: Node clusterpaghe01: interval 172120 ms
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: Status update for node clusterpaghe01: status active
> Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_lstatus_callback: Status update: Ping node clusterpaghe01 now has status [up]
> Oct 22 22:43:38 clusterpaghe02 crmd: [3344]: notice: crmd_ha_status_callback: Status update: Node clusterpaghe01 now has status [active]
> Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [up]
> Oct 22 22:43:38 clusterpaghe02 haclient: on_event:evt:cib_changed
> Oct 22 22:43:38 clusterpaghe02 pingd: [3347]: notice: pingd_nstatus_callback: Status update: Ping node clusterpaghe01 now has status [active]
> Oct 22 22:43:38 clusterpaghe02 cib: [3340]: info: cib_diff_notify: Local-only Change (client:3344, call: 30): 0.80.1587 (ok)
> Oct 22 22:43:38 clusterpaghe02 tengine: [3635]: info: te_update_diff: Processing diff (cib_update): 0.80.1587 -> 0.80.1587
> Oct 22 22:43:38 clusterpaghe02 cib: [3706]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: 1ec8a6bf75fd6006aba055083aa85b4e)
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: info: all clients are now paused
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=863
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=864
> Oct 22 22:43:38 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=865
> Oct 22 22:43:39 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=866
> Oct 22 22:43:40 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=867
> Oct 22 22:43:41 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:42 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: flush message from clusterpaghe01
> Oct 22 22:43:42 clusterpaghe02 attrd: [3343]: info: attrd_ha_callback: Sent update 8: pingd=200000
> Oct 22 22:43:42 clusterpaghe02 cib: [3707]: info: write_cib_contents: Wrote version 0.80.1587 of the CIB to disk (digest: b197b4491faf083594aac256c6cb11ed)
> Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=868
> Oct 22 22:43:42 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:42 clusterpaghe02 cib: [3340]: WARN: cib_peer_callback: Discarding cib_apply_diff message (3b3) from clusterpaghe01: not in our membership
> Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug: hist->ackseq =762
> Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug: hist->lowseq =761, hist->hiseq=869
> Oct 22 22:43:43 clusterpaghe02 heartbeat: [3190]: debug:
> Oct 22 22:43:44 clusterpaghe02 heartbeat: [3190]: info: all clients are now resumed
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: WARN: cib_peer_callback: Discarding cib_apply_diff message (3b5) from clusterpaghe01: not in our membership
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: quorum plugin: twonodes
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=1, member_quorum_votes=100
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:43:44 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
> Oct 22 22:43:44 clusterpaghe02 ccm: [3339]: info: Break tie for 2 nodes cluster
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=1)
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=1, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3
> Oct 22 22:43:44 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    CURRENT: clusterpaghe02 [nodeid=1, born=1]
> Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: quorum plugin: majority
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: cluster:linux-ha, member_count=2, member_quorum_votes=200
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: no mbr_track info
> Oct 22 22:43:45 clusterpaghe02 ccm: [3339]: debug: total_node_count=2, total_quorum_votes=200
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe02
> Oct 22 22:43:45 clusterpaghe02 cib: [3340]: info: cib_ccm_msg_callback: PEER: clusterpaghe01
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: no mbr_track info
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    CURRENT: clusterpaghe02 [nodeid=1, born=1]
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    CURRENT: clusterpaghe01 [nodeid=0, born=2]
> Oct 22 22:43:45 clusterpaghe02 crmd: [3344]: info: ccm_event_detail:    NEW:     clusterpaghe01 [nodeid=0, born=2]
> 
> clusterpaghe02:~ #
> 

> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems



More information about the Linux-HA mailing list