[Linux-HA] heartbeat 2.0.2 problem on two nodes
Alberto
xagonzalezm at gmail.com
Tue Oct 11 09:11:54 MDT 2005
If I dont use CRM/CIB and use the old style heartbeat 1.x config I get the
same!! both nodes end up having the resources up. with this config on both
nodes. This worked ok on version 1.2.3
[root at node1 heartbeat]# cat /etc/ha.d/haresources
node1 IEL \
IPaddr::10.64.110.70/24/eth0 <http://10.64.110.70/24/eth0>
[root at node1 heartbeat]# cat /etc/ha.d/ha.cf
logfacility daemon
node node1 node2
keepalive 2
deadtime 10
ucast eth0 10.64.110.32 <http://10.64.110.32>
ping 10.64.110.254 <http://10.64.110.254>
auto_failback no
respawn hacluster /usr/lib/heartbeat/ipfail
Oct 11 17:04:38 node2 modprobe: modprobe: Can't locate module char-major-203
Oct 11 17:04:38 node2 last message repeated 3 times
Oct 11 17:04:42 node2 iel:arranque_servicio: Arrancada aplicacion iel
Oct 11 17:04:47 node2 iel:arranque_servicio: Arrancado Web iel
Oct 11 17:04:47 node2 mach_down[7253]: info: /usr/lib/heartbeat/mach_down:
nice_failback: foreign resources acquired
Oct 11 17:04:47 node2 heartbeat: [6700]: info: mach_down takeover complete.
Oct 11 17:04:47 node2 mach_down[7253]: info: mach_down takeover complete for
node node1.
Oct 11 17:05:08 node2 heartbeat: [6700]: info: Heartbeat restart on node
node1
Oct 11 17:05:08 node2 heartbeat: [6700]: info: Link node1:eth0 up.
Oct 11 17:05:08 node2 heartbeat: [6700]: info: Status update for node node1:
status init
Oct 11 17:05:08 node2 ipfail: [6709]: info: Link Status update: Link
node1/eth0 now has status up
Oct 11 17:05:08 node2 ipfail: [6709]: info: Status update: Node node1 now
has status init
Oct 11 17:05:08 node2 heartbeat: [6700]: info: Status update for node node1:
status up
Oct 11 17:05:08 node2 ipfail: [6709]: info: Status update: Node node1 now
has status up
Oct 11 17:05:08 node2 harc[7659]: info: Running /etc/ha.d/rc.d/status status
Oct 11 17:05:08 node2 harc[7669]: info: Running /etc/ha.d/rc.d/status status
Oct 11 17:05:27 node2 heartbeat: [6700]: WARN: 1 lost packet(s) for [node1]
[14:16]
Oct 11 17:05:27 node2 heartbeat: [6700]: info: Status update for node node1:
status active
Oct 11 17:05:27 node2 ipfail: [6709]: info: Status update: Node node1 now
has status active
Oct 11 17:05:27 node2 heartbeat: [6700]: info: No pkts missing from node1!
Oct 11 17:05:27 node2 harc[7681]: info: Running /etc/ha.d/rc.d/status status
Oct 11 17:05:27 node2 heartbeat: [6700]: ERROR: Both machines own our
resources!
Oct 11 17:05:27 node2 ipfail: [6709]: info: Asking other side for ping node
count.
Oct 11 17:05:31 node2 heartbeat: [6700]: info: remote resource transition
completed.
Oct 11 17:05:31 node2 heartbeat: [6700]: ERROR: Both machines own our
resources!
Oct 11 17:05:31 node2 heartbeat: [6700]: ERROR: Both machines own foreign
resources!
Oct 11 17:05:31 node2 heartbeat: [6700]: ERROR: Both machines own our
resources!
Oct 11 17:05:31 node2 heartbeat: [6700]: ERROR: Both machines own foreign
resources!
Oct 11 17:05:37 node2 heartbeat: [6700]: ERROR: Both machines own our
resources!
Oct 11 17:05:37 node2 heartbeat: [6700]: ERROR: Both machines own foreign
resources!
On 10/11/05, Alberto <xagonzalezm at gmail.com> wrote:
>
> Both nodes have the same CIB I posted before. I start heartbeat on node 1
> and I get resource eth0:0 up, then start node 2 and stop node1, node 2
> starts and get resource ipaddr up, then I start heartbeat again in node 1
> and it starts resource again and node 2 keeps resource up too, having both
> nodes the resource up and running.
>
> node1
> --------
>
> Oct 11 16:31:19 node1 logd: [26212]: info: logd_term_write_action:
> received SIGTERM
> Oct 11 16:31:19 node1 logd: [26212]: info: ha_logd: Exiting write process
> Oct 11 16:31:20 node1 logd: [26437]: info: Pid 26211 exited
> Oct 11 16:32:01 node1 logd: [26459]: info: logd started with default
> configuration.
> Oct 11 16:32:01 node1 logd: [26460]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:01 node1 logd: [26459]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:01 node1 heartbeat: [26481]: info: Enabling logging daemon
> Oct 11 16:32:01 node1 heartbeat: [26481]: info: logfile and debug file are
> those specifiedin logd config file (default /etc/logd.cf)
> Oct 11 16:32:01 node1 heartbeat: [26481]: info: AUTH: i=1: key =
> 0x89e5cf0, auth=0x2d9658, authname=sha1
> Oct 11 16:32:02 node1 heartbeat: [26481]: info: **************************
> Oct 11 16:32:02 node1 heartbeat: [26481]: info: Configuration validated.
> Starting heartbeat 2.0.2
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: heartbeat: version 2.0.2
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: Heartbeat generation: 77
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: Removing
> /var/run/heartbeat/rsctmp failed, recreating.
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: glib: ucast: write socket
> priority set to IPTOS_LOWDELAY on eth0
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: glib: ucast: bound send
> socket to device: eth0
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: glib: ucast: bound receive
> socket to device: eth0
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: glib: ucast: started on
> port 694 interface eth0 to 10.64.110.32 <http://10.64.110.32>
> Oct 11 16:32:02 node1 heartbeat: [26482]: info: glib: ping heartbeat
> started.
> Oct 11 16:32:03 node1 heartbeat: [26482]: info: G_main_add_SignalHandler:
> Added signal handler for signal 17
> Oct 11 16:32:03 node1 heartbeat: [26482]: info: pid 26482 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26482]: info: Local status now set to:
> 'up'
> Oct 11 16:32:03 node1 heartbeat: [26489]: info: pid 26489 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26490]: info: pid 26490 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26493]: info: pid 26493 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26492]: info: pid 26492 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26491]: info: pid 26491 locked in
> memory.
> Oct 11 16:32:03 node1 heartbeat: [26482]: info: Link 10.64.110.254:10<http://10.64.110.254:10>.64.110.254
> up.
> Oct 11 16:32:03 node1 heartbeat: [26482]: info: Status update for node
> 10.64.110.254 <http://10.64.110.254>: status ping
> Oct 11 16:32:22 node1 heartbeat: [26482]: WARN: node node2: is dead
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Local status now set to:
> 'active'
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Starting child client
> "/usr/lib/heartbeat/ccm" (90,90)
> Oct 11 16:32:22 node1 heartbeat: [26495]: info: Starting
> "/usr/lib/heartbeat/ccm" as uid 90 gid 90 (pid 26495)
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Starting child client
> "/usr/lib/heartbeat/cib" (90,90)
> Oct 11 16:32:22 node1 heartbeat: [26496]: info: Starting
> "/usr/lib/heartbeat/cib" as uid 90 gid 90 (pid 26496)
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Starting child client
> "/usr/lib/heartbeat/stonithd" (0,0)
> Oct 11 16:32:22 node1 heartbeat: [26497]: info: Starting
> "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 26497)
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Starting child client
> "/usr/lib/heartbeat/lrmd" (0,0)
> Oct 11 16:32:22 node1 heartbeat: [26498]: info: Starting
> "/usr/lib/heartbeat/lrmd" as uid 0 gid 0 (pid 26498)
> Oct 11 16:32:22 node1 heartbeat: [26482]: info: Starting child client
> "/usr/lib/heartbeat/crmd" (90,90)
> Oct 11 16:32:22 node1 heartbeat: [26499]: info: Starting
> "/usr/lib/heartbeat/crmd" as uid 90 gid 90 (pid 26499)
> Oct 11 16:32:22 node1 ccm: [26495]: info: Enable using logging daemon
> Oct 11 16:32:22 node1 ccm: [26495]: info: PID=26495
> Oct 11 16:32:22 node1 ccm: [26495]: info: Signing in with Heartbeat
> Oct 11 16:32:23 node1 ccm: [26495]: info: Hostname: node1
> Oct 11 16:32:23 node1 ccm: [26495]: info: total node number is 2
> Oct 11 16:32:23 node1 ccm: [26495]: info: node 0 =node1, status=active
> Oct 11 16:32:23 node1 ccm: [26495]: info: node 1 =node2, status=dead
> Oct 11 16:32:23 node1 ccm: [26495]: info: change state from CCM_STATE_NONE
> to CCM_STATE_NONE, current leader is none
> Oct 11 16:32:23 node1 ccm: [26495]: info: change state from CCM_STATE_NONE
> to CCM_STATE_JOINED, current leader is node1
> Oct 11 16:32:23 node1 ccm: [26495]: info: Counting nodes(dead nodes are
> not shown):
> Oct 11 16:32:23 node1 ccm: [26495]: info: node=node1 status=active
> Oct 11 16:32:23 node1 ccm: [26495]: info: n_member=1, nodecount=2,
> inactive_count=0
> Oct 11 16:32:23 node1 ccm: [26495]: info: Asserting quorum for two node
> cluster!
> Oct 11 16:32:23 node1 ccm: [26495]: info: delivering new membership to 0
> clients:
> Oct 11 16:32:23 node1 ccm: [26495]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:23 node1 stonithd: [26497]: info: Enable using logging daemon
> Oct 11 16:32:23 node1 stonithd: [26497]: info: G_main_add_SignalHandler:
> Added signal handler for signal 10
> Oct 11 16:32:23 node1 stonithd: [26497]: info: G_main_add_SignalHandler:
> Added signal handler for signal 12
> Oct 11 16:32:23 node1 stonithd: [26497]: info: pid 26497 locked in memory.
> Oct 11 16:32:24 node1 stonithd: [26497]: info: Signing in with heartbeat.
> Oct 11 16:32:24 node1 stonithd: [26497]: notice:
> /usr/lib/heartbeat/stonithd start up successfully.
> Oct 11 16:32:24 node1 stonithd: [26497]: info: G_main_add_SignalHandler:
> Added signal handler for signal 17
> Oct 11 16:32:24 node1 lrmd: [26498]: info: Enable using logging daemon
> Oct 11 16:32:24 node1 lrmd: [26498]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:24 node1 lrmd: [26498]: info: G_main_add_SignalHandler: Added
> signal handler for signal 17
> Oct 11 16:32:24 node1 lrmd: [26498]: info: G_main_add_SignalHandler: Added
> signal handler for signal 10
> Oct 11 16:32:24 node1 lrmd: [26498]: info: G_main_add_SignalHandler: Added
> signal handler for signal 12
> Oct 11 16:32:24 node1 lrmd: [26498]: info: Started.
> Oct 11 16:32:24 node1 cib: [26496]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:24 node1 cib: [26496]: info: mask(main.c:cib_register_ha):
> Signing in with Heartbeat
> Oct 11 16:32:25 node1 cib: [26496]: info: mask(main.c:cib_register_ha):
> FSA Hostname: node1
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <node uname="node1" type="member"
> id="0d31dc26-cd56-4b92-8f24-7bb3eea98fae"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> 0d31dc26-cd56-4b92-8f24-7bb3eea98fae
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <node uname="node2" type="member"
> id="c56c020e-16c6-4bdd-b26a-b39596291e3d"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> c56c020e-16c6-4bdd-b26a-b39596291e3d
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <nvpair name="ip" value="10.64.110.70 <http://10.64.110.70>"
> id="238d9132-f07d-4b5f-96a5-08f74efe98c5"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> 238d9132-f07d-4b5f-96a5-08f74efe98c5
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <nvpair name="netmask" value="24"
> id="c7641cea-fa13-4690-a121-6d86772e47e0"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> c7641cea-fa13-4690-a121-6d86772e47e0
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <nvpair name="nic" value="eth0"
> id="b50e2ff4-24b7-4d9b-8d4a-4762906d770d"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> b50e2ff4-24b7-4d9b-8d4a-4762906d770d
> Oct 11 16:32:25 node1 cib: [26496]: WARN: mask(assign_uuid): Updated
> object <expression attribute="#uname" operation="eq" value="node1"
> id="3c7840f5-456c-4dd3-b492-606a233c3f58"/>
> Oct 11 16:32:25 node1 cib: [26496]: ERROR: mask(xml.c:do_id_check): Object
> with attributes but no ID field detected. Assigned:
> 3c7840f5-456c-4dd3-b492-606a233c3f58
> Oct 11 16:32:25 node1 cib: [26496]: notice: mask(io.c:initializeCib):
> Disabling CIB disk writes
> Oct 11 16:32:25 node1 cib: [26496]: info: mask(main.c:startCib): CIB
> Initialization completed successfully
> Oct 11 16:32:25 node1 cib: [26496]: info: mask(main.c:init_start):
> Starting cib mainloop
> Oct 11 16:32:25 node1 crmd: [26499]: info: mask(main.c:init_start):
> Starting crmd
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(control.c:register_with_ha):
> FSA Hostname: node1
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(control.c:do_startup):
> Register Signal Handler
> Oct 11 16:32:26 node1 crmd: [26499]: info: G_main_add_SignalHandler: Added
> signal handler for signal 15
> Oct 11 16:32:26 node1 crmd: [26499]: info: G_main_add_TriggerHandler:
> Added signal manual handler
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(control.c:do_startup):
> Init server comms
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(control.c:do_startup):
> Creating CIB object
> Oct 11 16:32:26 node1 crmd: [26499]: info: G_main_add_SignalHandler: Added
> signal handler for signal 17
> Oct 11 16:32:26 node1 crmd: [26499]: info:
> mask(cib_native.c:cib_native_signon): Connection to CIB successful
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(ccm.c:do_ccm_control): CCM
> Activation passed... all set to go!
> Oct 11 16:32:26 node1 crmd: [26499]: info: mask(control.c:do_started):
> Delaying start, CCM (0000000000100000) not connected
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(main.c:init_start):
> Starting crmd's mainloop
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(control.c:do_started):
> Delaying start, CCM (0000000000100000) not connected
> Oct 11 16:32:27 node1 ccm: [26495]: info: ccm from node1 started
> Oct 11 16:32:27 node1 cib: [26496]: info: mem_handle_event: Got an event
> OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 11 16:32:27 node1 cib: [26496]: info: mem_handle_event: instance=1,
> nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
> Oct 11 16:32:27 node1 crmd: [26499]: info: mem_handle_event: Got an event
> OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 11 16:32:27 node1 cib: [26496]: info: mask(
> callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW MEMBERSHIP (id=1)
> Oct 11 16:32:27 node1 crmd: [26499]: info: mem_handle_event: instance=1,
> nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(
> callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after event=NEW
> MEMBERSHIP (id=1)
> Oct 11 16:32:27 node1 cib: [26496]: info: mask(
> callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after event=NEW
> MEMBERSHIP (id=1)
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(ccm.c:ccm_event_detail):
> NEW MEMBERSHIP: trans=1, nodes=1, new=1, lost=0 n_idx=0, new_idx=0,
> old_idx=3
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(ccm.c:ccm_event_detail):
> NEW: node1 [nodeid=0, born=1]
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(control.c:do_started): The
> local CRM is operational
> Oct 11 16:32:27 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_STARTING -> S_PENDING [ input=I_PENDING
> cause=C_CCM_CALLBACK origin=do_started ]
> Oct 11 16:32:47 node1 crmd: [26499]: info: mask(utils.c:crm_timer_popped):
> Election Trigger (I_DC_TIMEOUT) just popped!
> Oct 11 16:32:47 node1 crmd: [26499]: WARN: mask(misc.c:do_log): [[FSA]]
> Input I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING)
> Oct 11 16:32:47 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Oct 11 16:32:59 node1 crmd: [26499]: info: mask(utils.c:crm_timer_popped):
> Election Timeout (I_ELECTION_DC) just popped!
> Oct 11 16:32:59 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Oct 11 16:32:59 node1 crmd: [26499]: info: mask(
> subsystems.c:start_subsystem): Starting sub-system "tengine"
> Oct 11 16:32:59 node1 crmd: [26499]: info: mask(
> subsystems.c:start_subsystem): Starting sub-system "pengine"
> Oct 11 16:32:59 node1 cib: [26496]: info: mask(
> messages.c:cib_process_readwrite): We are now in R/W mode
> Oct 11 16:32:59 node1 tengine: [26511]: info: G_main_add_SignalHandler:
> Added signal handler for signal 15
> Oct 11 16:32:59 node1 crmd: [26499]: info: mask(election.c:do_dc_takeover):
> Taking over DC status for this partition
> Oct 11 16:33:00 node1 crmd: [26499]: info:
> mask(join_dc.c:do_dc_join_offer_all): 0) Offering membership to 1 clients
> Oct 11 16:33:00 node1 crmd: [26499]: notice: mask(
> callbacks.c:crmd_client_status_callback): Status update: Client node1/crmd
> now has status [online]
> Oct 11 16:33:00 node1 tengine: [26511]: info:
> mask(cib_native.c:cib_native_signon): Connection to CIB successful
> Oct 11 16:33:00 node1 cib: [26496]: info: mask(
> callbacks.c:cib_null_callback): Setting cib_diff_notify callbacks for
> tengine: on
> Oct 11 16:33:00 node1 tengine: [26511]: info: mask(main.c:init_start):
> Starting tengine
> Oct 11 16:33:00 node1 tengine: [26511]: info: mask(
> tengine.c:initialize_graph): Registering TE UUID:
> 26274c92-5444-4ce9-a1f9-3b5e9f78cbbd
> Oct 11 16:33:00 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 11 16:33:00 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes responded to the join offer.
> Oct 11 16:33:01 node1 crmd: [26499]: info:
> mask(join_dc.c:process_join_ack_msg): 4) Updating node state to member for
> node1
> Oct 11 16:33:01 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 1 - Transition status: Triggered by CIB update: Non-status change
> Oct 11 16:33:01 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 11 16:33:01 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:33:01 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 1 - Transition status: Stopped: te_abort_confirmed
> Oct 11 16:33:01 node1 crmd: [26499]: info: mask(pengine.c:do_pe_invoke):
> Waiting for the PE to connect
> Oct 11 16:33:02 node1 crmd: [26499]: info: mask(pengine.c:do_pe_invoke):
> Waiting for the PE to connect
> Oct 11 16:33:02 node1 pengine: [26512]: info: G_main_add_SignalHandler:
> Added signal handler for signal 15
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(main.c:init_start):
> Starting pengine
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="5" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="1" origin="node1" ccm_transition="1"
> dc_uuid="6e892e64-3ad0-4111-8a2e-0b1a6928c8aa"
> debug_source="finalize_join"/>
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(
> native.c:native_create_actions): Start resource IEL:IEL_IPaddr_70 (node1)
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node1)
> Oct 11 16:33:02 node1 pengine: [26512]: info: mask(stages.c:stage8):
> Creating transition graph 0.
> Oct 11 16:33:02 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:33:03 node1 tengine: [26511]: info: mask(unpack.c:unpack_graph):
> Beginning transition 0 : timeout set to 120000ms
> Oct 11 16:33:03 node1 tengine: [26511]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:33:03 node1 tengine: [26511]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:33:03 node1 tengine: [26511]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (3): start on (null)
> Oct 11 16:33:03 node1 heartbeat: [26482]: WARN: Performed 1 more
> non-realtime malloc calls.
> Oct 11 16:33:03 node1 heartbeat: [26482]: info: Total non-realtime malloc
> bytes: 135168
> Oct 11 16:33:03 node1 tengine: [26511]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node1
> Oct 11 16:33:03 node1 crmd: [26499]: WARN: lrm_get_rsc(653): got a return
> code HA_FAIL from a reply message of getrsc with function get_ret_from_msg.
> Oct 11 16:33:03 node1 crmd: [26499]: WARN: lrm_get_rsc(653): got a return
> code HA_FAIL from a reply message of getrsc with function get_ret_from_msg.
> Oct 11 16:33:03 node1 crmd: [26499]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:33:03 node1 IPaddr[26513]: [26553]: INFO: /sbin/ifconfig eth0:0
> 10.64.110.70 <http://10.64.110.70> netmask 255.255.255.0z
> Oct 11 16:33:03 node1 IPaddr[26513]: [26558]: INFO: Sending Gratuitous Arp
> for 10.64.110.70 <http://10.64.110.70> on eth0:0 [eth0]�
> Oct 11 16:33:03 node1 IPaddr[26513]: [26559]: INFO:
> /usr/lib/heartbeat/send_arp -i 500 -r 10 -p
> /var/run/heartbeat/rsctmp/send_arp/send_arp-10.64.110.70<http://10.64.110.70>eth0
> 10.64.110.70 <http://10.64.110.70> auto 10.64.110.70 <http://10.64.110.70>ffffffffffffX
> Oct 11 16:33:03 node1 send_arp: [26562]: info: Enable using logging daemon
> Oct 11 16:33:04 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> Starting abort timer: 120000ms
> Oct 11 16:33:04 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 0 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:33:04 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 0 - Delay abort until 0 updates and 1 actions complete (state=2).
> Oct 11 16:34:59 node1 tengine: [26511]: WARN: mask(utils.c:timer_callback):
> Timer popped in state=2
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:timer_callback):
> Timer popped in state=2
> Oct 11 16:35:00 node1 tengine: [26511]: ERROR: mask(utils.c:timer_callback):
> Transition abort timeout reached... marking transition complete.
> Oct 11 16:35:00 node1 tengine: [26511]: ERROR: mask(utils.c:send_complete):
> 0 - Transition status: Abort timed out after 120000ms
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_state):
> Synapse 0 was confirmed
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_state):
> Synapse 1 is pending
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> [Action 4] Pending (cannot fail)
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> Pseudo Op: running
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <pseudo_event id="4" rsc_id="IEL" operation="running"
> operation_key="IEL_running_0" allow_fail="false">
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <attributes/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: </pseudo_event>
> Oct 11 16:35:00 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_input):
> [Input 1] Pending (rsc)
> Oct 11 16:35:00 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_input): Raw
> input: <rsc_op id="1" rsc_id="IEL:IEL_IPaddr_70" operation="start"
> operation_key="IEL:IEL_IPaddr_70_start_0" on_node="node1"
> on_node_uuid="0d31dc26-cd56-4b92-8f24-7bb3eea98fae" allow_fail="false"/>
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_state):
> Synapse 2 was executed
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> [Action 1] In-flight (cannot fail)
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> Resource Op: IEL:IEL_IPaddr_70/start on node1
> (0d31dc26-cd56-4b92-8f24-7bb3eea98fae)
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <rsc_op id="1" rsc_id="IEL:IEL_IPaddr_70" operation="start"
> operation_key="IEL:IEL_IPaddr_70_start_0" on_node="node1"
> on_node_uuid="0d31dc26-cd56-4b92-8f24-7bb3eea98fae" allow_fail="false"
> transition_key="0:26274c92-5444-4ce9-a1f9-3b5e9f78cbbd">
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <primitive class="ocf" id="IEL:IEL_IPaddr_70" provider="heartbeat"
> type="IPaddr"/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <attributes>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="ip" value="10.64.110.70 <http://10.64.110.70>"/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="netmask" value="24"/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="nic" value="eth0"/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: </attributes>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: </rsc_op>
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_state):
> Synapse 3 is pending
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> [Action 2] Pending (cannot fail)
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> Resource Op: IEL:IEL_IPaddr_70/monitor on node1
> (0d31dc26-cd56-4b92-8f24-7bb3eea98fae)
> Oct 11 16:35:00 node1 tengine: [26511]: WARN: mask(utils.c:print_action):
> timeout=5000, timer=-1
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <rsc_op id="2" rsc_id="IEL:IEL_IPaddr_70" operation="monitor"
> operation_key="IEL:IEL_IPaddr_70_monitor_5000" on_node="node1"
> on_node_uuid="0d31dc26-cd56-4b92-8f24-7bb3eea98fae" allow_fail="false">
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <primitive class="ocf" id="IEL:IEL_IPaddr_70" provider="heartbeat"
> type="IPaddr"/>
> Oct 11 16:35:00 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <attributes>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="timeout" value="5s"/>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="interval" value="5s"/>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="ip" value="10.64.110.70 <http://10.64.110.70>"/>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="netmask" value="24"/>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: <nvpair name="nic" value="eth0"/>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: </attributes>
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_action): Raw
> action: </rsc_op>
> Oct 11 16:35:01 node1 tengine: [26511]: WARN: mask(utils.c:print_input):
> [Input 1] Pending (rsc)
> Oct 11 16:35:01 node1 tengine: [26511]: info: mask(print_input): Raw
> input: <rsc_op id="1" rsc_id="IEL:IEL_IPaddr_70" operation="start"
> operation_key="IEL:IEL_IPaddr_70_start_0" on_node="node1"
> on_node_uuid="0d31dc26-cd56-4b92-8f24-7bb3eea98fae" allow_fail="false"/>
> Oct 11 16:35:01 node1 pengine: [26512]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="7" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="1" origin="node1" ccm_transition="1"
> dc_uuid="6e892e64-3ad0-4111-8a2e-0b1a6928c8aa"
> debug_source="finalize_join"/>
> Oct 11 16:35:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:02 node1 pengine: [26512]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:02 node1 pengine: [26512]: info: mask(
> native.c:native_create_actions): Stop resource IEL:IEL_IPaddr_70 (node1)
> Oct 11 16:35:02 node1 pengine: [26512]: WARN: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (<null>) (cancelled : start un-runnable)
> Oct 11 16:35:02 node1 pengine: [26512]: info: mask(stages.c:stage8):
> Creating transition graph 1.
> Oct 11 16:35:02 node1 crmd: [26499]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:02 node1 tengine: [26511]: info: mask(unpack.c:unpack_graph):
> Beginning transition 1 : timeout set to 120000ms
> Oct 11 16:35:02 node1 tengine: [26511]: info: mask(unpack.c:unpack_graph):
> Unpacked 5 actions in 5 synapses
> Oct 11 16:35:02 node1 tengine: [26511]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:35:02 node1 tengine: [26511]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (6): stop on (null)
> Oct 11 16:35:02 node1 tengine: [26511]: info: mask(
> tengine.c:cib_action_updated): Initiating action 2: stop IEL:IEL_IPaddr_70
> on node1
> Oct 11 16:35:02 node1 crmd: [26499]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op stop on IEL:IEL_IPaddr_70
> Oct 11 16:35:02 node1 IPaddr[26598]: [26616]: INFO: /sbin/route -n del
> -host 10.64.110.70\
> Oct 11 16:35:02 node1 IPaddr[26598]: [26618]: INFO: /sbin/ifconfig eth0:0
> downb
> Oct 11 16:35:02 node1 IPaddr[26598]: [26621]: INFO: IP Address
> 10.64.110.70 <http://10.64.110.70> releasedV
> Oct 11 16:35:03 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> Starting abort timer: 120000ms
> Oct 11 16:35:03 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 1 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:03 node1 tengine: [26511]: info: mask(utils.c:send_complete):
> 1 - Delay abort until 0 updates and 1 actions complete (state=2).
>
>
>
>
> node2
> --------
> Oct 11 16:34:56 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:34:57 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:34:57 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:34:57 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 18 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:34:57 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:34:57 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:34:57 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:34:57 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:34:58 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:34:58 node2 pengine: [24985]: info: mask(
> native.c:native_create_actions): Leave resource IEL:IEL_IPaddr_70 (node2)
> Oct 11 16:34:58 node2 pengine: [24985]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node2)
> Oct 11 16:34:58 node2 pengine: [24985]: info: mask(stages.c:stage8):
> Creating transition graph 23.
> Oct 11 16:34:58 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:34:58 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="53" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:34:58 node2 lrmd: [24902]: ERROR: cl_log: 107 messages were
> dropped
> Oct 11 16:34:58 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:34:59 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 19 : timeout set to 120000ms
> Oct 11 16:34:59 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:34:59 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:34:59 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:34:59 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:34:59 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (4): start on (null)
> Oct 11 16:35:00 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:00 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:01 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:01 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:01 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:01 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:01 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:01 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:01 node2 lrmd: [24902]: ERROR: cl_log: 108 messages were
> dropped
> Oct 11 16:35:02 node2 pengine: [24985]: info: mask(
> native.c:native_create_actions): Leave resource IEL:IEL_IPaddr_70 (node2)
> Oct 11 16:35:02 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:02 node2 pengine: [24985]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node2)
> Oct 11 16:35:02 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:02 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 19 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:02 node2 pengine: [24985]: info: mask(stages.c:stage8):
> Creating transition graph 24.
> Oct 11 16:35:02 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="55" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:03 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:03 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:04 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:04 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:04 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 20 : timeout set to 120000ms
> Oct 11 16:35:04 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:35:04 node2 lrmd: [24902]: ERROR: cl_log: 164 messages were
> dropped
> Oct 11 16:35:04 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:35:04 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:04 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:04 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (4): start on (null)
> Oct 11 16:35:05 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:05 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:05 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:05 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:05 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:05 node2 pengine: [24985]: info: mask(
> native.c:native_create_actions): Leave resource IEL:IEL_IPaddr_70 (node2)
> Oct 11 16:35:06 node2 pengine: [24985]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node2)
> Oct 11 16:35:06 node2 pengine: [24985]: info: mask(stages.c:stage8):
> Creating transition graph 25.
> Oct 11 16:35:06 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:06 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="57" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:06 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:06 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:07 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:07 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 20 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:07 node2 lrmd: [24902]: ERROR: cl_log: 229 messages were
> dropped
> Oct 11 16:35:07 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:08 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:08 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:08 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:08 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:09 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:09 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:09 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:09 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 21 : timeout set to 120000ms
> Oct 11 16:35:09 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:35:09 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:35:09 node2 pengine: [24985]: info: mask(
> native.c:native_create_actions): Leave resource IEL:IEL_IPaddr_70 (node2)
> Oct 11 16:35:09 node2 pengine: [24985]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node2)
> Oct 11 16:35:09 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:09 node2 pengine: [24985]: info: mask(stages.c:stage8):
> Creating transition graph 26.
> Oct 11 16:35:10 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (4): start on (null)
> Oct 11 16:35:10 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="59" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:10 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:10 node2 lrmd: [24902]: ERROR: cl_log: 39 messages were
> dropped
> Oct 11 16:35:10 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:11 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:11 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:11 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:11 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:12 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 21 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:12 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:12 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:12 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:12 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:12 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:13 node2 lrmd: [24902]: ERROR: cl_log: 164 messages were
> dropped
> Oct 11 16:35:13 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:13 node2 pengine: [24985]: info: mask(
> native.c:native_create_actions): Leave resource IEL:IEL_IPaddr_70 (node2)
> Oct 11 16:35:13 node2 pengine: [24985]: info: mask(
> native.c:create_recurring_actions): IEL:IEL_IPaddr_70_monitor_5000:
> (node2)
> Oct 11 16:35:13 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:13 node2 pengine: [24985]: info: mask(stages.c:stage8):
> Creating transition graph 27.
> Oct 11 16:35:13 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="61" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:13 node2 crmd: [24903]: ERROR: cl_log: 282 messages were
> dropped
> Oct 11 16:35:13 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:14 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:14 node2 tengine: [24984]: ERROR: cl_log: 1164 messages were
> dropped
> Oct 11 16:35:14 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:14 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:15 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 48 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:15 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:15 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:15 node2 lrmd: [24902]: ERROR: cl_log: 159 messages were
> dropped
> Oct 11 16:35:16 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:16 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:16 node2 pengine: [24985]: ERROR: cl_log: 648 messages were
> dropped
> Oct 11 16:35:16 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:16 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="103" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:16 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:16 node2 crmd: [24903]: ERROR: cl_log: 471 messages were
> dropped
> Oct 11 16:35:17 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:17 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 49 : timeout set to 120000ms
> Oct 11 16:35:17 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:35:17 node2 tengine: [24984]: ERROR: cl_log: 1110 messages were
> dropped
> Oct 11 16:35:17 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:18 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:18 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:18 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:18 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:18 node2 lrmd: [24902]: ERROR: cl_log: 99 messages were
> dropped
> Oct 11 16:35:19 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:19 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:19 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
> Oct 11 16:35:19 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:19 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:19 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 74 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:19 node2 pengine: [24985]: ERROR: cl_log: 1159 messages were
> dropped
> Oct 11 16:35:19 node2 crmd: [24903]: ERROR: cl_log: 462 messages were
> dropped
> Oct 11 16:35:19 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="177" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:20 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:20 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:20 node2 cib: [24900]: ERROR: cl_log: 115 messages were
> dropped
> Oct 11 16:35:20 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:20 node2 tengine: [24984]: ERROR: cl_log: 1433 messages were
> dropped
> Oct 11 16:35:20 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:21 node2 lrmd: [24902]: ERROR: cl_log: 158 messages were
> dropped
> Oct 11 16:35:21 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:21 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:21 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 107 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:22 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:22 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:22 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:22 node2 pengine: [24985]: ERROR: cl_log: 1032 messages were
> dropped
> Oct 11 16:35:22 node2 crmd: [24903]: ERROR: cl_log: 300 messages were
> dropped
> Oct 11 16:35:22 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="243" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:22 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:23 node2 cib: [24900]: ERROR: cl_log: 200 messages were
> dropped
> Oct 11 16:35:23 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:23 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:23 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 108 : timeout set to 120000ms
> Oct 11 16:35:23 node2 tengine: [24984]: ERROR: cl_log: 1382 messages were
> dropped
> Oct 11 16:35:24 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:24 node2 lrmd: [24902]: ERROR: cl_log: 151 messages were
> dropped
> Oct 11 16:35:24 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:25 node2 tengine: [24984]: info: mask(utils.c:send_complete):
> 139 - Transition status: Aborted by CIB update: Event not matched
> Oct 11 16:35:25 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:25 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:25 node2 pengine: [24985]: ERROR: cl_log: 1353 messages were
> dropped
> Oct 11 16:35:25 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:25 node2 crmd: [24903]: ERROR: cl_log: 451 messages were
> dropped
> Oct 11 16:35:25 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="329" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:25 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:26 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:26 node2 cib: [24900]: ERROR: cl_log: 334 messages were
> dropped
> Oct 11 16:35:26 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:26 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:26 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 140 : timeout set to 120000ms
> Oct 11 16:35:27 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:35:27 node2 lrmd: [24902]: ERROR: cl_log: 211 messages were
> dropped
> Oct 11 16:35:27 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:35:27 node2 tengine: [24984]: ERROR: cl_log: 2327 messages were
> dropped
> Oct 11 16:35:27 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:27 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Beginning transition 193 : timeout set to 120000ms
> Oct 11 16:35:28 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:28 node2 tengine: [24984]: info: mask(unpack.c:unpack_graph):
> Unpacked 4 actions in 4 synapses
> Oct 11 16:35:28 node2 pengine: [24985]: ERROR: cl_log: 300 messages were
> dropped
> Oct 11 16:35:28 node2 pengine: [24985]: info: mask(process_pe_message):
> [generation] <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> epoch="1" num_updates="349" have_quorum="true" last_written="Mon Oct 10
> 20:04:01 2005" num_peers="2" origin="node2" ccm_transition="1"
> dc_uuid="5932dc6d-50c1-4287-b00f-ff6565b6f590"
> debug_source="finalize_join"/>
> Oct 11 16:35:28 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_transition): Initating transition
> Oct 11 16:35:28 node2 crmd: [24903]: ERROR: cl_log: 437 messages were
> dropped
> Oct 11 16:35:28 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:28 node2 tengine: [24984]: info: mask(
> tengine.c:initiate_action): Executing pseudo-event (4): start on (null)
> Oct 11 16:35:28 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:29 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:29 node2 crmd: [24903]: info: mask(lrm.c:do_lrm_rsc_op):
> Performing op start on IEL:IEL_IPaddr_70
> Oct 11 16:35:29 node2 cib: [24900]: ERROR: cl_log: 116 messages were
> dropped
> Oct 11 16:35:30 node2 tengine: [24984]: info: mask(
> tengine.c:cib_action_updated): Initiating action 1: start
> IEL:IEL_IPaddr_70 on node2
> Oct 11 16:35:30 node2 lrmd: [24902]: ERROR: cl_log: 32 messages were
> dropped
> Oct 11 16:35:30 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_IPC_MESSAGE origin=do_msg_route ]
> Oct 11 16:35:30 node2 crmd: [24903]: info: mask(fsa.c:do_state_transition):
> All 1 cluster nodes are eligable to run resources.
> Oct 11 16:35:30 node2 tengine: [24984]: ERROR: cl_log: 1315 messages were
> dropped
> Oct 11 16:35:30 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> STONITH of failed nodes is disabled
> Oct 11 16:35:30 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> Cluster is symmetric - resources can run anywhere by default
> Oct 11 16:35:30 node2 pengine: [24985]: info: mask(unpack.c:unpack_config):
> On loss of CCM Quorum: Stop ALL resources
>
>
> On 10/11/05, Andrew Beekhof <beekhof at gmail.com> wrote:
> >
> > On 10/11/05, Alberto <xagonzalezm at gmail.com> wrote:
> > > Hi!
> > >
> > > I am running heartbeat 2.0.2 with this config on two nodes. I start
> > > heartbeat on node1 and it starts IPaddr resource, but then I start
> > heartbeat
> > > on node2 and stop node1 and node2 doesnt start resources, it doesnt
> > discover
> > > failure and cibdamin -Q show a different cib.xml than on node1. Why I
> > dont
> > > get two nodes synced? If I started node2 with cib.xml empty and it
> > generates
> > > an empty one. If I copy node1 cib.xml to node2.xml then I have the
> > IPaddr
> > > resource up on both nodes!
> >
> > no logs == no answer possible
> >
> > it would also be nice to the CIBs from both nodes to see the differences
> >
> >
> > >
> > > Another question is good to have suppress_cib_writes to true, is this
> > just
> > > for updating and addind uncomplete id fields to xml?
> >
> > no, it means no changes are ever written to disk
> >
> > > The problem is that the
> > > cib.xml get to confused with uids, states, etc. how can this be
> > solved?
> >
> > the combination of tag name + id must be unique. if you ensure that
> > before starting the cluster everything should be fine.
> >
> > >
> > > Thanks!!
> > >
> > > *** ha.cf <http://ha.cf>
> > >
> > > debugfile /var/log/ha/ha.debug
> > > logfile /var/log/ha/ha.log
> > > node node1
> > > node node2
> > > keepalive 2
> > > deadtime 10
> > > ucast eth0 10.64.110.32 <http://10.64.110.32>
> > > ping 10.64.110.254 <http://10.64.110.254>
> > > auto_failback no
> > > crm yes
> > > use_logd yes
> > > debug 1
> > >
> > >
> > >
> > > *** cib.xml:
> > >
> > > <cib generated="true" cib_feature_revision="1" admin_epoch="0"
> > epoch="0"
> > > num_updates="0" have_quorum="false" last_written="Mon Oct 10 20:04:01
> > 2005">
> > > <configuration>
> > > <crm_config>
> > > <nvpair id="transition_idle_timeout" name="transition_idle_timeout"
> > > value="120s"/>
> > > <nvpair id="symmetric_cluster" name="symmetric_cluster"
> > > value="true"/>
> > > <nvpair id="no_quorum_policy" name="no_quorum_policy" value="stop"/>
> > > <nvpair id="suppress_cib_writes" name="suppress_cib_writes"
> > > value="true"/>
> > > <nvpair id="stonith_enabled" name="stonith_enabled" value="false"/>
> > > <nvpair id="default_resource_stickiness"
> > > name="default_resource_stickiness" value="INFINITY"/>
> > > </crm_config>
> > > <nodes>
> > > <node uname="node1" type="member"/>
> > > <node uname="node2" type="member"/>
> > > </nodes>
> > > <resources>
> > > <group id="IEL">
> > > <primitive class="ocf" id="IEL_IPaddr_70" provider="heartbeat"
> > > type="IPaddr">
> > > <operations>
> > > <op id="1" interval="5s" name="monitor" timeout="5s"/>
> > > </operations>
> > > <instance_attributes>
> > > <attributes>
> > > <nvpair name="ip" value="10.64.110.70 <http://10.64.110.70>"/>
> > > <nvpair name="netmask" value="24"/>
> > > <nvpair name="nic" value="eth0"/>
> > > </attributes>
> > > </instance_attributes>
> > > </primitive>
> > > </group>
> > > </resources>
> > > <constraints>
> > > <rsc_location id="rsc_location_IEL" rsc="IEL">
> > > <rule id="prefered_location_IEL" score="100">
> > > <expression attribute="#uname" operation="eq"
> > > value="node1"/>
> > > </rule>
> > > </rsc_location>
> > > </constraints>
> > > </configuration>
> > > <status/>
> > > </cib>
> > >
> > >
> > > _______________________________________________
> > > Linux-HA mailing list
> > > Linux-HA at lists.linux-ha.org
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> > >
> > >
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA at lists.linux-ha.org
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.community.tummy.com/pipermail/linux-ha/attachments/20051011/7de672d3/attachment.html
More information about the Linux-HA
mailing list