[Linux-HA] Tow running Designated Coordinator

Andrew Beekhof abeekhof at suse.de
Fri Feb 10 05:30:46 MST 2006


According to this, the logs, the CCM hasn't told us the node came back.

Can you wait until Monday and see if 2.0.3 has the same behavior please?

zhenh, you might want to take a look at this too.

On Feb 10, 2006, at 1:17 PM, Ulrich Weber wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>>> However after unplugging the sync interface, both systems get DC.
>>> Waiting a few seconds and replug the sync interface both systems
>>> stay DC forever! Log says "ERROR: Cluster node ha1 returning
>>> after partition."
>>>
>>> Shouldnt one DC shutdown ?
>>
>>
>> Definitely. Logs?
>
> please finde attached the correspondig log files.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFD7IRL22t2oTuElzoRAjcOAJwLgtd/5NFQguan7fTnnJ1a7tBn/gCfVqqt
> VfyTNmjjJdX+36jptGQMJVA=
> =GXtl
> -----END PGP SIGNATURE-----
>
>
> 2006:02:10-14:09:00 (none) [local7:info] logd: [8037]: info: logd  
> started with default configuration.
> 2006:02:10-14:09:00 (none) [local7:warning] logd: [8037]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:09:00 (none) [local7:warning] logd: [8037]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:09:00 (none) [local7:info] logd: [8038]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:00 (none) [local7:info] logd: [8037]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: Logging daemon is disabled --enabling logging daemon is  
> recommended
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: File /etc/ha.d/haresources exists.
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: This file is not used because crm is enabled
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: Consider setting /proc/sys/kernel/core_uses_pid (or  
> equivalent) to 1 for maximum supportability
> 2006:02:10-14:09:00 (none) [daemon:warning] heartbeat: [8059]:  
> WARN: Logging daemon is disabled --enabling logging daemon is  
> recommended
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8059]: info:  
> **************************
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8059]: info:  
> Configuration validated. Starting heartbeat 2.0.2
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> heartbeat: version 2.0.2
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> Heartbeat generation: 14
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> Removing /var/run/heartbeat/rsctmp failed, recreating.
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> pid 8060 locked in memory.
> 2006:02:10-14:09:00 (none) [daemon:info] heartbeat: [8060]: info:  
> Local status now set to: 'up'
> 2006:02:10-14:09:01 (none) [daemon:info] heartbeat: [8065]: info:  
> pid 8065 locked in memory.
> 2006:02:10-14:09:01 (none) [daemon:info] heartbeat: [8067]: info:  
> pid 8067 locked in memory.
> 2006:02:10-14:09:01 (none) [daemon:info] heartbeat: [8066]: info:  
> pid 8066 locked in memory.
> 2006:02:10-14:09:01 (none) [daemon:info] heartbeat: [8060]: info:  
> Link ha1:eth1 up.
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Link ha2:eth1 up.
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Local status now set to: 'active'
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Starting child client "/usr/lib/heartbeat/ccm" (90,90)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Starting child client "/usr/lib/heartbeat/cib" (90,90)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Starting child client "/usr/lib/heartbeat/stonithd" (0,0)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Starting child client "/usr/lib/heartbeat/lrmd" (0,0)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Starting child client "/usr/lib/heartbeat/crmd" (90,90)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8068]: info:  
> Starting "/usr/lib/heartbeat/ccm" as uid 90  gid 90 (pid 8068)
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: Disable  
> using logging daemon
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: PID=8068
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: Signing  
> in with Heartbeat
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8068, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x814fd28 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8068, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x814a528
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8068, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x8143ba8 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info:  
> Switched to heartbeat syslog facility: 24
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info:  
> Hostname: ha1
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: total  
> node number is 2
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: node 0  
> =ha1, status=active
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: node 1  
> =ha2, status=init
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_NONE, current leader is none
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_NONE, current leader is none
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8069]: info:  
> Starting "/usr/lib/heartbeat/cib" as uid 90  gid 90 (pid 8069)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Status update for node ha2: status up
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG:  
> Dumping message with 12 fields
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [1] : [st=up]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [2] : [dt=1388]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [3] : [protocol=1]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [4] : [src=ha2]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [5] : [(1)srcuuid=0x805c468]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [6] : [seq=2]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [7] : [hg=9]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [8] : [ts=43ec9047]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [9] : [ld=0.01 0.03 0.00 3/71 8226]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [10] : [ttl=3]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [11] : [auth=1 7308a0607fa6e62954465aee8e9f203d]
> 2006:02:10-14:09:12 (none) [daemon:info] cib: [8069]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:12 (none) [daemon:info] cib: [8069]: info: mask 
> (main.c:cib_register_ha): Signing in with Heartbeat
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> Status update for node ha2: status active
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG:  
> Dumping message with 12 fields
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [1] : [st=active]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [2] : [dt=1388]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [3] : [protocol=1]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [4] : [src=ha2]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [5] : [(1)srcuuid=0x805cdf0]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [6] : [seq=5]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [7] : [hg=9]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [8] : [ts=43ec9048]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [9] : [ld=0.01 0.03 0.00 3/71 8226]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [10] : [ttl=3]
> 2006:02:10-14:09:12 (none) [daemon:info] ccm: [8068]: info: MSG 
> [11] : [auth=1 cc4ac70265e05bad37c64138ae81b90e]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8069, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x814fcf0 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8069, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x814aaa0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8069, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x8141ae8 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:info] cib: [8069]: info: mask 
> (main.c:cib_register_ha): FSA Hostname: ha1
> 2006:02:10-14:09:12 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:12 (none) [daemon:info] cib: [8069]: info: mask 
> (main.c:startCib): CIB Initialization completed successfully
> 2006:02:10-14:09:12 (none) [daemon:info] cib: [8069]: info: mask 
> (main.c:init_start): Starting cib mainloop
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8070]: info:  
> Starting "/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 8070)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8072]: info:  
> Starting "/usr/lib/heartbeat/crmd" as uid 90  gid 90 (pid 8072)
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8071]: info:  
> Starting "/usr/lib/heartbeat/lrmd" as uid 0  gid 0 (pid 8071)
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info:  
> Disable using logging daemon
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:09:12 (none) [daemon:warning] lrmd: [8071]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:09:12 (none) [daemon:warning] lrmd: [8071]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 10
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 12
> 2006:02:10-14:09:12 (none) [daemon:info] lrmd: [8071]: info: Started.
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> Disable using logging daemon
> 2006:02:10-14:09:12 (none) [daemon:warning] stonithd: [8070]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:09:12 (none) [daemon:warning] stonithd: [8070]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 10
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 12
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> pid 8070 locked in memory.
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8070, (0:0)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x8158788 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8070, (0:0)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x8157028
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8070, (0:0)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x8142ad8 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> Signing in with heartbeat.
> 2006:02:10-14:09:12 (none) [daemon:debug] stonithd: [8070]: debug:  
> Setting message filter mode
> 2006:02:10-14:09:12 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: 1 lost packet(s) for [ha2] [8:10]
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> No pkts missing from ha2!
> 2006:02:10-14:09:12 (none) [daemon:notice] stonithd: [8070]:  
> notice: /usr/lib/heartbeat/stonithd start up successfully.
> 2006:02:10-14:09:12 (none) [daemon:info] stonithd: [8070]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:09:12 (none) [daemon:warning] crmd: [8072]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:09:12 (none) [daemon:warning] crmd: [8072]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info: mask 
> (main.c:init_start): Starting crmd
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8072, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x81571b0 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8072, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x8159f10
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> SO_PEERCRED returned [8072, (90:90)]
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:12 (none) [daemon:debug] heartbeat: [8060]: debug:  
> Verifying authentication: uidptr=0x8142bd8 gidptr=0x0
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:register_with_ha): FSA Hostname: ha1
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:do_startup): Register Signal Handler
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info:  
> G_main_add_TriggerHandler: Added signal manual handler
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:do_startup): Init server comms
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:do_startup): Creating CIB object
> 2006:02:10-14:09:12 (none) [daemon:info] crmd: [8072]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:09:12 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: 1 lost packet(s) for [ha2] [13:15]
> 2006:02:10-14:09:12 (none) [daemon:info] heartbeat: [8060]: info:  
> No pkts missing from ha2!
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_VERSION_REQUEST, current  
> leader is none
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_VERSION_REQUEST to  
> CCM_STATE_NEW_NODE_WAIT_FOR_MEM_LIST, current leader is none
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG:  
> Dumping message with 18 fields
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [0] : [t=CCM_TYPE_PROTOVERSION_RESP]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [1] : [ccmproto=0]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [2] : [ccmmajor=1]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [3] : [ccmclsize=1]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [4] : [ccmcookie=>bI4k>I5=-u$%l]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [5] : [dest=ha1]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [6] : [from_id=ccm]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [7] : [to_id=ccm]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [8] : [client_gen=0]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [9] : [(1)destuuid=0x805c628]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [10] : [src=ha2]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [11] : [(1)srcuuid=0x805c430]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [12] : [seq=13]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [13] : [hg=9]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [14] : [ts=43ec9049]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [15] : [ld=0.01 0.03 0.00 4/76 8231]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [16] : [ttl=3]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [17] : [auth=1 33311a2a9c75eaf0de2b417ff27b6a8a]
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> ccm_state_new_node_wait_for_mem_list: leader  changed  cookie
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_NEW_NODE_WAIT_FOR_MEM_LIST to  
> CCM_STATE_JOINED, current leader is ha2
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG:  
> Dumping message with 18 fields
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [0] : [t=CCM_TYPE_MEM_LIST]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [1] : [ccmmajor=1]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [2] : [ccmminor=0]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [3] : [ccmcookie=>bI4k>I5=-u$%l]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [4] : [ccmmemlist=AwAAAAAAAAAAAAAAAA==]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [5] :[(3)ccmuptimelist=33554432 16777216 -1520786086 42330 -1 -1 -1  
> -1]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [6] : [ccmnewcookie=B_jCC(|`YxBWh^]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [7] : [from_id=ccm]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [8] : [to_id=ccm]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [9] : [client_gen=0]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [10] : [src=ha2]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [11] : [(1)srcuuid=0x805c580]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [12] : [seq=14]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [13] : [hg=9]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [14] : [ts=43ec9049]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [15] : [ld=0.01 0.03 0.00 2/76 8231]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [16] : [ttl=3]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [17] : [auth=1 5d795ef47d043b60e3dea9219a97c96f]
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: 		  
> the following are the members of the group of transition=2
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: 		 
> nodename=ha1 bornon=2
> 2006:02:10-14:09:13 (none) [daemon:debug] ccm: [8068]: debug: 		 
> nodename=ha2 bornon=1
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> Counting nodes(dead nodes are not shown):
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> node=ha1  status=active
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> node=ha2  status=active
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> n_member=2, nodecount=2, inactive_count=0
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> Asserting quorum for two node cluster!
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info:  
> delivering new membership to 1 clients:
> 2006:02:10-14:09:13 (none) [daemon:info] ccm: [8068]: info: client:  
> pid =8069
> 2006:02:10-14:09:13 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:09:13 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0,  
> new_idx=0, old_idx=4
> 2006:02:10-14:09:13 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW  
> MEMBERSHIP (id=2)
> 2006:02:10-14:09:13 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=2)
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (cib_native.c:cib_native_signon): Connection to CIB successful
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> SO_PEERCRED returned [8072, (90:90)]
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> Verifying authentication: uidptr=0x80587e0 gidptr=0x0
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> SO_PEERCRED returned [8072, (90:90)]
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:09:13 (none) [daemon:debug] lrmd: [8071]: debug:  
> Verifying authentication: uidptr=0x80587e0 gidptr=0x0
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:do_ccm_control): CCM Activation passed... all set to go!
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:do_started): Delaying start, CCM (0000000000100000) not  
> connected
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (main.c:init_start): Starting crmd's mainloop
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0,  
> new_idx=0, old_idx=4
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=2)
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:ccm_event_detail): NEW MEMBERSHIP: trans=2, nodes=2, new=2,  
> lost=0 n_idx=0, new_idx=0, old_idx=4
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:ccm_event_detail): 	NEW:     ha2 [nodeid=1, born=1]
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:ccm_event_detail): 	NEW:     ha1 [nodeid=0, born=2]
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (control.c:do_started): The local CRM is operational
> 2006:02:10-14:09:13 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_STARTING ->  
> S_PENDING [ input=I_PENDING cause=C_CCM_CALLBACK origin=do_started ]
> 2006:02:10-14:09:25 (none) [daemon:info] crmd: [8072]: info: mask 
> (utils.c:crm_timer_popped): Election Trigger (I_DC_TIMEOUT) just  
> popped!
> 2006:02:10-14:09:25 (none) [daemon:warning] crmd: [8072]: WARN: mask 
> (misc.c:do_log): [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped()  
> received in state (S_PENDING)
> 2006:02:10-14:09:25 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_PENDING ->  
> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED  
> origin=crm_timer_popped ]
> 2006:02:10-14:09:25 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_ELECTION ->  
> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL  
> origin=do_election_count_vote ]
> 2006:02:10-14:09:25 (none) [daemon:info] crmd: [8072]: info: mask 
> (election.c:do_dc_release): DC role released
> 2006:02:10-14:09:31 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: 1 lost packet(s) for [ha2] [42:44]
> 2006:02:10-14:09:31 (none) [daemon:info] heartbeat: [8060]: info:  
> No pkts missing from ha2!
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (messages.c:cib_process_diff): Diff 0.10.68 -> 0.10.69 not applied  
> to 0.11.73: current "epoch" is greater than required
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (notify.c:do_cib_notify): cib_apply_diff of <diff > FAILED:  
> Application of an update diff failed
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (notify.c:do_cib_notify): cib_apply_diff of <diff > FAILED:  
> Application of an update diff failed
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (callbacks.c:cib_process_request): cib_apply_diff operation failed:  
> Application of an update diff failed
> 2006:02:10-14:09:31 (none) [daemon:info] crmd: [8072]: info: mask 
> (join_client.c:do_cl_join_request): Set DC to ha2
> 2006:02:10-14:09:31 (none) [daemon:info] cib: [8069]: info: mask 
> (messages.c:sync_our_cib): Syncing CIB to all peers
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:31 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_PENDING -> S_NOT_DC  
> [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_result ]
> 2006:02:10-14:09:31 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:31 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: Performed 2 more non-realtime malloc calls.
> 2006:02:10-14:09:31 (none) [daemon:info] heartbeat: [8060]: info:  
> Total non-realtime malloc bytes: 131072
> 2006:02:10-14:09:32 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:32 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:20 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: node ha2: is dead
> 2006:02:10-14:10:20 (none) [daemon:info] heartbeat: [8060]: info:  
> Link ha2:eth1 dead.
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG:  
> Dumping message with 5 fields
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG 
> [1] : [seq=1]
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG 
> [2] : [ts=43ec90bc]
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG 
> [3] : [src=ha2]
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: MSG 
> [4] : [st=dead]
> 2006:02:10-14:10:20 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_JOINED to CCM_STATE_JOINING, current leader is  
> ha2
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG:  
> Dumping message with 5 fields
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [0] : [t=CCM_TYPE_LEAVE]
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [1] : [src=ha2]
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [2] : [ccmmajor=2]
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [3] : [ccmcookie=B_jCC(|`YxBWh^]
> 2006:02:10-14:10:20 (none) [daemon:debug] ccm: [8068]: debug: MSG 
> [4] : [ccmminor=0]
> 2006:02:10-14:10:20 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
> 2006:02:10-14:10:20 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0,  
> new_idx=0, old_idx=4
> 2006:02:10-14:10:20 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NOT PRIMARY  
> (id=2)
> 2006:02:10-14:10:20 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
> 2006:02:10-14:10:20 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0,  
> new_idx=0, old_idx=4
> 2006:02:10-14:10:20 (none) [daemon:info] crmd: [8072]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum lost after event=NOT  
> PRIMARY (id=2)
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> ccm_joining_to_joined: cookie changed
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info: change  
> state from CCM_STATE_JOINING to CCM_STATE_JOINED, current leader is  
> ha1
> 2006:02:10-14:10:30 (none) [daemon:debug] ccm: [8068]: debug: 		  
> the following are the members of the group of transition=3
> 2006:02:10-14:10:30 (none) [daemon:debug] ccm: [8068]: debug: 		 
> nodename=ha1 bornon=3
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> Counting nodes(dead nodes are not shown):
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> node=ha1  status=active
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> n_member=1, nodecount=2, inactive_count=0
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> Asserting quorum for two node cluster!
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info:  
> delivering new membership to 2 clients:
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info: client:  
> pid =8072
> 2006:02:10-14:10:30 (none) [daemon:info] ccm: [8068]: info: client:  
> pid =8069
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info:  
> mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0,  
> new_idx=1, old_idx=3
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=3)
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:ccm_event_detail): NEW MEMBERSHIP: trans=3, nodes=1, new=0,  
> lost=1 n_idx=0, new_idx=1, old_idx=3
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info: mask 
> (ccm.c:ccm_event_detail): 	LOST:    ha2 [nodeid=1, born=1]
> 2006:02:10-14:10:30 (none) [daemon:warning] crmd: [8072]: WARN: mask 
> (ccm.c:do_ccm_event): Our DC node (ha2) left the cluster
> 2006:02:10-14:10:30 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_NOT_DC ->  
> S_ELECTION [ input=I_ELECTION cause=C_CCM_CALLBACK  
> origin=do_ccm_event ]
> 2006:02:10-14:10:30 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:10:30 (none) [daemon:info] cib: [8069]: info:  
> mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0,  
> new_idx=1, old_idx=3
> 2006:02:10-14:10:30 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW  
> MEMBERSHIP (id=3)
> 2006:02:10-14:10:30 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=3)
> 2006:02:10-14:10:30 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (utils.c:crm_timer_popped): Election Timeout (I_ELECTION_DC) just  
> popped!
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_ELECTION ->  
> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED  
> origin=crm_timer_popped ]
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (subsystems.c:start_subsystem): Starting sub-system "tengine"
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (subsystems.c:start_subsystem): Starting sub-system "pengine"
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (election.c:do_dc_takeover): Taking over DC status for this partition
> 2006:02:10-14:10:36 (none) [daemon:info] cib: [8069]: info: mask 
> (messages.c:cib_process_readwrite): We are now in R/W mode
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (cib_native.c:cib_native_signon): Connection to CIB successful
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info: mask 
> (main.c:init_start): Starting pengine
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (join_dc.c:do_dc_join_offer_all): 0) Offering membership to 1 clients
> 2006:02:10-14:10:36 (none) [daemon:notice] crmd: [8072]: notice:  
> mask(callbacks.c:crmd_client_status_callback): Status update:  
> Client ha1/crmd now has status [online]
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_INTEGRATION ->  
> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL  
> origin=check_join_state ]
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): All 1 cluster nodes responded to the  
> join offer.
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_null_callback): Setting cib_diff_notify callbacks  
> for tengine: on
> 2006:02:10-14:10:36 (none) [daemon:debug] stonithd: [8070]: debug:  
> SO_PEERCRED returned [8119, (90:90)]
> 2006:02:10-14:10:36 (none) [daemon:debug] stonithd: [8070]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:10:36 (none) [daemon:debug] stonithd: [8070]: debug:  
> Verifying authentication: uidptr=0x805ae78 gidptr=0x0
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (main.c:init_start): Starting tengine
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (tengine.c:initialize_graph): Registering TE UUID:  
> 4f063d85-0e3f-42a2-8ee8-b86d992078b4
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (join_dc.c:process_join_ack_msg): 4) Updating node state to member  
> for ha1
> 2006:02:10-14:10:36 (none) [daemon:warning] cib: [8069]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_FINALIZE_JOIN ->  
> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL  
> origin=check_join_state ]
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): All 1 cluster nodes are eligable to  
> run resources.
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (tengine.c:do_te_invoke): Waiting for the TE to connect before  
> action A_TE_CANCEL
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (utils.c:send_complete): 1 - Transition status: Stopped:  
> te_abort_confirmed
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info: mask 
> (process_pe_message): [generation] <cib admin_epoch="0"  
> have_quorum="true" num_peers="2" origin="ha1"  
> cib_feature_revision="1" last_written="Fri Feb 10 14:10:36 2006"  
> debug_source="finalize_join" ccm_transition="3"  
> dc_uuid="70718d52-92b1-453e-a64d-b526d759bd42" generated="true"  
> epoch="13" num_updates="84"/>
> 2006:02:10-14:10:36 (none) [daemon:warning] pengine: [8120]: WARN:  
> mask(unpack.c:param_value): Option default_resource_stickiness not set
> 2006:02:10-14:10:36 (none) [daemon:warning] pengine: [8120]: WARN:  
> mask(unpack.c:param_value): Option stonith_enabled not set
> 2006:02:10-14:10:36 (none) [daemon:warning] pengine: [8120]: WARN:  
> mask(unpack.c:param_value): Option no_quorum_policy not set
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info: mask 
> (unpack.c:unpack_config): STONITH of failed nodes is disabled
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info: mask 
> (unpack.c:unpack_config): On loss of CCM Quorum: Stop ALL resources
> 2006:02:10-14:10:36 (none) [daemon:warning] pengine: [8120]: WARN:  
> mask(unpack.c:unpack_rsc_location): No resource  
> (con=switch_ha_mode, rsc=ha_master)
> 2006:02:10-14:10:36 (none) [daemon:warning] pengine: [8120]: WARN:  
> mask(native.c:native_color): Resource master:ha_master cannot run  
> anywhere
> 2006:02:10-14:10:36 (none) [daemon:info] pengine: [8120]: info: mask 
> (stages.c:stage8): Creating transition graph 0.
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_POLICY_ENGINE ->  
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (unpack.c:unpack_graph): Beginning transition 0 : timeout set to  
> 10000ms
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (unpack.c:unpack_graph): Unpacked 0 actions in 0 synapses
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (tengine.c:initiate_transition): Initating transition
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (tengine.c:check_for_completion): Transition complete
> 2006:02:10-14:10:36 (none) [daemon:info] tengine: [8119]: info: mask 
> (utils.c:send_complete): 0 - Transition status: Complete: complete
> 2006:02:10-14:10:36 (none) [daemon:info] crmd: [8072]: info: mask 
> (fsa.c:do_state_transition): State transition S_TRANSITION_ENGINE - 
> > S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:10:48 (none) [daemon:err] heartbeat: [8060]: ERROR:  
> Cluster node ha2 returning after partition.
> 2006:02:10-14:10:48 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: Deadtime value may be too small.
> 2006:02:10-14:10:48 (none) [daemon:info] heartbeat: [8060]: info:  
> See documentation for information on tuning deadtime.
> 2006:02:10-14:10:48 (none) [daemon:info] heartbeat: [8060]: info:  
> Link ha2:eth1 up.
> 2006:02:10-14:10:48 (none) [daemon:warning] heartbeat: [8060]:  
> WARN: Late heartbeat: Node ha2: interval 31230 ms
> 2006:02:10-14:10:48 (none) [daemon:info] heartbeat: [8060]: info:  
> Status update for node ha2: status active
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG:  
> Dumping message with 12 fields
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [1] : [st=active]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [2] : [dt=bb8]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [3] : [protocol=1]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [4] : [src=ha2]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [5] : [(1)srcuuid=0x806a618]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [6] : [seq=89]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [7] : [hg=9]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [8] : [ts=43ec90a8]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [9] : [ld=0.12 0.06 0.01 1/80 8261]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [10] : [ttl=3]
> 2006:02:10-14:10:48 (none) [daemon:info] ccm: [8068]: info: MSG 
> [11] : [auth=1 d55496da21a35add55354971333986b5]
> 2006:02:10-14:12:48 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_process_disconnect): Cleaning up after cib_ro  
> channel disconnect from client (0x806c2f8)  
> ce2376bc-9626-4021-8197-5e1ee1802c3/<null>
> 2006:02:10-14:12:48 (none) [daemon:info] cib: [8069]: info: mask 
> (callbacks.c:cib_process_disconnect): Cleaning up after  
> cib_callback channel disconnect from client (0x806eb98)  
> 9d5d46de-313a-4560-a548-5e5aa6e9890/crm_mon
>
> 2006:02:10-14:08:22 (none) [local7:info] logd: [8196]: info: logd  
> started with default configuration.
> 2006:02:10-14:08:22 (none) [local7:warning] logd: [8196]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:08:22 (none) [local7:warning] logd: [8196]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:08:22 (none) [local7:info] logd: [8199]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:22 (none) [local7:info] logd: [8196]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: Logging daemon is disabled --enabling logging daemon is  
> recommended
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: File /etc/ha.d/haresources exists.
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: This file is not used because crm is enabled
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: Consider setting /proc/sys/kernel/core_uses_pid (or  
> equivalent) to 1 for maximum supportability
> 2006:02:10-14:08:22 (none) [daemon:warning] heartbeat: [8218]:  
> WARN: Logging daemon is disabled --enabling logging daemon is  
> recommended
> 2006:02:10-14:08:22 (none) [daemon:info] heartbeat: [8218]: info:  
> **************************
> 2006:02:10-14:08:22 (none) [daemon:info] heartbeat: [8218]: info:  
> Configuration validated. Starting heartbeat 2.0.2
> 2006:02:10-14:08:22 (none) [daemon:info] heartbeat: [8219]: info:  
> heartbeat: version 2.0.2
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> Heartbeat generation: 9
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> Removing /var/run/heartbeat/rsctmp failed, recreating.
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> pid 8219 locked in memory.
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8219]: info:  
> Local status now set to: 'up'
> 2006:02:10-14:08:23 (none) [daemon:info] heartbeat: [8224]: info:  
> pid 8224 locked in memory.
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8225]: info:  
> pid 8225 locked in memory.
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8226]: info:  
> pid 8226 locked in memory.
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Link ha1:eth1 up.
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Status update for node ha1: status up
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Local status now set to: 'active'
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Starting child client "/usr/lib/heartbeat/ccm" (90,90)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Starting child client "/usr/lib/heartbeat/cib" (90,90)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Starting child client "/usr/lib/heartbeat/stonithd" (0,0)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Starting child client "/usr/lib/heartbeat/lrmd" (0,0)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Starting child client "/usr/lib/heartbeat/crmd" (90,90)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Status update for node ha1: status active
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> Link ha2:eth1 up.
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8227]: info:  
> Starting "/usr/lib/heartbeat/ccm" as uid 90  gid 90 (pid 8227)
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: Disable  
> using logging daemon
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: PID=8227
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: Signing  
> in with Heartbeat
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8227, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x814df48 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8227, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x814d630
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8227, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8143ba8 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: 1 lost packet(s) for [ha1] [28:30]
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> No pkts missing from ha1!
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info:  
> Switched to heartbeat syslog facility: 24
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info:  
> Hostname: ha2
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: total  
> node number is 2
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: node 0  
> =ha1, status=active
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: node 1  
> =ha2, status=active
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_NONE, current leader is none
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_NONE, current leader is none
> 2006:02:10-14:08:24 (none) [daemon:info] ccm: [8227]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8228]: info:  
> Starting "/usr/lib/heartbeat/cib" as uid 90  gid 90 (pid 8228)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8229]: info:  
> Starting "/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 8229)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8231]: info:  
> Starting "/usr/lib/heartbeat/crmd" as uid 90  gid 90 (pid 8231)
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8230]: info:  
> Starting "/usr/lib/heartbeat/lrmd" as uid 0  gid 0 (pid 8230)
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info:  
> Disable using logging daemon
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:08:24 (none) [daemon:warning] lrmd: [8230]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:08:24 (none) [daemon:warning] lrmd: [8230]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 10
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 12
> 2006:02:10-14:08:24 (none) [daemon:info] lrmd: [8230]: info: Started.
> 2006:02:10-14:08:24 (none) [daemon:info] cib: [8228]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:24 (none) [daemon:info] cib: [8228]: info: mask 
> (main.c:cib_register_ha): Signing in with Heartbeat
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8228, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x814d780 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8228, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x814ddf8
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8228, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8141ae8 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:info] cib: [8228]: info: mask 
> (main.c:cib_register_ha): FSA Hostname: ha2
> 2006:02:10-14:08:24 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:24 (none) [daemon:info] cib: [8228]: info: mask 
> (main.c:startCib): CIB Initialization completed successfully
> 2006:02:10-14:08:24 (none) [daemon:info] cib: [8228]: info: mask 
> (main.c:init_start): Starting cib mainloop
> 2006:02:10-14:08:24 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: 1 lost packet(s) for [ha1] [34:36]
> 2006:02:10-14:08:24 (none) [daemon:info] heartbeat: [8219]: info:  
> No pkts missing from ha1!
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> Disable using logging daemon
> 2006:02:10-14:08:24 (none) [daemon:warning] stonithd: [8229]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:08:24 (none) [daemon:warning] stonithd: [8229]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 10
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 12
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> pid 8229 locked in memory.
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8229, (0:0)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8153880 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8229, (0:0)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x8153688
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8229, (0:0)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=0 cred.gid=0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8142ad8 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> Signing in with heartbeat.
> 2006:02:10-14:08:24 (none) [daemon:debug] stonithd: [8229]: debug:  
> Setting message filter mode
> 2006:02:10-14:08:24 (none) [daemon:notice] stonithd: [8229]:  
> notice: /usr/lib/heartbeat/stonithd start up successfully.
> 2006:02:10-14:08:24 (none) [daemon:info] stonithd: [8229]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:08:24 (none) [daemon:warning] crmd: [8231]: WARN:  
> Core dumps could be lost if multiple dumps occur
> 2006:02:10-14:08:24 (none) [daemon:warning] crmd: [8231]: WARN:  
> Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to  
> 1 for maximum supportability
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (main.c:init_start): Starting crmd
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8231, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8153c00 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8231, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x0 gidptr=0x8153998
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> SO_PEERCRED returned [8231, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] heartbeat: [8219]: debug:  
> Verifying authentication: uidptr=0x8142bd8 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:register_with_ha): FSA Hostname: ha2
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_startup): Register Signal Handler
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info:  
> G_main_add_TriggerHandler: Added signal manual handler
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_startup): Init server comms
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_startup): Creating CIB object
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 17
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (cib_native.c:cib_native_signon): Connection to CIB successful
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> SO_PEERCRED returned [8231, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> Verifying authentication: uidptr=0x80587e0 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> SO_PEERCRED returned [8231, (90:90)]
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:24 (none) [daemon:debug] lrmd: [8230]: debug:  
> Verifying authentication: uidptr=0x80587e0 gidptr=0x0
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:do_ccm_control): CCM Activation passed... all set to go!
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_started): Delaying start, CCM (0000000000100000) not  
> connected
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (main.c:init_start): Starting crmd's mainloop
> 2006:02:10-14:08:24 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_started): Delaying start, CCM (0000000000100000) not  
> connected
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: change  
> state from CCM_STATE_NONE to CCM_STATE_VERSION_REQUEST, current  
> leader is none
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: change  
> state from CCM_STATE_VERSION_REQUEST to CCM_STATE_JOINED, current  
> leader is ha2
> 2006:02:10-14:08:25 (none) [daemon:debug] ccm: [8227]: debug: 		  
> the following are the members of the group of transition=1
> 2006:02:10-14:08:25 (none) [daemon:debug] ccm: [8227]: debug: 		 
> nodename=ha2 bornon=1
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> Counting nodes(dead nodes are not shown):
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> node=ha1  status=active
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> node=ha2  status=active
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> n_member=1, nodecount=2, inactive_count=0
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> Asserting quorum for two node cluster!
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> delivering new membership to 2 clients:
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8231
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8228
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_started): Delaying start, CCM (0000000000100000) not  
> connected
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0,  
> new_idx=0, old_idx=3
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=1)
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): NEW MEMBERSHIP: trans=1, nodes=1, new=1,  
> lost=0 n_idx=0, new_idx=0, old_idx=3
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): 	NEW:     ha2 [nodeid=1, born=1]
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (control.c:do_started): The local CRM is operational
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_STARTING ->  
> S_PENDING [ input=I_PENDING cause=C_CCM_CALLBACK origin=do_started ]
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0,  
> new_idx=0, old_idx=3
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW  
> MEMBERSHIP (id=1)
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=1)
> 2006:02:10-14:08:25 (none) [daemon:debug] ccm: [8227]: debug: 		  
> the following are the members of the group of transition=2
> 2006:02:10-14:08:25 (none) [daemon:debug] ccm: [8227]: debug: 		 
> nodename=ha2 bornon=1
> 2006:02:10-14:08:25 (none) [daemon:debug] ccm: [8227]: debug: 		 
> nodename=ha1 bornon=2
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> Counting nodes(dead nodes are not shown):
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> node=ha1  status=active
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> node=ha2  status=active
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> n_member=2, nodecount=2, inactive_count=0
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> Asserting quorum for two node cluster!
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info:  
> delivering new membership to 2 clients:
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8231
> 2006:02:10-14:08:25 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8228
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0,  
> new_idx=2, old_idx=4
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=2)
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): NEW MEMBERSHIP: trans=2, nodes=2, new=1,  
> lost=0 n_idx=0, new_idx=2, old_idx=4
> 2006:02:10-14:08:25 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): 	NEW:     ha1 [nodeid=0, born=2]
> 2006:02:10-14:08:25 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:25 (none) [daemon:warning] ccm: [8227]: WARN:  
> ccm_state_joined: received message with unknown cookie, just dropping
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0,  
> new_idx=2, old_idx=4
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW  
> MEMBERSHIP (id=2)
> 2006:02:10-14:08:25 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=2)
> 2006:02:10-14:08:37 (none) [daemon:info] crmd: [8231]: info: mask 
> (utils.c:crm_timer_popped): Election Trigger (I_DC_TIMEOUT) just  
> popped!
> 2006:02:10-14:08:37 (none) [daemon:warning] crmd: [8231]: WARN: mask 
> (misc.c:do_log): [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped()  
> received in state (S_PENDING)
> 2006:02:10-14:08:37 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_PENDING ->  
> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED  
> origin=crm_timer_popped ]
> 2006:02:10-14:08:37 (none) [daemon:info] crmd: [8231]: info: mask 
> (election.c:do_election_count_vote): Election won over ha1
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (utils.c:crm_timer_popped): Election Timeout (I_ELECTION_DC) just  
> popped!
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_ELECTION ->  
> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED  
> origin=crm_timer_popped ]
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (subsystems.c:start_subsystem): Starting sub-system "tengine"
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (subsystems.c:start_subsystem): Starting sub-system "pengine"
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (cib_native.c:cib_native_signon): Connection to CIB successful
> 2006:02:10-14:08:43 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_null_callback): Setting cib_diff_notify callbacks  
> for tengine: on
> 2006:02:10-14:08:43 (none) [daemon:debug] stonithd: [8229]: debug:  
> SO_PEERCRED returned [8235, (90:90)]
> 2006:02:10-14:08:43 (none) [daemon:debug] stonithd: [8229]: debug:  
> Verifying authentication: cred.uid=90 cred.gid=90
> 2006:02:10-14:08:43 (none) [daemon:debug] stonithd: [8229]: debug:  
> Verifying authentication: uidptr=0x805ae78 gidptr=0x0
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (main.c:init_start): Starting tengine
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (tengine.c:initialize_graph): Registering TE UUID: e5f1c57e- 
> dd6e-4144-887e-ddcc1d88d9e7
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info:  
> G_main_add_SignalHandler: Added signal handler for signal 15
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info: mask 
> (main.c:init_start): Starting pengine
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (election.c:do_dc_takeover): Taking over DC status for this partition
> 2006:02:10-14:08:43 (none) [daemon:info] cib: [8228]: info: mask 
> (messages.c:cib_process_readwrite): We are now in R/W mode
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (join_dc.c:do_dc_join_offer_all): 0) Offering membership to 2 clients
> 2006:02:10-14:08:43 (none) [daemon:notice] crmd: [8231]: notice:  
> mask(callbacks.c:crmd_client_status_callback): Status update:  
> Client ha2/crmd now has status [online]
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:notice] crmd: [8231]: notice:  
> mask(callbacks.c:crmd_client_status_callback): Status update:  
> Client ha1/crmd now has status [online]
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_INTEGRATION ->  
> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL  
> origin=check_join_state ]
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): All 2 cluster nodes responded to the  
> join offer.
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (join_dc.c:do_dc_join_finalize): Asking ha1 for its copy of the CIB
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (join_dc.c:process_join_ack_msg): 4) Updating node state to member  
> for ha1
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (join_dc.c:process_join_ack_msg): 4) Updating node state to member  
> for ha2
> 2006:02:10-14:08:43 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_FINALIZE_JOIN ->  
> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL  
> origin=check_join_state ]
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): All 2 cluster nodes are eligable to  
> run resources.
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (tengine.c:do_te_invoke): Waiting for the TE to connect before  
> action A_TE_CANCEL
> 2006:02:10-14:08:43 (none) [daemon:warning] crmd: [8231]: WARN: mask 
> (messages.c:register_fsa_input_adv): do_te_invoke stalled the FSA  
> with pending inputs
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (utils.c:send_complete): 1 - Transition status: Stopped:  
> te_abort_confirmed
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info: mask 
> (process_pe_message): [generation] <cib admin_epoch="0"  
> have_quorum="true" num_peers="1" origin="ha2"  
> cib_feature_revision="1" last_written="Fri Feb 10 14:08:43 2006"  
> dc_uuid="a45993cf-40fc-42ef-9b98-a8360f1db7fa"  
> debug_source="finalize_join" generated="true" epoch="12"  
> num_updates="79" ccm_transition="2"/>
> 2006:02:10-14:08:43 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option default_resource_stickiness not set
> 2006:02:10-14:08:43 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option stonith_enabled not set
> 2006:02:10-14:08:43 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option no_quorum_policy not set
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info: mask 
> (unpack.c:unpack_config): STONITH of failed nodes is disabled
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info: mask 
> (unpack.c:unpack_config): On loss of CCM Quorum: Stop ALL resources
> 2006:02:10-14:08:43 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:unpack_rsc_location): No resource  
> (con=switch_ha_mode, rsc=ha_master)
> 2006:02:10-14:08:43 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(native.c:native_color): Resource master:ha_master cannot run  
> anywhere
> 2006:02:10-14:08:43 (none) [daemon:info] pengine: [8236]: info: mask 
> (stages.c:stage8): Creating transition graph 0.
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_POLICY_ENGINE ->  
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (unpack.c:unpack_graph): Beginning transition 0 : timeout set to  
> 10000ms
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (unpack.c:unpack_graph): Unpacked 0 actions in 0 synapses
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (tengine.c:initiate_transition): Initating transition
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (tengine.c:check_for_completion): Transition complete
> 2006:02:10-14:08:43 (none) [daemon:info] tengine: [8235]: info: mask 
> (utils.c:send_complete): 0 - Transition status: Complete: complete
> 2006:02:10-14:08:43 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_TRANSITION_ENGINE - 
> > S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:08:44 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: Performed 1 more non-realtime malloc calls.
> 2006:02:10-14:08:44 (none) [daemon:info] heartbeat: [8219]: info:  
> Total non-realtime malloc bytes: 135168
> 2006:02:10-14:09:32 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: node ha1: is dead
> 2006:02:10-14:09:32 (none) [daemon:info] heartbeat: [8219]: info:  
> Link ha1:eth1 dead.
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG:  
> Dumping message with 5 fields
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG 
> [1] : [seq=1]
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG 
> [2] : [ts=43ec908c]
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG 
> [3] : [src=ha1]
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: MSG 
> [4] : [st=dead]
> 2006:02:10-14:09:32 (none) [daemon:debug] ccm: [8227]: debug: 		  
> the following are the members of the group of transition=3
> 2006:02:10-14:09:32 (none) [daemon:debug] ccm: [8227]: debug: 		 
> nodename=ha2 bornon=3
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info:  
> Counting nodes(dead nodes are not shown):
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info:  
> node=ha2  status=active
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info:  
> n_member=1, nodecount=2, inactive_count=0
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info:  
> Asserting quorum for two node cluster!
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info:  
> delivering new membership to 2 clients:
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8231
> 2006:02:10-14:09:32 (none) [daemon:info] ccm: [8227]: info: client:  
> pid =8228
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info:  
> mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0,  
> new_idx=1, old_idx=3
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (callbacks.c:crmd_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=3)
> 2006:02:10-14:09:32 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
> 2006:02:10-14:09:32 (none) [daemon:info] cib: [8228]: info:  
> mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0,  
> new_idx=1, old_idx=3
> 2006:02:10-14:09:32 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Process CCM event=NEW  
> MEMBERSHIP (id=3)
> 2006:02:10-14:09:32 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_ccm_msg_callback): Quorum (re)attained after  
> event=NEW MEMBERSHIP (id=3)
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): NEW MEMBERSHIP: trans=3, nodes=1, new=0,  
> lost=1 n_idx=0, new_idx=1, old_idx=3
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (ccm.c:ccm_event_detail): 	LOST:    ha1 [nodeid=0, born=2]
> 2006:02:10-14:09:32 (none) [daemon:warning] ccm: [8227]: WARN:  
> ccm_state_joined: received message with unknown cookie, just dropping
> 2006:02:10-14:09:32 (none) [daemon:warning] cib: [8228]: WARN: mask 
> (io.c:initializeCib): Option suppress_cib_writes not set
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (utils.c:send_complete): 0 - Transition status: Triggered by CIB  
> update: Stonith/shutdown event not matched
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_IDLE ->  
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:09:32 (none) [daemon:warning] crmd: [8231]: WARN: mask 
> (fsa.c:do_state_transition): Only 2 of 1 cluster nodes are eligable  
> to run resources
> 2006:02:10-14:09:32 (none) [daemon:info] pengine: [8236]: info: mask 
> (process_pe_message): [generation] <cib admin_epoch="0"  
> have_quorum="true" num_peers="1" origin="ha2"  
> cib_feature_revision="1" last_written="Fri Feb 10 14:09:32 2006"  
> dc_uuid="a45993cf-40fc-42ef-9b98-a8360f1db7fa"  
> debug_source="finalize_join" generated="true" epoch="12"  
> num_updates="79" ccm_transition="3"/>
> 2006:02:10-14:09:32 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option default_resource_stickiness not set
> 2006:02:10-14:09:32 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option stonith_enabled not set
> 2006:02:10-14:09:32 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:param_value): Option no_quorum_policy not set
> 2006:02:10-14:09:32 (none) [daemon:info] pengine: [8236]: info: mask 
> (unpack.c:unpack_config): STONITH of failed nodes is disabled
> 2006:02:10-14:09:32 (none) [daemon:info] pengine: [8236]: info: mask 
> (unpack.c:unpack_config): On loss of CCM Quorum: Stop ALL resources
> 2006:02:10-14:09:32 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(unpack.c:unpack_rsc_location): No resource  
> (con=switch_ha_mode, rsc=ha_master)
> 2006:02:10-14:09:32 (none) [daemon:warning] pengine: [8236]: WARN:  
> mask(native.c:native_color): Resource master:ha_master cannot run  
> anywhere
> 2006:02:10-14:09:32 (none) [daemon:info] pengine: [8236]: info: mask 
> (stages.c:stage8): Creating transition graph 1.
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_POLICY_ENGINE ->  
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (unpack.c:unpack_graph): Beginning transition 1 : timeout set to  
> 10000ms
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (unpack.c:unpack_graph): Unpacked 0 actions in 0 synapses
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (tengine.c:initiate_transition): Initating transition
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (tengine.c:check_for_completion): Transition complete
> 2006:02:10-14:09:32 (none) [daemon:info] tengine: [8235]: info: mask 
> (utils.c:send_complete): 1 - Transition status: Complete: complete
> 2006:02:10-14:09:32 (none) [daemon:info] crmd: [8231]: info: mask 
> (fsa.c:do_state_transition): State transition S_TRANSITION_ENGINE - 
> > S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE  
> origin=do_msg_route ]
> 2006:02:10-14:10:00 (none) [daemon:err] heartbeat: [8219]: ERROR:  
> Cluster node ha1 returning after partition.
> 2006:02:10-14:10:00 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: Deadtime value may be too small.
> 2006:02:10-14:10:00 (none) [daemon:info] heartbeat: [8219]: info:  
> See documentation for information on tuning deadtime.
> 2006:02:10-14:10:00 (none) [daemon:info] heartbeat: [8219]: info:  
> Link ha1:eth1 up.
> 2006:02:10-14:10:00 (none) [daemon:warning] heartbeat: [8219]:  
> WARN: Late heartbeat: Node ha1: interval 30920 ms
> 2006:02:10-14:10:00 (none) [daemon:info] heartbeat: [8219]: info:  
> Status update for node ha1: status active
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG:  
> Dumping message with 12 fields
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [0] : [t=status]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [1] : [st=active]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [2] : [dt=bb8]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [3] : [protocol=1]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [4] : [src=ha1]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [5] : [(1)srcuuid=0x805ef18]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [6] : [seq=9c]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [7] : [hg=e]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [8] : [ts=43ec90d8]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [9] : [ld=0.29 0.17 0.05 1/81 8120]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [10] : [ttl=3]
> 2006:02:10-14:10:00 (none) [daemon:info] ccm: [8227]: info: MSG 
> [11] : [auth=1 47078e408272601cdd21ea810e8f35b0]
> 2006:02:10-14:12:09 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_process_disconnect): Cleaning up after cib_ro  
> channel disconnect from client (0x8068c40) 898de504-15d2-4385- 
> b2a9-8ff170280b4/<null>
> 2006:02:10-14:12:09 (none) [daemon:info] cib: [8228]: info: mask 
> (callbacks.c:cib_process_disconnect): Cleaning up after  
> cib_callback channel disconnect from client (0x8070770)  
> 0c0250dc-532c-4332-8736-fd152d58e58/crm_mon

--
Andrew Beekhof

"Too much knowledge leads to confusion; Too many guitar lessons lead  
to jazz-fusion!" - TISM






More information about the Linux-HA mailing list