[Linux-HA] Heartbeat reboots machine after "generic plugin load
failed" error - please help
Luis Motta Campos
luismottacampos at yahoo.co.uk
Mon Mar 3 05:16:15 MST 2008
Hi Linux-HA list :)
I'm running CentOS 5 (Linux 2.6.18-53.1.6.el5 #1 SMP Wed Jan 23
11:28:47 EST 2008 x86_64 GNU/Linux), and Heartbeat from the packages:
heartbeat.x86_64 2.1.3-3.el5.centos
heartbeat-pils.x86_64 2.1.3-3.el5.centos
heartbeat-stonith.x86_64 2.1.3-3.el5.centos
I'm trying to set up my second cluster, and it's giving me some
headaches already.
It should have two machines, and will support a DRBD filesystem and an
IP address as resources. No resources are configured at the moment, I
just got the two CentOS machines and I'm trying to start heartbeat.
When I issue "service heartbeat start", just after successful packages
installation, I got those error messages multiple times, and then
heartbeat (or some of it's components) just decide that rebooting the
machine is the right thing to do. :(
(Full logfiles for a single session, from startup to the first error
messages is attached)
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin
load failed
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin:
init plugin universe failed
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR:
ccm_calculate_quorum: loading plugin majority failed
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin
load failed
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin:
init plugin universe failed
/var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR:
ccm_calculate_quorum: loading plugin twonodes failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin
load failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin:
init plugin universe failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR:
ccm_calculate_quorum: loading plugin majority failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin
load failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin:
init plugin universe failed
/var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR:
ccm_calculate_quorum: loading plugin twonodes failed
I just ran out of alternatives, and the error messages aren't
descriptive enough to point me to the next clue. Does someone have
suggestions?
Thank you all in advance for your help.
Kindest regards.
--
Luis Motta Campos (a.k.a. Monsieur Champs) is a software engineer,
Perl fanatic evangelist, and amateur {cook, photographer}
-------------- next part --------------
logd[3763]: 2008/03/03_13:00:07 info: logd started with /etc/logd.cf.
logd[3771]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15
logd[3763]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15
heartbeat[3784]: 2008/03/03_13:00:07 info: Enabling logging daemon
heartbeat[3784]: 2008/03/03_13:00:07 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
heartbeat[3784]: 2008/03/03_13:00:07 info: Version 2 support: on
heartbeat[3784]: 2008/03/03_13:00:07 info: **************************
heartbeat[3784]: 2008/03/03_13:00:07 info: Configuration validated. Starting heartbeat 2.1.3
heartbeat[3785]: 2008/03/03_13:00:07 info: heartbeat: version 2.1.3
heartbeat[3785]: 2008/03/03_13:00:07 info: Heartbeat generation: 1203522985
heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2
heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[3785]: 2008/03/03_13:00:07 info: Local status now set to: 'up'
heartbeat[3785]: 2008/03/03_13:00:38 WARN: node db-mysql-test4-ha.ripe.net: is dead
heartbeat[3785]: 2008/03/03_13:00:38 info: Comm_now_up(): updating status to active
heartbeat[3785]: 2008/03/03_13:00:38 info: Local status now set to: 'active'
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/ccm" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/cib" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/lrmd -r" (0,0)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/stonithd" (0,0)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/attrd" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/crmd" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/mgmtd -v" (0,0)
heartbeat[3925]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/ccm" as uid 498 gid 496 (pid 3925)
heartbeat[3926]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/cib" as uid 498 gid 496 (pid 3926)
heartbeat[3927]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3927)
heartbeat[3928]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/stonithd" as uid 0 gid 0 (pid 3928)
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
heartbeat[3929]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/attrd" as uid 498 gid 496 (pid 3929)
heartbeat[3930]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/crmd" as uid 498 gid 496 (pid 3930)
heartbeat[3931]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3931)
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
mgmtd[3931]: 2008/03/03_13:00:38 debug: Enabling coredumps
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
ccm[3925]: 2008/03/03_13:00:38 info: Hostname: db-mysql-test3-ha.ripe.net
mgmtd[3931]: 2008/03/03_13:00:38 WARN: lrm_signon: can not initiate connection
mgmtd[3931]: 2008/03/03_13:00:38 info: login to lrm: 0, ret:0
stonithd[3928]: 2008/03/03_13:00:38 info: Signing in with heartbeat.
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
cib[3926]: 2008/03/03_13:00:38 info: main: Retrieval of a per-action CIB: disabled
cib[3926]: 2008/03/03_13:00:38 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="false" admin_epoch="0" epoch="1" num_updates="1" have_quorum="false" ignore_dtd="false" num_peers="0" cib_feature_revision="2.0" cib-last-written="Mon Mar 3 12:30:41 2008">
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <configuration>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <crm_config>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cluster_property_set id="cib-bootstrap-options">
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <attributes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </attributes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cluster_property_set>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </crm_config>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nodes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <node id="0f24958a-37b1-495a-84fc-911fe61720de" uname="db-mysql-test3-ha.ripe.net" type="normal"/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </nodes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <resources/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <constraints/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </configuration>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <status/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cib>
cib[3926]: 2008/03/03_13:00:38 info: startCib: CIB Initialization completed successfully
cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: Signing in with Heartbeat
cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: FSA Hostname: db-mysql-test3-ha.ripe.net
cib[3926]: 2008/03/03_13:00:38 info: ccm_connect: Registering with CCM...
cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Activation failed
cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Connection failed 1 times (30 max)
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
lrmd[3927]: 2008/03/03_13:00:38 info: Started.
attrd[3929]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net
crmd[3930]: 2008/03/03_13:00:38 info: main: CRM Hg Version: node: 552305612591183b1628baa5bc6e903e0f1e26a3
crmd[3930]: 2008/03/03_13:00:38 info: crmd_init: Starting crmd
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
stonithd[3928]: 2008/03/03_13:00:38 notice: /usr/lib64/heartbeat/stonithd start up successfully.
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de
mgmtd[3931]: 2008/03/03_13:00:39 info: init_crm
ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed
ccm[3925]: 2008/03/03_13:00:39 info: G_main_add_SignalHandler: Added signal handler for signal 15
heartbeat[3785]: 2008/03/03_13:00:40 info: killing /usr/lib64/heartbeat/mgmtd -v process group 3931 with signal 15
cib[3926]: 2008/03/03_13:00:41 info: ccm_connect: Registering with CCM...
cib[3926]: 2008/03/03_13:00:41 info: cib_init: Starting cib mainloop
cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
cib[3926]: 2008/03/03_13:00:41 info: cib_ccm_msg_callback: PEER: db-mysql-test3-ha.ripe.net
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last)
cib[3926]: 2008/03/03_13:00:42 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on
cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [join]
cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [online]
mgmtd[3931]: 2008/03/03_13:00:43 debug: main: run the loop...
mgmtd[3931]: 2008/03/03_13:00:43 info: Started.
mgmtd[3931]: 2008/03/03_13:00:43 info: mgmtd is shutting down
cib[3926]: 2008/03/03_13:00:43 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on
crmd[3930]: 2008/03/03_13:00:43 info: do_cib_control: CIB connection established
cib[3926]: 2008/03/03_13:00:43 ERROR: generic plugin load failed
cib[3926]: 2008/03/03_13:00:43 ERROR: cl_compress_field: loading compression module failed
cib[3926]: 2008/03/03_13:00:43 ERROR: uncompress2compress: compressing 6th field failed
mgmtd[3931]: 2008/03/03_13:00:43 debug: [mgmtd] stopped
heartbeat[3785]: 2008/03/03_13:00:43 info: killing /usr/lib64/heartbeat/crmd process group 3930 with signal 15
crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net
cib[3952]: 2008/03/03_13:00:43 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 330f8d72be5851c99d7ce2e9a8b976a4)
cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last)
crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de
crmd[3930]: 2008/03/03_13:00:44 info: populate_cib_nodes: Requesting the list of configured nodes
crmd[3930]: 2008/03/03_13:00:44 WARN: get_uuid: Could not calculate UUID for db-mysql-test4-ha.ripe.net
crmd[3930]: 2008/03/03_13:00:44 WARN: populate_cib_nodes: Node db-mysql-test4-ha.ripe.net: no uuid found
crmd[3930]: 2008/03/03_13:00:45 notice: populate_cib_nodes: Node: db-mysql-test3-ha.ripe.net (uuid: 0f24958a-37b1-495a-84fc-911fe61720de)
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Connected to Heartbeat
cib[3926]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
cib[3926]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
cib[3926]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 info: do_ccm_control: CCM connection established... waiting for first callback
crmd[3930]: 2008/03/03_13:00:45 info: do_started: Delaying start, CCM (0000000000100000) not connected
crmd[3930]: 2008/03/03_13:00:45 info: crmd_init: Starting crmd's mainloop
crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Requesting shutdown
crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Using default shutdown escalation: 1200000ms
crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online]
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online]
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 WARN: do_log: [[FSA]] Input I_SHUTDOWN from crm_shutdown() received in state (S_STARTING)
crmd[3930]: 2008/03/03_13:00:45 info: do_state_transition: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
crmd[3930]: 2008/03/03_13:00:45 info: do_shutdown: All subsystems stopped, continuing
crmd[3930]: 2008/03/03_13:00:45 info: do_lrm_control: Disconnected from the LRM
crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Disconnected from Heartbeat
crmd[3930]: 2008/03/03_13:00:45 info: do_cib_control: Disconnecting CIB
crmd[3930]: 2008/03/03_13:00:45 info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
crmd[3930]: 2008/03/03_13:00:45 info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
crmd[3930]: 2008/03/03_13:00:45 info: do_exit: [crmd] stopped (0)
ccm[3925]: 2008/03/03_13:00:45 info: client (pid=3930) removed from ccm
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/attrd process group 3929 with signal 15
cib[3926]: 2008/03/03_13:00:45 WARN: send_via_callback_channel: Cannot find client for token a2d999f1-a039-4587-93db-746028050fec
cib[3926]: 2008/03/03_13:00:45 WARN: do_local_notify: A-Sync reply to 3930 failed: client left before we could send reply
attrd[3929]: 2008/03/03_13:00:45 info: main: Starting mainloop...
attrd[3929]: 2008/03/03_13:00:45 info: attrd_shutdown: Exiting
attrd[3929]: 2008/03/03_13:00:45 info: main: Exiting...
attrd[3929]: 2008/03/03_13:00:45 info: attrd_cib_connection_destroy: Connection to the CIB terminated...
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/stonithd process group 3928 with signal 15
stonithd[3928]: 2008/03/03_13:00:45 notice: /usr/lib64/heartbeat/stonithd normally quit.
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/lrmd -r process group 3927 with signal 15
lrmd[3927]: 2008/03/03_13:00:45 info: lrmd is shutting down
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/cib process group 3926 with signal 15
cib[3926]: 2008/03/03_13:00:45 info: cib_shutdown: Disconnected 0 clients
cib[3926]: 2008/03/03_13:00:45 info: cib_process_disconnect: All clients disconnected...
cib[3926]: 2008/03/03_13:00:45 info: terminate_ha_connection: initiate_exit: Disconnecting heartbeat
cib[3926]: 2008/03/03_13:00:45 info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting
cib[3926]: 2008/03/03_13:00:45 info: main: Done
ccm[3925]: 2008/03/03_13:00:46 info: client (pid=3926) removed from ccm
heartbeat[3785]: 2008/03/03_13:00:46 info: killing /usr/lib64/heartbeat/ccm process group 3925 with signal 15
ccm[3925]: 2008/03/03_13:00:46 info: received SIGTERM, going to shut down
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBFIFO process 3790 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBWRITE process 3791 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBREAD process 3792 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3792 exited. 3 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3790 exited. 2 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3791 exited. 1 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: db-mysql-test3-ha.ripe.net Heartbeat shutdown complete.
logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: received SIGTERM
logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: waiting for 0 messages to be read by write process
logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: sending SIGTERM to write process
logd[3771]: 2008/03/03_13:00:47 info: logd_term_write_action: received SIGTERM
logd[3771]: 2008/03/03_13:00:47 debug: Writing out 0 messages then quitting
logd[3771]: 2008/03/03_13:00:47 info: Exiting write process
-------------- next part --------------
logd[3763]: 2008/03/03_13:00:07 info: logd started with /etc/logd.cf.
logd[3771]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15
logd[3763]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15
heartbeat[3784]: 2008/03/03_13:00:07 info: Enabling logging daemon
heartbeat[3784]: 2008/03/03_13:00:07 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
heartbeat[3784]: 2008/03/03_13:00:07 info: Version 2 support: on
heartbeat[3784]: 2008/03/03_13:00:07 info: **************************
heartbeat[3784]: 2008/03/03_13:00:07 info: Configuration validated. Starting heartbeat 2.1.3
heartbeat[3785]: 2008/03/03_13:00:07 info: heartbeat: version 2.1.3
heartbeat[3785]: 2008/03/03_13:00:07 info: Heartbeat generation: 1203522985
heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2
heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler
heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 17
heartbeat[3785]: 2008/03/03_13:00:07 info: Local status now set to: 'up'
heartbeat[3785]: 2008/03/03_13:00:38 WARN: node db-mysql-test4-ha.ripe.net: is dead
heartbeat[3785]: 2008/03/03_13:00:38 info: Comm_now_up(): updating status to active
heartbeat[3785]: 2008/03/03_13:00:38 info: Local status now set to: 'active'
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/ccm" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/cib" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/lrmd -r" (0,0)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/stonithd" (0,0)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/attrd" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/crmd" (498,496)
heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/mgmtd -v" (0,0)
heartbeat[3925]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/ccm" as uid 498 gid 496 (pid 3925)
heartbeat[3926]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/cib" as uid 498 gid 496 (pid 3926)
heartbeat[3927]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3927)
heartbeat[3928]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/stonithd" as uid 0 gid 0 (pid 3928)
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
heartbeat[3929]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/attrd" as uid 498 gid 496 (pid 3929)
heartbeat[3930]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/crmd" as uid 498 gid 496 (pid 3930)
heartbeat[3931]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3931)
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
ccm[3925]: 2008/03/03_13:00:38 info: Hostname: db-mysql-test3-ha.ripe.net
mgmtd[3931]: 2008/03/03_13:00:38 WARN: lrm_signon: can not initiate connection
mgmtd[3931]: 2008/03/03_13:00:38 info: login to lrm: 0, ret:0
stonithd[3928]: 2008/03/03_13:00:38 info: Signing in with heartbeat.
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler
cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
cib[3926]: 2008/03/03_13:00:38 info: main: Retrieval of a per-action CIB: disabled
cib[3926]: 2008/03/03_13:00:38 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="false" admin_epoch="0" epoch="1" num_updates="1" have_quorum="false" ignore_dtd="false" num_peers="0" cib_feature_revision="2.0" cib-last-written="Mon Mar 3 12:30:41 2008">
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <configuration>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <crm_config>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cluster_property_set id="cib-bootstrap-options">
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <attributes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </attributes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cluster_property_set>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </crm_config>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nodes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <node id="0f24958a-37b1-495a-84fc-911fe61720de" uname="db-mysql-test3-ha.ripe.net" type="normal"/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </nodes>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <resources/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <constraints/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </configuration>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <status/>
cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cib>
cib[3926]: 2008/03/03_13:00:38 info: startCib: CIB Initialization completed successfully
cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: Signing in with Heartbeat
cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: FSA Hostname: db-mysql-test3-ha.ripe.net
cib[3926]: 2008/03/03_13:00:38 info: ccm_connect: Registering with CCM...
cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Activation failed
cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Connection failed 1 times (30 max)
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10
lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12
lrmd[3927]: 2008/03/03_13:00:38 info: Started.
attrd[3929]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net
crmd[3930]: 2008/03/03_13:00:38 info: main: CRM Hg Version: node: 552305612591183b1628baa5bc6e903e0f1e26a3
crmd[3930]: 2008/03/03_13:00:38 info: crmd_init: Starting crmd
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler
crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
stonithd[3928]: 2008/03/03_13:00:38 notice: /usr/lib64/heartbeat/stonithd start up successfully.
stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17
attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de
mgmtd[3931]: 2008/03/03_13:00:39 info: init_crm
ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed
ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed
ccm[3925]: 2008/03/03_13:00:39 info: G_main_add_SignalHandler: Added signal handler for signal 15
heartbeat[3785]: 2008/03/03_13:00:40 info: killing /usr/lib64/heartbeat/mgmtd -v process group 3931 with signal 15
cib[3926]: 2008/03/03_13:00:41 info: ccm_connect: Registering with CCM...
cib[3926]: 2008/03/03_13:00:41 info: cib_init: Starting cib mainloop
cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
cib[3926]: 2008/03/03_13:00:41 info: cib_ccm_msg_callback: PEER: db-mysql-test3-ha.ripe.net
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last)
cib[3926]: 2008/03/03_13:00:42 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on
cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [join]
cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [online]
mgmtd[3931]: 2008/03/03_13:00:43 info: Started.
mgmtd[3931]: 2008/03/03_13:00:43 info: mgmtd is shutting down
cib[3926]: 2008/03/03_13:00:43 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on
crmd[3930]: 2008/03/03_13:00:43 info: do_cib_control: CIB connection established
cib[3926]: 2008/03/03_13:00:43 ERROR: generic plugin load failed
cib[3926]: 2008/03/03_13:00:43 ERROR: cl_compress_field: loading compression module failed
cib[3926]: 2008/03/03_13:00:43 ERROR: uncompress2compress: compressing 6th field failed
heartbeat[3785]: 2008/03/03_13:00:43 info: killing /usr/lib64/heartbeat/crmd process group 3930 with signal 15
crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net
cib[3952]: 2008/03/03_13:00:43 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 330f8d72be5851c99d7ce2e9a8b976a4)
cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last)
crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de
crmd[3930]: 2008/03/03_13:00:44 info: populate_cib_nodes: Requesting the list of configured nodes
crmd[3930]: 2008/03/03_13:00:44 WARN: get_uuid: Could not calculate UUID for db-mysql-test4-ha.ripe.net
crmd[3930]: 2008/03/03_13:00:44 WARN: populate_cib_nodes: Node db-mysql-test4-ha.ripe.net: no uuid found
crmd[3930]: 2008/03/03_13:00:45 notice: populate_cib_nodes: Node: db-mysql-test3-ha.ripe.net (uuid: 0f24958a-37b1-495a-84fc-911fe61720de)
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Connected to Heartbeat
cib[3926]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
cib[3926]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
cib[3926]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 info: do_ccm_control: CCM connection established... waiting for first callback
crmd[3930]: 2008/03/03_13:00:45 info: do_started: Delaying start, CCM (0000000000100000) not connected
crmd[3930]: 2008/03/03_13:00:45 info: crmd_init: Starting crmd's mainloop
crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Requesting shutdown
crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Using default shutdown escalation: 1200000ms
crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online]
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online]
crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed
crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed
crmd[3930]: 2008/03/03_13:00:45 WARN: do_log: [[FSA]] Input I_SHUTDOWN from crm_shutdown() received in state (S_STARTING)
crmd[3930]: 2008/03/03_13:00:45 info: do_state_transition: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
crmd[3930]: 2008/03/03_13:00:45 info: do_shutdown: All subsystems stopped, continuing
crmd[3930]: 2008/03/03_13:00:45 info: do_lrm_control: Disconnected from the LRM
crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Disconnected from Heartbeat
crmd[3930]: 2008/03/03_13:00:45 info: do_cib_control: Disconnecting CIB
crmd[3930]: 2008/03/03_13:00:45 info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
crmd[3930]: 2008/03/03_13:00:45 info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
crmd[3930]: 2008/03/03_13:00:45 info: do_exit: [crmd] stopped (0)
ccm[3925]: 2008/03/03_13:00:45 info: client (pid=3930) removed from ccm
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/attrd process group 3929 with signal 15
cib[3926]: 2008/03/03_13:00:45 WARN: send_via_callback_channel: Cannot find client for token a2d999f1-a039-4587-93db-746028050fec
cib[3926]: 2008/03/03_13:00:45 WARN: do_local_notify: A-Sync reply to 3930 failed: client left before we could send reply
attrd[3929]: 2008/03/03_13:00:45 info: main: Starting mainloop...
attrd[3929]: 2008/03/03_13:00:45 info: attrd_shutdown: Exiting
attrd[3929]: 2008/03/03_13:00:45 info: main: Exiting...
attrd[3929]: 2008/03/03_13:00:45 info: attrd_cib_connection_destroy: Connection to the CIB terminated...
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/stonithd process group 3928 with signal 15
stonithd[3928]: 2008/03/03_13:00:45 notice: /usr/lib64/heartbeat/stonithd normally quit.
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/lrmd -r process group 3927 with signal 15
lrmd[3927]: 2008/03/03_13:00:45 info: lrmd is shutting down
heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/cib process group 3926 with signal 15
cib[3926]: 2008/03/03_13:00:45 info: cib_shutdown: Disconnected 0 clients
cib[3926]: 2008/03/03_13:00:45 info: cib_process_disconnect: All clients disconnected...
cib[3926]: 2008/03/03_13:00:45 info: terminate_ha_connection: initiate_exit: Disconnecting heartbeat
cib[3926]: 2008/03/03_13:00:45 info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting
cib[3926]: 2008/03/03_13:00:45 info: main: Done
ccm[3925]: 2008/03/03_13:00:46 info: client (pid=3926) removed from ccm
heartbeat[3785]: 2008/03/03_13:00:46 info: killing /usr/lib64/heartbeat/ccm process group 3925 with signal 15
ccm[3925]: 2008/03/03_13:00:46 info: received SIGTERM, going to shut down
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBFIFO process 3790 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBWRITE process 3791 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBREAD process 3792 with signal 15
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3792 exited. 3 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3790 exited. 2 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3791 exited. 1 remaining
heartbeat[3785]: 2008/03/03_13:00:47 info: db-mysql-test3-ha.ripe.net Heartbeat shutdown complete.
logd[3771]: 2008/03/03_13:00:47 info: logd_term_write_action: received SIGTERM
logd[3771]: 2008/03/03_13:00:47 info: Exiting write process
More information about the Linux-HA
mailing list