[Linux-HA] Filesystem resource agent will not mount device, "ERROR: unpack_rsc_op: Hard error"

[Martin Oswald]

> 
> Any more log around the errors? Perhaps the output of the RA itself just above 
> the first entry?
> 

Here are the logs from the following situation:
- device not mounted, the GUI says the resource is "not running"
- I start the resource via the GUI

-- snip --
Oct 27 10:16:27 db10 mgmtd: [8515]: info: (delete)xml:<nvpair 
id="mount_ora37_data2_instattr_target_role">
Oct 27 10:16:28 db10 mgmtd: [8515]: info: on_set_target_role:<group 
id="group_ORA37"><primitive id="mount_ora37_data2"><meta_attributes 
id="mount_ora37_data2_meta_attrs"><attributes><nvpair 
id="mount_ora37_data2_metaattr_target_role" name="target_role" 
value="started"/></attributes></meta_attributes></primitive></group>
Oct 27 10:16:28 db10 cib: [24610]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:28 db10 cib: [24610]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:28 db10 cib: [24610]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 10:16:28 db10 cib: [24610]: info: write_cib_contents: Wrote 
version 0.710.7 of the CIB to disk (digest: 
7afd7cf461dac9f3fd14a53c6cb5d540)
Oct 27 10:16:28 db10 cib: [24610]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:28 db10 cib: [24610]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 10:16:29 db10 cib: [24615]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:29 db10 haclient: on_event:evt:cib_changed
Oct 27 10:16:29 db10 haclient: on_event:evt:cib_changed
Oct 27 10:16:29 db10 cib: [24615]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:29 db10 cib: [24615]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 10:16:29 db10 cib: [24615]: info: write_cib_contents: Wrote 
version 0.711.1 of the CIB to disk (digest: 
42b4adad823c89216673776e6389ca88)
Oct 27 10:16:29 db10 cib: [24615]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 10:16:29 db10 cib: [24615]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 10:16:29 db10 mgmtd: [8515]: ERROR: unpack_rsc_op: Hard error: 
mount_ora37_data2_monitor_0 failed with rc=2.
Oct 27 10:16:29 db10 mgmtd: [8515]: ERROR: unpack_rsc_op:   Preventing 
mount_ora37_data2 from re-starting anywhere in the cluster
-- snip --

After the start, the device is not mounted but the gui adds an attribute 
  "target_role started" as can be seen in the logfile.



I have another couple of logs, based on the following situation:
- the resource had been started correctly via the ocf Filesystem 
Resource Agent (as shown in my last post)
- the gui showed although that the resource was "not running"
- I triggered a "cleanup resource" via the gui

-- snip --
Oct 27 08:40:32 db10 crmd: [8514]: info: do_lrm_invoke: Removing 
resource mount_ora37_data2 from the LRM
Oct 27 08:40:32 db10 crmd: [8514]: info: send_direct_ack: ACK'ing 
resource op mount_ora37_data2_delete_0 from mgmtd-8515: 
lrm_invoke-lrmd-1288161632-5
Oct 27 08:40:32 db10 mgmtd: [8515]: info: Delete fail-count for 
mount_ora37_data2 from db10
Oct 27 08:40:32 db10 crmd: [8514]: info: do_lrm_invoke: Forcing a local 
LRM refresh
Oct 27 08:40:33 db10 lrmd: [8511]: info: rsc:mount_ora37_data2: monitor
Oct 27 08:40:33 db10 crmd: [8514]: info: do_lrm_rsc_op: Performing 
op=mount_ora37_data2_monitor_0 
key=11:28:95b86501-375f-4df4-9db6-135ae0d53f7a)
Oct 27 08:40:33 db10 crmd: [8514]: info: process_lrm_event: LRM 
operation mount_ora37_data2_monitor_0 (call=43, rc=0) complete
Oct 27 08:40:33 db10 cib: [8510]: info: apply_xml_diff: Digest 
mis-match: expected f1412af1da404176a1dc7c9d63228668, calculated 
458e2a6f38667f7ad95a7b26b3026a81
Oct 27 08:40:33 db10 cib: [8510]: info: cib_process_diff: Diff 0.710.1 
-> 0.710.2 not applied to 0.710.1: Failed application of a global 
update.  Requesting full refresh.
Oct 27 08:40:33 db10 cib: [8510]: info: cib_process_diff: Requesting 
re-sync from peer: Failed application of a global update.  Requesting 
full refresh.
Oct 27 08:40:33 db10 cib: [8510]: WARN: do_cib_notify: cib_apply_diff of 
<diff > FAILED: Application of an update diff failed, requesting a full 
refresh
Oct 27 08:40:33 db10 cib: [8510]: WARN: cib_process_request: 
cib_apply_diff operation failed: Application of an update diff failed, 
requesting a full refresh
Oct 27 08:40:33 db10 cib: [17546]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:33 db10 cib: [17546]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:33 db10 cib: [17546]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:33 db10 cib: [17546]: info: write_cib_contents: Wrote 
version 0.710.1 of the CIB to disk (digest: 
bf505fb2ace2c69dc74b575efb34f725)
Oct 27 08:40:33 db10 cib: [17546]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:33 db10 cib: [17546]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:33 db10 cib: [8510]: WARN: cib_process_diff: Not applying 
diff 0.710.2 -> 0.710.3 (sync in progress)
Oct 27 08:40:33 db10 cib: [8510]: WARN: do_cib_notify: cib_apply_diff of 
<diff > FAILED: Application of an update diff failed, requesting a full 
refresh
Oct 27 08:40:33 db10 cib: [8510]: WARN: cib_process_request: 
cib_apply_diff operation failed: Application of an update diff failed, 
requesting a full refresh
Oct 27 08:40:34 db10 crmd: [8514]: info: do_lrm_rsc_op: Performing 
op=mount_ora37_data2_stop_0 key=52:29:95b86501-375f-4df4-9db6-135ae0d53f7a)
Oct 27 08:40:34 db10 lrmd: [8511]: info: rsc:mount_ora37_data2: stop
Oct 27 08:40:34 db10 cib: [8510]: info: cib_replace_notify: Replaced: 
0.710.1 -> 0.710.3 from <null>
Oct 27 08:40:34 db10 crmd: [8514]: info: populate_cib_nodes: Requesting 
the list of configured nodes
Oct 27 08:40:34 db10 cib: [17578]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:34 db10 Filesystem[17547]: [17577]: INFO: Running stop for 
/dev/vx/dsk/dg_ora37/data2_ora37 on /data2/ora37
Oct 27 08:40:34 db10 Filesystem[17547]: [17588]: INFO: Trying to unmount 
/data2/ora37
Oct 27 08:40:34 db10 cib: [17578]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:34 db10 cib: [17578]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:34 db10 cib: [17578]: info: write_cib_contents: Wrote 
version 0.710.4 of the CIB to disk (digest: 
73fb050ab7d1c4393346c3a984fd13c3)
Oct 27 08:40:34 db10 cib: [17578]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:34 db10 cib: [17578]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:34 db10 Filesystem[17547]: [17591]: INFO: unmounted 
/data2/ora37 successfully
Oct 27 08:40:35 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:35 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:35 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:35 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:35 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:35 db10 cib: [17597]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:35 db10 cib: [17597]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:35 db10 cib: [17597]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:35 db10 cib: [17597]: info: write_cib_contents: Wrote 
version 0.710.5 of the CIB to disk (digest: 
c2fb1ed565bc849a3b7c187eb389f141)
Oct 27 08:40:35 db10 cib: [17597]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:35 db10 cib: [17597]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:35 db10 mgmtd: [8515]: ERROR: unpack_rsc_op: Hard error: 
mount_ora37_data2_monitor_0 failed with rc=2.
Oct 27 08:40:35 db10 mgmtd: [8515]: ERROR: unpack_rsc_op:   Preventing 
mount_ora37_data2 from re-starting anywhere in the cluster
Oct 27 08:40:35 db10 crmd: [8514]: notice: populate_cib_nodes: Node: 
db10 (uuid: 93359856-99d6-4e18-8892-482959b53983)
Oct 27 08:40:35 db10 crmd: [8514]: notice: populate_cib_nodes: Node: 
db09 (uuid: 67c511c2-6adb-49af-b18f-502ee2e0eb70)
Oct 27 08:40:35 db10 crmd: [8514]: info: process_lrm_event: LRM 
operation mount_ora37_data2_stop_0 (call=44, rc=0) complete
Oct 27 08:40:36 db10 haclient: on_event:evt:cib_changed
Oct 27 08:40:36 db10 cib: [17604]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:36 db10 cib: [17604]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:36 db10 cib: [17604]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:36 db10 cib: [17604]: info: write_cib_contents: Wrote 
version 0.710.6 of the CIB to disk (digest: 
fdcf0df98bcccb10c4aa19db68bd6680)
Oct 27 08:40:36 db10 cib: [17604]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
/var/lib/heartbeat/crm/cib.xml.sig)
Oct 27 08:40:36 db10 cib: [17604]: info: retrieveCib: Reading cluster 
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
/var/lib/heartbeat/crm/cib.xml.sig.last)
Oct 27 08:40:37 db10 mgmtd: [8515]: ERROR: unpack_rsc_op: Hard error: 
mount_ora37_data2_monitor_0 failed with rc=2.
Oct 27 08:40:37 db10 mgmtd: [8515]: ERROR: unpack_rsc_op:   Preventing 
mount_ora37_data2 from re-starting anywhere in the cluster
-- snip --



Should I use the cli instead of the gui, whould that be a safer solution?