[Linux-HA] Couldn't mount filesystem /dev/drbd0 on /shared

Dejan Muhamedagic dejanmm at fastmail.fm
Fri Oct 5 13:13:12 MDT 2007


Hi,

On Fri, Oct 05, 2007 at 12:51:36PM -0500, Alejandro Rios Peña wrote:
> Hello.
> 
> DRBD partition can't be properly configured by heartbeat on master node, 
> causing filesystem mounting to fail.
> 
> How can I tell heartbeat to do a  ''drbdadm primary all'' or 
> something?

You can't and you shouldn't need to.

> Doing it by hand and mounting the filesystem works without 
> problem.
> 
> Maybe I configured something wrong, could somebody help me pointing it out?
> 
> I'm using:
> heartbeat 2.0.7-2

You could upgrade to 2.1.2.

> drbd 0.7.21
> drbdlinks 1.07
> 
> ---------------------------------------------------------------
> "crm_mon -n -1 -r "  shows:
> 
> ============
> Last updated: Fri Oct  5 12:26:57 2007
> Current DC: hal-9000 (9f275006-1303-4c16-b5c6-3f52c4ed0865)
> 2 Nodes configured.
> 4 Resources configured.
> ============
> 
> Node: hal-9000 (9f275006-1303-4c16-b5c6-3f52c4ed0865): online
>        r0:1    (heartbeat::ocf:drbd)
> Node: flexo (d66a3fcc-da10-4807-afec-950187f4c084): online
>        IPaddr_192_168_0_101    (heartbeat::ocf:IPaddr)
>        samba_3 (lsb:samba)
>        r0:0    (heartbeat::ocf:drbd)
> 
> Inactive resources:
> fs0     (heartbeat::ocf:Filesystem):    Stopped
> ---------------------------------------------------------------
> 
> Here's relevant part of syslog (extended debug output is at 
> http://paste.debian.net/38881):
> 
> [snip]

>From where is this log (looks as if it's from flexo)? I don't see
any problems here (beware: I don't have much experience with
drbd). The other log (from the DC node) could be more
interesting.

Thanks,

Dejan


> Oct  5 11:45:46 localhost crmd: [7163]: info: do_lrm_rsc_op:lrm.c 
> Performing op monitor on r0:1 (interval=0ms, 
> key=3:190e3e58-5139-4b1f-a9a1-6faa888e5adc)
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 156692 bytes (was 154400)
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 1760, call:2): 0.17.585 -> 0.17.586 (ok)
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 156588 bytes (was 156692)
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 7163, call:28): 0.17.586 -> 0.17.587 (ok)
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7520]: DEBUG: r0: Calling 
> /sbin/drbdadm -c /etc/drbd.conf state r0
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7524]: DEBUG: r0: Exit code 0
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7525]: DEBUG: r0: Command output: 
> Secondary/Secondary
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7533]: DEBUG: r0: Calling 
> /sbin/drbdadm -c /etc/drbd.conf cstate r0
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 158880 bytes (was 156588)
> #
> Oct  5 11:45:46 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 1715, call:36): 0.17.587 -> 0.17.588 (ok)
> #
> Oct  5 11:45:46 localhost cib: [7539]: info: write_cib_contents:io.c 
> Wrote version 0.17.588 of the CIB to disk (digest: 
> d794dd901148deb7f0af574270f73c00)
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7544]: DEBUG: r0: Exit code 0
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7545]: DEBUG: r0: Command output: 
> Connected
> #
> Oct  5 11:45:46 localhost drbd[7513]: [7546]: DEBUG: r0 status: 
> Secondary/Secondary Secondary Secondary Connected
> #
> Oct  5 11:45:46 localhost crmd: [7163]: info: process_lrm_event:lrm.c 
> LRM operation (13) monitor_0 on r0:1 complete
> #
> Oct  5 11:45:48 localhost crmd: [7163]: info: do_lrm_rsc_op:lrm.c 
> Performing op stop on r0:1 (interval=0ms, 
> key=4:190e3e58-5139-4b1f-a9a1-6faa888e5adc)
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7554]: DEBUG: r0: Calling 
> /sbin/drbdadm -c /etc/drbd.conf state r0
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7558]: DEBUG: r0: Exit code 0
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7559]: DEBUG: r0: Command output: 
> Secondary/Secondary
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7567]: DEBUG: r0: Calling 
> /sbin/drbdadm -c /etc/drbd.conf cstate r0
> #
> Oct  5 11:45:48 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 158776 bytes (was 158880)
> #
> Oct  5 11:45:48 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 7163, call:29): 0.17.588 -> 0.17.589 (ok)
> #
> Oct  5 11:45:48 localhost cib: [7572]: info: write_cib_contents:io.c 
> Wrote version 0.17.589 of the CIB to disk (digest: 
> 18831e94c442d16a54c1a9659314d39e)
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7584]: DEBUG: r0: Exit code 0
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7585]: DEBUG: r0: Command output: 
> Connected
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7586]: DEBUG: r0 status: 
> Secondary/Secondary Secondary Secondary Connected
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7587]: DEBUG: r0: Calling 
> /sbin/drbdadm -c /etc/drbd.conf down r0
> #
> Oct  5 11:45:48 localhost kernel: drbd0: drbdsetup [7590]: cstate 
> Connected --> Unconnected
> #
> Oct  5 11:45:48 localhost kernel: drbd0: drbd0_receiver [7508]: cstate 
> Unconnected --> BrokenPipe
> #
> Oct  5 11:45:48 localhost kernel: drbd0: short read expecting header on 
> sock: r=-512
> #
> Oct  5 11:45:48 localhost kernel: drbd0: worker terminated
> #
> Oct  5 11:45:48 localhost kernel: drbd0: asender terminated
> #
> Oct  5 11:45:48 localhost kernel: drbd0: drbd0_receiver [7508]: cstate 
> BrokenPipe --> StandAlone
> #
> Oct  5 11:45:48 localhost kernel: drbd0: Connection lost.
> #
> Oct  5 11:45:48 localhost kernel: drbd0: receiver terminated
> #
> Oct  5 11:45:48 localhost kernel: drbd0: drbdsetup [7590]: cstate 
> StandAlone --> StandAlone
> #
> Oct  5 11:45:48 localhost kernel: drbd0: drbdsetup [7590]: cstate 
> StandAlone --> Unconfigured
> #
> Oct  5 11:45:48 localhost kernel: drbd0: worker terminated
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7592]: DEBUG: r0: Exit code 0
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7593]: DEBUG: r0: Command output:
> #
> Oct  5 11:45:48 localhost lrmd: [7160]: info: RA output: (r0:1:stop:stdout)
> #
> Oct  5 11:45:48 localhost drbd[7547]: [7594]: DEBUG: r0 stop: drbdadm 
> down succeeded.
> #
> Oct  5 11:45:48 localhost crmd: [7163]: info: process_lrm_event:lrm.c 
> LRM operation (14) stop_0 on r0:1 complete
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 161068 bytes (was 158776)
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 1715, call:38): 0.17.589 -> 0.17.590 (ok)
> #
> Oct  5 11:45:49 localhost cib: [7598]: info: write_cib_contents:io.c 
> Wrote version 0.17.590 of the CIB to disk (digest: 
> 0b9ae0f4610a1af2e1c8a843608192e5)
> #
> Oct  5 11:45:49 localhost crmd: [7163]: info: do_lrm_rsc_op:lrm.c 
> Performing op start on fs0 (interval=0ms, 
> key=5:190e3e58-5139-4b1f-a9a1-6faa888e5adc)
> #
> Oct  5 11:45:49 localhost Filesystem[7599]: [7605]: INFO: Running start 
> for /dev/drbd0 on /shared
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 163360 bytes (was 161068)
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 7163, call:30): 0.17.590 -> 0.17.591 (ok)
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 165652 bytes (was 163360)
> #
> Oct  5 11:45:49 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 1715, call:40): 0.17.591 -> 0.17.592 (ok)
> #
> Oct  5 11:45:49 localhost cib: [7611]: info: write_cib_contents:io.c 
> Wrote version 0.17.592 of the CIB to disk (digest: 
> 7750e155947fcba58272b13b572530cf)
> #
> Oct  5 11:45:49 localhost Filesystem[7599]: [7615]: ERROR: Couldn't 
> mount filesystem /dev/drbd0 on /shared
> #
> Oct  5 11:45:49 localhost crmd: [7163]: WARN: process_lrm_event:lrm.c 
> LRM operation (15) start_0 on fs0 Error: (1) unknown error
> #
> Oct  5 11:45:49 localhost lrmd: [7160]: info: RA output: 
> (fs0:start:stderr) BLKFLSBUF: Inappropriate ioctl for device mount: 
> block device /dev/drbd0 is write-protected, mounting read-only mount: 
> /dev/drbd0 already mounted or /shared busy
> #
> Oct  5 11:45:51 localhost crmd: [7163]: info: do_lrm_rsc_op:lrm.c 
> Performing op stop on fs0 (interval=0ms, 
> key=6:190e3e58-5139-4b1f-a9a1-6faa888e5adc)
> #
> Oct  5 11:45:51 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 167944 bytes (was 165652)
> #
> Oct  5 11:45:51 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 7163, call:31): 0.17.592 -> 0.17.593 (ok)
> #
> Oct  5 11:45:51 localhost cib: [7619]: info: write_cib_contents:io.c 
> Wrote version 0.17.593 of the CIB to disk (digest: 
> 69c7df548815877a47420fece9960bf1)
> #
> Oct  5 11:45:51 localhost Filesystem[7616]: [7623]: INFO: Running stop 
> for /dev/drbd0 on /shared
> #
> Oct  5 11:45:51 localhost lrmd: [7160]: info: RA output: 
> (fs0:stop:stderr) BLKFLSBUF: Inappropriate ioctl for device
> #
> Oct  5 11:45:51 localhost crmd: [7163]: info: process_lrm_event:lrm.c 
> LRM operation (16) stop_0 on fs0 complete
> #
> Oct  5 11:45:53 localhost cib: [7159]: info: activateCibXml:io.c CIB 
> size is 170236 bytes (was 167944)
> #
> Oct  5 11:45:53 localhost cib: [7159]: info: cib_diff_notify:notify.c 
> Update (client: 7163, call:32): 0.17.593 -> 0.17.594 (ok)
> #
> Oct  5 11:45:53 localhost cib: [7633]: info: write_cib_contents:io.c 
> Wrote version 0.17.594 of the CIB to disk (digest: 
> 885fea462a3e6c8e73f0e1d62973f572)
> 
> 
> ---------------------------------------------------------------
> Here's my cib.xml file:
> <cib admin_epoch="0" have_quorum="true" num_peers="2" 
> cib_feature_revision="1.3" ccm_transition="2" generated="true" 
> dc_uuid="9f275006-1303-4c16-b5c6-3f52c4ed0865" epoch="17" 
> num_updates="594" cib-last-written="Fri Oct  5 11:45:53 2007">
>   <configuration>
>     <crm_config>
>       <cluster_property_set id="cib-bootstrap-options">
>         <attributes>
>           <nvpair id="cib-bootstrap-options-symmetric_cluster" 
> name="symmetric_cluster" value="true"/>
>           <nvpair id="cib-bootstrap-options-no_quorum_policy" 
> name="no_quorum_policy" value="stop"/>
>           <nvpair 
> id="cib-bootstrap-options-default_resource_stickiness" 
> name="default_resource_stickiness" value="0"/>
>           <nvpair 
> id="cib-bootstrap-options-default_resource_failure_stickiness" 
> name="default_resource_failure_stickiness" value="0"/>
>           <nvpair id="cib-bootstrap-options-stonith_enabled" 
> name="stonith_enabled" value="false"/>
>           <nvpair id="cib-bootstrap-options-stonith_action" 
> name="stonith_action" value="reboot"/>
>           <nvpair id="cib-bootstrap-options-stop_orphan_resources" 
> name="stop_orphan_resources" value="true"/>
>           <nvpair id="cib-bootstrap-options-stop_orphan_actions" 
> name="stop_orphan_actions" value="true"/>
>           <nvpair id="cib-bootstrap-options-remove_after_stop" 
> name="remove_after_stop" value="false"/>
>           <nvpair id="cib-bootstrap-options-short_resource_names" 
> name="short_resource_names" value="true"/>
>           <nvpair id="cib-bootstrap-options-transition_idle_timeout" 
> name="transition_idle_timeout" value="5min"/>
>           <nvpair id="cib-bootstrap-options-default_action_timeout" 
> name="default_action_timeout" value="5s"/>
>           <nvpair id="cib-bootstrap-options-is_managed_default" 
> name="is_managed_default" value="true"/>
>         </attributes>
>       </cluster_property_set>
>     </crm_config>
>     <nodes>
>       <node uname="hal-9000" type="normal" 
> id="9f275006-1303-4c16-b5c6-3f52c4ed0865">
>         <instance_attributes 
> id="master-9f275006-1303-4c16-b5c6-3f52c4ed0865">
>           <attributes/>
>         </instance_attributes>
>       </node>
>       <node uname="flexo" type="normal" 
> id="d66a3fcc-da10-4807-afec-950187f4c084">
>         <instance_attributes 
> id="master-d66a3fcc-da10-4807-afec-950187f4c084">
>           <attributes/>
>         </instance_attributes>
>       </node>
>     </nodes>
>     <resources>
>       <group id="group_1">
>         <primitive class="ocf" id="IPaddr_192_168_0_101" 
> provider="heartbeat" type="IPaddr">
>           <instance_attributes id="IPaddr_192_168_0_101_inst_attr">
>             <attributes>
>               <nvpair id="IPaddr_192_168_0_101_attr_0" name="ip" 
> value="192.168.0.101"/>
>             </attributes>
>           </instance_attributes>
>         </primitive>
>         <primitive class="lsb" id="samba_3" provider="heartbeat" 
> type="samba">
>           <operations>
>             <op id="samba_1_start" name="stop" timeout="5"/>
>             <op id="samba_2_stop" name="start" timeout="5s"/>
>           </operations>
>         </primitive>
>       </group>
>       <master_slave id="ms-r0">
>         <meta_attributes id="ma-ms-r0">
>           <attributes>
>             <nvpair id="ma-ms-r0-1" name="clone_max" value="2"/>
>             <nvpair id="ma-ms-r0-2" name="clone_node_max" value="1"/>
>             <nvpair id="ma-ms-r0-3" name="master_max" value="1"/>
>             <nvpair id="ma-ms-r0-4" name="master_node_max" value="1"/>
>             <nvpair id="ma-ms-r0-5" name="notify" value="yes"/>
>             <nvpair id="ma-ms-r0-6" name="globally_unique" value="false"/>
>             <nvpair id="ma-ms-r0-7" name="target_role" value="#default"/>
>           </attributes>
>         </meta_attributes>
>         <primitive id="r0" class="ocf" provider="heartbeat" type="drbd">
>           <instance_attributes id="ia-r0">
>             <attributes>
>               <nvpair id="ia-r0-1" name="drbd_resource" value="r0"/>
>             </attributes>
>           </instance_attributes>
>         </primitive>
>       </master_slave>
>       <primitive class="ocf" provider="heartbeat" type="Filesystem" 
> id="fs0">
>         <meta_attributes id="ma-fs0">
>           <attributes>
>             <nvpair name="target_role" id="ma-fs0-1" value="#default"/>
>           </attributes>
>         </meta_attributes>
>         <instance_attributes id="ia-fs0">
>           <attributes>
>             <nvpair id="ia-fs0-1" name="fstype" value="ext3"/>
>             <nvpair id="ia-fs0-2" name="directory" value="/shared"/>
>             <nvpair id="ia-fs0-3" name="device" value="/dev/drbd0"/>
>           </attributes>
>         </instance_attributes>
>       </primitive>
>     </resources>
>     <constraints>
>       <rsc_location id="rsc_location_r0" rsc="ms-r0">
>         <rule id="prefered_location_r0" score="100">
>           <expression attribute="#uname" id="prefered_location_r0_expr" 
> operation="eq" value="flexo"/>
>         </rule>
>       </rsc_location>
>       <rsc_order id="r0_before_fs0" from="fs0" action="start" to="ms-r0"/>
>       <rsc_colocation id="fs0_on_r0" to="ms-r0" from="fs0" 
> score="INFINITY"/>
>       <rsc_location id="rsc_location_group_1" rsc="group_1">
>         <rule id="prefered_location_group_1" score="100">
>           <expression attribute="#uname" 
> id="prefered_location_group_1_expr" operation="eq" value="flexo"/>
>         </rule>
>       </rsc_location>
>     </constraints>
>   </configuration>
> </cib>
> ---------------------------------------------------------------
> drbd.conf:
> 
> resource r0 {
>  protocol C;
>  incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; 
> halt -f";
> 
>  startup {
>    degr-wfc-timeout 120;    # 2 minutes.
>  }
> 
>  disk {
>    on-io-error   detach;
>  }
> 
>  net {
>    # TODO: Should these timeouts be relative to some heartbeat settings?
>    # timeout       60;    #  6 seconds  (unit = 0.1 seconds)
>    # connect-int   10;    # 10 seconds  (unit = 1 second)
>    # ping-int      10;    # 10 seconds  (unit = 1 second)
>    on-disconnect reconnect;
>  }
> 
>  syncer {
>    rate 10M;
>    group 1;
>    al-extents 257;
>  }
> 
>  on flexo {
>    device     /dev/drbd0;
>    disk       /dev/sda4;
>    address    192.168.0.105:7788;
>    meta-disk  internal;
>  }
> 
>  on hal-9000 {
>    device    /dev/drbd0;
>    disk      /dev/sda7;
>    address   192.168.0.100:7788;
>    meta-disk internal;
>  }
> }
> 
> 
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems


More information about the Linux-HA mailing list