[Linux-HA] Xen EVMS-HA mini-howto and locking mechanism remark

Thu Jan 4 06:55:08 MST 2007

On 1/4/07, Tijl Van den Broeck <subspawn at gmail.com> wrote:
> Vini,
>
> You have a good point, this should be on a site somewhere. I took some
> time to make it a little bit more readable and added a page to the
> Xensource wiki:
> http://wiki.xensource.com/xenwiki/EVMS-HAwSAN-SLES10
>
> Perhaps it can be linked from the Linux-HA site and from the EVMS
> site? In order to keep the documentation centralised on one location.

the LHA site is built from wiki.linux-ha.org... feel free to add it
somewhere appropriate

>
> greetings
>
> Tijl Van den Broeck
>
>
> On 1/3/07, vini.bill at gmail.com <vini.bill at gmail.com> wrote:
> > Great! I was thinking about implementing somthing like that here at the
> > company, now I got some guidance.
> >
> > Thank you very much.
> >
> > p.s.: Isn't  that worth being the website?
> >
> > On 1/3/07, Tijl Van den Broeck <subspawn at gmail.com> wrote:
> > >
> > > The following is a mini-howto, skip to the end of this mail for some
> > > EVMS specific questions/problems.
> > >
> > > So far I've not seen anybody yet on the Xen list talking about a
> > > succesfull setup of EVMS with heartbeat 2. Setting up EVMS-HA is not
> > > that terribly difficul.
> > >
> > > If you're not experienced with heartbeat 2, perhaps read some basic
> > > material on it at www.linux-ha.org especially on the administration
> > > tools. Now you've 2 direct options you'd want to use EVMS-HA for:
> > > 2-node, DRDB-sync + EVMS-HA resource failover (possibly with HA-Xen
> > > scripts): I haven't tested this yet, but it should be possible as far
> > > as I've read.
> > > n-node, iSCSI/FC-SAN + EVMS-HA; my current testing environment.
> > >
> > > Notice there's a big difference, afaik when using DRDB you must
> > > actually failover your resources to the other node. In a SAN based env
> > > this is not the case as all nodes constantly have full I/O access,
> > > which is why EVMS-HA should be usefull (at least I thought, read my
> > > remarks on that at the end of this mail).
> > >
> > > I installed plain SLES-10 copies (NOT using EVMS at installation
> > > time), booted into Xen kernel in which all is configured. My intention
> > > is/was only to use EVMS for Xen DomU volume management, not for local
> > > disk management. Just for the ease of administration to keep those
> > > strictly separated.
> > >
> > > So to begin, make sure you've got all I/O access on all nodes to the
> > > same resources (preferably with a unique name on all nodes for the
> > > ease of administration, but this isn't necessary as EVMS can fix
> > > this).
> > >
> > > As local disk configuration is not EVMS aware, exclude the local disks
> > > from EVMS management (in this case cciss mirrored disks, but this
> > > could well be your hda/sda disks).
> > > /etc/evms.conf:
> > > sysfs_devices {
> > > include = [ * ]
> > > exclude =  [ cciss* ]
> > > ...
> > > }
> > > Make sure admin_mode is off on all nodes, admin_mode has little to do
> > > with admininistration but more with recovery/maintenance if things
> > > have gone bad. More on this in the evms user guide:
> > > http://evms.sourceforge.net/user_guide/
> > >
> > > Setup heartbeat on both clusters to enable EVMS cluster awareness,
> > > next to the usual ip and node configuration (which can be done using
> > > yast in SLES10), just adding 2 lines to /etc/ha.d/ha.cf will do:
> > > respawn root /sbin/evmsd
> > > apiauth evms uid=hacluster,root
> > >
> > > Start the cluster node by node:
> > > /etc/init.d/hearbeat start
> > >
> > > Make sure both sync and come up (keep an eye on /var/log/messages).
> > > You can use the crmadmin tool to query the state of master and nodes.
> > > Also usefull is cl_status for checking link & daemon status.
> > > Note: If you're using bonding, you can run into some trouble here. Use
> > > unicast for node sync, not multicast as somehow the Xen software
> > > bridge doesn't fully cope with that yet (at least I didn't get it to
> > > work, perhaps someone who did?).
> > >
> > > When all nodes are up and running, start evmsgui (or evmsn, whichever
> > > you prefer) on one of the nodes. If you click the settings menu and
> > > find the option "Node administered" enabled, congratulations you've
> > > got an cluster aware EVMS). Be sure to know some essentials of EVMS
> > > (it's a little different from plain LVM2).
> > >
> > > Create a Containter with the Cluster Segm Manager, select your
> > > attached SAN storage objects (could be named sdb, sdc ...), choose
> > > whichever node name, type shared storage and name the container
> > > "c_sanstorage" for example.
> > >
> > > You can patch through the SAN disks to EVMS volumes (see the disk list
> > > in available objects). Don't do that, as these volumes will be fixed
> > > in size (as they were originally presented from the SAN), instead use
> > > EVMS for storage management. For this, create another container, this
> > > time a LVM2 Region Manager, in which you store all the objects from
> > > the CCM c_sanstorage (objects will have a name like c_sanstorage/sdb,
> > > c_sanstorage/sdc, ...). Choose the extent size at will and name it
> > > (vgsan for example).
> > >
> > > Go into the Region tab, and create a region from the LVM2 freespace
> > > named and sized at your will, for example domu01stor, domu02stor, ...
> > >
> > > Save the configuration, all settings will now be applied and you will
> > > find your correctly sized+named volumes in /dev/evms/c_sanstorage/
> > >
> > > Now partition (if wanted) the evms disks like you used to
> > > (fdisk),format,place domU's on it and launch.
> > >
> > >
> > >
> > >
> > >
> > > As for the problems & remarks I've seen with this setup.
> > > For the EVMS configuration to be updated at all nodes, you have to
> > > select each node in "node administered" and select save for each node
> > > (as only then the correct devices will be created on the node).
> > >
> > > This could be a structural question, but... being cluster aware,
> > > shouldn't the EVMS-HA combination (with CCM) provide locking on
> > > volumes created beneath CCM? It is perfectly possible for me to
> > > corrupt data on an EVMS volume on node 2 which volume is also mounted
> > > on node 1. I expected some kind of locking to step up:
> > > dom0_2# mount /dev/evms/c_sanstorage/domu01stor /mnt
> > > failure: volume domu01stor is already mounted on node dom0_1
> > >
> > > Or something amongst those lines. My initial thoughts: it had to do
> > > with my CCM being "shared". But when creating a CCM as "private", the
> > > same issues were possible! And even more remarkable, if I create a CCM
> > > as private on node dom0_1 and I launch evmsgui om dom0_2 it recognizes
> > > the CCM as private owned by dom0_2 ?!? This strikes me as very odd.
> > > Are these problems due to faults in my procedure, if so let me know
> > > please, or are they of a more structural nature (or perhaps SLES10
> > > specific)?
> > > They are kind of essential with Xen domains, you wouldn't want to boot
> > > the same domain twice (one copy on dom0_1 and another running on
> > > dom0_2) as data corruption is garanteed.
> > >
> > > That is why this mail is crossposted at all 3 lists:
> > > information for xen-users
> > > technical questions for evms and linux-ha.
> > >
> > > greetings
> > >
> > > Tijl Van den Broeck
> > > _______________________________________________
> > > Linux-HA mailing list
> > > Linux-HA at lists.linux-ha.org
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> > >
> >
> >
> >
> > --
> > ... Vinicius Menezes ...
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA at lists.linux-ha.org
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>