[Linux-HA] colocating resources on failed restart
:CRM-Stable-4a0d4e40eeb0
Alex and Gill Strachan
asgks at yahoo.com
Mon Nov 6 08:37:47 MST 2006
okay, i'll send tomorrow; going to catch a little sleep.
----- Original Message ----
From: Andrew Beekhof <beekhof at gmail.com>
To: General Linux-HA mailing list <linux-ha at lists.linux-ha.org>
Sent: Tuesday, 7 November, 2006 1:23:36 AM
Subject: Re: Re: Re: [Linux-HA] colocating resources on failed restart :CRM-Stable-4a0d4e40eeb0
On 11/6/06, Alex and Gill Strachan <asgks at yahoo.com> wrote:
> The resource was managed at the time. I had previously made it unmanaged; I think it may be due to the monitoring continuing when unmanaged maybe ?
possible.
can you send attachments? the whole log might be helpful (both from
the DC and from the node where aims is being restarted)
>
> (If I knew how to configure Yahoo for inline replies - I WOULD :-) It must be an option )
>
> On 11/6/06, Alex and Gill Strachan <asgks at yahoo.com> wrote:
> > Spoke to soon. Now the resource_sinfds3A_aims
> > starts
> > stop
> > starts
> > stop
> > starts ... (very repititive!)
>
> This was when resource_sinfds3A_aims was unmanaged
>
> >
> > It looks like it's going to be a late night, already spent all day on this (AUS timezone) and I have run out of time to have a working hb2 solution. This is very disappointing :-(
> >
> > There are no report failcounts, crm_verify shows nothing wrong.
> >
> > Hopefully in the few hours you guys can help me.
> >
> > Node: sinfids3b1 (338afa76-8997-4d66-8381-fc36ec4b456b): online
> > resource_sinfids3B_vip (heartbeat::ocf:IPaddr)
> > Node: sinfids3a2 (ec74bd17-2016-4d32-a694-0f6983121cd9): online
> > resource_sinfids3A_aims (lsb:aims)
> > resource_sinfids3_vip (heartbeat::ocf:IPaddr)
> > resource_sinfids3A_drbd (heartbeat:drbddisk)
> > resource_sinfids3A_fs (heartbeat::ocf:Filesystem)
> > resource_sinfids3A_vip (heartbeat::ocf:IPaddr)
> > resource_sinfids3A_oracle (heartbeat::ocf:oracle)
> > resource_sinfids3A_smb (lsb:smb)
> > resource_sinfids3A_oralsnr (heartbeat::ocf:oralsnr)
> > Node: sinfids3a1 (b757aece-0e47-41e5-92b7-6a80b4f3eea7): online
> >
> >
> > [root at sinfids3b1 ~]# cibadmin -Q |egrep "managed"
> > <nvpair id="cib-bootstrap-options-is-managed-default" name="is-managed-default" value="true"/>
> > <group id="group_sinfids3" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > <group id="group_sinfids3A" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > <nvpair id="resource_sinfids3A_aims-is_managed" name="is_managed" value="true"/>
> > <group id="group_sinfids3B" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> >
> > I removed the nvpair attribute for the resource, hoping that maybe it was causing some confusion.
> >
> > [root at sinfids3b1 ~]# cibadmin -Q |egrep "managed"
> > <nvpair id="cib-bootstrap-options-is-managed-default" name="is-managed-default" value="true"/>
> > <group id="group_sinfids3" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > <group id="group_sinfids3A" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > <group id="group_sinfids3B" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > [root at sinfids3b1 ~]# crm_verify -L -VVV
> > crm_verify[27339]: 2006/11/06_21:50:40 info: main: =#=#=#=#= Getting XML =#=#=#=#=
> > crm_verify[27339]: 2006/11/06_21:50:40 info: main: Reading XML from: live cluster
> > crm_verify[27339]: 2006/11/06_21:50:40 notice: main: Required feature set: 1.1
> > crm_verify[27339]: 2006/11/06_21:50:40 notice: unpack_config: On loss of CCM Quorum: Ignore
> > crm_verify[27339]: 2006/11/06_21:50:40 info: determine_online_status: Node sinfids3b1 is online
> > crm_verify[27339]: 2006/11/06_21:50:40 info: determine_online_status: Node sinfids3a1 is online
> > crm_verify[27339]: 2006/11/06_21:50:40 info: determine_online_status: Node sinfids3a2 is online
> > [root at sinfids3b1 ~]# /usr/lib/heartbeat/ptest -L -VVVVVVVVVVVVVVV 2>&1 | egrep assign
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3B_vip, Node[0] sinfids3b1: 10100
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3B_vip, Node[1] sinfids3a1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3B_vip, Node[2] sinfids3a2: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3b1 to resource_sinfids3B_vip
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_aims, Node[0] sinfids3a2: 60700
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_aims, Node[1] sinfids3a1: 700
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_aims, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_aims
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oralsnr, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oralsnr, Node[1] sinfids3a1: 600
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oralsnr, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_oralsnr
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oracle, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oracle, Node[1] sinfids3a1: 500
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_oracle, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_oracle
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_smb, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_smb, Node[1] sinfids3a1: 400
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_smb, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_smb
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_fs, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_fs, Node[1] sinfids3a1: 300
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_fs, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_fs
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_drbd, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_drbd, Node[1] sinfids3a1: 200
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_drbd, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_drbd
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_vip, Node[0] sinfids3a2: 1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_vip, Node[1] sinfids3a1: 100
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3A_vip, Node[2] sinfids3b1: -1000000
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3A_vip
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3_vip, Node[0] sinfids3a2: 90100
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3_vip, Node[1] sinfids3b1: 3100
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Color resource_sinfids3_vip, Node[2] sinfids3a1: 100
> > ptest[27348]: 2006/11/06_21:50:51 debug: native_assign_node: Assigning sinfids3a2 to resource_sinfids3_vip
> >
> >
> >
> > ----- Original Message ----
> > From: Andreas Kurz <akurz at sms.at>
> > To: General Linux-HA mailing list <linux-ha at lists.linux-ha.org>
> > Sent: Monday, 6 November, 2006 6:38:56 PM
> > Subject: Re: [Linux-HA] colocating resources on failed restart :CRM-Stable-4a0d4e40eeb0
> >
> > Hello!
> >
> > For system upgrades I prefer disabling the management (and so the
> > monitoring) of the affected resource. Works fine for me ... in your case
> > e.g:
> >
> > crm_resource -p is_managed -r resource_sinfids3A_aims -t primitive -v off
> > ... do your maintenance ....
> > crm_resource -p is_managed -r resource_sinfids3A_aims -t primitive -v on
> >
> > Regards,
> > Andi
> >
> > > I converted the cib.xml back to using groups which fixes the colocation problem but how to do perform system upgrades.
> > >
> > > e.g.
> > > I run command
> > > crm_resource -r resource_sinfids3A_aims -p target_role -v stopped
> > > my hope here is that hb will stop the resource and stop monitoring, allowing the aims software to be manually updated/started/checked/stopped; then run
> > > crm_resource -r resource_sinfids3A_aims -p target_role -v started
> > > hb will start the resource if necessary then continue to monitor.
> > >
> > > What actually happens is that hb stops ALL resources within the group_sinfids3A - aarrrgh.
> > >
> > >
> > > I realise that I am very confused on how to configure the primitive restart_type, and the on_fail for the monitor operation. Any help on this would be wonderful.
> > >
> > >
> > > The scenarios are
> > >
> > > crm_resource -r resource_sinfids3A_aims -p target_role -v stopped [as above]
> > > stop aims - allow manual updates...
> > >
> > > crm_resource -r resource_sinfids3A_oralsnr -p target_role -v stopped
> > > stop oracle listener - allow manaul updates
> > >
> > >
> > >
> > > crm_resource -r resource_sinfids3A_oracle -p target_role -v stopped
> > >
> > > stop aims, oracle listener, then oracle
> > >
> > >
> > > crm_resource -r resource_sinfids3A_smb -p target_role -v stopped
> > > stop samba - allow manual updates
> > >
> > >
> > > Node: sinfids3b1 (338afa76-8997-4d66-8381-fc36ec4b456b): online
> > > Node: sinfids3a2 (ec74bd17-2016-4d32-a694-0f6983121cd9): online
> > > Node: sinfids3a1 (b757aece-0e47-41e5-92b7-6a80b4f3eea7): online
> > >
> > > Resource Group: group_sinfids3
> > > resource_sinfids3_vip (heartbeat::ocf:IPaddr): Started sinfids3a1
> > > Resource Group: group_sinfids3A
> > > resource_sinfids3A_vip (heartbeat::ocf:IPaddr): Started sinfids3a1
> > > resource_sinfids3A_drbd (heartbeat:drbddisk): Started sinfids3a1
> > > resource_sinfids3A_fs (heartbeat::ocf:Filesystem): Started sinfids3a1
> > > resource_sinfids3A_smb (lsb:smb): Started sinfids3a1
> > > resource_sinfids3A_oracle (heartbeat::ocf:oracle): Started sinfids3a1
> > > resource_sinfids3A_oralsnr (heartbeat::ocf:oralsnr): Started sinfids3a1
> > > resource_sinfids3A_aims (lsb:aims): Started sinfids3a1
> > > Resource Group: group_sinfids3B
> > > resource_sinfids3B_vip (heartbeat::ocf:IPaddr): Started sinfids3b1
> > >
> > >
> > > My current settings for the primitives, res_order and monitor:
> > >
> > > [root at sinfids3b1 ~]# egrep "primitive |monitor|rsc_ord|group " saved-cib.xml
> > > <group id="group_sinfids3" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > > <primitive id="resource_sinfids3_vip" class="ocf" type="IPaddr" provider="heartbeat" is_managed="true" restart_type="restart">
> > > <op id="IPaddr_sinfids3_vip_mon" interval="60s" name="monitor" timeout="15s" on_fail="restart"/>
> > > <group id="group_sinfids3A" ordered="true" collocated="true" is_managed="true" restart_type="ignore">
> > > <primitive id="resource_sinfids3A_vip" class="ocf" type="IPaddr" provider="heartbeat" is_managed="true" restart_type="restart">
> > > <op id="IPaddr_sinfids3A_vip_mon" interval="60s" name="monitor" timeout="15s" on_fail="restart"/>
> > > <primitive id="resource_sinfids3A_drbd" class="heartbeat" type="drbddisk" provider="heartbeat" is_managed="true" restart_type="restart">
> > > <op id="drbddisk_sinfids3A_drbd_mon" name="monitor" interval="60s" timeout="60s" on_fail="restart"/>
> > > <primitive class="ocf" type="Filesystem" provider="heartbeat" id="resource_sinfids3A_fs" is_managed="true" restart_type="restart">
> > > <op name="monitor" timeout="60s" id="Filesystem_sinfids3A_fs_mon" interval="60s" on_fail="restart"/>
> > > <primitive class="lsb" type="smb" id="resource_sinfids3A_smb" is_managed="true" restart_type="restart">
> > > <op name="monitor" timeout="60s" id="smb_sinfids3A_smb_mon" interval="30s" on_fail="restart"/>
> > > <primitive class="ocf" type="oracle" provider="heartbeat" id="resource_sinfids3A_oracle" is_managed="true" restart_type="restart">
> > > <op name="monitor" timeout="60s" id="oracle_sinfids3A_oracle_mon" interval="300s" on_fail="restart"/>
> > > <primitive class="ocf" type="oralsnr" provider="heartbeat" id="resource_sinfids3A_oralsnr" is_managed="true" restart_type="restart">
> > > <op name="monitor" timeout="60s" id="oralsnr_sinfids3A_oralsnr_mon" interval="300s" on_fail="restart"/>
> > > <primitive class="lsb" type="aims" id="resource_sinfids3A_aims" is_managed="true" restart_type="ignore">
> > > <op name="monitor" timeout="240s" id="aims_sinfids3A_aims_mon" interval="180s" on_fail="restart"/>
> > > <group id="group_sinfids3B" ordered="true" collocated="true" is_managed="true" restart_type="restart">
> > > <primitive id="resource_sinfids3B_vip" class="ocf" type="IPaddr" provider="heartbeat" is_managed="true" restart_type="restart">
> > > <op id="IPaddr_sinfids3B_vip_mon" interval="60s" name="monitor" timeout="15s" on_fail="restart"/>
> > > <rsc_order id="order_sinfids3_sinfids3A" from="group_sinfids3" action="start" type="after" to="group_sinfids3A" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3_sinfids3B" from="group_sinfids3" action="start" type="after" to="group_sinfids3B" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_vip" from="resource_sinfids3A_vip" action="start" type="before" to="resource_sinfids3A_drbd" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_drbd" from="resource_sinfids3A_drbd" action="start" type="after" to="resource_sinfids3A_vip" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_fs" from="resource_sinfids3A_fs" action="start" type="after" to="resource_sinfids3A_drbd" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_smb" from="resource_sinfids3A_smb" action="start" type="after" to="resource_sinfids3A_fs" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_oracle" from="resource_sinfids3A_oracle" action="start" type="after" to="resource_sinfids3A_fs" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_oralsnr" from="resource_sinfids3A_oralsnr" action="start" type="after" to="resource_sinfids3A_oracle" symmetrical="true"/>
> > > <rsc_order id="order_sinfids3A_aims" from="resource_sinfids3A_aims" action="start" type="after" to="resource_sinfids3A_oralsnr" symmetrical="true"/>
> > >
> > >
> > >
> > >
> > > ----- Original Message ----
> > > From: Serge Dubrouski <sergeyfd at gmail.com>
> > > To: General Linux-HA mailing list <linux-ha at lists.linux-ha.org>
> > > Sent: Sunday, 5 November, 2006 11:47:05 PM
> > > Subject: Re: [Linux-HA] colocating resources on failed restart :CRM-Stable-4a0d4e40eeb0
> > >
> > > Why not to ombine your resources into a group with collcated=true. In
> > > this case they'll always stick together for all operations:
> > > start/stop/move etc...
> > >
> > > On 11/4/06, Alex and Gill Strachan <asgks at yahoo.com> wrote:
> > >> I have a group of resources linked by the name 3A, these resources must always run together so I allocated large co-location scores.
> > >>
> > >> When the resource_sinfids3A_aims fails and it is moved to another node I need all of the 3A resources to move with it and to start before.
> > >>
> > >> e.g.
> > >> resource_sinfids3A_aims fails on node 3a2
> > >> hb restarts and reduces node weight for that node..
> > >> resource_sinfids3A_aims fails on node 3a2
> > >> hb is unable to restart on node 3a2 so decides to relocate to 3a1
> > >>
> > >> ...How do I inform hb to stop all the other 3A resources on 3a2 and move
> > >> ...everything to 3a1, also starting in a particular order.
> > >>
> > >> Why didn't the colocation scores help in keeping the 3A resources together?
> > >>
> > >>
> > >> I originally had colocation scores of INFINITY for the 3A group but this then prevents the ability to specifiy that resource smb can fail 3 times while resource aims can only fail once.
> > >>
> > >>
> > >> I originally had this working by using groups and on_fail="fence" but it doesn't offer enough flexibility.
> > >>
> > >> e.g.
> > >> I would like heartbeat to restart smb on failure 3 times before moving to another node; using resource_stickiness. When using groups the restart of smb would trigger a stop of all higher resources, then start smb followed by start the higher resources. This behaviour was not wanted.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> ============
> > >> Last updated: Sun Nov 5 14:02:46 2006
> > >> Current DC: sinfids3a2 (ec74bd17-2016-4d32-a694-0f6983121cd9)
> > >> 3 Nodes configured.
> > >> 9 Resources configured.
> > >> ============
> > >>
> > >> Node: sinfids3b1 (338afa76-8997-4d66-8381-fc36ec4b456b): online
> > >> resource_sinfids3B_vip (heartbeat::ocf:IPaddr)
> > >> Node: sinfids3a2 (ec74bd17-2016-4d32-a694-0f6983121cd9): online
> > >> resource_sinfids3A_drbd (heartbeat:drbddisk)
> > >> resource_sinfids3A_fs (heartbeat::ocf:Filesystem)
> > >> resource_sinfids3A_smb (lsb:smb)
> > >> resource_sinfids3A_vip (heartbeat::ocf:IPaddr)
> > >> resource_sinfids3A_oralsnr (heartbeat::ocf:oralsnr)
> > >> resource_sinfids3_vip (heartbeat::ocf:IPaddr)
> > >> resource_sinfids3A_oracle (heartbeat::ocf:oracle)
> > >> resource_sinfids3A_aims (lsb:aims)
> > >> Node: sinfids3a1 (b757aece-0e47-41e5-92b7-6a80b4f3eea7): online
> > >>
> > >>
> > >>
> > >> <rsc_order id="order_sinfids3_sinfids3A" from="resource_sinfids3_vip" type="after" to="resource_sinfids3A_vip"/>
> > >> <rsc_order id="order_sinfids3_sinfids3B" from="resource_sinfids3_vip" type="after" to="resource_sinfids3B_vip"/>
> > >> <rsc_order id="order_sinfids3A_drbd" from="resource_sinfids3A_drbd" type="after" to="resource_sinfids3A_vip"/>
> > >> <rsc_order id="order_sinfids3A_fs" from="resource_sinfids3A_fs" type="after" to="resource_sinfids3A_drbd"/>
> > >> <rsc_order id="order_sinfids3A_smb" from="resource_sinfids3A_smb" type="after" to="resource_sinfids3A_fs"/>
> > >> <rsc_order id="order_sinfids3A_oracle" from="resource_sinfids3A_oracle" type="after" to="resource_sinfids3A_fs"/>
> > >> <rsc_order id="order_sinfids3A_oralsnr" from="resource_sinfids3A_oralsnr" type="after" to="resource_sinfids3A_oracle"/>
> > >> <rsc_order id="order_sinfids3A_aims" from="resource_sinfids3A_aims" type="after" to="resource_sinfids3A_oralsnr"/>
> > >>
> > >> <rsc_colocation id="colocation_sinfids3_sinfids3A" from="resource_sinfids3_vip" to="resource_sinfids3A_vip" score="9000"/>
> > >> <rsc_colocation id="colocation_sinfids3_sinfids3B" from="resource_sinfids3_vip" to="resource_sinfids3B_vip" score="3000"/>
> > >>
> > >> <rsc_colocation id="colocation_sinfids3A_drbd" from="resource_sinfids3A_drbd" to="resource_sinfids3A_vip" score="100000"/>
> > >> <rsc_colocation id="colocation_sinfids3A_fs" from="resource_sinfids3A_fs" to="resource_sinfids3A_drbd" score="100000"/>
> > >> <rsc_colocation id="colocation_sinfids3A_smb" from="resource_sinfids3A_smb" to="resource_sinfids3A_fs" score="100000"/>
> > >> <rsc_colocation id="colocation_sinfids3A_oracle" from="resource_sinfids3A_oracle" to="resource_sinfids3A_fs" score="100000"/>
> > >> <rsc_colocation id="colocation_sinfids3A_oralsnr" from="resource_sinfids3A_oralsnr" to="resource_sinfids3A_oracle" score="100000"/>
> > >> <rsc_colocation id="colocation_sinfids3A_aims" from="resource_sinfids3A_aims" to="resource_sinfids3A_oralsnr" score="100000"/>
> > >>
> > >>
> > >> <primitive class="lsb" type="aims" id="resource_sinfids3A_aims" restart_type="restart">
> > >> <operations>
> > >> <op name="monitor" timeout="240s" id="aims_sinfids3A_aims_mon" interval="180s"/>
> > >> </operations>
> > >> <instance_attributes id="resource_sinfids3A_aims">
> > >> <attributes>
> > >> <nvpair id="resource_sinfids3A_aims-target_role" name="target_role" value="started"/>
> > >> </attributes>
> > >> </instance_attributes>
> > >> </primitive>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Send instant messages to your online friends http://au.messenger.yahoo.com
> > >> _______________________________________________
> > >> Linux-HA mailing list
> > >> Linux-HA at lists.linux-ha.org
> > >> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > >> See also: http://linux-ha.org/ReportingProblems
> > >>
> > > _______________________________________________
> > > Linux-HA mailing list
> > > Linux-HA at lists.linux-ha.org
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> > >
> > >
> > >
> > >
> > >
> > >
> > > Send instant messages to your online friends http://au.messenger.yahoo.com
> > > _______________________________________________
> > > Linux-HA mailing list
> > > Linux-HA at lists.linux-ha.org
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> >
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA at lists.linux-ha.org
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> >
> >
> >
> >
> >
> > Send instant messages to your online friends http://au.messenger.yahoo.com
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA at lists.linux-ha.org
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
>
>
>
>
>
> Send instant messages to your online friends http://au.messenger.yahoo.com
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA at lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Send instant messages to your online friends http://au.messenger.yahoo.com
More information about the Linux-HA
mailing list