[Linux-HA] a strange case of mixing ping nodes and member nodes

Thu Sep 7 09:45:45 MDT 2006

***********************
Warning: Your file, sapcl01.xml.bz2, appears to be a compressed file but is corrupt. It was not scanned by InterScan MSS.
***********************

Hello,

The cluster has three nodes (sapcl01,02,03) and sapcl01 has
since last night a rather strange status section. The
node_state of sapcl02 has vanished and in it's stead there's
now a node_state of one of the ping nodes. This is the
crm_diff of the global CIB and the sapcl01 CIB:

 <diff>
   <diff-removed>
     <cib>
       <status>
         <node_state uname="9.158.30.40" crmd="offline" in_ccm="false" ha="dead" join="down" id="bdcbaad6-5fdc-4880-a309-ccaaf70db357"/>
       </status>
     </cib>
   </diff-removed>
   <diff-added>
     <cib>
       <status>
         <node_state uname="sapcl02" crmd="online" in_ccm="true" ha="active" join="member" id="bdcbaad6-5fdc-4880-a309-ccaaf70db357">
           <transient_attributes id="bdcbaad6-5fdc-4880-a309-ccaaf70db357" __crm_diff_marker__="added:top">
             <instance_attributes id="status-bdcbaad6-5fdc-4880-a309-ccaaf70db357">
               <attributes>
                 <nvpair id="status-bdcbaad6-5fdc-4880-a309-ccaaf70db357-pingd" name="pingd" value="300"/>
                 <nvpair id="status-bdcbaad6-5fdc-4880-a309-ccaaf70db357-probe_complete" name="probe_complete" value="true"/>
               </attributes>
             </instance_attributes>
           </transient_attributes>
         </node_state>
       </status>
     </cib>
   </diff-added>
 </diff>

crm_mon on sapcl01 shows both that sapcl02 is the current DC
and that it's offline.

The ping node 9.158.30.40 is running AIX and has no
heartbeat installed.

I'll attach the relevant stuff, though I can't see any clues
in the logs. Hope that somebody out there will have more
luck.

Cheers,

Dejan

-------------- next part --------------
traditional_compression false
coredumps true
use_logd yes
keepalive 2
warntime 6
deadtime 8
initdead 10
deadping 6
mcast   eth0 225.0.0.1 694 1 0
mcast   eth1 225.0.0.2 694 1 0
auto_failback	legacy
node sapcl01
node sapcl02
node sapcl03
#node lingws
ping 9.158.3.144 9.158.29.46 9.158.30.40
crm	on
respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s
debug 1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ha-debug.bz2
Type: application/octet-stream
Size: 2086 bytes
Desc: not available
Url : http://lists.community.tummy.com/pipermail/linux-ha/attachments/20060907/3ee1670d/ha-debug.obj
-------------- next part --------------

============
Last updated: Thu Sep  7 17:28:50 2006
Current DC: sapcl02 (bdcbaad6-5fdc-4880-a309-ccaaf70db357)
3 Nodes configured.
6 Resources configured.
============

Node: sapcl02 (bdcbaad6-5fdc-4880-a309-ccaaf70db357): OFFLINE
Node: sapcl01 (cf68c349-b7ca-4495-9aaf-5e158969efef): online
Node: sapcl03 (4b288449-8f13-4f2b-9e96-ba8c45f3ed8d): online

Resource Group: a1
    IPaddr_10_1_1_22	(heartbeat::ocf:IPaddr2):	Started sapcl03
    IPaddr_192_168_1_22	(heartbeat::ocf:IPaddr2):	Started sapcl03
    apache_a1	(heartbeat::ocf:apache):	Started sapcl03
Resource Group: a2
    IPaddr_10_1_1_23	(heartbeat::ocf:IPaddr2):	Started sapcl01
    IPaddr_192_168_1_23	(heartbeat::ocf:IPaddr2):	Started sapcl01
    apache_a2	(heartbeat::ocf:apache):	Started sapcl01
Resource Group: a4
    IPaddr_10_1_1_25	(heartbeat::ocf:IPaddr2):	Started sapcl03
    IPaddr_192_168_1_25	(heartbeat::ocf:IPaddr2):	Started sapcl03
    apache_a4	(heartbeat::ocf:apache):	Started sapcl03
Resource Group: a5
    IPaddr_10_1_1_26	(heartbeat::ocf:IPaddr2):	Started sapcl01
    IPaddr_192_168_1_26	(heartbeat::ocf:IPaddr2):	Started sapcl01
    apache_a5	(heartbeat::ocf:apache):	Started sapcl01
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cib.xml.bz2
Type: application/octet-stream
Size: 4081 bytes
Desc: not available
Url : http://lists.community.tummy.com/pipermail/linux-ha/attachments/20060907/3ee1670d/cib.xml.obj
-------------- next part --------------
BZh91AY&SY??T? ?_?j????/l? ????`?`?X?????y<B?f???j?mM???R?c??8AD?B??AR??@:TF?z???i?=L??CF? ???D??h	???????h #0	????L&i??OU%FP?   4 @ ??L?'?M
???  ? ? R? @&??????e???]???:j??$?????????@0@?>???????#$???) ?RD??,I"?%#$???) ?RD??,Im??m??1N?`"? ?P?)	 ?Q
E	?????4?
??gm?N??sj?km\F??Y5H???++??1?b?_???_?i??8??Vc????o\W'3????????w??;?]|????>M>]3????n????o}1?
w?*???+[??Mk??????5?:?|???}s?<??)V???)W?Q??R(??????????<?i??1? $%2A?5?yZ?eo???m??g?,??}?zl{i???9????_?6$???|??-lVa???2?????y?`?1wwwbHbB??}zy?t???y<??"?c?v??2`?l???mD??
Z??e?&VM???6vl???X??$BLAe??K??e)?A????tYh?]??4?=?>>+?p??????w??=M?oL?q????????'?P????_j??>x??1?K?w???ll????y/P??Wz??+?N-?oZ??}?{????z?Q?\?????$?]??s????x??>h?????Z????~>J)}W?;?_?\?#jz/???.????????v?V?m????????gf~??5???6?(^B?m???EI?HZbz????|FD?S????????#???x?n?????vj?????m?D[F?????-m?U?R???c2g?l???p\????#m??]E?????O?????
\f???C??[.K?D?q/%????]:??,??>?0???W???y??*??
L?uX?gi??r?0[v?.???<>U?????R?#5????.5F???46???]??2????????bw??M????L??,[?uG???a??z??G ?	??2?a`eV	qDn??u?^??zM????????I?{#?9?pw????W??g??????[?VYWi????M?'5?c2?:?Ww?=??)?u??
?l?????\\Q????=???p<???E8??8?5?F@??Z??y???f)?6???I%?H??yCy???OU?z?^?iv????K??S}?T?j?U??Z?^p
??aiQ????e???y=????+?q\4?Mv^K???Y<?N?'?????????*?`??S(eP??2?
??yL?F????Ki?????:??&?<q??X?w?f?s?r??ir?
?j????0Z??^^??????p]?????sZ??{????S??E?b??61?Yc2??gy??W??????
???Q????:N<?J??[y??{??we?|X??i?{w?6?R??$??U??wZ???N???.K?:9??? ?IT? ??9?W??V(?R?7?????n???-tK?`/?????c?KU??;??b?<??9?%?(X14??w???$!2????i?[?????5???E?)?<W?q?M*?M'<???-?s?? ??!?????x?+??V???Lc??a?????)m:???v)y??\i5??V?Y?E??b??MN?,?M?xM???9NUoQ1k?x???kM>$???u????????<??7???r\?&??Sz??'9???o????WYtY;FN???y?c#??.K???\?p?R??
??ls?/???[M ?X???LN+???kV?z?+E?s?r??????`?N?????#G1?t?b??35?????.?I????C"??????'d?:?p????:<??
??b??2????,?f)??Vn?U-k]u????QFM??V?5?8????????NNP?\Y?3m?%?6?~j?Z??????[?8????????-??6?iq?u??????X\Vr??kJZ????.4??1??qG+K??????????W?O?K?"??v?}??8e??	????c2??S????6??[?]SYa#9??sM??V?[?s??h???r-?S???lb?hf?YMs?rk???um?????n?h?????-???-?-?????&?6T5F?t?XabbbL?Hh)P?Z??????|??Be8?????k5J? ?chLX?l?R???6,????Z?W????9?s1?33&f2ffd?fe2f?\Fk\kp?????:??k??????????)?_<??)??????J???&W+m|v??m??m\?? ???j?"@$?@?$ $ ???$\?T?????mzsci4T?5&??#0c&`?;6}?7l???_????????$????r?6
?m ?|?HX????f???}w?????????x??8???!xB?lllll?H^Fs???1?`???`???(??L`?(ps^0?
!gF???K????KD?K?x??????K:u??RL2BC?E?$/?A???j+\??9y???K^Zz-S????}?/?Z?knS?p???/????\C??p?_}
??W?d??N.8??zNs?s4?	?`???p??y??z????G-?kYo3???0?3???S?~~3??.????>??&/????Ft???????W?????f%??????G?#i}	s??VfL?aa{,???2?d1?
e?.qW????_e???<????Y??????}???????:??I?s[?K??VG???<?U?~nx??????v_??yt?u??J??9??????~???>!??K?q?5?s_%???N|}Z'????8@???U???>>??q?k?qg?]*?u[I;??K????1;??_??????1s?%}???????Sa?\~F?]??N????w:?}U.Wz.???m?<????q5^~???I??U??j?|/?hiw.??]?/?t]??b?}'9?="{%k?I???/	???K?N?9??r_*?:??Y.S??????~E/??<??????A~S??ffk???^?
?[??K?wT_?;*}
??????O?s?_N?yn?K?p{?g'VZ???>K????yTO???'?|????K?>???_?)zV?????{/Ut?d?????0YM?Q?M,?1?c??2????>W????T?2?6J?kr]e.+7g'j??C?A?J^?? _y???^????y?}?Jd?-Vs?v?u/Sz??Rk?i?w??????_#?3???*G?~Zo*:?3?a??Z???Tj?^(?3?K
??K????W;????q~?W????)~???O?T??????}G?u?}?*?s?J??Z?~u?;?w??3???q[????5?I?[V?y?U???{????????M?M?Z)`/????g?o????p%}g&??d}??2?:??GC??	:??K@X?9?j????~?????o?~???+?H?k??V?t??????gR?#@??9????K
???bX??D?R???,?{L'??y????
?7??k??&
0??`???"?}B??R??????4qD???????????'(?T:=???kV????Pue??????|%d???U???a?'?O;??Zf?n???\J4?s6????????|?x?q?E,-???nv/?feP?
?????u-??x{???b??h?ccE????2\????)a???n?Oz??tZ?/+??????9????S?????G%?=?/a??H???i0.??'???>B??????qm?sZ.+K???l????^?oz????f??t{/`???emTieQ?E???(?V???,?4???j???R[`??9????U?1v?_ ?b?????u?,???v^?x???"?$z??i?s\m?G??}?Ko???&?3g?3m{?a??????/?$??#?p??P?bb0????N?????U.??\.????R?????R???-:T?????/n??~f?m?*?1Z?y;Ii??;I	??t ez?	#`?#?KrF$?
??#???cX#}p??Z?|???sU\$o?G^^?#??G%??EJZ?s*??????Uy??x?6?;t???RY;?#???E??H??=?~????b?3?3?.k?????????????????e?????G????????%????Lh\?????W?rE8P???T?