[Linux-HA] crm_mon showing different status on cluster nodes

[Andrew Beekhof]

On Nov 19, 2007, at 7:53 AM, Alex Strachan wrote:

> In a production cluster (four nodes) the crm_mon command is showing
> different statuses.
>
>
>
> How to resync the status - without affecting services!  Not sure how  
> the
> sync option in cibadmin works.

pick a node that shows the correct output, use cibadmin -S -H node- 
with-correct-mon-output

> crm_mon output from the four nodes, basically
>
>  3 resources running on sinfids3a1 are being reported as stopped  
> although
> they are running OKAY.
>
>  1 node is reported as being offline - although it is running the  
> resources
> as reported by other nodes
>
> Very puzzling.

Indeed.  Version? Logs?

>
>
>  All nodes report sinfids3b2 as the DC
>
>
>
>
>
> [root at sinfids3a1 init.d]#
>
> ============
>
> Last updated: Mon Nov 19 14:15:35 2007
>
> Current DC: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb)
>
> 4 Nodes configured.
>
> 18 Resources configured.
>
> ============
>
>
>
> Node: sinfids3a1 (3e940b7a-0e41-4df4-9279-71205d5038e8): online
>
> Node: sinfids3a2 (8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4): online
>
> Node: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb): online
>
> Node: sinfids3b1 (ac809f97-d6b7-4c14-ab50-a8aa6246e132): online
>
>
>
> Resource Group: group_sinfids3a
>
>    resource_sinfids3a_drbd     (heartbeat:sinfids3a_drbddisk): Started
> sinfids3a1
>
>    resource_sinfids3a_fs       (heartbeat::ocf:sinfids3a_Filesystem):
> Started sinfids3a1
>
>    resource_sinfids3a_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3a1
>
>    resource_sinfids3a_oracle   (heartbeat::ocf:sinfids3a_oracle):
> Stopped
>
>    resource_sinfids3a_oralsnr  (heartbeat::ocf:sinfids3a_oralsnr):
> Stopped
>
>    resource_sinfids3a_aims     (lsb:sinfids3a_aims):   Started  
> sinfids3a1
>
>    resource_sinfids3a_apache   (heartbeat::ocf:sinfids3a_apache):
> Stopped
>
>    resource_sinfids3a_smb      (lsb:sinfids3a_smb):    Started  
> sinfids3a1
>
> Resource Group: group_sinfids3b
>
>    resource_sinfids3b_drbd     (heartbeat:sinfids3b_drbddisk): Started
> sinfids3b1
>
>    resource_sinfids3b_fs       (heartbeat::ocf:sinfids3b_Filesystem):
> Started sinfids3b1
>
>    resource_sinfids3b_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3b1
>
>    resource_sinfids3b_oracle   (heartbeat::ocf:sinfids3b_oracle):
> Started sinfids3b1
>
>    resource_sinfids3b_oralsnr  (heartbeat::ocf:sinfids3b_oralsnr):
> Started sinfids3b1
>
>    resource_sinfids3b_aims     (lsb:sinfids3b_aims):   Started  
> sinfids3b1
>
>    resource_sinfids3b_apache   (heartbeat::ocf:sinfids3b_apache):
> Started sinfids3b1
>
>    resource_sinfids3b_smb      (lsb:sinfids3b_smb):    Started  
> sinfids3b1
>
> resource_sinfids3_vip   (heartbeat::ocf:IPaddr):        Started  
> sinfids3a1
>
> resource_sinfids3_skel1 (heartbeat::ocf:sinfids3_ISskel):        
> Started
> sinfids3a1
>
> resource_sinfids3_gatemid1      (heartbeat::ocf:sinfids3_ISgatemid):
> Started sinfids3a1
>
> resource_sinfids3_itsixcats01   (heartbeat::ocf:sinfids3_ITSIXcats):
> Started sinfids3a1
>
> resource_sinfids3_dummy (heartbeat::ocf:Dummy): Started sinfids3a1
>
> resource_sinfids3a_itsixatob    (heartbeat::ocf:sinfids3a_ITSIXatob):
> Started sinfids3a1
>
> resource_sinfids3a_apache_nagios
> (heartbeat::ocf:sinfids3a_apache_nagios):       Started sinfids3a1
>
> resource_sinfids3a_nagios       (lsb:sinfids3a_nagios): Started  
> sinfids3a1
>
> resource_sinfids3b_apache_nagios
> (heartbeat::ocf:sinfids3b_apache_nagios):       Started sinfids3b1
>
> resource_sinfids3b_nagios       (lsb:sinfids3b_nagios): Started  
> sinfids3b1
>
> resource_sinfids3a_fids_flight  (lsb:sinfids3a_fids_flight):     
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_weather (lsb:sinfids3a_fids_weather):    
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_proxy   (lsb:sinfids3a_fids_proxy):      
> Started
> sinfids3a2
>
> resource_sinfids3b_fids_flight  (lsb:sinfids3b_fids_flight):     
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_weather (lsb:sinfids3b_fids_weather):    
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_proxy   (lsb:sinfids3b_fids_proxy):      
> Started
> sinfids3b2
>
>
>
>
>
>
>
> [root at sinfids3a2 ~]#
>
>
>
> ============
>
> Last updated: Mon Nov 19 14:15:51 2007
>
> Current DC: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb)
>
> 4 Nodes configured.
>
> 18 Resources configured.
>
> ============
>
>
>
> Node: sinfids3a1 (3e940b7a-0e41-4df4-9279-71205d5038e8): online
>
> Node: sinfids3a2 (8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4): online
>
> Node: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb): online
>
> Node: sinfids3b1 (ac809f97-d6b7-4c14-ab50-a8aa6246e132): online
>
>
>
> Resource Group: group_sinfids3a
>
>    resource_sinfids3a_drbd     (heartbeat:sinfids3a_drbddisk): Started
> sinfids3a1
>
>    resource_sinfids3a_fs       (heartbeat::ocf:sinfids3a_Filesystem):
> Started sinfids3a1
>
>    resource_sinfids3a_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3a1
>
>    resource_sinfids3a_oracle   (heartbeat::ocf:sinfids3a_oracle):
> Stopped
>
>    resource_sinfids3a_oralsnr  (heartbeat::ocf:sinfids3a_oralsnr):
> Stopped
>
>    resource_sinfids3a_aims     (lsb:sinfids3a_aims):   Started  
> sinfids3a1
>
>    resource_sinfids3a_apache   (heartbeat::ocf:sinfids3a_apache):
> Stopped
>
>    resource_sinfids3a_smb      (lsb:sinfids3a_smb):    Started  
> sinfids3a1
>
> Resource Group: group_sinfids3b
>
>    resource_sinfids3b_drbd     (heartbeat:sinfids3b_drbddisk): Started
> sinfids3b1
>
>    resource_sinfids3b_fs       (heartbeat::ocf:sinfids3b_Filesystem):
> Started sinfids3b1
>
>    resource_sinfids3b_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3b1
>
>    resource_sinfids3b_oracle   (heartbeat::ocf:sinfids3b_oracle):
> Started sinfids3b1
>
>    resource_sinfids3b_oralsnr  (heartbeat::ocf:sinfids3b_oralsnr):
> Started sinfids3b1
>
>    resource_sinfids3b_aims     (lsb:sinfids3b_aims):   Started  
> sinfids3b1
>
>    resource_sinfids3b_apache   (heartbeat::ocf:sinfids3b_apache):
> Started sinfids3b1
>
>    resource_sinfids3b_smb      (lsb:sinfids3b_smb):    Started  
> sinfids3b1
>
> resource_sinfids3_vip   (heartbeat::ocf:IPaddr):        Started  
> sinfids3a1
>
> resource_sinfids3_skel1 (heartbeat::ocf:sinfids3_ISskel):        
> Started
> sinfids3a1
>
> resource_sinfids3_gatemid1      (heartbeat::ocf:sinfids3_ISgatemid):
> Started sinfids3a1
>
> resource_sinfids3_itsixcats01   (heartbeat::ocf:sinfids3_ITSIXcats):
> Started sinfids3a1
>
> resource_sinfids3_dummy (heartbeat::ocf:Dummy): Started sinfids3a1
>
> resource_sinfids3a_itsixatob    (heartbeat::ocf:sinfids3a_ITSIXatob):
> Started sinfids3a1
>
> resource_sinfids3a_apache_nagios
> (heartbeat::ocf:sinfids3a_apache_nagios):       Started sinfids3a1
>
> resource_sinfids3a_nagios       (lsb:sinfids3a_nagios): Started  
> sinfids3a1
>
> resource_sinfids3b_apache_nagios
> (heartbeat::ocf:sinfids3b_apache_nagios):       Started sinfids3b1
>
> resource_sinfids3b_nagios       (lsb:sinfids3b_nagios): Started  
> sinfids3b1
>
> resource_sinfids3a_fids_flight  (lsb:sinfids3a_fids_flight):     
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_weather (lsb:sinfids3a_fids_weather):    
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_proxy   (lsb:sinfids3a_fids_proxy):      
> Started
> sinfids3a2
>
> resource_sinfids3b_fids_flight  (lsb:sinfids3b_fids_flight):     
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_weather (lsb:sinfids3b_fids_weather):    
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_proxy   (lsb:sinfids3b_fids_proxy):      
> Started
> sinfids3b2
>
>
>
>
>
>
>
> [root at sinfids3b1 ~]#
>
> ============
>
> Last updated: Mon Nov 19 14:16:15 2007
>
> Current DC: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb)
>
> 4 Nodes configured.
>
> 18 Resources configured.
>
> ============
>
>
>
> Node: sinfids3a1 (3e940b7a-0e41-4df4-9279-71205d5038e8): OFFLINE
>
> Node: sinfids3a2 (8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4): online
>
> Node: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb): online
>
> Node: sinfids3b1 (ac809f97-d6b7-4c14-ab50-a8aa6246e132): online
>
>
>
> Resource Group: group_sinfids3b
>
>    resource_sinfids3b_drbd     (heartbeat:sinfids3b_drbddisk): Started
> sinfids3b1
>
>    resource_sinfids3b_fs       (heartbeat::ocf:sinfids3b_Filesystem):
> Started sinfids3b1
>
>    resource_sinfids3b_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3b1
>
>    resource_sinfids3b_oracle   (heartbeat::ocf:sinfids3b_oracle):
> Started sinfids3b1
>
>    resource_sinfids3b_oralsnr  (heartbeat::ocf:sinfids3b_oralsnr):
> Started sinfids3b1
>
>    resource_sinfids3b_aims     (lsb:sinfids3b_aims):   Started  
> sinfids3b1
>
>    resource_sinfids3b_apache   (heartbeat::ocf:sinfids3b_apache):
> Started sinfids3b1
>
>    resource_sinfids3b_smb      (lsb:sinfids3b_smb):    Started  
> sinfids3b1
>
> resource_sinfids3b_apache_nagios
> (heartbeat::ocf:sinfids3b_apache_nagios):       Started sinfids3b1
>
> resource_sinfids3b_nagios       (lsb:sinfids3b_nagios): Started  
> sinfids3b1
>
> resource_sinfids3a_fids_proxy   (lsb:sinfids3a_fids_proxy):      
> Started
> sinfids3a2
>
> resource_sinfids3b_fids_flight  (lsb:sinfids3b_fids_flight):     
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_weather (lsb:sinfids3b_fids_weather):    
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_proxy   (lsb:sinfids3b_fids_proxy):      
> Started
> sinfids3b2
>
>
>
>
>
> [root at sinfids3b2 ~]#
>
> ============
>
> Last updated: Mon Nov 19 14:16:38 2007
>
> Current DC: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb)
>
> 4 Nodes configured.
>
> 18 Resources configured.
>
> ============
>
>
>
> Node: sinfids3a1 (3e940b7a-0e41-4df4-9279-71205d5038e8): online
>
> Node: sinfids3a2 (8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4): online
>
> Node: sinfids3b2 (b641a173-b7a1-4cf9-a8d3-a90cea2d7beb): online
>
> Node: sinfids3b1 (ac809f97-d6b7-4c14-ab50-a8aa6246e132): online
>
>
>
> Resource Group: group_sinfids3a
>
>    resource_sinfids3a_drbd     (heartbeat:sinfids3a_drbddisk): Started
> sinfids3a1
>
>    resource_sinfids3a_fs       (heartbeat::ocf:sinfids3a_Filesystem):
> Started sinfids3a1
>
>    resource_sinfids3a_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3a1
>
>    resource_sinfids3a_oracle   (heartbeat::ocf:sinfids3a_oracle):
> Stopped
>
>    resource_sinfids3a_oralsnr  (heartbeat::ocf:sinfids3a_oralsnr):
> Stopped
>
>    resource_sinfids3a_aims     (lsb:sinfids3a_aims):   Started  
> sinfids3a1
>
>    resource_sinfids3a_apache   (heartbeat::ocf:sinfids3a_apache):
> Stopped
>
>    resource_sinfids3a_smb      (lsb:sinfids3a_smb):    Started  
> sinfids3a1
>
> Resource Group: group_sinfids3b
>
>    resource_sinfids3b_drbd     (heartbeat:sinfids3b_drbddisk): Started
> sinfids3b1
>
>    resource_sinfids3b_fs       (heartbeat::ocf:sinfids3b_Filesystem):
> Started sinfids3b1
>
>    resource_sinfids3b_vip      (heartbeat::ocf:IPaddr):        Started
> sinfids3b1
>
>    resource_sinfids3b_oracle   (heartbeat::ocf:sinfids3b_oracle):
> Started sinfids3b1
>
>    resource_sinfids3b_oralsnr  (heartbeat::ocf:sinfids3b_oralsnr):
> Started sinfids3b1
>
>    resource_sinfids3b_aims     (lsb:sinfids3b_aims):   Started  
> sinfids3b1
>
>    resource_sinfids3b_apache   (heartbeat::ocf:sinfids3b_apache):
> Started sinfids3b1
>
>    resource_sinfids3b_smb      (lsb:sinfids3b_smb):    Started  
> sinfids3b1
>
> resource_sinfids3_vip   (heartbeat::ocf:IPaddr):        Started  
> sinfids3a1
>
> resource_sinfids3_skel1 (heartbeat::ocf:sinfids3_ISskel):        
> Started
> sinfids3a1
>
> resource_sinfids3_gatemid1      (heartbeat::ocf:sinfids3_ISgatemid):
> Started sinfids3a1
>
> resource_sinfids3_itsixcats01   (heartbeat::ocf:sinfids3_ITSIXcats):
> Started sinfids3a1
>
> resource_sinfids3_dummy (heartbeat::ocf:Dummy): Started sinfids3a1
>
> resource_sinfids3a_itsixatob    (heartbeat::ocf:sinfids3a_ITSIXatob):
> Started sinfids3a1
>
> resource_sinfids3a_apache_nagios
> (heartbeat::ocf:sinfids3a_apache_nagios):       Started sinfids3a1
>
> resource_sinfids3a_nagios       (lsb:sinfids3a_nagios): Started  
> sinfids3a1
>
> resource_sinfids3b_apache_nagios
> (heartbeat::ocf:sinfids3b_apache_nagios):       Started sinfids3b1
>
> resource_sinfids3b_nagios       (lsb:sinfids3b_nagios): Started  
> sinfids3b1
>
> resource_sinfids3a_fids_flight  (lsb:sinfids3a_fids_flight):     
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_weather (lsb:sinfids3a_fids_weather):    
> Started
> sinfids3a1
>
> resource_sinfids3a_fids_proxy   (lsb:sinfids3a_fids_proxy):      
> Started
> sinfids3a2
>
> resource_sinfids3b_fids_flight  (lsb:sinfids3b_fids_flight):     
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_weather (lsb:sinfids3b_fids_weather):    
> Started
> sinfids3b1
>
> resource_sinfids3b_fids_proxy   (lsb:sinfids3b_fids_proxy):      
> Started
> sinfids3b2
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems