[Linux-HA] HA Firewall

Tue Nov 20 02:57:58 MST 2007

Hi,

On Mon, Nov 19, 2007 at 08:29:23PM +0000, North Country Boy wrote:
> Hello again.  Please find attached my logs from both node1 and node2.
>  
> Using node1 logs as reference, here are the sequence of events for brevity20:52 start node1 & node220.55 node1 can no longer reach the external host.  Node 2 does not log anything for this event!20:59 node1 can now reach external host.  Node 2 does not log anything for this event!21:01 heartbeat (eth0) cable pulled out.  Both machines recognise and a failover begins, node 2 logs this and failover is successful21.03 heartbeat (eth0) plugged back in .  Both machines respond and failover completes successfully.
> It just seems that this will not failover unless the heartbeart connection fails!! 
>  
> Joris - thanks for the suggestion I will certainly take a look to see what other solutions people use. However I really want to crack this using heartbeat so that I can apply this to other situations such as webservers etc.  For the purpose of this firewall I am not really interested in state table failover because the idea is just to keep an online presence in the event of failure overnight.
> 
> 
> 
> > Subject: RE: [Linux-HA] HA Firewall> Date: Fri, 16 Nov 2007 01:14:58 +0100> From: Joris at familiedobbelsteen.nl> To: linux-ha at lists.linux-ha.org> > If you are looking for a highly available stateful firewall, check out> OpenBSD or FreeBSD with the PF firewall. It includes pfsync which allows> state synchronization. It also includes CARP for IP address failover.> > I have found nothing equivalent on Linux that provides the same> capabilities for high availability.> > Perhaps a good 'distribution' is pfsense, which packages it all> (FreeBSD+PF+CARP+more) including a web interface. There is plenty of> documentation on the web avaiable for such a setup...> > - Joris> > >-----Original Message-----> >From: linux-ha-bounces at lists.linux-ha.org > >[mailto:linux-ha-bounces at lists.linux-ha.org] On Behalf Of > >North Country Boy> >Sent: woensdag 14 november 2007 23:31> >To: General Linux-HA mailing list> >Subject: RE: [Linux-HA] HA Firewall> >> >I will just bump this the once. Does anybody have any > >suggestions that may help?Thanks in advance> >> >> From: northcountryboy79 at hotmail.com> To: > >linux-ha at lists.linux-ha.org> > >> Subject: RE: [Linux-HA] HA Firewall> Date: Sun, 4 Nov 2007 21:59:13 > >> +0000> > Sorry for the delay, > > Please find attached > >configs. Its a > >> curious problem...> > > > > Subject: Re: [Linux-HA] HA > >Firewall> From: > >> mzagrabe at d.umn.edu> To: linux-ha at lists.linux-ha.org> Date: > >Mon, 29 Oct > >> 2007 10:38:30 -0500> > On Thu, 2007-10-25 at 22:23 +0100, North > >> Country Boy wrote:> > Ok ok, I admit. I dont get it!!!!> > > > I am > >> trying to config a simple HA firewall and it just isnt > >working to how > >> I had imagined.> > > > Ok here is the deal.> > > > The Firewall has > >> two interfaces> > > > 1) Internal interface eth1 > >192.168.0.254> > > > > >> 2) External Interface eth0 195.63.63.100, 195.63.63.101, > >> 195.63.63.102> > > > The plan would be that in the event of failure, > >> these IP addresses as well as an iptables script would be brought > >> online on the second box.> > > > The story so far....> > > > > >Because I > >> am new to this, I wanted to take things nice and slowly and realise > >> the full solution in stages so that I could learn & understand. I > >> decided to test a simple failover with one ip just using the > >external > >> interface.> > > > I added a second nic to both machines (node1 & > >> node2) and got heartbeat working no problem. Using the verison 1 > >> haresource file, I added the following line> > > > node1 > >> 195.63.63.101> > > > In the ha.cf file I added> > > > ping > >> 195.63.63.254 (an external router accessible by both nodes)> > > > > >> Also I added the ipfail command.> > > > Ok so heartbeat all > >looks good > >> so far, the new address 195.63.63.101 is added as eth1:0 > > > >> > No I > >> prevent access to the external router from node1, it recognises that > >> it can no longer reach 195.63.63.254 in the logs, whilst node 2 says > >> and does nothing. huh????> > I thought that at this point, ipfail > >> flags a failure and the failover process begins????> > > > > >> Conicidentally, pulling the heartbeat cable causes the failover to > >> happen perfectly (which is nice to know).> > > > So now I am left > >> wondering... If my external eth0 card fails, this isnt > >enough to cause > >> failover?> > Yes, if things are configured correctly.> > I have been > >> dealing with v2 only, so I won't be able to help you with> your > >> configs, but I did play with v1 a tiny bit and I remember ipfail> > >> working fine.> > Speaking of configs, you should post your ha.cf and > >> haresources files> along with logs. I believe the list prefers > >> attachments rather than> inline.> > [...]> > -- > Matt Zagrabelny - > >> mzagrabe at d.umn.edu - (218) 726 8844> University of Minnesota Duluth> > >> Information Technology Systems & Services> PGP key 1024D/84E22DA2 > >> 2005-11-07> Fingerprint: 78F9 18B3 EF58 56F5 FC85 C5CA 53E7 > >887F 84E2 > >> 2DA2> > He is not a fool who gives up what he cannot keep to > >gain what > >> he cannot> lose.> -Jim Elliot> > > _______________________________________________> Linux-HA mailing list> Linux-HA at lists.linux-ha.org> http://lists.linux-ha.org/mailman/listinfo/linux-ha> See also: http://linux-ha.org/ReportingProblems
> _________________________________________________________________
> The next generation of MSN Hotmail has arrived - Windows Live Hotmail
> http://www.newhotmail.co.uk
> Nov 19 20:52:04 node1 logd: [9992]: info: setting log facility to daemon
> Nov 19 20:52:04 node1 logd: [9992]: info: logd started with /etc/logd.cf.
> Nov 19 20:52:04 node1 logd: [9992]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 19 20:52:04 node1 logd: [9992]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 19 20:52:04 node1 logd: [9992]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 19 20:52:04 node1 logd: [9993]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Nov 19 20:52:04 node1 logd: [9992]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Nov 19 20:52:04 node1 heartbeat: [10053]: info: Enabling logging daemon 
> Nov 19 20:52:04 node1 heartbeat: [10053]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
> Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 19 20:52:04 node1 heartbeat: [10053]: info: Version 2 support: false
> Nov 19 20:52:04 node1 heartbeat: [10053]: info: **************************
> Nov 19 20:52:04 node1 heartbeat: [10053]: info: Configuration validated. Starting heartbeat 2.1.2
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: heartbeat: version 2.1.2
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: Heartbeat generation: 1193181887
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: ping heartbeat started.
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_SignalHandler: Added signal handler for signal 17
> Nov 19 20:52:04 node1 heartbeat: [10054]: info: Local status now set to: 'up'
> Nov 19 20:52:05 node1 heartbeat: [10054]: info: Link node1:eth0 up.
> Nov 19 20:52:06 node1 heartbeat: [10054]: info: Link externalhost:externalhost up.
> Nov 19 20:52:06 node1 heartbeat: [10054]: info: Status update for node externalhost: status ping
> Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: node node2: is dead
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: Comm_now_up(): updating status to active
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: Local status now set to: 'active'
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
> Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: No STONITH device configured.
> Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Shared disks are not protected.
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: Resources being acquired from node2.
> Nov 19 20:52:35 node1 heartbeat: [10064]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 10064)
> Nov 19 20:52:35 node1 heartbeat: [10065]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:52:35 node1 ipfail: [10064]: debug: PID=10064
> Nov 19 20:52:35 node1 ipfail: [10064]: debug: Signing in with heartbeat
> Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Client [ipfail] pid 10064 failed authorization [client failed authorization]
> Nov 19 20:52:35 node1 heartbeat: [10054]: ERROR: api_process_registration_msg: cannot add client(ipfail)

Your ipfail never connects to the Heartbeat. No wonder that it
won't work.

Any chance to change your mail user agent? This one, whatever it
is, sucks.

Thanks,

Dejan

> Nov 19 20:52:35 node1 ipfail: [10064]: ERROR: No managed resources
> Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Exiting /usr/lib/heartbeat/ipfail process 10064 returned rc 100.
> Nov 19 20:52:35 node1 harc[10065]: [10073]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 20:52:35 node1 mach_down[10085]: [10123]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
> Nov 19 20:52:35 node1 mach_down[10085]: [10135]: info: mach_down takeover complete for node node2.
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: mach_down takeover complete.
> Nov 19 20:52:35 node1 heartbeat: [10054]: info: Initial resource acquisition complete (mach_down)
> Nov 19 20:52:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 20:52:35 node1 IPaddr[10139]: [10170]: INFO:  Resource is stopped
> Nov 19 20:52:35 node1 heartbeat: [10066]: info: Local Resource acquisition completed.
> Nov 19 20:52:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 20:52:35 node1 heartbeat: [10174]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:52:35 node1 harc[10174]: [10180]: info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
> Nov 19 20:52:35 node1 ip-request-resp[10174]: [10186]: received ip-request-resp 195.63.63.101 OK yes
> Nov 19 20:52:35 node1 ResourceManager[10187]: [10198]: info: Acquiring resource group: node1 195.63.63.101
> Nov 19 20:52:35 node1 IPaddr[10210]: [10241]: INFO:  Resource is stopped
> Nov 19 20:52:35 node1 ResourceManager[10187]: [10257]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 19 20:52:35 node1 ResourceManager[10187]: [10258]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 19 20:52:35 node1 IPaddr[10276]: [10307]: INFO: Using calculated nic for 195.63.63.101: eth1
> Nov 19 20:52:35 node1 IPaddr[10276]: [10312]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
> Nov 19 20:52:35 node1 IPaddr[10276]: [10317]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
> Nov 19 20:52:35 node1 IPaddr[10276]: [10334]: INFO: eval ifconfig eth1:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
> Nov 19 20:52:36 node1 IPaddr[10276]: [10339]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth1:0 [eth1]
> Nov 19 20:52:36 node1 IPaddr[10259]: [10353]: INFO:  Success
> Nov 19 20:52:36 node1 ResourceManager[10187]: [10354]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
> Nov 19 20:52:45 node1 heartbeat: [10054]: info: Local Resource acquisition completed. (none)
> Nov 19 20:52:45 node1 heartbeat: [10054]: info: local resource transition completed.
> Nov 19 20:53:35 node1 heartbeat: [10054]: info: Link node2:eth0 up.
> Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status init
> Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status up
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: get_delnodelist: delnodelist= 
> Nov 19 20:53:35 node1 heartbeat: [10356]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:53:35 node1 harc[10356]: [10363]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 20:53:35 node1 heartbeat: [10369]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:53:35 node1 harc[10369]: [10375]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 20:53:35 node1 heartbeat: [10054]: info: all clients are now paused
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->ackseq =5
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=106
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug:  
> Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status active
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->ackseq =5
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=107
> Nov 19 20:53:35 node1 heartbeat: [10054]: debug:  
> Nov 19 20:53:35 node1 heartbeat: [10381]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:53:35 node1 harc[10381]: [10387]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 20:53:36 node1 heartbeat: [10054]: info: remote resource transition completed.
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=108
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=109
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
> Nov 19 20:53:36 node1 heartbeat: [10054]: info: node1 wants to go standby [foreign]
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=110
> Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
> Nov 19 20:53:36 node1 heartbeat: [10054]: info: all clients are now resumed
> Nov 19 20:53:36 node1 heartbeat: [10054]: info: standby: node2 can take our foreign resources
> Nov 19 20:53:36 node1 heartbeat: [10393]: info: give up foreign HA resources (standby).
> Nov 19 20:53:36 node1 heartbeat: [10393]: info: foreign HA resource release completed (standby).
> Nov 19 20:53:36 node1 heartbeat: [10054]: info: Local standby process completed [foreign].
> Nov 19 20:53:37 node1 heartbeat: [10054]: WARN: 1 lost packet(s) for [node2] [13:15]
> Nov 19 20:53:37 node1 heartbeat: [10054]: info: remote resource transition completed.
> Nov 19 20:53:37 node1 heartbeat: [10054]: info: No pkts missing from node2!
> Nov 19 20:53:37 node1 heartbeat: [10054]: info: Other node completed standby takeover of foreign resources.
> Nov 19 20:55:44 node1 heartbeat: [10054]: WARN: node externalhost: is dead
> Nov 19 20:55:44 node1 heartbeat: [10054]: info: Link externalhost:externalhost dead.
> Nov 19 20:55:44 node1 heartbeat: [10408]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 20:55:44 node1 harc[10408]: [10414]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 20:59:26 node1 heartbeat: [10054]: info: Link externalhost:externalhost up.
> Nov 19 20:59:26 node1 heartbeat: [10054]: WARN: Late heartbeat: Node externalhost: interval 252090 ms
> Nov 19 20:59:26 node1 heartbeat: [10054]: info: Status update for node externalhost: status ping
> Nov 19 21:00:17 node1 syslog-ng[2337]: STATS: dropped 838
> Nov 19 21:01:50 node1 kernel: tg3: eth0: Link is down.
> Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: node node2: is dead
> Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: No STONITH device configured.
> Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: Shared disks are not protected.
> Nov 19 21:02:20 node1 heartbeat: [10054]: info: Resources being acquired from node2.
> Nov 19 21:02:20 node1 heartbeat: [10054]: info: Link node2:eth0 dead.
> Nov 19 21:02:20 node1 heartbeat: [10451]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:02:20 node1 harc[10451]: [10461]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 21:02:20 node1 mach_down[10473]: [10514]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
> Nov 19 21:02:20 node1 mach_down[10473]: [10524]: info: mach_down takeover complete for node node2.
> Nov 19 21:02:20 node1 heartbeat: [10054]: info: mach_down takeover complete.
> Nov 19 21:02:20 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 21:02:20 node1 IPaddr[10525]: [10556]: INFO:  Running OK
> Nov 19 21:02:20 node1 heartbeat: [10452]: info: Local Resource acquisition completed.
> Nov 19 21:03:36 node1 kernel: tg3: eth0: Link is up at 100 Mbps, full duplex.
> Nov 19 21:03:36 node1 kernel: tg3: eth0: Flow control is on for TX and on for RX.
> Nov 19 21:03:38 node1 heartbeat: [10054]: CRIT: Cluster node node2 returning after partition.
> Nov 19 21:03:38 node1 heartbeat: [10054]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
> Nov 19 21:03:38 node1 heartbeat: [10054]: WARN: Deadtime value may be too small.
> Nov 19 21:03:38 node1 heartbeat: [10054]: info: See FAQ for information on tuning deadtime.
> Nov 19 21:03:38 node1 heartbeat: [10054]: info: URL: http://linux-ha.org/FAQ#heavy_load
> Nov 19 21:03:38 node1 heartbeat: [10054]: info: Link node2:eth0 up.
> Nov 19 21:03:38 node1 heartbeat: [10054]: WARN: Late heartbeat: Node node2: interval 108070 ms
> Nov 19 21:03:38 node1 heartbeat: [10054]: info: Status update for node node2: status active
> Nov 19 21:03:38 node1 heartbeat: [10559]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:03:38 node1 harc[10559]: [10565]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 21:03:39 node1 heartbeat: [10054]: info: Received shutdown notice from 'node2'.
> Nov 19 21:03:39 node1 heartbeat: [10054]: info: Resources being acquired from node2.
> Nov 19 21:03:39 node1 heartbeat: [10571]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:03:39 node1 harc[10571]: [10579]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 21:03:39 node1 mach_down[10590]: [10629]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
> Nov 19 21:03:39 node1 mach_down[10590]: [10640]: info: mach_down takeover complete for node node2.
> Nov 19 21:03:39 node1 heartbeat: [10054]: info: mach_down takeover complete.
> Nov 19 21:03:39 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 21:03:39 node1 IPaddr[10645]: [10676]: INFO:  Running OK
> Nov 19 21:03:39 node1 heartbeat: [10572]: info: Local Resource acquisition completed.
> Nov 19 21:03:40 node1 heartbeat: [10054]: info: Heartbeat shutdown in progress. (10054)
> Nov 19 21:03:40 node1 heartbeat: [10677]: info: Giving up all HA resources.
> Nov 19 21:03:40 node1 ResourceManager[10690]: [10701]: info: Releasing resource group: node1 195.63.63.101
> Nov 19 21:03:40 node1 ResourceManager[10690]: [10718]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
> Nov 19 21:03:40 node1 ResourceManager[10690]: [10719]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
> Nov 19 21:03:40 node1 IPaddr[10737]: [10752]: INFO: ifconfig eth1:0 down
> Nov 19 21:03:40 node1 heartbeat: [10677]: info: All HA resources relinquished.
> Nov 19 21:03:40 node1 IPaddr[10720]: [10755]: INFO:  Success
> Nov 19 21:03:40 node1 ResourceManager[10690]: [10756]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop done. RC=0
> Nov 19 21:03:41 node1 heartbeat: [10054]: info: client [/usr/lib/heartbeat/ipfail] is not running.
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBFIFO process 10056 with signal 15
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBWRITE process 10057 with signal 15
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBREAD process 10058 with signal 15
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBWRITE process 10059 with signal 15
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBREAD process 10060 with signal 15
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10060 exited. 5 remaining
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10059 exited. 4 remaining
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10058 exited. 3 remaining
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10057 exited. 2 remaining
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10056 exited. 1 remaining
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: node1 Heartbeat shutdown complete.
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Heartbeat restart triggered.
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Restarting heartbeat.
> Nov 19 21:03:42 node1 heartbeat: [10054]: info: Performing heartbeat restart exec.
> Nov 19 21:04:13 node1 heartbeat: [10054]: info: Enabling logging daemon 
> Nov 19 21:04:13 node1 heartbeat: [10054]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
> Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 19 21:04:13 node1 heartbeat: [10054]: info: Version 2 support: false
> Nov 19 21:04:13 node1 heartbeat: [10054]: info: **************************
> Nov 19 21:04:13 node1 heartbeat: [10054]: info: Configuration validated. Starting heartbeat 2.1.2
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: heartbeat: version 2.1.2
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: Heartbeat generation: 1193181888
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: ping heartbeat started.
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_SignalHandler: Added signal handler for signal 17
> Nov 19 21:04:13 node1 heartbeat: [10757]: info: Local status now set to: 'up'
> Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link node2:eth0 up.
> Nov 19 21:04:14 node1 heartbeat: [10757]: info: Status update for node node2: status up
> Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link node1:eth0 up.
> Nov 19 21:04:14 node1 heartbeat: [10765]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link externalhost:externalhost up.
> Nov 19 21:04:14 node1 heartbeat: [10757]: info: Status update for node externalhost: status ping
> Nov 19 21:04:14 node1 harc[10765]: [10772]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 21:04:15 node1 heartbeat: [10757]: debug: get_delnodelist: delnodelist= 
> Nov 19 21:04:15 node1 heartbeat: [10757]: info: Comm_now_up(): updating status to active
> Nov 19 21:04:15 node1 heartbeat: [10757]: info: Local status now set to: 'active'
> Nov 19 21:04:15 node1 heartbeat: [10757]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
> Nov 19 21:04:15 node1 heartbeat: [10778]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 10778)
> Nov 19 21:04:15 node1 ipfail: [10778]: debug: PID=10778
> Nov 19 21:04:15 node1 ipfail: [10778]: debug: Signing in with heartbeat
> Nov 19 21:04:15 node1 heartbeat: [10757]: WARN: Client [ipfail] pid 10778 failed authorization [client failed authorization]
> Nov 19 21:04:15 node1 heartbeat: [10757]: ERROR: api_process_registration_msg: cannot add client(ipfail)
> Nov 19 21:04:15 node1 ipfail: [10778]: ERROR: No managed resources
> Nov 19 21:04:15 node1 heartbeat: [10757]: WARN: Exiting /usr/lib/heartbeat/ipfail process 10778 returned rc 100.
> Nov 19 21:04:16 node1 heartbeat: [10757]: info: Status update for node node2: status active
> Nov 19 21:04:16 node1 heartbeat: [10779]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:04:16 node1 harc[10779]: [10785]: info: Running /etc/ha.d/rc.d/status status
> Nov 19 21:04:26 node1 heartbeat: [10757]: info: remote resource transition completed.
> Nov 19 21:04:26 node1 heartbeat: [10757]: info: remote resource transition completed.
> Nov 19 21:04:26 node1 heartbeat: [10757]: info: Initial resource acquisition complete (T_RESOURCES(us))
> Nov 19 21:04:27 node1 IPaddr[10827]: [10858]: INFO:  Resource is stopped
> Nov 19 21:04:27 node1 heartbeat: [10791]: info: Local Resource acquisition completed.
> Nov 19 21:04:27 node1 heartbeat: [10757]: debug: StartNextRemoteRscReq(): child count 1
> Nov 19 21:04:27 node1 heartbeat: [10862]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 19 21:04:27 node1 harc[10862]: [10868]: info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
> Nov 19 21:04:27 node1 ip-request-resp[10862]: [10874]: received ip-request-resp 195.63.63.101 OK yes
> Nov 19 21:04:27 node1 ResourceManager[10875]: [10886]: info: Acquiring resource group: node1 195.63.63.101
> Nov 19 21:04:27 node1 IPaddr[10898]: [10929]: INFO:  Resource is stopped
> Nov 19 21:04:27 node1 ResourceManager[10875]: [10945]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 19 21:04:27 node1 ResourceManager[10875]: [10946]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 19 21:04:27 node1 IPaddr[10964]: [10995]: INFO: Using calculated nic for 195.63.63.101: eth1
> Nov 19 21:04:27 node1 IPaddr[10964]: [11000]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
> Nov 19 21:04:27 node1 IPaddr[10964]: [11005]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
> Nov 19 21:04:27 node1 IPaddr[10964]: [11022]: INFO: eval ifconfig eth1:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
> Nov 19 21:04:27 node1 IPaddr[10964]: [11027]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth1:0 [eth1]
> Nov 19 21:04:27 node1 IPaddr[10947]: [11041]: INFO:  Success
> Nov 19 21:04:27 node1 ResourceManager[10875]: [11042]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
> Nov 19 21:06:14 node1 sshd[11043]: Accepted keyboard-interactive/pam for root from 195.63.63.254 port 1322 ssh2
> Nov 19 21:06:17 node1 sshd[11043]: subsystem request for sftp

> Nov 18 19:52:45 node2 logd: [16885]: info: setting log facility to daemon
> Nov 18 19:52:45 node2 logd: [16885]: info: logd started with /etc/logd.cf.
> Nov 18 19:52:45 node2 logd: [16885]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 18 19:52:45 node2 logd: [16885]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 18 19:52:45 node2 logd: [16885]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 18 19:52:45 node2 logd: [16886]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Nov 18 19:52:45 node2 logd: [16885]: info: G_main_add_SignalHandler: Added signal handler for signal 15
> Nov 18 19:52:45 node2 heartbeat: [16946]: info: Enabling logging daemon 
> Nov 18 19:52:45 node2 heartbeat: [16946]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
> Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 18 19:52:45 node2 heartbeat: [16946]: info: Version 2 support: false
> Nov 18 19:52:45 node2 heartbeat: [16946]: info: **************************
> Nov 18 19:52:45 node2 heartbeat: [16946]: info: Configuration validated. Starting heartbeat 2.1.2
> Nov 18 19:52:45 node2 heartbeat: [16947]: info: heartbeat: version 2.1.2
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: Heartbeat generation: 1193174534
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: ping heartbeat started.
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_SignalHandler: Added signal handler for signal 17
> Nov 18 19:52:46 node2 heartbeat: [16947]: info: Local status now set to: 'up'
> Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link node1:eth0 up.
> Nov 18 19:52:47 node2 heartbeat: [16947]: info: Status update for node node1: status active
> Nov 18 19:52:47 node2 heartbeat: [16955]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 18 19:52:47 node2 harc[16955]: [16962]: info: Running /etc/ha.d/rc.d/status status
> Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link node2:eth0 up.
> Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link externalhost:externalhost up.
> Nov 18 19:52:47 node2 heartbeat: [16947]: info: Status update for node externalhost: status ping
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: Comm_now_up(): updating status to active
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: Local status now set to: 'active'
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
> Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 370 ms (> 50 ms) (GSource: 0x81014c0)
> Nov 18 19:52:48 node2 heartbeat: [16968]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 16968)
> Nov 18 19:52:48 node2 ipfail: [16968]: debug: PID=16968
> Nov 18 19:52:48 node2 ipfail: [16968]: debug: Signing in with heartbeat
> Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: Client [ipfail] pid 16968 failed authorization [client failed authorization]
> Nov 18 19:52:48 node2 heartbeat: [16947]: ERROR: api_process_registration_msg: cannot add client(ipfail)
> Nov 18 19:52:48 node2 ipfail: [16968]: ERROR: No managed resources
> Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: Exiting /usr/lib/heartbeat/ipfail process 16968 returned rc 100.
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: remote resource transition completed.
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: remote resource transition completed.
> Nov 18 19:52:48 node2 heartbeat: [16947]: info: Local Resource acquisition completed. (none)
> Nov 18 19:52:49 node2 heartbeat: [16947]: info: node1 wants to go standby [foreign]
> Nov 18 19:52:49 node2 heartbeat: [16947]: info: standby: acquire [foreign] resources from node1
> Nov 18 19:52:49 node2 heartbeat: [16969]: info: acquire local HA resources (standby).
> Nov 18 19:52:49 node2 heartbeat: [16969]: info: local HA resource acquisition completed (standby).
> Nov 18 19:52:49 node2 heartbeat: [16947]: info: Standby resource acquisition done [foreign].
> Nov 18 19:52:49 node2 heartbeat: [16947]: info: Initial resource acquisition complete (auto_failback)
> Nov 18 19:52:50 node2 heartbeat: [16947]: info: remote resource transition completed.
> Nov 18 19:58:36 node2 syslog-ng[2236]: STATS: dropped 0
> Nov 18 20:01:04 node2 kernel: e100: eth0: e100_watchdog: link down
> Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: node node1: is dead
> Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: No STONITH device configured.
> Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: Shared disks are not protected.
> Nov 18 20:01:32 node2 heartbeat: [16947]: info: Resources being acquired from node1.
> Nov 18 20:01:32 node2 heartbeat: [16947]: info: Link node1:eth0 dead.
> Nov 18 20:01:32 node2 heartbeat: [17005]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 18 20:01:32 node2 harc[17005]: [17024]: info: Running /etc/ha.d/rc.d/status status
> Nov 18 20:01:32 node2 heartbeat: [17006]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys node2] to acquire.
> Nov 18 20:01:32 node2 heartbeat: [16947]: debug: StartNextRemoteRscReq(): child count 1
> Nov 18 20:01:32 node2 mach_down[17030]: [17051]: info: Taking over resource group 195.63.63.101
> Nov 18 20:01:32 node2 ResourceManager[17052]: [17063]: info: Acquiring resource group: node1 195.63.63.101
> Nov 18 20:01:32 node2 IPaddr[17075]: [17106]: INFO:  Resource is stopped
> Nov 18 20:01:32 node2 ResourceManager[17052]: [17122]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 18 20:01:32 node2 ResourceManager[17052]: [17123]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
> Nov 18 20:01:33 node2 IPaddr[17141]: [17172]: INFO: Using calculated nic for 195.63.63.101: eth2
> Nov 18 20:01:33 node2 IPaddr[17141]: [17177]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
> Nov 18 20:01:33 node2 IPaddr[17141]: [17182]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
> Nov 18 20:01:33 node2 IPaddr[17141]: [17199]: INFO: eval ifconfig eth2:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
> Nov 18 20:01:33 node2 IPaddr[17141]: [17204]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth2:0 [eth2]
> Nov 18 20:01:33 node2 IPaddr[17124]: [17218]: INFO:  Success
> Nov 18 20:01:33 node2 ResourceManager[17052]: [17219]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
> Nov 18 20:01:33 node2 mach_down[17030]: [17220]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
> Nov 18 20:01:33 node2 mach_down[17030]: [17224]: info: mach_down takeover complete for node node1.
> Nov 18 20:01:33 node2 heartbeat: [16947]: info: mach_down takeover complete.
> Nov 18 20:01:33 node2 kernel: NET: Registered protocol family 17
> Nov 18 20:02:49 node2 heartbeat: [16947]: CRIT: Cluster node node1 returning after partition.
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
> Nov 18 20:02:49 node2 heartbeat: [16947]: WARN: Deadtime value may be too small.
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: See FAQ for information on tuning deadtime.
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: URL: http://linux-ha.org/FAQ#heavy_load
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: Link node1:eth0 up.
> Nov 18 20:02:49 node2 heartbeat: [16947]: WARN: Late heartbeat: Node node1: interval 106990 ms
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: Status update for node node1: status active
> Nov 18 20:02:49 node2 heartbeat: [17230]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 18 20:02:49 node2 harc[17230]: [17236]: info: Running /etc/ha.d/rc.d/status status
> Nov 18 20:02:49 node2 heartbeat: [16947]: info: all clients are now paused
> Nov 18 20:02:49 node2 heartbeat: [16947]: debug: hist->ackseq =520
> Nov 18 20:02:49 node2 heartbeat: [16947]: debug: hist->lowseq =519, hist->hiseq=621
> Nov 18 20:02:49 node2 heartbeat: [16947]: debug:  
> Nov 18 20:02:50 node2 kernel: e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
> Nov 18 20:02:50 node2 heartbeat: [16947]: debug: hist->ackseq =520
> Nov 18 20:02:50 node2 heartbeat: [16947]: debug: hist->lowseq =519, hist->hiseq=622
> Nov 18 20:02:50 node2 heartbeat: [16947]: debug:  
> Nov 18 20:02:51 node2 heartbeat: [16947]: info: all clients are now resumed
> Nov 18 20:02:51 node2 heartbeat: [16947]: info: Heartbeat shutdown in progress. (16947)
> Nov 18 20:02:51 node2 heartbeat: [17242]: info: Giving up all HA resources.
> Nov 18 20:02:51 node2 ResourceManager[17255]: [17266]: info: Releasing resource group: node1 195.63.63.101
> Nov 18 20:02:51 node2 ResourceManager[17255]: [17283]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
> Nov 18 20:02:51 node2 ResourceManager[17255]: [17284]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
> Nov 18 20:02:51 node2 IPaddr[17302]: [17317]: INFO: ifconfig eth2:0 down
> Nov 18 20:02:51 node2 IPaddr[17285]: [17320]: INFO:  Success
> Nov 18 20:02:51 node2 ResourceManager[17255]: [17321]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop done. RC=0
> Nov 18 20:02:51 node2 heartbeat: [17242]: info: All HA resources relinquished.
> Nov 18 20:02:52 node2 heartbeat: [16947]: info: client [/usr/lib/heartbeat/ipfail] is not running.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Received shutdown notice from 'node1'.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Resource takeover cancelled - shutdown in progress.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBREAD process 16951 with signal 15
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBWRITE process 16952 with signal 15
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBREAD process 16953 with signal 15
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBFIFO process 16949 with signal 15
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBWRITE process 16950 with signal 15
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16953 exited. 5 remaining
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16952 exited. 4 remaining
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16951 exited. 3 remaining
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16950 exited. 2 remaining
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16949 exited. 1 remaining
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: node2 Heartbeat shutdown complete.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Heartbeat restart triggered.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Restarting heartbeat.
> Nov 18 20:02:53 node2 heartbeat: [16947]: info: Performing heartbeat restart exec.
> Nov 18 20:03:24 node2 heartbeat: [16947]: info: Enabling logging daemon 
> Nov 18 20:03:24 node2 heartbeat: [16947]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
> Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Core dumps could be lost if multiple dumps occur.
> Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
> Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
> Nov 18 20:03:24 node2 heartbeat: [16947]: info: Version 2 support: false
> Nov 18 20:03:24 node2 heartbeat: [16947]: info: **************************
> Nov 18 20:03:24 node2 heartbeat: [16947]: info: Configuration validated. Starting heartbeat 2.1.2
> Nov 18 20:03:24 node2 heartbeat: [17322]: info: heartbeat: version 2.1.2
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: Heartbeat generation: 1193174535
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_TriggerHandler: Added signal manual handler
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: ping heartbeat started.
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_SignalHandler: Added signal handler for signal 17
> Nov 18 20:03:25 node2 heartbeat: [17322]: info: Local status now set to: 'up'
> Nov 18 20:03:26 node2 heartbeat: [17322]: info: Link node2:eth0 up.
> Nov 18 20:03:27 node2 heartbeat: [17322]: info: Link externalhost:externalhost up.
> Nov 18 20:03:27 node2 heartbeat: [17322]: info: Status update for node externalhost: status ping
> Nov 18 20:03:27 node2 heartbeat: [17322]: info: Link node1:eth0 up.
> Nov 18 20:03:27 node2 heartbeat: [17322]: info: Status update for node node1: status up
> Nov 18 20:03:27 node2 heartbeat: [17322]: debug: get_delnodelist: delnodelist= 
> Nov 18 20:03:27 node2 heartbeat: [17330]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 18 20:03:27 node2 harc[17330]: [17337]: info: Running /etc/ha.d/rc.d/status status
> Nov 18 20:03:28 node2 heartbeat: [17322]: info: Comm_now_up(): updating status to active
> Nov 18 20:03:28 node2 heartbeat: [17322]: info: Local status now set to: 'active'
> Nov 18 20:03:28 node2 heartbeat: [17322]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
> Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 480 ms (> 50 ms) (GSource: 0x8101d38)
> Nov 18 20:03:28 node2 heartbeat: [17322]: info: Status update for node node1: status active
> Nov 18 20:03:28 node2 heartbeat: [17343]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 17343)
> Nov 18 20:03:28 node2 heartbeat: [17344]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
> Nov 18 20:03:28 node2 harc[17344]: [17350]: info: Running /etc/ha.d/rc.d/status status
> Nov 18 20:03:28 node2 ipfail: [17343]: debug: PID=17343
> Nov 18 20:03:28 node2 ipfail: [17343]: debug: Signing in with heartbeat
> Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: Client [ipfail] pid 17343 failed authorization [client failed authorization]
> Nov 18 20:03:28 node2 heartbeat: [17322]: ERROR: api_process_registration_msg: cannot add client(ipfail)
> Nov 18 20:03:28 node2 ipfail: [17343]: ERROR: No managed resources
> Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: Exiting /usr/lib/heartbeat/ipfail process 17343 returned rc 100.
> Nov 18 20:03:38 node2 heartbeat: [17322]: info: local resource transition completed.
> Nov 18 20:03:38 node2 heartbeat: [17322]: info: Initial resource acquisition complete (T_RESOURCES(us))
> Nov 18 20:03:39 node2 heartbeat: [17356]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys node2] to acquire.
> Nov 18 20:03:39 node2 heartbeat: [17322]: info: remote resource transition completed.
> Nov 18 20:07:04 node2 sshd[17369]: Accepted keyboard-interactive/pam for root from 195.63.63.254 port 1324 ssh2
> Nov 18 20:07:08 node2 sshd[17369]: subsystem request for sftp

> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems