[Linux-HA] HA Firewall

[North Country Boy]

Hello again.  Please find attached my logs from both node1 and node2.
 
Using node1 logs as reference, here are the sequence of events for brevity20:52 start node1 & node220.55 node1 can no longer reach the external host.  Node 2 does not log anything for this event!20:59 node1 can now reach external host.  Node 2 does not log anything for this event!21:01 heartbeat (eth0) cable pulled out.  Both machines recognise and a failover begins, node 2 logs this and failover is successful21.03 heartbeat (eth0) plugged back in .  Both machines respond and failover completes successfully.
It just seems that this will not failover unless the heartbeart connection fails!! 
 
Joris - thanks for the suggestion I will certainly take a look to see what other solutions people use. However I really want to crack this using heartbeat so that I can apply this to other situations such as webservers etc.  For the purpose of this firewall I am not really interested in state table failover because the idea is just to keep an online presence in the event of failure overnight.



> Subject: RE: [Linux-HA] HA Firewall> Date: Fri, 16 Nov 2007 01:14:58 +0100> From: Joris at familiedobbelsteen.nl> To: linux-ha at lists.linux-ha.org> > If you are looking for a highly available stateful firewall, check out> OpenBSD or FreeBSD with the PF firewall. It includes pfsync which allows> state synchronization. It also includes CARP for IP address failover.> > I have found nothing equivalent on Linux that provides the same> capabilities for high availability.> > Perhaps a good 'distribution' is pfsense, which packages it all> (FreeBSD+PF+CARP+more) including a web interface. There is plenty of> documentation on the web avaiable for such a setup...> > - Joris> > >-----Original Message-----> >From: linux-ha-bounces at lists.linux-ha.org > >[mailto:linux-ha-bounces at lists.linux-ha.org] On Behalf Of > >North Country Boy> >Sent: woensdag 14 november 2007 23:31> >To: General Linux-HA mailing list> >Subject: RE: [Linux-HA] HA Firewall> >> >I will just bump this the once. Does anybody have any > >suggestions that may help?Thanks in advance> >> >> From: northcountryboy79 at hotmail.com> To: > >linux-ha at lists.linux-ha.org> > >> Subject: RE: [Linux-HA] HA Firewall> Date: Sun, 4 Nov 2007 21:59:13 > >> +0000> > Sorry for the delay, > > Please find attached > >configs. Its a > >> curious problem...> > > > > Subject: Re: [Linux-HA] HA > >Firewall> From: > >> mzagrabe at d.umn.edu> To: linux-ha at lists.linux-ha.org> Date: > >Mon, 29 Oct > >> 2007 10:38:30 -0500> > On Thu, 2007-10-25 at 22:23 +0100, North > >> Country Boy wrote:> > Ok ok, I admit. I dont get it!!!!> > > > I am > >> trying to config a simple HA firewall and it just isnt > >working to how > >> I had imagined.> > > > Ok here is the deal.> > > > The Firewall has > >> two interfaces> > > > 1) Internal interface eth1 > >192.168.0.254> > > > > >> 2) External Interface eth0 195.63.63.100, 195.63.63.101, > >> 195.63.63.102> > > > The plan would be that in the event of failure, > >> these IP addresses as well as an iptables script would be brought > >> online on the second box.> > > > The story so far....> > > > > >Because I > >> am new to this, I wanted to take things nice and slowly and realise > >> the full solution in stages so that I could learn & understand. I > >> decided to test a simple failover with one ip just using the > >external > >> interface.> > > > I added a second nic to both machines (node1 & > >> node2) and got heartbeat working no problem. Using the verison 1 > >> haresource file, I added the following line> > > > node1 > >> 195.63.63.101> > > > In the ha.cf file I added> > > > ping > >> 195.63.63.254 (an external router accessible by both nodes)> > > > > >> Also I added the ipfail command.> > > > Ok so heartbeat all > >looks good > >> so far, the new address 195.63.63.101 is added as eth1:0 > > > >> > No I > >> prevent access to the external router from node1, it recognises that > >> it can no longer reach 195.63.63.254 in the logs, whilst node 2 says > >> and does nothing. huh????> > I thought that at this point, ipfail > >> flags a failure and the failover process begins????> > > > > >> Conicidentally, pulling the heartbeat cable causes the failover to > >> happen perfectly (which is nice to know).> > > > So now I am left > >> wondering... If my external eth0 card fails, this isnt > >enough to cause > >> failover?> > Yes, if things are configured correctly.> > I have been > >> dealing with v2 only, so I won't be able to help you with> your > >> configs, but I did play with v1 a tiny bit and I remember ipfail> > >> working fine.> > Speaking of configs, you should post your ha.cf and > >> haresources files> along with logs. I believe the list prefers > >> attachments rather than> inline.> > [...]> > -- > Matt Zagrabelny - > >> mzagrabe at d.umn.edu - (218) 726 8844> University of Minnesota Duluth> > >> Information Technology Systems & Services> PGP key 1024D/84E22DA2 > >> 2005-11-07> Fingerprint: 78F9 18B3 EF58 56F5 FC85 C5CA 53E7 > >887F 84E2 > >> 2DA2> > He is not a fool who gives up what he cannot keep to > >gain what > >> he cannot> lose.> -Jim Elliot> > > _______________________________________________> Linux-HA mailing list> Linux-HA at lists.linux-ha.org> http://lists.linux-ha.org/mailman/listinfo/linux-ha> See also: http://linux-ha.org/ReportingProblems
_________________________________________________________________
The next generation of MSN Hotmail has arrived - Windows Live Hotmail
http://www.newhotmail.co.uk
-------------- next part --------------
Nov 19 20:52:04 node1 logd: [9992]: info: setting log facility to daemon
Nov 19 20:52:04 node1 logd: [9992]: info: logd started with /etc/logd.cf.
Nov 19 20:52:04 node1 logd: [9992]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 19 20:52:04 node1 logd: [9992]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 19 20:52:04 node1 logd: [9992]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 19 20:52:04 node1 logd: [9993]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Nov 19 20:52:04 node1 logd: [9992]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Nov 19 20:52:04 node1 heartbeat: [10053]: info: Enabling logging daemon 
Nov 19 20:52:04 node1 heartbeat: [10053]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 19 20:52:04 node1 heartbeat: [10053]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 19 20:52:04 node1 heartbeat: [10053]: info: Version 2 support: false
Nov 19 20:52:04 node1 heartbeat: [10053]: info: **************************
Nov 19 20:52:04 node1 heartbeat: [10053]: info: Configuration validated. Starting heartbeat 2.1.2
Nov 19 20:52:04 node1 heartbeat: [10054]: info: heartbeat: version 2.1.2
Nov 19 20:52:04 node1 heartbeat: [10054]: info: Heartbeat generation: 1193181887
Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 19 20:52:04 node1 heartbeat: [10054]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Nov 19 20:52:04 node1 heartbeat: [10054]: info: glib: ping heartbeat started.
Nov 19 20:52:04 node1 heartbeat: [10054]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Nov 19 20:52:04 node1 heartbeat: [10054]: info: Local status now set to: 'up'
Nov 19 20:52:05 node1 heartbeat: [10054]: info: Link node1:eth0 up.
Nov 19 20:52:06 node1 heartbeat: [10054]: info: Link externalhost:externalhost up.
Nov 19 20:52:06 node1 heartbeat: [10054]: info: Status update for node externalhost: status ping
Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: node node2: is dead
Nov 19 20:52:35 node1 heartbeat: [10054]: info: Comm_now_up(): updating status to active
Nov 19 20:52:35 node1 heartbeat: [10054]: info: Local status now set to: 'active'
Nov 19 20:52:35 node1 heartbeat: [10054]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: No STONITH device configured.
Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Shared disks are not protected.
Nov 19 20:52:35 node1 heartbeat: [10054]: info: Resources being acquired from node2.
Nov 19 20:52:35 node1 heartbeat: [10064]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 10064)
Nov 19 20:52:35 node1 heartbeat: [10065]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:52:35 node1 ipfail: [10064]: debug: PID=10064
Nov 19 20:52:35 node1 ipfail: [10064]: debug: Signing in with heartbeat
Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Client [ipfail] pid 10064 failed authorization [client failed authorization]
Nov 19 20:52:35 node1 heartbeat: [10054]: ERROR: api_process_registration_msg: cannot add client(ipfail)
Nov 19 20:52:35 node1 ipfail: [10064]: ERROR: No managed resources
Nov 19 20:52:35 node1 heartbeat: [10054]: WARN: Exiting /usr/lib/heartbeat/ipfail process 10064 returned rc 100.
Nov 19 20:52:35 node1 harc[10065]: [10073]: info: Running /etc/ha.d/rc.d/status status
Nov 19 20:52:35 node1 mach_down[10085]: [10123]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Nov 19 20:52:35 node1 mach_down[10085]: [10135]: info: mach_down takeover complete for node node2.
Nov 19 20:52:35 node1 heartbeat: [10054]: info: mach_down takeover complete.
Nov 19 20:52:35 node1 heartbeat: [10054]: info: Initial resource acquisition complete (mach_down)
Nov 19 20:52:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 20:52:35 node1 IPaddr[10139]: [10170]: INFO:  Resource is stopped
Nov 19 20:52:35 node1 heartbeat: [10066]: info: Local Resource acquisition completed.
Nov 19 20:52:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 20:52:35 node1 heartbeat: [10174]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:52:35 node1 harc[10174]: [10180]: info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
Nov 19 20:52:35 node1 ip-request-resp[10174]: [10186]: received ip-request-resp 195.63.63.101 OK yes
Nov 19 20:52:35 node1 ResourceManager[10187]: [10198]: info: Acquiring resource group: node1 195.63.63.101
Nov 19 20:52:35 node1 IPaddr[10210]: [10241]: INFO:  Resource is stopped
Nov 19 20:52:35 node1 ResourceManager[10187]: [10257]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 19 20:52:35 node1 ResourceManager[10187]: [10258]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 19 20:52:35 node1 IPaddr[10276]: [10307]: INFO: Using calculated nic for 195.63.63.101: eth1
Nov 19 20:52:35 node1 IPaddr[10276]: [10312]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
Nov 19 20:52:35 node1 IPaddr[10276]: [10317]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
Nov 19 20:52:35 node1 IPaddr[10276]: [10334]: INFO: eval ifconfig eth1:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
Nov 19 20:52:36 node1 IPaddr[10276]: [10339]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth1:0 [eth1]
Nov 19 20:52:36 node1 IPaddr[10259]: [10353]: INFO:  Success
Nov 19 20:52:36 node1 ResourceManager[10187]: [10354]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
Nov 19 20:52:45 node1 heartbeat: [10054]: info: Local Resource acquisition completed. (none)
Nov 19 20:52:45 node1 heartbeat: [10054]: info: local resource transition completed.
Nov 19 20:53:35 node1 heartbeat: [10054]: info: Link node2:eth0 up.
Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status init
Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status up
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: get_delnodelist: delnodelist= 
Nov 19 20:53:35 node1 heartbeat: [10356]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:53:35 node1 harc[10356]: [10363]: info: Running /etc/ha.d/rc.d/status status
Nov 19 20:53:35 node1 heartbeat: [10369]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:53:35 node1 harc[10369]: [10375]: info: Running /etc/ha.d/rc.d/status status
Nov 19 20:53:35 node1 heartbeat: [10054]: info: all clients are now paused
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->ackseq =5
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=106
Nov 19 20:53:35 node1 heartbeat: [10054]: debug:  
Nov 19 20:53:35 node1 heartbeat: [10054]: info: Status update for node node2: status active
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->ackseq =5
Nov 19 20:53:35 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=107
Nov 19 20:53:35 node1 heartbeat: [10054]: debug:  
Nov 19 20:53:35 node1 heartbeat: [10381]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:53:35 node1 harc[10381]: [10387]: info: Running /etc/ha.d/rc.d/status status
Nov 19 20:53:36 node1 heartbeat: [10054]: info: remote resource transition completed.
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=108
Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=109
Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
Nov 19 20:53:36 node1 heartbeat: [10054]: info: node1 wants to go standby [foreign]
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->ackseq =5
Nov 19 20:53:36 node1 heartbeat: [10054]: debug: hist->lowseq =4, hist->hiseq=110
Nov 19 20:53:36 node1 heartbeat: [10054]: debug:  
Nov 19 20:53:36 node1 heartbeat: [10054]: info: all clients are now resumed
Nov 19 20:53:36 node1 heartbeat: [10054]: info: standby: node2 can take our foreign resources
Nov 19 20:53:36 node1 heartbeat: [10393]: info: give up foreign HA resources (standby).
Nov 19 20:53:36 node1 heartbeat: [10393]: info: foreign HA resource release completed (standby).
Nov 19 20:53:36 node1 heartbeat: [10054]: info: Local standby process completed [foreign].
Nov 19 20:53:37 node1 heartbeat: [10054]: WARN: 1 lost packet(s) for [node2] [13:15]
Nov 19 20:53:37 node1 heartbeat: [10054]: info: remote resource transition completed.
Nov 19 20:53:37 node1 heartbeat: [10054]: info: No pkts missing from node2!
Nov 19 20:53:37 node1 heartbeat: [10054]: info: Other node completed standby takeover of foreign resources.
Nov 19 20:55:44 node1 heartbeat: [10054]: WARN: node externalhost: is dead
Nov 19 20:55:44 node1 heartbeat: [10054]: info: Link externalhost:externalhost dead.
Nov 19 20:55:44 node1 heartbeat: [10408]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 20:55:44 node1 harc[10408]: [10414]: info: Running /etc/ha.d/rc.d/status status
Nov 19 20:59:26 node1 heartbeat: [10054]: info: Link externalhost:externalhost up.
Nov 19 20:59:26 node1 heartbeat: [10054]: WARN: Late heartbeat: Node externalhost: interval 252090 ms
Nov 19 20:59:26 node1 heartbeat: [10054]: info: Status update for node externalhost: status ping
Nov 19 21:00:17 node1 syslog-ng[2337]: STATS: dropped 838
Nov 19 21:01:50 node1 kernel: tg3: eth0: Link is down.
Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: node node2: is dead
Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: No STONITH device configured.
Nov 19 21:02:20 node1 heartbeat: [10054]: WARN: Shared disks are not protected.
Nov 19 21:02:20 node1 heartbeat: [10054]: info: Resources being acquired from node2.
Nov 19 21:02:20 node1 heartbeat: [10054]: info: Link node2:eth0 dead.
Nov 19 21:02:20 node1 heartbeat: [10451]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:02:20 node1 harc[10451]: [10461]: info: Running /etc/ha.d/rc.d/status status
Nov 19 21:02:20 node1 mach_down[10473]: [10514]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Nov 19 21:02:20 node1 mach_down[10473]: [10524]: info: mach_down takeover complete for node node2.
Nov 19 21:02:20 node1 heartbeat: [10054]: info: mach_down takeover complete.
Nov 19 21:02:20 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 21:02:20 node1 IPaddr[10525]: [10556]: INFO:  Running OK
Nov 19 21:02:20 node1 heartbeat: [10452]: info: Local Resource acquisition completed.
Nov 19 21:03:36 node1 kernel: tg3: eth0: Link is up at 100 Mbps, full duplex.
Nov 19 21:03:36 node1 kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 19 21:03:38 node1 heartbeat: [10054]: CRIT: Cluster node node2 returning after partition.
Nov 19 21:03:38 node1 heartbeat: [10054]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Nov 19 21:03:38 node1 heartbeat: [10054]: WARN: Deadtime value may be too small.
Nov 19 21:03:38 node1 heartbeat: [10054]: info: See FAQ for information on tuning deadtime.
Nov 19 21:03:38 node1 heartbeat: [10054]: info: URL: http://linux-ha.org/FAQ#heavy_load
Nov 19 21:03:38 node1 heartbeat: [10054]: info: Link node2:eth0 up.
Nov 19 21:03:38 node1 heartbeat: [10054]: WARN: Late heartbeat: Node node2: interval 108070 ms
Nov 19 21:03:38 node1 heartbeat: [10054]: info: Status update for node node2: status active
Nov 19 21:03:38 node1 heartbeat: [10559]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:03:38 node1 harc[10559]: [10565]: info: Running /etc/ha.d/rc.d/status status
Nov 19 21:03:39 node1 heartbeat: [10054]: info: Received shutdown notice from 'node2'.
Nov 19 21:03:39 node1 heartbeat: [10054]: info: Resources being acquired from node2.
Nov 19 21:03:39 node1 heartbeat: [10571]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:03:39 node1 harc[10571]: [10579]: info: Running /etc/ha.d/rc.d/status status
Nov 19 21:03:39 node1 mach_down[10590]: [10629]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Nov 19 21:03:39 node1 mach_down[10590]: [10640]: info: mach_down takeover complete for node node2.
Nov 19 21:03:39 node1 heartbeat: [10054]: info: mach_down takeover complete.
Nov 19 21:03:39 node1 heartbeat: [10054]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 21:03:39 node1 IPaddr[10645]: [10676]: INFO:  Running OK
Nov 19 21:03:39 node1 heartbeat: [10572]: info: Local Resource acquisition completed.
Nov 19 21:03:40 node1 heartbeat: [10054]: info: Heartbeat shutdown in progress. (10054)
Nov 19 21:03:40 node1 heartbeat: [10677]: info: Giving up all HA resources.
Nov 19 21:03:40 node1 ResourceManager[10690]: [10701]: info: Releasing resource group: node1 195.63.63.101
Nov 19 21:03:40 node1 ResourceManager[10690]: [10718]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
Nov 19 21:03:40 node1 ResourceManager[10690]: [10719]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
Nov 19 21:03:40 node1 IPaddr[10737]: [10752]: INFO: ifconfig eth1:0 down
Nov 19 21:03:40 node1 heartbeat: [10677]: info: All HA resources relinquished.
Nov 19 21:03:40 node1 IPaddr[10720]: [10755]: INFO:  Success
Nov 19 21:03:40 node1 ResourceManager[10690]: [10756]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop done. RC=0
Nov 19 21:03:41 node1 heartbeat: [10054]: info: client [/usr/lib/heartbeat/ipfail] is not running.
Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBFIFO process 10056 with signal 15
Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBWRITE process 10057 with signal 15
Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBREAD process 10058 with signal 15
Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBWRITE process 10059 with signal 15
Nov 19 21:03:42 node1 heartbeat: [10054]: info: killing HBREAD process 10060 with signal 15
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10060 exited. 5 remaining
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10059 exited. 4 remaining
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10058 exited. 3 remaining
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10057 exited. 2 remaining
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Core process 10056 exited. 1 remaining
Nov 19 21:03:42 node1 heartbeat: [10054]: info: node1 Heartbeat shutdown complete.
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Heartbeat restart triggered.
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Restarting heartbeat.
Nov 19 21:03:42 node1 heartbeat: [10054]: info: Performing heartbeat restart exec.
Nov 19 21:04:13 node1 heartbeat: [10054]: info: Enabling logging daemon 
Nov 19 21:04:13 node1 heartbeat: [10054]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 19 21:04:13 node1 heartbeat: [10054]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 19 21:04:13 node1 heartbeat: [10054]: info: Version 2 support: false
Nov 19 21:04:13 node1 heartbeat: [10054]: info: **************************
Nov 19 21:04:13 node1 heartbeat: [10054]: info: Configuration validated. Starting heartbeat 2.1.2
Nov 19 21:04:13 node1 heartbeat: [10757]: info: heartbeat: version 2.1.2
Nov 19 21:04:13 node1 heartbeat: [10757]: info: Heartbeat generation: 1193181888
Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 19 21:04:13 node1 heartbeat: [10757]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Nov 19 21:04:13 node1 heartbeat: [10757]: info: glib: ping heartbeat started.
Nov 19 21:04:13 node1 heartbeat: [10757]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Nov 19 21:04:13 node1 heartbeat: [10757]: info: Local status now set to: 'up'
Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link node2:eth0 up.
Nov 19 21:04:14 node1 heartbeat: [10757]: info: Status update for node node2: status up
Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link node1:eth0 up.
Nov 19 21:04:14 node1 heartbeat: [10765]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:04:14 node1 heartbeat: [10757]: info: Link externalhost:externalhost up.
Nov 19 21:04:14 node1 heartbeat: [10757]: info: Status update for node externalhost: status ping
Nov 19 21:04:14 node1 harc[10765]: [10772]: info: Running /etc/ha.d/rc.d/status status
Nov 19 21:04:15 node1 heartbeat: [10757]: debug: get_delnodelist: delnodelist= 
Nov 19 21:04:15 node1 heartbeat: [10757]: info: Comm_now_up(): updating status to active
Nov 19 21:04:15 node1 heartbeat: [10757]: info: Local status now set to: 'active'
Nov 19 21:04:15 node1 heartbeat: [10757]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
Nov 19 21:04:15 node1 heartbeat: [10778]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 10778)
Nov 19 21:04:15 node1 ipfail: [10778]: debug: PID=10778
Nov 19 21:04:15 node1 ipfail: [10778]: debug: Signing in with heartbeat
Nov 19 21:04:15 node1 heartbeat: [10757]: WARN: Client [ipfail] pid 10778 failed authorization [client failed authorization]
Nov 19 21:04:15 node1 heartbeat: [10757]: ERROR: api_process_registration_msg: cannot add client(ipfail)
Nov 19 21:04:15 node1 ipfail: [10778]: ERROR: No managed resources
Nov 19 21:04:15 node1 heartbeat: [10757]: WARN: Exiting /usr/lib/heartbeat/ipfail process 10778 returned rc 100.
Nov 19 21:04:16 node1 heartbeat: [10757]: info: Status update for node node2: status active
Nov 19 21:04:16 node1 heartbeat: [10779]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:04:16 node1 harc[10779]: [10785]: info: Running /etc/ha.d/rc.d/status status
Nov 19 21:04:26 node1 heartbeat: [10757]: info: remote resource transition completed.
Nov 19 21:04:26 node1 heartbeat: [10757]: info: remote resource transition completed.
Nov 19 21:04:26 node1 heartbeat: [10757]: info: Initial resource acquisition complete (T_RESOURCES(us))
Nov 19 21:04:27 node1 IPaddr[10827]: [10858]: INFO:  Resource is stopped
Nov 19 21:04:27 node1 heartbeat: [10791]: info: Local Resource acquisition completed.
Nov 19 21:04:27 node1 heartbeat: [10757]: debug: StartNextRemoteRscReq(): child count 1
Nov 19 21:04:27 node1 heartbeat: [10862]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 19 21:04:27 node1 harc[10862]: [10868]: info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp
Nov 19 21:04:27 node1 ip-request-resp[10862]: [10874]: received ip-request-resp 195.63.63.101 OK yes
Nov 19 21:04:27 node1 ResourceManager[10875]: [10886]: info: Acquiring resource group: node1 195.63.63.101
Nov 19 21:04:27 node1 IPaddr[10898]: [10929]: INFO:  Resource is stopped
Nov 19 21:04:27 node1 ResourceManager[10875]: [10945]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 19 21:04:27 node1 ResourceManager[10875]: [10946]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 19 21:04:27 node1 IPaddr[10964]: [10995]: INFO: Using calculated nic for 195.63.63.101: eth1
Nov 19 21:04:27 node1 IPaddr[10964]: [11000]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
Nov 19 21:04:27 node1 IPaddr[10964]: [11005]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
Nov 19 21:04:27 node1 IPaddr[10964]: [11022]: INFO: eval ifconfig eth1:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
Nov 19 21:04:27 node1 IPaddr[10964]: [11027]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth1:0 [eth1]
Nov 19 21:04:27 node1 IPaddr[10947]: [11041]: INFO:  Success
Nov 19 21:04:27 node1 ResourceManager[10875]: [11042]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
Nov 19 21:06:14 node1 sshd[11043]: Accepted keyboard-interactive/pam for root from 195.63.63.254 port 1322 ssh2
Nov 19 21:06:17 node1 sshd[11043]: subsystem request for sftp
-------------- next part --------------
Nov 18 19:52:45 node2 logd: [16885]: info: setting log facility to daemon
Nov 18 19:52:45 node2 logd: [16885]: info: logd started with /etc/logd.cf.
Nov 18 19:52:45 node2 logd: [16885]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 18 19:52:45 node2 logd: [16885]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 18 19:52:45 node2 logd: [16885]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 18 19:52:45 node2 logd: [16886]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Nov 18 19:52:45 node2 logd: [16885]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Nov 18 19:52:45 node2 heartbeat: [16946]: info: Enabling logging daemon 
Nov 18 19:52:45 node2 heartbeat: [16946]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 18 19:52:45 node2 heartbeat: [16946]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 18 19:52:45 node2 heartbeat: [16946]: info: Version 2 support: false
Nov 18 19:52:45 node2 heartbeat: [16946]: info: **************************
Nov 18 19:52:45 node2 heartbeat: [16946]: info: Configuration validated. Starting heartbeat 2.1.2
Nov 18 19:52:45 node2 heartbeat: [16947]: info: heartbeat: version 2.1.2
Nov 18 19:52:46 node2 heartbeat: [16947]: info: Heartbeat generation: 1193174534
Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 18 19:52:46 node2 heartbeat: [16947]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Nov 18 19:52:46 node2 heartbeat: [16947]: info: glib: ping heartbeat started.
Nov 18 19:52:46 node2 heartbeat: [16947]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Nov 18 19:52:46 node2 heartbeat: [16947]: info: Local status now set to: 'up'
Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link node1:eth0 up.
Nov 18 19:52:47 node2 heartbeat: [16947]: info: Status update for node node1: status active
Nov 18 19:52:47 node2 heartbeat: [16955]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 18 19:52:47 node2 harc[16955]: [16962]: info: Running /etc/ha.d/rc.d/status status
Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link node2:eth0 up.
Nov 18 19:52:47 node2 heartbeat: [16947]: info: Link externalhost:externalhost up.
Nov 18 19:52:47 node2 heartbeat: [16947]: info: Status update for node externalhost: status ping
Nov 18 19:52:48 node2 heartbeat: [16947]: info: Comm_now_up(): updating status to active
Nov 18 19:52:48 node2 heartbeat: [16947]: info: Local status now set to: 'active'
Nov 18 19:52:48 node2 heartbeat: [16947]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 370 ms (> 50 ms) (GSource: 0x81014c0)
Nov 18 19:52:48 node2 heartbeat: [16968]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 16968)
Nov 18 19:52:48 node2 ipfail: [16968]: debug: PID=16968
Nov 18 19:52:48 node2 ipfail: [16968]: debug: Signing in with heartbeat
Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: Client [ipfail] pid 16968 failed authorization [client failed authorization]
Nov 18 19:52:48 node2 heartbeat: [16947]: ERROR: api_process_registration_msg: cannot add client(ipfail)
Nov 18 19:52:48 node2 ipfail: [16968]: ERROR: No managed resources
Nov 18 19:52:48 node2 heartbeat: [16947]: WARN: Exiting /usr/lib/heartbeat/ipfail process 16968 returned rc 100.
Nov 18 19:52:48 node2 heartbeat: [16947]: info: remote resource transition completed.
Nov 18 19:52:48 node2 heartbeat: [16947]: info: remote resource transition completed.
Nov 18 19:52:48 node2 heartbeat: [16947]: info: Local Resource acquisition completed. (none)
Nov 18 19:52:49 node2 heartbeat: [16947]: info: node1 wants to go standby [foreign]
Nov 18 19:52:49 node2 heartbeat: [16947]: info: standby: acquire [foreign] resources from node1
Nov 18 19:52:49 node2 heartbeat: [16969]: info: acquire local HA resources (standby).
Nov 18 19:52:49 node2 heartbeat: [16969]: info: local HA resource acquisition completed (standby).
Nov 18 19:52:49 node2 heartbeat: [16947]: info: Standby resource acquisition done [foreign].
Nov 18 19:52:49 node2 heartbeat: [16947]: info: Initial resource acquisition complete (auto_failback)
Nov 18 19:52:50 node2 heartbeat: [16947]: info: remote resource transition completed.
Nov 18 19:58:36 node2 syslog-ng[2236]: STATS: dropped 0
Nov 18 20:01:04 node2 kernel: e100: eth0: e100_watchdog: link down
Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: node node1: is dead
Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: No STONITH device configured.
Nov 18 20:01:32 node2 heartbeat: [16947]: WARN: Shared disks are not protected.
Nov 18 20:01:32 node2 heartbeat: [16947]: info: Resources being acquired from node1.
Nov 18 20:01:32 node2 heartbeat: [16947]: info: Link node1:eth0 dead.
Nov 18 20:01:32 node2 heartbeat: [17005]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 18 20:01:32 node2 harc[17005]: [17024]: info: Running /etc/ha.d/rc.d/status status
Nov 18 20:01:32 node2 heartbeat: [17006]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys node2] to acquire.
Nov 18 20:01:32 node2 heartbeat: [16947]: debug: StartNextRemoteRscReq(): child count 1
Nov 18 20:01:32 node2 mach_down[17030]: [17051]: info: Taking over resource group 195.63.63.101
Nov 18 20:01:32 node2 ResourceManager[17052]: [17063]: info: Acquiring resource group: node1 195.63.63.101
Nov 18 20:01:32 node2 IPaddr[17075]: [17106]: INFO:  Resource is stopped
Nov 18 20:01:32 node2 ResourceManager[17052]: [17122]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 18 20:01:32 node2 ResourceManager[17052]: [17123]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 start
Nov 18 20:01:33 node2 IPaddr[17141]: [17172]: INFO: Using calculated nic for 195.63.63.101: eth2
Nov 18 20:01:33 node2 IPaddr[17141]: [17177]: INFO: Using calculated netmask for 195.63.63.101: 255.255.255.0
Nov 18 20:01:33 node2 IPaddr[17141]: [17182]: DEBUG: Using calculated broadcast for 195.63.63.101: 195.63.63.255
Nov 18 20:01:33 node2 IPaddr[17141]: [17199]: INFO: eval ifconfig eth2:0 195.63.63.101 netmask 255.255.255.0 broadcast 195.63.63.255
Nov 18 20:01:33 node2 IPaddr[17141]: [17204]: DEBUG: Sending Gratuitous Arp for 195.63.63.101 on eth2:0 [eth2]
Nov 18 20:01:33 node2 IPaddr[17124]: [17218]: INFO:  Success
Nov 18 20:01:33 node2 ResourceManager[17052]: [17219]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 start done. RC=0
Nov 18 20:01:33 node2 mach_down[17030]: [17220]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Nov 18 20:01:33 node2 mach_down[17030]: [17224]: info: mach_down takeover complete for node node1.
Nov 18 20:01:33 node2 heartbeat: [16947]: info: mach_down takeover complete.
Nov 18 20:01:33 node2 kernel: NET: Registered protocol family 17
Nov 18 20:02:49 node2 heartbeat: [16947]: CRIT: Cluster node node1 returning after partition.
Nov 18 20:02:49 node2 heartbeat: [16947]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Nov 18 20:02:49 node2 heartbeat: [16947]: WARN: Deadtime value may be too small.
Nov 18 20:02:49 node2 heartbeat: [16947]: info: See FAQ for information on tuning deadtime.
Nov 18 20:02:49 node2 heartbeat: [16947]: info: URL: http://linux-ha.org/FAQ#heavy_load
Nov 18 20:02:49 node2 heartbeat: [16947]: info: Link node1:eth0 up.
Nov 18 20:02:49 node2 heartbeat: [16947]: WARN: Late heartbeat: Node node1: interval 106990 ms
Nov 18 20:02:49 node2 heartbeat: [16947]: info: Status update for node node1: status active
Nov 18 20:02:49 node2 heartbeat: [17230]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 18 20:02:49 node2 harc[17230]: [17236]: info: Running /etc/ha.d/rc.d/status status
Nov 18 20:02:49 node2 heartbeat: [16947]: info: all clients are now paused
Nov 18 20:02:49 node2 heartbeat: [16947]: debug: hist->ackseq =520
Nov 18 20:02:49 node2 heartbeat: [16947]: debug: hist->lowseq =519, hist->hiseq=621
Nov 18 20:02:49 node2 heartbeat: [16947]: debug:  
Nov 18 20:02:50 node2 kernel: e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
Nov 18 20:02:50 node2 heartbeat: [16947]: debug: hist->ackseq =520
Nov 18 20:02:50 node2 heartbeat: [16947]: debug: hist->lowseq =519, hist->hiseq=622
Nov 18 20:02:50 node2 heartbeat: [16947]: debug:  
Nov 18 20:02:51 node2 heartbeat: [16947]: info: all clients are now resumed
Nov 18 20:02:51 node2 heartbeat: [16947]: info: Heartbeat shutdown in progress. (16947)
Nov 18 20:02:51 node2 heartbeat: [17242]: info: Giving up all HA resources.
Nov 18 20:02:51 node2 ResourceManager[17255]: [17266]: info: Releasing resource group: node1 195.63.63.101
Nov 18 20:02:51 node2 ResourceManager[17255]: [17283]: info: Running /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
Nov 18 20:02:51 node2 ResourceManager[17255]: [17284]: debug: Starting /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop
Nov 18 20:02:51 node2 IPaddr[17302]: [17317]: INFO: ifconfig eth2:0 down
Nov 18 20:02:51 node2 IPaddr[17285]: [17320]: INFO:  Success
Nov 18 20:02:51 node2 ResourceManager[17255]: [17321]: debug: /etc/ha.d/resource.d/IPaddr 195.63.63.101 stop done. RC=0
Nov 18 20:02:51 node2 heartbeat: [17242]: info: All HA resources relinquished.
Nov 18 20:02:52 node2 heartbeat: [16947]: info: client [/usr/lib/heartbeat/ipfail] is not running.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Received shutdown notice from 'node1'.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Resource takeover cancelled - shutdown in progress.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBREAD process 16951 with signal 15
Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBWRITE process 16952 with signal 15
Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBREAD process 16953 with signal 15
Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBFIFO process 16949 with signal 15
Nov 18 20:02:53 node2 heartbeat: [16947]: info: killing HBWRITE process 16950 with signal 15
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16953 exited. 5 remaining
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16952 exited. 4 remaining
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16951 exited. 3 remaining
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16950 exited. 2 remaining
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Core process 16949 exited. 1 remaining
Nov 18 20:02:53 node2 heartbeat: [16947]: info: node2 Heartbeat shutdown complete.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Heartbeat restart triggered.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Restarting heartbeat.
Nov 18 20:02:53 node2 heartbeat: [16947]: info: Performing heartbeat restart exec.
Nov 18 20:03:24 node2 heartbeat: [16947]: info: Enabling logging daemon 
Nov 18 20:03:24 node2 heartbeat: [16947]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Core dumps could be lost if multiple dumps occur.
Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Nov 18 20:03:24 node2 heartbeat: [16947]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Nov 18 20:03:24 node2 heartbeat: [16947]: info: Version 2 support: false
Nov 18 20:03:24 node2 heartbeat: [16947]: info: **************************
Nov 18 20:03:24 node2 heartbeat: [16947]: info: Configuration validated. Starting heartbeat 2.1.2
Nov 18 20:03:24 node2 heartbeat: [17322]: info: heartbeat: version 2.1.2
Nov 18 20:03:25 node2 heartbeat: [17322]: info: Heartbeat generation: 1193174535
Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 18 20:03:25 node2 heartbeat: [17322]: info: Removing /var/run/heartbeat/rsctmp failed, recreating.
Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Nov 18 20:03:25 node2 heartbeat: [17322]: info: glib: ping heartbeat started.
Nov 18 20:03:25 node2 heartbeat: [17322]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Nov 18 20:03:25 node2 heartbeat: [17322]: info: Local status now set to: 'up'
Nov 18 20:03:26 node2 heartbeat: [17322]: info: Link node2:eth0 up.
Nov 18 20:03:27 node2 heartbeat: [17322]: info: Link externalhost:externalhost up.
Nov 18 20:03:27 node2 heartbeat: [17322]: info: Status update for node externalhost: status ping
Nov 18 20:03:27 node2 heartbeat: [17322]: info: Link node1:eth0 up.
Nov 18 20:03:27 node2 heartbeat: [17322]: info: Status update for node node1: status up
Nov 18 20:03:27 node2 heartbeat: [17322]: debug: get_delnodelist: delnodelist= 
Nov 18 20:03:27 node2 heartbeat: [17330]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 18 20:03:27 node2 harc[17330]: [17337]: info: Running /etc/ha.d/rc.d/status status
Nov 18 20:03:28 node2 heartbeat: [17322]: info: Comm_now_up(): updating status to active
Nov 18 20:03:28 node2 heartbeat: [17322]: info: Local status now set to: 'active'
Nov 18 20:03:28 node2 heartbeat: [17322]: info: Starting child client "/usr/lib/heartbeat/ipfail" (0,0)
Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 480 ms (> 50 ms) (GSource: 0x8101d38)
Nov 18 20:03:28 node2 heartbeat: [17322]: info: Status update for node node1: status active
Nov 18 20:03:28 node2 heartbeat: [17343]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 0  gid 0 (pid 17343)
Nov 18 20:03:28 node2 heartbeat: [17344]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Nov 18 20:03:28 node2 harc[17344]: [17350]: info: Running /etc/ha.d/rc.d/status status
Nov 18 20:03:28 node2 ipfail: [17343]: debug: PID=17343
Nov 18 20:03:28 node2 ipfail: [17343]: debug: Signing in with heartbeat
Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: Client [ipfail] pid 17343 failed authorization [client failed authorization]
Nov 18 20:03:28 node2 heartbeat: [17322]: ERROR: api_process_registration_msg: cannot add client(ipfail)
Nov 18 20:03:28 node2 ipfail: [17343]: ERROR: No managed resources
Nov 18 20:03:28 node2 heartbeat: [17322]: WARN: Exiting /usr/lib/heartbeat/ipfail process 17343 returned rc 100.
Nov 18 20:03:38 node2 heartbeat: [17322]: info: local resource transition completed.
Nov 18 20:03:38 node2 heartbeat: [17322]: info: Initial resource acquisition complete (T_RESOURCES(us))
Nov 18 20:03:39 node2 heartbeat: [17356]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys node2] to acquire.
Nov 18 20:03:39 node2 heartbeat: [17322]: info: remote resource transition completed.
Nov 18 20:07:04 node2 sshd[17369]: Accepted keyboard-interactive/pam for root from 195.63.63.254 port 1324 ssh2
Nov 18 20:07:08 node2 sshd[17369]: subsystem request for sftp