Problem with IP failover on caching LAN switch

[Meyer, Craig H]

Greetings Linux-HA list,
I am relatively new to this community and I find the technologies quite
interesting.  Here's a little item I ran into while setting up an
experimental Linux-HA cluster.  Perhaps it will help others who may run into
the same or similar problems.  I found that the instructions for setting up
the HA cluster were quite clear and helpful.  In short order I had it up and
running.  The mail lists have been quite helpful as long as I check them
carefully (I encountered the problem with the serial port lock file and
patched it myself before realizing it had been reported and fixed a week
earlier on the list).  Now however, I have the following problem with
failovers:

If I take a node down, the virtual IP address fails over along with the
associated services.  I can ping the cluster and get services (e.g., Web
pages) from within the same subnet.  However, when I try to access the
cluster from a client that is on a different subnet the cluster appears
dead.  If I wait patiently, don't send anything for about 5 minutes and then
try again, it appears to wake up.  If I am impatient or ping it
continuously, it takes a very long time (hours) to wake up.  I found that
there is a bug in the Cisco Catalyst 6509 switch causing it to not update
its route cache, so that in these cases there is a stale IP address /
Ethernet address route pair.  There is a 256 second cache timeout which
explains why it starts to work in about 5 minutes.  My network wizard
explains that he used this bug to determine who the fast installers were (if
they replaced a desktop served by this switch in under 5 minutes they would
report the problem :-).  Cisco has developed a patch for this condition
which is expected to be formally released in the very near future.  Note
also that if I bring the failed node back up in under five minutes, the
failover is the expected 8 to 10 seconds for a Web server (since the cache
is again correct).  If I wait for the switch cache timeout and then bring
the node back, I'm stuck with waiting the full 5 minutes again, since the
switch has now cached the failover node's Ethernet address.

I'm not sure anything can or should be done about this on the Linux-HA side,
but I expect that network switching technologies like cached switches,
spanning tree, and VLAN will become more prevalent, and they will likely
have consequences for HA cluster failovers.  Perhaps the subject at least
merits a line or two in the troubleshooting section of the HOWTO.

Keep up the good work!
Best regards,
-- Craig

Craig H. Meyer
Lockheed Martin Naval Electronics & Surveillance Systems
3333 Pilot Knob Rd.
Eagan, MN  55121
(651) 456-7446