[Linux-HA] Re: [Linux-ha-dev] If suicide is the answer, you're asking the wrong question

[Andrew Beekhof]

On Nov 8, 2007, at 4:18 PM, Lars Marowsky-Bree wrote:

> On 2007-11-08T16:04:25, Andrew Beekhof <abeekhof at suse.de> wrote:
>
>> The attached table^ attempts to explain why node suicide, at least  
>> the "so
>> simple it can't possibly have a single bug" kind being proposed, is  
>> no
>> substitute for enabling stonith (even when no plugins are  
>> configured!).
>
> Agreed. Though under certain circumstances, "suicide" as a STONITH
> plugin can be made to work reliably "enough".

sure

>> From the discussion, it seems clear to me that many people are  
>> under the
>> impression that the proposal would protect their data in other (and  
>> far
>> more likely) failure situations.
>> This is not the case.
>
> That is correct, but a fail-fast mechanism (suicide, reboot on certain
> errors) is less likely to introduce errors than a process recovery
> attempt. That's well documented in the literature.
>
> _In combination_ with STONITH this can ensure that, for example, an
> internal CRM failure keeps the node active at the heartbeat/ccm level
> and thus fencing does not occur right away (it'll eventually occur due
> to eventually "stop" timing out, but that is not quite the same).
>
> Fail-fast error escalation does not replace STONITH.
>
>> There is also the issue that when stonith is enabled, node suicide
>> guarantees the worst case (avoidable service outage) will always  
>> occur.
>> I object quite strongly to that.
>
> Hm? That is not true. Node suicice does not automatically cause  
> service
> outage.

Yes it does - that was the initial point, to address 1762.

> The assumption is that node suicide only occurs under conditions
> so severe where fencing would occur eventually anyway.


My understanding of the proposal was that "any child process exiting,  
ever" is counted as "so severe"
At least it would have to be that way to make it relevant to bug 1762.

So with suicide enabled, the node is always rebooted, stopping all  
services it has running.
With only stonith enabled, the cluster will normally recover  
sufficiently fast enough to avoid fencing.

Therefor there are times when the node would commit suicide when  
stonith would not have been needed.
That qualifies as avoidable service outage.


Rebooting because processes are respawning themselves several times a  
second or returning 100 (as if to say, i can't continue)... yeah, that  
i see adding value over stonith on its own.
But not any process and every time and stonith is enabled.


> Fail-fast is about bubbling errors up faster, so that STONITH can  
> occur
> more reliably (and possibly faster) - it escalates partial, localized
> but not-yet-recoverable errors

Assuming that a process exiting is a "not-yet-recoverable" error.
However for the mostly likely cases, at least in the crm, this is not  
true.

> to full node failures, which the cluster
> can then recover (using STONITH).
>
> If the node suicide is combined with the unsolicited "I will have
> committed suicide within 5s" message, the remaining cluster very  
> likely
> can skip an additional power cycle of the node.
>
>> So to summarize: if you care about your data, enable stonith.
>
> That summary is entirely correct, and I agree that people should not
> consider "node suicide" a panacea and as a replacement for STONITH.
>
> But that does not mean that fail-fast escalation is not still a good
> idea ;-)

True.

But I don't like this particular proposal, as it has so far been  
explained, nor do I see the point of it when split-brain is a hole  
wide enough to drive a truck through and far more likely to occur.

>> Perhaps the question should instead be: why haven't we already made  
>> stonith
>> mandatory?
>
> That's a good question indeed. Possibly because of non-shared storage
> environments, but in reality, those are the minority. We probably  
> ought
> to change the default.