Retry Budgets, Hysteresis, and Kill Switches for Self-Healing

Introduction to Remediation Loops

A remediation loop repeatedly evaluates system health, applies corrective actions, and re‑evaluates until a desired state is observed or a termination criterion is met. Its purpose is to restore service‑level objectives (SLOs) without continuous human intervention while limiting blast‑radius and avoiding runaway behavior when automation cannot fully resolve the underlying problem.

Stability prevents oscillations that amplify faults or waste resources. Convergence guarantees that, when the fault is remediable, the loop reaches a steady state where no further action is needed. Without explicit bounds on retries, hysteresis, stop conditions, and manual intervention, a loop can thrash configuration, exhaust quotas, mask root causes, or create a denial‑of‑service on the remediation system itself. Defining retry budgets, hysteresis windows, stop conditions, and manual kill switches provides observable, operable safeguards that keep the loop within a known risk envelope.

Retry Budgets

Definition

A retry budget caps the number of remediation attempts for a given symptom within a defined observation window, expressed as a maximum count or rate. It limits the frequency of corrective actions regardless of each attempt’s success.

Configuration Parameters

Parameter	Meaning	Typical Unit
`max_retries`	Upper bound on total retry attempts in the window	count
`window_seconds`	Sliding time window over which the budget is measured	seconds
`budget_type`	`fixed` (hard limit) or `token_bucket` (refill rate)	enum
`refill_rate`	For token‑bucket: tokens added per second	tokens/second
`burst`	Maximum tokens that can be accumulated	tokens

When exhausted, the loop enters a back‑off state or triggers a stop condition, preventing further retries until the window slides forward or a manual reset occurs.

Example Configuration

CLI (using remctl):

# Fixed retry budget: 5 attempts per 5‑minute window
remctl set-retry-budget \
    --symptom high_cpu \
    --max-retries 5 \
    --window-seconds 300 \
    --budget-type fixed

YAML (Kubernetes‑based operator):

apiVersion: remediation.example.com/v1alpha1
kind: RetryBudget
metadata:
  name: high-cpu-budget
spec:
  symptom: high_cpu
  maxRetries: 5
  windowSeconds: 300
  budgetType: fixed   # or token_bucket
  # token_bucket example:
  # refillRate: 0.0167   # ~1 token/min → 5 tokens/300s
  # burst: 5

Python (token‑bucket implementation):

import time
from collections import deque

class TokenBucket:
    def __init__(self, rate_per_sec, burst):
        self.rate = rate_per_sec          # tokens added per second
        self.burst = burst                # max tokens
        self.tokens = burst               # start full
        self.timestamp = time.monotonic()

    def _add_tokens(self):
        now = time.monotonic()
        elapsed = now - self.timestamp
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
        self.timestamp = now

    def consume(self, tokens=1):
        self._add_tokens()
        if self.tokens >= tokens:
            self.tokens -= tokens
            return True
        return False

# 5 retries per 5 min → rate = 5/300 ≈ 0.0167 tps, burst = 5
budget = TokenBucket(rate_per_sec=0.0167, burst=5)

def attempt_remediation():
    if budget.consume():
        print("Retrying remediation...")
        return True
    print("Retry budget exhausted – backing off")
    return False

Best Practices

Base the budget on observed failure frequency (e.g., 5 min window for short‑lived CPU spikes).
Prefer token‑bucket to allow bursts after quiet periods while limiting long‑term average rate.
Align the window with the remediation action’s effect latency to avoid premature exhaustion.
Expose remaining tokens and refill time to monitoring/alerting.
Document exhaustion behavior (cool‑down, escalation, manual reset).

Hysteresis Windows

Definition

A hysteresis window introduces a delay or dead‑band between observing a symptom change and deciding to trigger or suppress remediation, preventing chattering on momentary fluctuations.

Configuration Parameters

Parameter	Meaning
`trigger_threshold`	Metric value that initiates remediation (e.g., CPU > 85%).
`clear_threshold`	Metric value indicating recovery (often lower than trigger).
`persist_seconds`	Minimum time metric must stay beyond `trigger_threshold` before acting.
`clear_persist_seconds`	Minimum time metric must stay below `clear_threshold` before considering symptom cleared.
`action_cooldown`	Optional global cool‑down after an action, independent of metric state.

Example Configuration

CLI (remctl):

remctl set-hysteresis \
    --symptom high_cpu \
    --trigger-threshold 85 \
    --clear-threshold 70 \
    --persist-seconds 60 \
    --clear-persist-seconds 30 \
    --action-cooldown 120

YAML:

apiVersion: remediation.example.com/v1alpha1
kind: HysteresisPolicy
metadata:
  name: high-cpu-hysteresis
spec:
  symptom: high_cpu
  triggerThreshold: 85   # percent
  clearThreshold: 70
  persistSeconds: 60
  clearPersistSeconds: 30
  actionCooldownSeconds: 120

Go (state machine):

type Hysteresis struct {
    trigger      float64
    clear        float64
    persist      time.Duration
    clearPersist time.Duration
    cooldown     time.Duration

    state        string // "ok", "triggered", "recovering"
    triggerStart time.Time
    clearStart   time.Time
    lastAction   time.Time
}

func (h *Hysteresis) Evaluate(value float64, now time.Time) bool {
    if now.Sub(h.lastAction) < h.cooldown {
        return false
    }
    switch h.state {
    case "ok":
        if value > h.trigger {
            h.triggerStart = now
            h.state = "triggered"
        }
    case "triggered":
        if value <= h.trigger {
            h.state = "ok"
        } else if now.Sub(h.triggerStart) >= h.persist {
            h.lastAction = now
            h.state = "recovering"
            h.clearStart = now
            return true // trigger remediation
        }
    case "recovering":
        if value < h.clear {
            h.clearStart = now
        } else if now.Sub(h.clearStart) >= h.clearPersist {
            h.state = "ok"
        }
    }
    return false
}

Best Practices

Set clear_threshold lower than trigger_threshold to create a dead‑band.
Base persist durations on expected remediation effect latency.
Avoid excessively long hysteresis that delays genuine fault response; monitor MTTD/MTTR.
Log state transitions for audit and post‑mortem analysis.
Consider adaptive hysteresis during known maintenance or after repeated failures.

Stop Conditions

Definition

A stop condition is a predicate that, when true, halts further remediation attempts for a symptom. It can be threshold‑based, time‑based, resource‑based, dependency‑based, or manual.

Configuration Parameters

Field	Description
`maxElapsedSeconds`	Hard ceiling on total remediation duration.
`maxRetryRatePerMinute`	Maximum allowed remediation actions per minute (distinct from retry budget).
`criticalMetric`	If a related metric crosses a severity threshold, stop.
`quotaConsumption`	Maximum allowed consumption of a shared resource (e.g., API quota).
`dependencyHealthCheck`	Name of a health endpoint that must be healthy; if unhealthy, stop.
`evaluationIntervalSeconds`	How often the stop condition is re‑checked.

Example Configuration

CLI (remctl):

remctl set-stop-condition \
    --symptom high_cpu \
    --max-elapsed-seconds 900 \          # 15 min max
    --max-retry-rate 2 \                # ≤2 actions/min
    --critical-latency-ms 5000 \        # stop if 95th‑pct latency >5 s
    --api-quota-limit-percent 80 \      # stop if >80% of remediation API quota used
    --dependency-health-check db-health \
    --eval-interval-seconds 15

YAML:

apiVersion: remediation.example.com/v1alpha1
kind: StopCondition
metadata:
  name: high-cpu-stop
spec:
  symptom: high_cpu
  maxElapsedSeconds: 900
  maxRetryRatePerMinute: 2
  criticalMetric:
    name: request_latency_p95
    thresholdMs: 5000
    operator: gt
  quotaConsumption:
    resource: remediation_api
    limitPercent: 80
  dependencyHealthCheck: db-health
  evaluationIntervalSeconds: 15

Python (evaluated in a loop):

import time
import requests
from collections import deque

class StopEvaluator:
    def __init__(self, cfg):
        self.cfg = cfg
        self.start_time = time.monotonic()
        self.action_times = deque()  # timestamps of recent actions

    def record_action(self):
        now = time.monotonic()
        self.action_times.append(now)
        while self.action_times and now - self.action_times[0] > 60:
            self.action_times.popleft()

    def should_stop(self, metrics):
        # 1. max elapsed
        if time.monotonic() - self.start_time > self.cfg['maxElapsedSeconds']:
            return True, "max elapsed exceeded"

        # 2. max retry rate
        if len(self.action_times) > self.cfg['maxRetryRatePerMinute']:
            return True, "retry rate exceeded"

        # 3. critical metric
        crit = self.cfg['criticalMetric']
        if metrics.get(crit['name'], 0) > crit['thresholdMs']:
            return True, f"critical metric {crit['name']} exceeded"

        # 4. quota consumption (example)
        quota_used = self._get_quota_used()
        if quota_used > self.cfg['quotaConsumption']['limitPercent']:
            return True, f"quota usage {quota_used}% > limit"

        # 5. dependency health
        dep = self.cfg['dependencyHealthCheck']
        dep_ok = requests.get(f"http://{dep}/health", timeout=2).ok
        if not dep_ok:
            return True, f"dependency {dep} unhealthy"

        return False, ""

    def _get_quota_used(self):
        # placeholder: query internal metrics store
        return 75  # percent

Best Practices

Layer stop conditions: combine a hard time ceiling with rate‑ or metric‑based limits.
Emit a distinct metric (e.g., remediation_stop_triggered_total) and log the reason.
Avoid circular dependencies where the stop metric is directly altered by the remediation action.
Test stop conditions with synthetic faults before production rollout.
Document the expected post‑stop state (safe degraded mode, manual rollback, etc.).

Manual Kill Switches

Definition

A manual kill switch provides an operator‑initiated, immediate means to halt all remediation activity for a symptom, component, or globally. Unlike automated stop conditions, it is a deliberate human intervention point used when automated logic may be flawed, during known changes, for forensic analysis, or to satisfy compliance break‑glass requirements.

Desired Properties

Atomic – stops new remediation attempts instantly; in‑flight actions may finish or be terminated per design.
Auditable – logs initiator, timestamp, and reason.
Reversible – can be reset after the situation is resolved, allowing the loop to resume (subject to remaining budgets or stop conditions).

Implementation Examples

REST API (OpenAPI snippet):

/openapi: 3.0.0
paths:
  /remediation/kill-switch:
    put:
      summary: Activate or deactivate the kill switch for a symptom
      operationId: setKillSwitch
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [symptom, enabled, operator, reason]
              properties:
                symptom:
                  type: string
                  example: high_cpu
                enabled:
                  type: boolean
                  description: true to activate kill switch
                operator:
                  type: string
                  example: alice@example.com
                reason:
                  type: string
                  example: "Investigating memory leak"
      responses:
        '200':
          description: Kill switch state updated

CLI (remctl):

# Activate kill switch for high_cpu symptom
remctl set-kill-switch \
    --symptom high_cpu \
    --enabled true \
    --operator alice@example.com \
    --reason "Investigating memory leak"

# Deactivate after investigation
remctl set-kill-switch \
    --symptom high_cpu \
    --enabled false \
    --operator alice@example.com \
    --reason "Issue resolved, resuming remediation"

YAML (declarative policy):

apiVersion: remediation.example.com/v1alpha1
kind: KillSwitch
metadata:
  name: high-cpu-kill
spec:
  symptom: high_cpu
  enabled: false   # default off
  # operator and reason are recorded in status/audit logs upon change

Best Practices

Protect the switch with authentication and authorization (e.g., RBAC, MFA).
Emit an audit event to a secure log or SIEM on every state change.
Provide a clear UI or CLI command with confirmation prompts to avoid accidental activation.
Document the process for resetting the switch and any prerequisite checks.
Test the kill switch in a staging environment to verify it halts remediation as expected.