Safely disabling topology hints during a live incident

##Introduction to Staged Remediation

Understanding Topology Hints

Topology hints are metadata that a node attaches to outgoing requests to steer traffic toward replicas that are topologically close (e.g., same rack, same availability zone). In distributed data stores such as Cassandra, ScyllaDB, and CockroachDB, hints reduce cross‑zone latency and improve read‑locality. When hints become stale or mis‑aligned—often after a network partition, node replacement, or zone‑level outage—they can cause request routing to fail repeatedly, manifesting as timeouts that appear application‑level but are actually hint‑driven mis‑routes.

Importance of Pre‑Checks

Before changing any routing‑influencing configuration, establish a known‑good baseline and verify that the system is not already in a failure cascade. Pre‑checks provide:

• Evidence that observed timeouts are hint‑related (rising hint counters, request traces showing mis‑directed hops).
• A measurable safety envelope (baseline latency, error rates, throughput) against which each remediation step can be judged.
• A clear rollback trigger (if any metric deviates beyond a pre‑defined threshold, halt and revert the change).

Skipping pre‑checks turns a controlled remediation into a gamble, risking a local zone problem becoming a cluster‑wide rebalance shock because hints are removed everywhere at once, forcing all nodes to re‑evaluate replica placement simultaneously.

Pre‑Remediation Checks

Network and System Health

1. Node‑level health – Confirm each node reports UP in gossip and none are DOWN or LEAVING.

   nodetool status | grep -E 'UN|UP'

2. Inter‑node latency – Measure 95th‑percentile RTT between zones using ping or tcptrace.

   ping -c 20 <node-in-zone-b> | tail -1

3. Resource saturation – CPU, memory, disk I/O, and network utilization < 70 % on all nodes (adjust per SLA). Use collectd, Prometheus node_exporter, or sar.
4. Hint statistics – Verify hint counters are non‑zero and trending upward in the affected zone.

   nodetool gossipinfo | grep hint
   # or via JMX: org.apache.cassandra.metrics:type=HintedHandoff,name=TotalHints

Traffic Patterns and Baselines

• Request latency – Capture 99th‑percentile read/write latency per client zone for the last 15 min.
• Error rate – Track HTTP 5xx / database timeout errors per zone.
• Throughput – Measure requests per second (RPS) per zone.
  Store these numbers in a time‑series dashboard (Grafana, CloudWatch) and export as CSV for later comparison.

Current Topology Hint Configuration

• Hint enabled flag – Ensure hints are globally enabled (hinted_handoff_enabled: true in cassandra.yaml).
• Per‑zone hint settings – Check for any datacenter‑specific overrides.
• Hint TTL – Note max_hint_window_in_ms to understand how long stale hints may linger.
• Recent changes – Review Git/ChatOps logs for hint‑related modifications in the past 24 h.

If any check fails (node down, latency > 2× baseline, hint counters zero despite timeouts), abort and investigate the underlying cause first.

Designing the Canary Scope

Selecting Representative Nodes

Pick a small, homogeneous subset that mirrors the failure zone:

Typical size: 3‑5 nodes (~5 % of a 60‑node cluster) per zone.

Isolating Canary Scope Traffic

1. Dedicated client profile – Point a small set of test clients (or a canary service) to the canary nodes only via a load‑balancer ACL or DNS override.
2. Tag the traffic – Use a custom HTTP header (X-Canary: true) or a distinct Cassandra username/password so metrics can be filtered.
3. Disable hint propagation from canary to non‑canary – Set hinted_handoff_throttle_in_kb: 0 on the canary nodes only (prevents them from sending hints to the rest of the cluster while testing removal).

   # Update cassandra.yaml on canary nodes, then restart
   sudo sed -i 's/^hinted_handoff_throttle_in_kb:.*/hinted_handoff_throttle_in_kb: 0/' /etc/cassandra/cassandra.yaml
   sudo systemctl restart cassandra

Monitoring and Feedback Mechanisms

• Real‑time dashboards – Graph hint counters, read/write latency, and timeout rates for the canary traffic stream.
• Alert thresholds – Pre‑define:
  • Latency increase > 20 % over baseline → pause rollout.
  • Timeout rate > 0.5 % of canary RPS → abort and rollback.
  • Hint drop rate > 90 % with no latency regression → proceed.
• Log‑based verification – Enable trace logging on canary nodes (log_level: DEBUG for org.apache.cassandra.service.StorageProxy) and grep for HintedHandoffManager.
• Automated verification script – A short Python/Bash loop that queries the metrics API every 10 s and exits with non‑zero if any threshold is breached.

Implementing Rollback Thresholds

Defining Threshold Metrics

These thresholds are operational (based on observed variance), not system guarantees; they rely on the operator to act when breached.

Automated Rollback Triggers

• Prometheus alert rule (example):

  - alert: TopologyHintRemediationLatencyHigh
    expr: histogram_quantile(0.99, sum(rate(read_latency_seconds_bucket[5m])) by (le)) > 0.00024  # 240 ms
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Read latency exceeded safe threshold on canary"
      description: "Latency is {{ $value }}s, baseline is {{ $baseline }}s."

• Alertmanager webhook triggers a rollback script that:
  1. Re‑applies the original cassandra.yaml hint settings on canary nodes.
  2. Restarts the Cassandra service.
  3. Sends a notification to the on‑call channel.

Manual Intervention Points

• Pre‑commit checkpoint – After updating canary nodes but before setting hint throttle to zero, run a manual verification (nodetool status + hint check) and sign off in the change ticket.
• Post‑step verification gate – After each batch of nodes, review the dashboard; if any metric is amber (approaching threshold), pause and investigate before proceeding.
• Emergency stop – A dedicated Slack command (/remediation-stop) that instantly runs the rollback script on all touched nodes.

Executing the Remediation Plan

Step‑by‑Step Procedure

Code/CLI Examples for Topology Hint Removal

Cassandra (nodetool)

# 1. Disable hint throttle on a node (yaml edit)
sudo sed -i 's/^hinted_handoff_throttle_in_kb:.*/hinted_handoff_throttle_in_kb: 0/' /etc/cassandra/cassandra.yaml
sudo systemctl restart cassandra   # reload if supported, otherwise restart

# 2. Disable hint persistence (max_hint_window)
sudo sed -i 's/^max_hint_window_in_ms:.*/max_hint_window_in_ms: 0/' /etc/cassandra/cassandra.yaml
sudo systemctl restart cassandra

# 3. Flush existing hints (immediate)
nodetool disablehints   # stops accepting new hints
nodetool flush          # forces memtable to SSTable; hints cleared on flush
# Verify
nodetool gossipinfo | grep Hint
# Expected: HintedHandoffManager: ActiveHints=0

ScyllaDB (scylla-cli)

# Truncate hints table
cqlsh -e "TRUNCATE system.hints;"
# Disable hinted handoff via scylla.yaml
sudo sed -i 's/^hinted_handoff_enabled:.*/hinted_handoff_enabled: false/' /etc/scylla/scylla.yaml
sudo systemctl restart scylla

CockroachDB – Cockroach does not use hinted handoff; the analogous knob is kv.rangefeed.enabled. For this article we assume a Cassandra‑style system.

Verification of Successful Removal

• JMX/MBean – org.apache.cassandra.metrics:type=HintedHandoff,name=WaitingHints should report 0.
• File system – ls -l /var/lib/cassandra/hints/ shows no .Hint files.
• Trace – Enable request tracing (cqlsh -e "TRACING ON"), run a query, and confirm the trace does not contain a HintedHandoff stage.
• Metrics – Graph cassandra_thread_pool_pending_tasks for HintedHandoffPool – should stay at zero.

Monitoring and Validation

Proof Points for Successful Remediation

1. Hint Count = 0 for ≥ 5 min across all nodes in the remediated zone.
2. 99th‑p latency ≤ baseline × 1.05 (no regression).
3. Timeout rate ≤ baseline × 1.05.
4. No new hint files created in the hint directory.
5. Gossip convergence – all nodes report UN (Up Normal) and no LEAVING/JOING states.

If any proof point fails, the rollback trigger must fire.

Identifying and Addressing Local Zone Problems

• Local hotspot – If a single rack shows higher latency after hint removal, check token distribution with nodetool ring.
• Repair backlog – Stale hints can mask anti‑entropy delays; run nodetool repair -pr on the affected rack to catch up.
• Client‑side retry storms – Increase client-side timeout or enable exponential backoff if retries surge after hint removal.

Preventing Cluster‑Wide Rebalance Shock

• Gradual scope – Never hint‑disable more than one failure zone at a time.
• Throttle rebalancing – Set stream_throughput_outbound_megabits_per_sec to a low value (e.g., 20 Mbps) during remediation to limit streaming traffic when nodes re‑evaluate replica placement.
• Pause automatic bootstraps – Disable auto_bootstrap: true temporarily (nodetool pausehandoff) to stop new nodes from streaming data while hints are cleared.
• Monitor PendingCompactions and PendingFlushes – spikes indicate catch‑up effort; if they exceed 2× baseline, pause further node batches.

Troubleshooting Common Issues

Identifying and Resolving Traffic Timeouts

End of document.