Turning ad hoc probes into an incident playbook

Introduction to bpftrace and Incident Playbooks

Overview of bpftrace

bpftrace is a high‑level tracing language that uses eBPF to attach probes to kernel and user‑space events. Its concise syntax lets you capture function calls, tracepoints, kprobes, uprobes, and more with minimal overhead. A typical one‑liner looks like:

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

Because bpftrace programs run in‑kernel, they pass the verifier’s safety checks, have bounded execution time, and can be attached or detached without rebooting. Ad‑hoc snippets, however, lack lifecycle management: they may be left running, waste resources, or generate noisy output that obscures real incidents.

Benefits of Incident Playbooks

Turning a one‑off bpftrace snippet into a governed playbook gives responders:

• Predictable execution – preflight checks confirm the environment is suitable before attaching probes.
• Resource safety – time‑to‑live (TTL) limits guarantee probes auto‑detach after a bounded window.
• Deterministic cleanup – verification steps ensure all bpftrace instances are removed.
• Clear stop criteria – responders know exactly when to halt collection and move to remediation.
• Auditability – each step logs its outcome for post‑mortem review and compliance.

Designing the Incident Playbook

Identifying Key Components of the bpftrace Snippet

Defining Preflight Checks

Preflight checks verify that the host can safely run the intended bpftrace program:

1. Kernel version – required eBPF features (e.g., BPF_CORE_READ) are present.
2. bpftrace binary availability – correct version installed.
3. Privilege level – root or CAP_SYS_ADMIN/CAP_PERFMON.
4. Available resources – enough memory for maps, not exceeding per‑cgroup limits.
5. No conflicting probes – ensure no other bpftrace instance is already attached to the same probe point.

Establishing Probe Time-to-Live Limits

TTL limits protect against runaway probes that continue after the incident is resolved. Strategies include:

• Hard timeout – bpftrace -t <seconds> to auto‑exit after N seconds.
• Soft timeout – embed a timer in the script (BEGIN { … } interval:s:30 { … exit(); }).
• External watchdog – a supervising process sends SIGTERM after a configured window.

Combining an internal timer with an external watchdog provides the safest guarantee.

Implementing Cleanup Verification

After the bpftrace program exits, verify:

• No bpftrace processes remain (pgrep -f bpftrace returns empty).
• All eBPF maps associated with the program are unpinned (bpftool map show shows none owned by the uid/gid).
• Any temporary files or perf buffers created are removed.

Setting Explicit Stop Criteria

Stop criteria define when data collection is sufficient to proceed to analysis or remediation:

• Event count threshold – stop after N occurrences of the target syscall.
• Time window – stop after M seconds of observation regardless of count.
• Pattern detection – stop when a specific anomaly (e.g., open of /etc/shadow by non‑root) is seen.
• Operator signal – a manual Ctrl‑C or the appearance of a file such as /tmp/stop_bpftrace.

Implementing Preflight Checks

Wrapper Script

#!/usr/bin/env bash
# preflight_bpftrace.sh
set -euo pipefail

REQUIRED_KERNEL="5.10"
REQUIRED_BPFTRACE_VERSION="0.15.0"
NEEDED_CAPS="CAP_SYS_ADMIN CAP_PERFMON"

# 1. Kernel version
KVER=$(uname -r)
if [[ "$(printf '%s\n%s' "$REQUIRED_KERNEL" "$KVER" | sort -V | head -n1)" != "$REQUIRED_KERNEL" ]]; then
  echo "ERROR: Kernel $KVER is older than required $REQUIRED_KERNEL"
  exit 1
fi

# 2. bpftrace presence and version
if ! command -v bpftrace >/dev/null 2>&1; then
  echo "ERROR: bpftrace not found in PATH"
  exit 1
fi
BPFTRACE_VER=$(bpftrace --version | awk '{print $2}')
if [[ "$(printf '%s\n%s' "$REQUIRED_BPFTRACE_VERSION" "$BPFTRACE_VER" | sort -V | head -n1)" != "$REQUIRED_BPFTRACE_VERSION" ]]; then
  echo "ERROR: bpftrace version $BPFTRACE_VER < $REQUIRED_BPFTRACE_VERSION"
  exit 1
fi

# 3. Privilege check
if [[ $EUID -ne 0 ]]; then
  echo "ERROR: Script must be run as root (or with CAP_SYS_ADMIN/CAP_PERFMON)"
  exit 1
fi

# 4. Resource check – max locked memory for BPF
MAX_LOCKED=$(grep MaxLocked /proc/self/status | awk '{print $2}')
if [[ $MAX_LOCKED -lt 256 ]]; then
  echo "WARN: MaxLocked memory low ($MAX_LOCKED KiB); may cause map allocation failures"
fi

# 5. Conflict detection – existing bpftrace on same probe
PROBE="tracepoint:syscalls:sys_enter_openat"
if pgrep -f "bpftrace.*$PROBE" >/dev/null; then
  echo "ERROR: Another bpftrace instance is already attached to $PROBE"
  exit 1
fi

echo "All preflight checks passed."
exec bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' "$@"

CLI Examples for Preflight Checks

# Run the wrapper
sudo ./preflight_bpftrace.sh

# Individual checks (optional)
uname -r | awk -v req="5.10" '{if ($0 < req) print "Kernel too old"}'
bpftrace --version | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+' | awk -v req="0.15.0" '{if ($0 < req) print "bpftrace too old"}'
[[ $EUID -eq 0 ]] || echo "Not root"
pgrep -f "bpftrace.*tracepoint:syscalls:sys_enter_openat" && echo "Conflict detected"

Troubleshooting Common Preflight Issues

Configuring Probe Time-to-Live Limits

Understanding TTL Limitations

bpftrace has no built‑in TTL map; TTL is enforced externally:

• Process lifetime – the program exits on SIGTERM/SIGINT or when an exit() statement runs.
• Interval timers – the interval:s:<sec> probe fires periodically, enabling self‑termination after a duration.
• External watchdog – a supervisor (systemd, cron, or custom daemon) sends a signal after a configured window.

The safest approach combines an internal timer (guarantees self‑cleanup if the supervisor fails) with an external watchdog (hard ceiling).

Setting TTL Limits in bpftrace

Hard limit example (60 seconds)

#!/usr/bin/env bash
# ttl_bpftrace.sh
DURATION=${1:-60}   # seconds, default 60
bpftrace -e "
BEGIN {
    printf(\"Starting trace for %d seconds\\n\", $DURATION);
}
interval:s:$DURATION {
    printf(\"%d seconds elapsed – stopping\\n\", $DURATION);
    exit();
}
tracepoint:syscalls:sys_enter_openat {
    printf(\"%s %s\\n\", comm, str(args->filename));
}
"

The interval:s:$DURATION probe fires once after $DURATION seconds, triggering exit().

Flexible version with warning and hard stop

bpftrace -e "
BEGIN {
    @start = nsecs;
    printf(\"Trace started\\n\");
}
interval:s:30 {
    printf(\"Warning: 30s elapsed\\n\");
}
interval:s:60 {
    printf(\"Hard limit reached – stopping\\n\");
    exit();
}
tracepoint:syscalls:sys_enter_openat {
    @[comm, str(args->filename)] = count();
}
END {
    printf(\"Final counts:\\n\");
    print(@);
}
"

CLI Commands for TTL Verification

# Run with an external timeout as a safety net
timeout 70s ./ttl_bpftrace.sh 60

# Verify no bpftrace processes remain
pgrep -f bpftrace || echo "No bpftrace processes remaining"

# Check exit code
./ttl_bpftrace.sh 60; echo "Exit code: $?"

A clean exit (0) indicates the internal timer fired; a non‑zero code from timeout indicates the external watchdog terminated the process.

Cleanup Verification and Explicit Stop Criteria

Designing Cleanup Verification Checks

After the bpftrace program ends, run the following verification:

#!/usr/bin/env bash
# verify_cleanup.sh
set -euo pipefail

USER_ID=$(id -u)
GROUP_ID=$(id -g)

# 1. No bpftrace processes
if pgrep -f bpftrace >/dev/null; then
    echo "ERROR: bpftrace processes still running:"
    pgrep -f bpftrace
    exit 1
fi

# 2. Maps owned by this user/group
MAPS=$(bpftool map show | awk -v uid="$USER_ID" -v gid="$GROUP_ID" '$5==uid && $6==gid {print $1}')
if [[ -n "$MAPS" ]]; then
    echo "ERROR: Leftover eBPF maps owned by uid=$USER_ID gid=$GROUP_ID:"
    echo "$MAPS"
    exit 1
fi

# 3. Pinned objects
PINNED=$(find /sys/fs/bpf -type f -uid "$USER_ID" -gid "$GROUP_ID" 2>/dev/null)
if [[ -n "$PINNED" ]]; then
    echo "ERROR: Pinned bpftrace objects remain:"
    echo "$PINNED"
    exit 1
fi

# 4. Temporary files in /tmp
TMPFILES=$(find /tmp -maxdepth 1 -name "bpftrace_*" -uid "$USER_ID" 2>/dev/null)
if [[ -n "$TMPFILES" ]]; then
    echo "ERROR: Temporary bpftrace files in /tmp:"
    echo "$TMPFILES"
    exit 1
fi

echo "Cleanup verification passed."
exit 0

Implementing Explicit Stop Criteria

Internal criteria – stop after a specific event count:

bpftrace -e "
tracepoint:syscalls:sys_enter_openat {
    @[comm, str(args->filename)] = count();
    if (@[comm, str(args->filename)] >= 100) {
        printf(\"Reached 100 events for %s %s\\n\", comm, str(args->filename));
        exit();
    }
}
"

External criteria – supervisor watches a stop file:

# Supervisor loop (run in background)
while [[ ! -f /tmp/stop_bpftrace ]]; do
    sleep 1
done
pkill -f "bpftrace.*tracepoint:syscalls:sys_enter_openat"

Combined Example (Preflight → TTL → Cleanup)

#!/usr/bin/env bash
# incident_playbook.sh
set -euo pipefail

# ---- Preflight -------------------------------------------------
source ./preflight_bpftrace.sh   # exits on failure

# ---- TTL‑limited execution ------------------------------------
DURATION=60
bpftrace -e "
BEGIN {
    printf(\"Trace started for %d seconds\\n\", $DURATION);
}
interval:s:$DURATION {
    printf(\"%d seconds elapsed – stopping\\n\", $DURATION);
    exit();
}
tracepoint:syscalls:sys_enter_openat {
    @[comm, str(args->filename)] = count();
    if (@[comm, str(args->filename)] >= 100) {
        printf(\"Reached 100 events for %s %s\\n\", comm, str(args->filename));
        exit();
    }
}
END {
    printf(\"Final counts:\\n\");
    print(@);
}
" &

BPFTRACE_PID=$!
wait $BPFTRACE_PID
BPFTRACE_EXIT=$?

# ---- Cleanup verification --------------------------------------
source ./verify_cleanup.sh
if [[ $? -ne 0 ]]; then
    echo "Cleanup failed – manual intervention required."
    exit 1
fi

echo "Incident playbook completed successfully (exit code $BPFTRACE_EXIT)."
exit $BPFTRACE_EXIT

This script demonstrates a full lifecycle: preflight validation, TTL‑bound data collection with an internal stop criterion, and automated cleanup verification—providing on‑call responders with a reliable, auditable incident playbook.