Introduction to Rollout Batching

Definition of Rollout Batching

Rollout batching groups configuration or software changes into discrete sets (batches) that are applied to a fleet of devices in a coordinated manner. Instead of pushing a change to every host sequentially, operators send the same change to N devices concurrently, wait for the batch to finish (or reach a success threshold), then proceed to the next batch. The goal is to amortize fixed overhead—such as connection setup, transaction logging, and human review—while limiting the blast‑radius of a faulty change.

Benefits of Rollout Batching

Common Challenges in Rollout Batching

1. Device commit queues become saturated – if the batch arrival rate exceeds the device’s commit service rate, queues grow and latency spikes.
2. SSH fan‑out limits – opening many simultaneous SSH connections can exhaust client‑side file descriptors, SSH daemon MaxStartups, or network NIC queues.
3. Operator review latency – human approval steps do not scale linearly; large batches may stall waiting for a single reviewer, negating parallelism gains.
4. Failure propagation – a faulty change in a large batch can affect many devices before detection, increasing blast‑radius.
5. Tooling overhead – orchestration frameworks (Ansible, Salt, custom scripts) add their own serialization points (e.g., playbook strategy linear vs free).

Understanding Bottlenecks in Rollout Batching

Device Commit Queues

Queue Architecture

Most network operating systems expose a commit queue that serializes configuration transactions to guarantee atomicity and consistency. Internally this is often implemented as a single‑threaded workqueue backed by a kernel FIFO (e.g., Linux kfifo or a proprietary ring buffer).

• Juniper JunOS – commit operations are dispatched to the commitd daemon; each commit acquires a mutex, writes candidate config to the internal DB, then runs validation and activation steps before releasing the lock.
• Cisco IOS XR – the commit command enters the sysdb transaction queue; a single sysdb worker processes transactions in FIFO order.
• Arista EOS – the ConfigSync service uses a std::queue protected by a std::mutex; each batch entry triggers a ConfigApply transaction.

The service rate (μ) is determined by the slowest stage (usually validation + hardware reprogramming). Typical μ values:

Queue Limitations

• Finite depth – many devices expose a configurable maximum queue length (e.g., set system commit queue-depth 100). Exceeding this drops new commits with commit failed: queue full.
• Head‑of‑line blocking – a slow commit (e.g., a large ACL rewrite) blocks all subsequent commits regardless of their size.
• Priority inversion – some platforms allow “commit confirm” or “commit check” to bypass the queue, but these are not suitable for production changes.
• Measurement latency – the time a commit spends waiting in queue (Wq) is not exposed via standard CLI; it must be inferred from timestamps or internal counters.

SSH Fan‑out

SSH Fan‑out Architecture

Parallel SSH execution is typically achieved via one of three patterns:

1. Process‑per‑host – tools like pssh, mssh, or ansible with strategy: free spawn an OS process (or thread) per target. Each process opens its own TCP socket to port 22 and performs the SSH handshake independently.
2. Multiplexed connections – ssh -M -S <control-path> creates a master connection; subsequent sessions reuse the same TCP channel, reducing handshake overhead but sharing a single SSH channel’s window flow control.
3. Connection pooling daemons – sshd with MaxSessions and MaxStartups tuned, or external proxies like haproxy in TCP mode, distribute incoming SSH SYNs across a pool of backend sshd workers.

Key parameters that affect fan‑out performance:

SSH Fan‑out Limitations

• Connection establishment overhead – each new SSH session incurs a full key exchange (typically Curve25519‑SHA256) and authentication (password/pubkey). This dominates latency for batches < 20 devices.
• Server‑side resource exhaustion – sshd forks a child process per session (unless UsePrivilegeSeparation=no and PermitTunnel=no). High fork rates can cause fork: Cannot allocate memory errors.
• Network congestion – the SYN‑ACK exchange for dozens of connections can saturate the NIC’s interrupt moderation or cause TCP retransmits if the server’s listen backlog is exceeded.
• Session multiplexing limits – reusing a master connection reduces handshake cost but serializes payloads over a single encrypted channel; large command outputs can cause head‑of‑line blocking inside the SSH channel itself.

Operator Review Latency

Review Process Overview

In many organizations a change must pass through a human gate before execution:

1. Change request creation – operator fills a ticket (Jira, ServiceNow) with description, CI/CD pipeline link, and rollback plan.
2. Peer review – one or more engineers examine the diff, run static checks (e.g., batfish, cisco‑parser), and leave comments.
3. Approval – a designated approver (often a network architect) signs off, sometimes requiring a second‑level approval for high‑risk changes.
4. Scheduled window – the change is queued for a maintenance window; the batch is only released when the window opens.

Each step introduces latency that is largely independent of batch size (except for the time to collate the diff).

Latency Factors

When the operator review latency (T_review) exceeds the time saved by parallelizing device commits (T_save), increasing batch size yields diminishing or negative returns.

Identifying the Point of Diminishing Returns

Metrics for Evaluation

The point of diminishing returns occurs when ∂T_batch/∂N ≈ 0 or when T_review dominates T_batch such that increasing N no longer reduces T_batch appreciably.

Data Collection Methods

Logging and Monitoring Tools

CLI Commands for Data Collection

Measuring SSH Connection Time

# Single shot measurement (average of 10 runs)
for i in {1..10}; do
    /usr/bin/time -p ssh -o BatchMode=yes -o ConnectTimeout=3 user@router01 true 2>&1 | grep real
done | awk '{sum+=$2} END{print "avg ssh connect:", sum/NR, "s"}'

• -o BatchMode=yes disables password prompts; fails fast if auth requires interaction.
• Output: real 0.27 → 270 ms handshake + auth.

Capturing Device Commit Queue Wait (Juniper example)

# Enable commit logging (if not already)
set system syslog file commit_log any info
commit and-quit

# After a batch, extract timestamps:
show log commit_log | match "commit started" | display timestamp
show log commit_log | match "commit completed" | display timestamp

Compute Wq = commit_start - commit_enqueue where commit_enqueue can be approximated by the time the orchestration host sent the RPC (logged by the automation tool).

Measuring Batch Completion Time with Ansible

# batch_playbook.yml
- hosts: all
  serial: "{{ batch_size | int }}"   # controls batch size
  tasks:
    - name: Apply config template
      template:
        src: templates/router.j2
        dest: /etc/router.conf
      notify: reload router

  handlers:
    - name: reload router
      ios_config:
        src: /etc/router.conf
        commit: yes

Run with:

ANSIBLE_FORCE_COLOR=1 ansible-playbook -i inventory.ini batch_playbook.yml -e "batch_size=20" \
  | tee run_20.log

Extract timing:

grep "PLAY RECAP" run_20.log | awk -F' ' '{for(i=1;i<=NF;i++) if($i ~ /ok=/) print $i}'

Or use callback_plugin: profile_tasks to get per‑task timing.

Pulling Review Time from Jira (bash + jq)

JIRA_URL="https://jira.example.com/rest/api/2/search"
JQL='project = NET AND status = Approved AND updated >= -7d'
curl -s -u "$JIRA_USER:$JIRA_TOKEN" -H "Content-Type: application/json" \
     -d "{\"jql\":\"$JQL\",\"fields\":[\"created\",\"resolved\"]}" "$JIRA_URL" |
jq -r '.issues[] | .fields.created, .fields.resolved' |
awk '
  NR%2{created=$1; next}
  {cmd="date -d \""created"\" +%s"; cmd | getline start; close(cmd)
   cmd="date -d \""$1"\" +%s"; cmd | getline end; close(cmd)
   print end-start}
' | awk '{sum+=$1; count++} END{print "avg review time (s):", sum/count}'

Troubleshooting Common Issues

Device Commit Queue Issues

Queue Overflow

Symptom: commit failed: queue full appears in device logs; orchestration logs show many retries.

Diagnosis

1. Check queue depth:

   # Juniper
   show system commit queue
   # Cisco IOS XR
   show