systemd Says Ready Network Says Not Yet

Introduction

Systemd starts units in parallel while respecting explicit ordering constraints (After=, Before=, Wants=, Requires=). Ordering only guarantees that a referenced unit has reached the active state; it does not wait for that unit to finish initializing internal resources. A service may therefore be reported as active before it is truly ready to accept traffic.

Linux network namespaces (netns) isolate the network stack. Systemd can create namespaces via a dedicated unit, the Namespace= option, or ExecStartPre/ExecStartPostscripts that invokeip netnsandip link`.

FRR (Free Range Routing) expects to run in the namespace where its interfaces reside. Its readiness is signaled by the Unix control socket (/var/run/frr/zserv.api) and the ability to respond to vtysh. FRR’s service unit typically uses Type=notify and a PIDFile= to inform systemd when the main process is up, but FRR may bind to interfaces before the routing table is fully populated, creating a transient active but not ready state.

Intended Startup State

Unit‑File Design

The intended state is expressed by the unit files that ship with FRR and any custom namespace‑creation units. A typical design:

# /etc/systemd/system/frr-namespace.service
[Unit]
Description=Create network namespace for FRR
DefaultDependencies=no
Before=frr.service
[Service]
Type=oneshot
ExecStart=/usr/sbin/ip netns add frr-ns
ExecStartPost=/usr/sbin/ip link set dev eth0 netns frr-ns
RemainAfterExit=yes

# /lib/systemd/system/frr.service (vendor unit, possibly overridden)
[Unit]
Description=FRR routing daemon
After=network.target frr-namespace.service
Wants=frr-namespace.service
[Service]
Type=notify
PIDFile=/var/run/frr/frr.pid
ExecStart=/usr/lib/frr/frrdaemon.sh
ExecStartPost=/usr/lib/frr/frr-reload.sh
PrivateTmp=true

Assumptions of the intended state

The namespace frr-ns exists and has the required interface (eth0) assigned.
The interface is up and has an IP address (static, DHCP, or pre‑configured).
FRR configuration files (/etc/frr/frr.conf, daemon‑specific files) are present and valid.
Systemd starts frr.service only after the namespace unit is active, guaranteeing that the interfaces FRR will bind to are already in the correct namespace.

Alternative: Using `Namespace=`

Systemd can create the namespace as part of service activation:

[Service]
Namespace=net
ExecStartPre=/usr/sbin/ip link set dev eth0 netns %N
ExecStart=/usr/lib/frr/frrdaemon.sh

Here %N expands to the service name, or an explicit netns path can be used. When Namespace= is used, an explicit After= on a separate namespace unit is unnecessary because the namespace is created before the service’s exec process starts.

Rendered Startup State: Where Theory Meets Practice

Ordering Limitations

Systemd’s ordering graph is a partial order; it does not wait for a unit to finish initializing beyond reaching the active state. For a oneshot unit without RemainAfterExit=yes, systemd marks it inactive immediately after the exec process exits, even though side effects (e.g., the namespace) persist.

If the namespace unit is:

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip netns add frr-ns

and lacks RemainAfterExit=yes, a subsequent frr.service with After=frr-namespace.service may start while the kernel is still finalizing the namespace’s internal data structures, causing sporadic ENODEV errors when moving an interface into the namespace.

Interaction with `ExecStartPre=` Scripts

Placing namespace creation and interface moves in ExecStartPre= keeps the ordering implicit but introduces timing risks:

[Service]
Type=notify
ExecStartPre=/usr/sbin/ip netns add frr-ns
ExecStartPre=/usr/sbin/ip link set dev eth0 netns frr-ns
ExecStart=/usr/lib/frr/frrdaemon.sh

Systemd runs the two ExecStartPre= commands, then starts the FRR daemon. Problems arise when:

The FRR wrapper backgrounds quickly (Type=forking or a script that forks and exits), causing systemd to consider the service started before child daemons finish initializing.
The wrapper internally calls unshare(CLONE_NEWNET), creating a new namespace and discarding the one created by ExecStartPre=.
Interface moves occur after the FRR process has already bound sockets (e.g., the wrapper starts daemons in a thread before returning to systemd).

Namespace=“ and Interface Placement

When using Namespace=net, the service’s main process already runs inside the new namespace. An ExecStartPre= that moves an interface must reference the interface in the host namespace; otherwise the command fails because the interface is not visible inside the child netns. A common pattern:

ExecStartPre=/usr/bin/nsenter --net=/var/run/netns/host ip link set dev eth0 netns frr-ns

If the host netns file descriptor is not ready, the service fails to start.

Observed Startup State: Symptoms and Diagnosis

Typical Failure Indicators

Even when systemctl status frr.service shows active (running), FRR may not be ready:

The control socket is missing:

$ ls -l /var/run/frr/zserv.api
ls: cannot access '/var/run/frr/zserv.api': No such file or directory

vtysh hangs or returns “% Socket: Connection refused”.
Journal entries show errors like %NETWORK_INTERFACE_NOT_FOUND: Interface eth0 not found.

Verifying Namespace Membership

$ sudo nsenter --target $(pidof frr) --net ip link

If the output lists the interface in the host namespace, the namespace move failed or was overridden.

Checking Unit State and Ordering

$ systemctl status frr-namespace.service
● frr-namespace.service - Create network namespace for FRR
   Loaded: loaded (/etc/systemd/system/frr-namespace.service; enabled)
   Active: active (exited) since Thu 2025-09-24 10:15:30 UTC; 3min ago
     Process: 567 ExecStart=/usr/sbin/ip netns add frr-ns (code=exited, status=0/SUCCESS)

If Active: active (exited) and RemainAfterExit=no, the unit is considered inactive after exit, breaking the ordering guarantee for any unit that relies on it being active.

Inspecting Namespace Links

$ ls -l /proc/$(pidof frr)/ns/net
lrwxrwxwx 1 root root 0 Sep 24 10:15 /proc/1234/ns/net -> net:[4026532293]
$ ls -l /var/run/netns/frr-ns
lrwxrwxwx 1 root root 0 Sep 24 10:15 /var/run/netns/frr-ns -> net:[4026532294]

A mismatch indicates the FRR process is not in the expected namespace.

Confirming Interface Placement

$ ip -n frr-ns link show eth0

If the command returns “Device not found”, the interface was never moved or was moved back.

Tracing Timing with Journal

$ journalctl -u frr.service --since "5 minutes ago"

Look for gaps between the service’s start timestamp and the appearance of the control socket or routing table population.

Best Practices to Align Intended, Rendered, and Observed States

Persist namespace units – Always add RemainAfterExit=yes to oneshot namespace creators so systemd treats them as active after exit.
Prefer Namespace= – Let systemd create the namespace before the service starts; use ExecStartPre= with nsenter or ip -n host to move interfaces from the host netns.
Ensure FRR does not bypass systemd’s notification – Verify the FRR wrapper calls sd_notify("READY=1") only after all child daemons have bound their sockets and parsed configuration. If the wrapper forks early, consider changing the service type to forking and providing a proper PIDFile= that points to the actual daemon’s PID.
Validate interface readiness – Add an ExecStartPost= that waits for the interface to be up and has an address (e.g., a loop that checks ip -n frr-ns -brief address show eth0).
Verify ordering graph – Run systemd-analyze verify frr.service and systemd-analyze plot to ensure no missing edges or cycles.
Log namespace and interface state – Include debug lines in the service’s ExecStartPre/ExecStartPostscripts thatlogger` the netns inode and interface location; this makes post‑mortem analysis straightforward.

By aligning the unit file design with the actual behavior of the kernel and the FRR wrapper, the gap between intended, rendered, and observed startup states can be eliminated, ensuring FRR is truly ready when systemd reports it as active.