Audit Trails That Can Reconstruct AI Decisions

Introduction

Forensic replay reconstructs the exact sequence of decisions, inputs, tool interactions, and human judgments that led to a network change. In an LLM‑augmented operator workbench, each change originates from a natural‑language prompt, is translated into tool calls, yields evidence, is filtered into selected configuration fields, passes automated validation, and receives operator approval before execution. Capturing every artifact in an immutable, queryable log enables post‑incident analysis: investigators can replay the decision tree, verify safety checks, and determine whether a failure stemmed from a faulty prompt, missing evidence, erroneous validation, or an inappropriate approval.

A purpose‑built action log schema provides:

Atomic traceability – each logical step (prompt, tool evidence, selection, validation, approval) is a distinct row with foreign‑key links, enabling precise reconstruction.
Immutable audit trail – append‑only storage with cryptographic hashes prevents tampering.
Query‑friendly structure – indexed columns (timestamp, operator ID, change ID) support rapid filtering during incident triage.
Extensibility – new evidence types or validation steps can be added as new tables without breaking existing queries.
Operational safety – the schema enforces that no change can be marked “executed” without a corresponding approval row, creating a hard gate.

Designing the Action Log Schema

Key Entities and Attributes

Entity	Primary Key	Key Attributes
PromptInput	`prompt_id` (UUID)	`change_id`, `operator_id`, `timestamp`, `raw_text`, `token_count`, `model_version`
ToolEvidence	`evidence_id` (UUID)	`prompt_id`, `tool_name`, `invocation_id`, `start_ts`, `end_ts`, `exit_code`, `stdout`, `stderr`, `structured_payload`
SelectedField	`field_id` (UUID)	`prompt_id`, `field_path`, `old_value`, `new_value`, `source_evidence_id`
ValidationResult	`validation_id` (UUID)	`prompt_id`, `rule_name`, `passed` (bool), `message`, `severity`, `checked_at`
OperatorApproval	`approval_id` (UUID)	`prompt_id`, `operator_id`, `decision` (approved/rejected/needs_review), `comment`, `approved_at`

All tables share a change_id (UUID) that groups the artifacts belonging to a single change request, enabling a single‑query view of an entire change lifecycle.

PostgreSQL‑Compatible DDL

-- Prompt Inputs
CREATE TABLE prompt_input (
    prompt_id      UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    change_id      UUID NOT NULL,
    operator_id    UUID NOT NULL,
    received_ts    TIMESTAMPTZ NOT NULL DEFAULT now(),
    raw_text       TEXT NOT NULL,
    token_count    INTEGER,
    model_version  VARCHAR(32),
    CONSTRAINT uq_prompt_change UNIQUE (change_id, prompt_id)
);
CREATE INDEX ix_prompt_input_change ON prompt_input(change_id);
CREATE INDEX ix_prompt_input_ts ON prompt_input(received_ts);

-- Tool Evidence
CREATE TABLE tool_evidence (
    evidence_id    UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    prompt_id      UUID NOT NULL REFERENCES prompt_input(prompt_id) ON DELETE CASCADE,
    tool_name      VARCHAR(64) NOT NULL,
    invocation_id  VARCHAR(128),
    start_ts       TIMESTAMPTZ NOT NULL,
    end_ts         TIMESTAMPTZ NOT NULL,
    exit_code      SMALLINT,
    stdout         TEXT,
    stderr         TEXT,
    structured_payload JSONB,
    CONSTRAINT chk_evidence_order CHECK (end_ts >= start_ts)
);
CREATE INDEX ix_tool_evidence_prompt ON tool_evidence(prompt_id);
CREATE INDEX ix_tool_evidence_ts ON tool_evidence(start_ts);

-- Selected Fields
CREATE TABLE selected_field (
    field_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    prompt_id      UUID NOT NULL REFERENCES prompt_input(prompt_id) ON DELETE CASCADE,
    field_path     TEXT NOT NULL,
    old_value      TEXT,
    new_value      TEXT NOT NULL,
    source_evidence_id UUID REFERENCES tool_evidence(evidence_id) ON DELETE SET NULL,
    CONSTRAINT uq_field_per_prompt UNIQUE (prompt_id, field_path)
);
CREATE INDEX ix_selected_field_prompt ON selected_field(prompt_id);

-- Validation Results
CREATE TABLE validation_result (
    validation_id  UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    prompt_id      UUID NOT NULL REFERENCES prompt_input(prompt_id) ON DELETE CASCADE,
    rule_name      VARCHAR(128) NOT NULL,
    passed         BOOLEAN NOT NULL,
    message        TEXT,
    severity       VARCHAR(16) CHECK (severity IN ('info','warning','error','critical')),
    checked_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ix_validation_prompt ON validation_result(prompt_id);
CREATE INDEX ix_validation_passed ON validation_result(passed);

-- Operator Approvals
CREATE TABLE operator_approval (
    approval_id    UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    prompt_id      UUID NOT NULL REFERENCES prompt_input(prompt_id) ON DELETE CASCADE,
    operator_id    UUID NOT NULL,
    decision       VARCHAR(16) NOT NULL CHECK (decision IN ('approved','rejected','needs_review')),
    comment        TEXT,
    approved_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ix_approval_prompt ON operator_approval(prompt_id);
CREATE INDEX ix_approval_decision ON operator_approval(decision);

Notes

prompt_input.change_id is not a foreign key to a separate changes table in this minimal schema; if a dedicated change‑request table exists, add the FK.
Tables are append‑only; corrections should insert a new row with a correction_flag and retain the original for audit.
A materialized view or SQL function can join the five tables to produce a flat “change timeline” for replay.

Capturing Prompt Inputs

Input Data Model

The prompt_input row stores the exact natural‑language utterance that the operator typed or spoke into the workbench.

raw_text – verbatim prompt, preserved for reproducibility.
token_count – optional, useful for cost analysis and detecting truncation.
model_version – identifies which LLM processed the prompt; critical when model upgrades change behavior.
operator_id – links to the identity provider (LDAP, OIDC) for non‑repudiation.
received_ts – monotonic clock; used for ordering and sharding.

Input Validation Rules

Before persisting a prompt, the workbench should enforce:

Length limit – e.g., LENGTH(raw_text) ≤ 4000 characters to prevent DoS via massive prompts.
Character set – allow UTF‑8 but reject control characters < 0x20 except \n, \t.
Rate limiting – per‑operator max N prompts per minute to curb abuse.
Prompt sanitization (optional) – strip known injection patterns if the LLM wrapper does not already do so; log the original and sanitized versions separately.

If any rule fails, the workbench must reject the prompt and return an error to the operator; no downstream tables are populated for that attempt.

Example Input Data

{
  "prompt_id": "a3f9c2e1-5b6d-4f9a-8c2e-1f7a9b3c4d5e",
  "change_id": "c1-2024-09-24-001",
  "operator_id": "op-42",
  "received_ts": "2024-09-24T14:03:12Z",
  "raw_text": "Configure interface GigabitEthernet0/1 with description UPLINK_TO_CORE and enable IPv6 address 2001:db8:1::1/64",
  "token_count": 22,
  "model_version": "llama3-70b-instruct-v0.1"
}

INSERT INTO prompt_input (prompt_id, change_id, operator_id, received_ts, raw_text, token_count, model_version)
VALUES (
    'a3f9c2e1-5b6d-4f9a-8c2e-1f7a9b3c4d5e'::uuid,
    'c1-2024-09-24-001'::uuid,
    'op-42'::uuid,
    '2024-09-24T14:03:12Z'::timestamptz,
    'Configure interface GigabitEthernet0/1 with description UPLINK_TO_CORE and enable IPv6 address 2001:db8:1::1/64',
    22,
    'llama3-70b-instruct-v0.1'
);

Integrating Tool Evidence

Evidence Data Model

Each tool invocation (CLI, NETCONF, RESTCONF, SNMP, custom script) yields a row in tool_evidence.

tool_name – e.g., ios_cli, netconf_put, restconf_patch.
invocation_id – correlation ID supplied by the orchestration layer.
start_ts / end_ts – monotonic timestamps; enable duration metrics.
exit_code – numeric status; non‑zero signals failure.
stdout / stderr – raw text output; stored as‑is for replay.
structured_payload – JSONB containing parsed fields (e.g., { "interface": "GigabitEthernet0/1", "ipv6": "2001:db8:1::1/64" }) to facilitate querying without parsing raw text.

Evidence Collection Methods

Orchestrator‑mediated capture – The automation engine wraps each tool call in a subprocess logger that records stdin/stdout/stderr, timestamps, and exit code before returning results to the LLM planner.
Side‑car agent – A lightweight agent on the target device streams command output via a secure channel (TLS) to a central collector, which writes directly to tool_evidence.
Proxy‑based interception – For RESTCONF/NETCONF, a transparent proxy logs the full request/response bodies and headers.
Sandboxed script execution – Custom validation or remediation scripts run in a confined environment; their output is captured via cgroups/ptrace and inserted as evidence.

All methods must guarantee atomicity: either the evidence row is written successfully, or the entire change transaction is rolled back.

Example Evidence Data

{
  "evidence_id": "9d2e4b7a-1c3f-4a9b-8d2e-6f1a2c3d4e5f",
  "prompt_id": "a3f9c2e1-5b6d-4f9a-8c2e-1f7a9b3c4d5e",
  "tool_name": "ios_cli",
  "invocation_id": "inv-20240924-001",
  "start_ts": "2024-09-24T14:03:15Z",
  "end_ts": "2024-09-24T14:03:18Z",
  "exit_code": 0,
  "stdout": "interface GigabitEthernet0/1\n description UPLINK_TO_CORE\n ipv6 address 2001:db8:1::1/64\n",
  "stderr": "",
  "structured_payload": {
    "interface": "GigabitEthernet0/1",
    "description": "UPLINK_TO_CORE",
    "ipv6_address": "2001:db8:1::1/64"
  }
}

INSERT INTO tool_evidence (
    evidence_id, prompt_id, tool_name, invocation_id, start_ts, end_ts,
    exit_code, stdout, stderr, structured_payload
) VALUES (
    '9d2e4b7a-1c3f-4a9b-8d2e-6f1a2c3d4e5f'::uuid,
    'a3f9c2e1-5b6d-4f9a-8c2e-1f7a9b3c4d5e'::uuid,
    'ios_cli',
    'inv-20240924-001',
    '2024-09-24T14:03:15Z'::timestamptz,
    '2024-09-24T14:03:18Z'::timestamptz,
    0,
    'interface GigabitEthernet0/1\n description UPLINK_TO_CORE\n ipv6 address 2001:db8:1::1/64\n',
    '',
    '{"interface":"GigabitEthernet0/1","description":"UPLINK_TO_CORE","ipv6_address":"2001:db8:1::1/64"}'::jsonb
);