Generating parsers from samples without blind trust

#Comparing Grammar Approaches for Platform Onboarding
Hand‑written grammars, parser‑generator libraries, and LLM‑generated extraction rules

Hand‑Written Grammars

Overview

Hand‑written grammars are explicit specifications of a language’s syntax authored directly by a developer, typically expressed in Backus‑Naur Form (BNF), Extended BNF (EBNF), or a custom DSL. The grammar defines terminals, non‑terminals, production rules, and optionally semantic actions. For platform onboarding, the grammar describes the structure of configuration files, API payloads, or CLI output that must be parsed before the data can be ingested into automation workflows.

Advantages & Disadvantages

Example Use Cases

• Parsing vendor‑specific CLI show output (e.g., show interface on a proprietary switch) where spacing is irregular and fields are optional.
• Interpreting legacy configuration files that mix INI‑style sections with custom directives not supported by standard parsers.
• Building a domain‑specific language (DSL) for intent‑based networking where the grammar encodes policy constructs that must be validated before translation to device‑level commands.

Parser‑Generator Libraries

Overview

Parser‑generator libraries take a formal grammar description (usually EBNF‑like) and automatically produce a parser (often recursive‑descent, LR, or PEG) in the target language. The developer writes the grammar once; the generator emits source code or a runtime parser object. This approach reduces manual parser implementation while preserving many benefits of a declarative grammar.

Popular Libraries

Advantages & Disadvantages

Example: Lark Grammar & Transformer

onboarding.lark

?start: device+

device: "device" NAME "{" interface* "}"
interface: "interface" NAME "{" ip_addr? "}"
ip_addr: "ip" ADDRESS "/" PREFIX_LEN

NAME: /[a-zA-Z0-9_]+/
ADDRESS: /(\d{1,3}\.){3}\d{1,3}/
PREFIX_LEN: /[0-9]+|[1-2][0-9]|3[0-2]/
%import common.WS
%ignore WS

Python transformer

from lark import Lark, Transformer

class OnboardingTransformer(Transformer):
    def device(self, items):
        name, *ifaces = items
        return {"device": str(name), "interfaces": [i for i in ifaces if isinstance(i, dict)]}
    def interface(self, items):
        name, *rest = items
        ip = rest[0] if rest and isinstance(rest[0], dict) else None
        return {"interface": str(name), "ip": ip}
    def ip_addr(self, items):
        addr, prefix = items
        return {"address": str(addr), "prefix": int(prefix)}
    def NAME(self, token): return str(token)
    def ADDRESS(self, token): return str(token)
    def PREFIX_LEN(self, token): return int(token)

def parse_onboarding(text: str):
    parser = Lark.open("onboarding.lark", parser="lalr", transformer=OnboardingTransformer())
    return parser.parse(text)

# Sample usage
sample = """
device router1 {
    interface eth0 {
        ip 10.0.0.1/24
    }
    interface eth1 {
    }
}
"""
result = parse_onboarding(sample)
print(result)

CLI Workflows

ANTLR (Java) – generate a Python parser and test it

# 1. Install ANTLR tool (requires Java)
wget https://www.antlr.org/download/antlr-4.13.1-complete.jar -O antlr.jar

# 2. Define grammar in Onboarding.g4
cat > Onboarding.g4 <<'EOF'
grammar Onboarding;
start: device+ ;
device: 'device' NAME '{' interface* '}' ;
interface: 'interface' NAME '{' ip_addr? '}' ;
ip_addr: 'ip' ADDRESS '/' PREFIX_LEN ;
NAME: [a-zA-Z0-9_]+ ;
ADDRESS: ('0'..'9')+ '.' ('0'..'9')+ '.' ('0'..'9')+ '.' ('0'..'9')+ ;
PREFIX_LEN: [0-9]+ ;
WS: [ \t\r\n]+ -> skip ;
EOF

# 3. Generate Python parser & listener
java -jar antlr.jar -Dlanguage=Python3 Onboarding.g4

# 4. Simple test driver (test_antlr.py)
cat > test_antlr.py <<'EOF'
from antlr4 import *
from OnboardingLexer import OnboardingLexer
from OnboardingParser import OnboardingParser
from OnboardingListener import OnboardingListener
import sys

class PrintListener(OnboardingListener):
    def enterDevice(self, ctx):
        print(f"Device: {ctx.NAME().getText()}")
    def enterInterface(self, ctx):
        print(f"  Interface: {ctx.NAME().getText()}")
    def enterIp_addr(self, ctx):
        print(f"    IP: {ctx.ADDRESS().getText()}/{ctx.PREFIX_LEN().getText()}")

def main():
    data = FileStream(sys.argv[1])
    lexer = OnboardingLexer(data)
    stream = CommonTokenStream(lexer)
    parser = OnboardingParser(stream)
    walker = ParseTreeWalker()
    walker.walk(PrintListener(), parser.start())

if __name__ == '__main__':
    main()
EOF

# 5. Run
python3 test_antlr.py sample.cfg

Lark – CLI utility

# Assuming lark is installed: pip install lark
lark -g onboarding.lark -e "pretty" sample.cfg

The command instantly reports parse errors (e.g., missing brace) with line/column numbers, facilitating rapid iteration.

Troubleshooting Common Issues

LLM‑Generated Extraction Rules

Overview

Instead of manually authoring a grammar or feeding it to a parser generator, a large language model (LLM) can be prompted to produce extraction rules directly from examples of the target format. These rules often take the form of regular expressions, simple parsing functions (e.g., Python snippets using re or pyparsing), or a provisional grammar that the LLM believes captures the structure. The LLM’s output is then used—often after light post‑processing—to extract fields from new configuration blobs during platform onboarding.

How LLMs Generate Extraction Rules

1. Prompt Construction – Supply a few labeled examples (e.g., raw CLI output with highlighted fields) and ask the model: “Write a Python function that extracts device_name, interface, and ip_address from the text below.”
2. Chain‑of‑Thought (CoT) Reasoning – The model may first explain observed patterns (delimiters, keyword positions) before emitting code.
3. Output Format – Returns a code block (commonly fenced with triple backticks) containing the extraction logic; optionally includes a short rationale.
4. Post‑Processing – A harness strips the fencing, validates syntax (e.g., python -m py_compile), and optionally runs unit tests on the supplied examples.
5. Iterative Refinement – If the initial rule fails on additional samples, feed the failures back to the model for correction.

Advantages & Disadvantages

Example Use Cases

• Extracting VLAN IDs and interface names from the output of show vlan brief on various switch models where column spacing varies.
• Parsing JSON‑like configuration snippets that include comments (non‑standard JSON) supplied by a network‑automation portal.
• Pulling key‑value pairs from vendor‑specific syslog messages that lack a formal schema but follow a recognizable pattern (e.g., %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up).

Sample LLM‑Generated Extractor (Python)

import re

def extract_interface_info(text: str):
    """
    Extracts interface name and IP address from lines like:
        interface GigabitEthernet0/1
          ip address 10.0.0.1 255.255.255.0
    Returns a list of dicts: [{'interface': 'GigabitEthernet0/1', 'ip': '10.0.0.1', 'mask': '255.255.255.0'}, ...]
    """
    pattern = re.compile(
        r"interface\s+(?P<iface>\S+)\s*\n\s*ip\s+address\s+(?P<ip>\d+\.\d+\.\d+\.\d+)\s+(?P<mask>\d+\.\d+\.\d+\.\d+)",
        re.IGNORECASE | re.MULTILINE,
    )
    return [m.groupdict() for m in pattern.finditer(text)]

# Example usage
sample = """
interface GigabitEthernet0/1
  ip address 10.0.0.1 255.255.255.0
interface GigabitEthernet0/2
  no ip address
"""
print(extract_interface_info(sample))

Verification Harness for Production‑Ready Output

Before any parsed or extracted data reaches production workflows, it must pass a rigorous verification harness. The harness combines static checks, dynamic testing, and safety guarantees to ensure correctness, security, and performance.

1. Static Analysis

• Syntax validation – Run the generated code through the language’s parser/compiler (python -m py_compile, javac, cargo check, etc.) to catch early errors.
• Linting & security scanning – Apply tools such as flake8, pylint, bandit (Python); ErrorProne, SpotBugs (Java); clippy (Rust) to detect unsafe patterns (e.g., eval, shell injection).
• Dependency verification – Ensure all third‑party libraries are approved, version‑pinned, and free of known vulnerabilities (using safety, dependabot, ossindex).

2. Unit‑Test Suite

• Positive cases