AI & AUTOMATION

Cloudflare Security Audit Skill: Multi-Agent Vulnerability Harness

08/11/20261 view0

Key Takeaways
Cloudflare’s security-audit-skill is an open-source coding-agent skill released under the MIT license in the cloudflare/security-audit-skill repository. Originally seeding Cloudflare’s internal, fleet-wide vulnerability discovery harness described in skills/security-audit/README.md and their engineering publication Build your own vulnerability harness, this repository formalizes automated security reviews into a deterministic six-phase pipeline. Rather than trusting a single language model prompt to audit code and grade its own findings, the architecture orchestrates isolated sub-agents through reconnaissance, coverage-led hunting waves, adversarial validation (where a fresh verifier attempts to disprove every candidate finding), structured JSON emission, independent record verification, and target-neutral reporting. The pipeline is enforced by two zero-dependency Node.js CommonJS engines –validate-coverage-ledger.cjs and validate-findings.cjs– alongside OS-enforced sandboxing and descriptor-pinned file traversal (O_NOFOLLOW, fstat) that protects parent audit artifacts against malicious target code.

Why single-prompt LLM security audits fail

Treating code auditing as a single-turn prompt conversation (“find security bugs in this repository”) consistently breaks down across production codebases. Large language models (LLMs) instructed without structural boundaries frequently display predictable failure modes:

Confirmation bias and hallucinations: An agent that proposes a vulnerability becomes biased toward defending it, mistaking theoretical framework behaviors or unconfigured options for exploitable production bugs.
Severity inflation: Stylistic issues, missing optional headers, or defense-in-depth gaps get labeled as “Critical” or “High” vulnerabilities without a demonstrable path from untrusted input to an affected sink.
Lack of coverage accounting: Unstructured agents examine obvious top-level files, declare a codebase “thoroughly reviewed,” and silently overlook secondary RPC boundaries, background queues, batch jobs, and deserialization paths.
Hostile target execution risk: Running tests or build scripts supplied by an untrusted target repository can compromise the auditor host via malicious symlinks, process escapes, or path traversals.

[Naive Single-Prompt AI Audit]
  │
  ├── Unbounded Prompt ────> Skims Surface Files & Ignores Complex Subsystems
  ├── Self-Grading Model ──> Inflates Minor Style Gaps to "Critical" Bugs
  ├── Hallucinated Sinks ──> Assumes Framework Flaws Without Trace Proof
  └── Host Environment ───> Executes Untrusted Code Directly on Host

==================================== VS ====================================

[Cloudflare Security-Audit Pipeline]
  │
  ├── Phase 1: Recon ──────> Seeds coverage-ledger.json With Canonical IDs
  ├── Phase 2: Waves ──────> Isolated Hunters Cover 10 Attack Domain Companions
  ├── Phase 3: Validation ─> Fresh Adversarial Verifiers Attempt to Disprove
  ├── Phase 4: Output ─────> Tri-Verdict findings.json (Zero-Dep Schema Validator)
  ├── Phase 5: Re-Check ───> Fresh Eyes Audit Final Source Lines & Invariants
  └── Sandbox & Promotion ─> POSIX Descriptor-Locked Traversal (O_NOFOLLOW / fstat)

Architecture: from an agent skill to Cloudflare’s fleet harness

As documented in skills/security-audit/README.md by Cloudflare’s security team, the security-audit-skill was the foundational starting point for their internal automated vulnerability discovery harness. While Cloudflare expanded the concept into an internal multi-stage distributed system monitoring corporate repositories, the core operational model remains codified in skills/security-audit/SKILL.md.

The workflow operates under two modes defined in skills/security-audit/SKILL.md:

Guidance mode (default): Intended for focused security questions, pull request triage, or targeted subsystem checks. It avoids creating output files or launching heavy multi-agent swarms unless explicitly directed.
Full audit mode: Activated by explicit requests to audit or penetration-test a repository. It enforces the full six-phase lifecycle and outputs durable, machine-verifiable artifacts.

Target Codebase / Repository
        │
        ▼
[Phase 1: Reconnaissance] ───> 4 Parallel Research Agents (Stack, Auth, Sinks, Sandbox)
        │                      Outputs: architecture.md & coverage-ledger.json
        ▼
[Phase 2: Hunting Waves]  ───> Isolated General Hunters + Coverage Critics
        │                      Injected Companions: Web, AI, Native, Cloud, RPC...
        ▼
[Phase 3: Validation]     ───> Adversarial Candidate Verifiers (Try to DISPROVE)
        │                      Bounded Sandbox Fixtures & Local Dummy Inputs
        ▼
[Phase 4: Structured JSON]───> findings.json (confirmed / needs_validation / rejected)
        │                      Verified by validate-findings.cjs & report-schema.json
        ▼
[Phase 5: Verification]   ───> Fresh Eyes Re-Verification of Line Numbers & Invariants
        │
        ▼
[Phase 6: Reporting]      ───> REPORT.md, FINDINGS-DETAIL.md, NEEDS-VALIDATION.md

Multi-agent roles and strict write isolation

The framework establishes agent-neutral roles in skills/security-audit/SKILL.md to coordinate work across any modern coding runtime (such as Claude Code, Google Antigravity, or OpenAI Codex):

Parent orchestrator: Owns shared state, tracks the cost budget, manages the coverage ledger, and acts as the sole trusted process permitted to promote files from agent scratch spaces.
Research agents (subagent_type: research): Read-only sub-agents deployed during Phase 1 to map technical stacks, identity boundaries, entry surfaces, and offline test viability without modifying filesystem state.
General hunters (subagent_type: general): Isolated agents assigned specific units from the coverage ledger. Each hunter receives an empty scratch/ directory, inspects source deeply, and returns one structured JSON result.
Adversarial verifiers: Independent general agents assigned in Phase 3. A fundamental tenet of the harness is: The agent that checks a finding is never the agent that found it. The verifier receives the candidate finding with an explicit prompt mandate to disprove it against the source code and bounded local fixtures.

Sandboxed execution and POSIX descriptor promotion

When a lead requires executing a test fixture, build script, or parser reproducer, skills/security-audit/SKILL.md mandates an OS-enforced sandbox:

No external networking (only an isolated loopback namespace for local client/server fixtures).
An empty, allowlisted environment with scratch-local HOME and temporary directories.
Read-only mounts for target source code and compiler toolchains.
Strict CPU, memory, process count, disk, and wall-clock execution limits.

To eliminate symlink race conditions (TOCTOU) and directory traversal exploits, skills/security-audit/SKILL.md mandates that hunters and verifiers write only to their private scratch/ directory. Trusted parent-side code inspects and promotes files to retained artifacts/ using descriptor-relative system calls:

Opens non-inheritable file descriptors for the scratch and artifact directories before execution.
Traverses path components strictly using directory-relative lookups without following symlinks (O_NOFOLLOW).
Executes fstat on the opened leaf descriptor, verifying that it is a regular file with a hard link count of exactly one (st_nlink == 1) and within strict byte limits.
Creates destination parents and files exclusively, refusing to copy FIFOs, character devices, sockets, or changing files.

The six-phase audit lifecycle

The skill executes six progressive phases. Moving to an arbitrary phase or abandoning the workflow mid-run is prohibited; a run must terminate either with passing Phase 6 reports or an explicit run_status: "incomplete" record.

Phase 1: Reconnaissance and budget gating

Before launching reconnaissance, the parent verifies the run’s token and agent budget. Four parallel research agents analyze the codebase:

Agent 1a (Product & Stack): Identifies frameworks, runtime dependencies, offline build commands, and entry points.
Agent 1b (Authority & IAM): Maps low-trust principals, authorization middleware, permission gates, and tenant scopes.
Agent 1c (Entry Surfaces & Sinks): Inventories external inputs (HTTP, RPC, CLI, files, webviews, message queues) and follows them to sensitive sinks.
Agent 1d (Sandbox & Execution Visibility): Determines offline test capabilities, dummy data fixtures, and environment requirements.

The parent synthesizes these observations into architecture.md and seeds the deterministic coverage-ledger.json.

Phase 2: Coverage-led hunting waves

The parent schedules planned ledger units across isolated general hunters. Units are prioritized by entry point trust: unauthenticated internet-facing surfaces run before authenticated internal interfaces.

In addition to core attack classes defined in skills/security-audit/ATTACK-CLASSES.md (Injection, Access Control, File Handling, Cryptography, Business Logic), the parent injects specific domain companions based on repository technology:

AI-AND-LLM.md: Indirect prompt injection, memory poisoning, MCP server/tool confusion, action-binding approvals, and tool-schema dispatcher disagreement.
MEMORY-SAFETY-AND-BINARY.md: Buffer overflows, use-after-free, integer wraps, concurrency races, and binary loaders for native C/C++/Rust code.
WEB-PROTOCOL-AND-AUTH.md: HTTP request smuggling, cache poisoning, JWT/OAuth flaws, and session fixation.
CLIENT-SIDE.md: DOM XSS, prototype pollution, cross-window messaging trust, and webview bridges.
SUPPLY-CHAIN-AND-RELEASE.md: CI workflows, dependency pinning, artifact signing, and plugin update channels.
CLOUD-AND-DEPLOYMENT.md: Infrastructure as Code, IAM role assume policies, container escapes, and ingress routing.
PROTOCOLS-RPC-AND-MESSAGING.md: gRPC, GraphQL, Protobuf/Cap’n Proto, webhook validation, and message queue serialization.
RESOURCE-EXHAUSTION-AND-AVAILABILITY.md: ReDoS, worker queue starvation, rate limit bypasses, and cloud spend amplification.
DATA-ISOLATION-AND-LIFECYCLE.md: Multi-tenant cache leaks, search index bleed, soft-delete bypasses, and backup leakage.
DESKTOP-MOBILE-AND-LOCAL-IPC.md: Deep links, Unix domain sockets, XPC, Binder, D-Bus, and local daemon privileges.

Coverage critics review completed hunter waves to detect blind spots, reopening missed paths into an append-only attempts archive.

Phase 3: Adversarial candidate validation

Every proposed candidate finding is assigned to a fresh verifier. The verifier re-traces every line of the source path, attempts to identify compensating controls, and reproduces the vulnerability using bounded dummy inputs. If an explicit control stops the attack, the verifier rejects the candidate. If a required deployment fact is invisible in source, the candidate transitions to needs_validation.

Phase 4: Structured output validation

The parent aggregates verified records into findings.json and evaluates the document against report-schema.json using skills/security-audit/validate-findings.cjs.

Phase 5: Independent record verification

Fresh agents re-verify line references, check code context for recent modifications, and ensure the proposed fix represents the smallest effective change at the last trusted decision point.

Phase 6: Target-neutral reporting

The parent derives three clean markdown reports from the validated JSON:

REPORT.md: Executive summary, coverage statistics, methodology, and risk rollup.
FINDINGS-DETAIL.md: Detailed breakdown of confirmed vulnerabilities, source traces, reproduction steps, invariant definitions, and remediation patches.
NEEDS-VALIDATION.md: Open hypotheses blocked by unobserved production infrastructure, detailing non-destructive verification plans for service owners.

The Tri-Verdict findings model and schema enforcement

The findings engine in skills/security-audit/validate-findings.cjs enforces a three-branch schema that rejects ambiguous security claims:

                     [Hunter Candidate Finding]
                                 │
                                 ▼
                     [Adversarial Verification]
                                 │
       ┌─────────────────────────┼─────────────────────────┐
       ▼                         ▼                         ▼
  "confirmed"            "needs_validation"           "rejected"
 ──────────────         ────────────────────         ────────────
 • Full source trace    • Source-grounded lead       • Disproved by source
 • Demonstrated impact  • Missing deployment fact    • Defense-in-depth gap
 • Tested bounded POC   • NO SEVERITY permitted      • Intended behavior
 • Concrete fix patch   • Owner validation plan      • Recorded rationale

confirmed: Requires an unbroken source trace from an untrusted entrypoint through propagation steps to a dangerous sink, demonstrated impact on dummy data, and a concrete code fix.
needs_validation: Represents a valid source-grounded vulnerability hypothesis blocked solely by an unobservable deployment detail (such as a cloud WAF rule, CDN cache header, or identity provider configuration). Crucially, the schema strictly prohibits assigning a severity score to unvalidated findings.
rejected: Documents candidates refuted by code analysis, compensating controls, or intended same-principal authorization. Recording rejected candidates prevents repetitive hunting loops across successive waves.

Severity floors and trace topology rules

To eliminate common audit anti-patterns, validate-findings.cjs programmatically validates:

Severity caps: The overall_severity property cannot exceed the demonstrated impact.score. A theoretical flaw with low demonstrable blast radius cannot be inflated to high severity.
Trace topology: For any multi-line vulnerability trace, the first element must be an entrypoint, the final element must be a sink, and all intermediate elements must be propagation steps.
Deterministic ordering: Findings must be sorted lexicographically by their unique SHA-256 fingerprint, guaranteeing consistent diffs across runs.

Deterministic coverage tracking with zero-dependency Node.js

Rather than reporting vague percentages, skills/security-audit/validate-coverage-ledger.cjs tracks audit progress across a formal state machine:

    ┌──────────┐
    │ planned  │
    └────┬─────┘
         │
         ▼
  ┌─────────────┐
  │ in_progress │
  └──┬───────┬──┘
     │       │
     ▼       ▼
┌─────────┐ ┌───────────┐ ┌─────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────┐
│ covered │ │ candidate │ │ blocked │ │ deferred │ │ out_of_scope │ │ not_app. │
└─────────┘ └───────────┘ └─────────┘ └──────────┘ └──────────────┘ └──────────┘

The script derives a collision-free coverage_id for every unit by percent-encoding its canonical tuple:

function canonicalCoverageId(refs) {
  const fields = hasOwn(refs, "lifecycle") 
    ? ["surface", "boundary", "subsystem", "attack_class", "lifecycle"] 
    : ["surface", "boundary", "subsystem", "attack_class"];
  return fields.map((field) => encodeCanonicalRef(refs[field])).join("::");
}

This encoding guarantees that distinct architectural surfaces never collide into identical ledger keys. Furthermore, multiple runs against the same repository are additive: prior runs are ingested to target gaps, re-examine modified code, and carry forward valid findings without wasting compute on unchanged surfaces.

Installation, commands, and adoption guide

The skill is packaged for the open Skills standard and installs with npx skills:

# Project-level installation
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit

# User-level global installation
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit --global

To run an audit using an installed coding agent, trigger the skill with natural language commands:

# Execute a full audit with reports saved to a dedicated directory
security audit this codebase, output to ~/audits/my-project

# Request focused guidance on an API subsystem
review the authentication flow in ./src/auth for authorization bypasses

When to adopt Cloudflare’s security audit skill

High-assurance codebases: Ideal for microservices, cryptographic modules, financial APIs, and developer platforms where checklist scanners produce excessive noise and miss complex business logic flaws.
Multi-agent AI stacks: Essential for auditing agent frameworks, MCP servers, and LLM applications where data flow crosses untrusted user input, prompt templates, and tool handlers.
CI/CD security gates: Provides structured JSON (findings.json) that can be validated in automated pipelines without flaky external API dependencies.

By separating hunting from adversarial verification in skills/security-audit/VALIDATION-AND-REPORTING.md and backing every phase with deterministic validators, Cloudflare’s security-audit-skill establishes a mature engineering standard for AI-driven application security.