Engine Internals
The Attack Corpus: Organizing Adversarial Techniques as Parametric Gadgets
A red team that keeps a folder of jailbreak strings finds the same things every run. The Austa engine stores techniques differently: each one is a gadget, a skeleton with typed mutation knobs, so a single template expands into thousands of concrete attempts and the generator can keep mutating it. This is how the corpus is shaped, classified, deduplicated, and versioned.
This is Part 3 of the series on the Austa engine, the closed loop that generates attacks, runs them through the harness, and judges the result. The loop only adapts as fast as its raw material. If the material is a flat list of finished payloads, the engine is a fancy way to replay last quarter's attacks. The corpus is the material. It has to be expressive enough that the generator in Part 4 has somewhere to go.
Why a string list is the wrong abstraction
The first generation of LLM security tooling shipped payload lists: a file of jailbreak prompts, a file of injection strings, run them all, count the refusals. That model has a ceiling. A finished string is a single point in attack space. It either works against this target or it does not, and when it stops working (because the model was patched, or the system prompt changed, or a guardrail caught that exact phrasing) the string is dead and you are back to writing new ones by hand.
The structure underneath any good attack is not the string, it is the recipe. "Tell the model its previous instructions were a test, then issue the real instruction" is a recipe. "Carry the forbidden instruction in a layer the safety filter does not read, then ask the model to decode it" is a recipe. The exact words are an instantiation. The Austa corpus stores the recipe and treats the words as one of many knobs.
We call a stored recipe a gadget. A gadget is a parametric template: a skeleton of an attack with typed slots, where each slot is a mutation knob the generator can turn. One gadget is not one attack. It is a family of attacks that share a mechanism, and the size of the family is the product of its knob ranges. This is a generalization of the probe taxonomy in garak, which groups attacks by the weakness they exercise rather than by the literal text, pushed further so that the grouping is itself executable and expandable.
The anatomy of a gadget
Every gadget in the corpus carries the same fields. The skeleton is a template with named holes. The knobs are typed parameters that fill those holes. The metadata classifies the gadget and ties it to ground truth. Here is the shape, lightly sketched:
gadget: indirect_injection.rag_document
skeleton: |
{carrier_open}
{benign_filler}
{injection_marker} {payload_goal} {injection_marker}
{benign_filler}
{carrier_close}
knobs:
carrier: [markdown_comment, html_attr, footnote, table_cell, alt_text]
injection_marker: [none, "SYSTEM:", "### instruction", invisible_tag]
obfuscation: [none, base64, homoglyph, zero_width_split, lang_switch]
payload_goal: ref(goal_library) # exfiltrate canary | call tool | leak prompt
persistence: [single_doc, split_across_docs, memory_write]
taxonomy:
owasp_llm: LLM01 # prompt injection
owasp_asi: ASI02 # context injection
surface: retrieved_context
ground_truth: canary_required
provenance: ["garak/promptinject", "internal/2026-Q2"]
version: 3The skeleton holds the mechanism: a payload wrapped in a carrier so it arrives through a data surface rather than the user turn. The knobs are where the expansion lives. carrier has five values, injection_marker four, obfuscation five, persistence three, and payload_goal draws from a shared goal library that might hold dozens of concrete objectives. The Cartesian product is already in the thousands before the attacker model rewrites any natural-language slot. That is the whole point: one curated gadget is a large, structured search space, and the generator walks it under judge feedback instead of enumerating it blindly.
Knobs are typed, which is what makes mutation safe and seeded. An obfuscation knob is an enum drawn from a registry of pure, deterministic transforms (the same transform layer Part 4 leans on). A payload-goal knob is a reference into the goal library, never a free string baked into the gadget. Because every knob has a declared type and range, the generator can sample a knob, mutate one axis while holding the rest fixed, or recombine knob settings across gadgets without producing nonsense.
Four axes, and the two standards on top
A gadget needs more than a mechanism to be useful in a campaign. It needs to be findable: an operator scoping a run wants "everything that targets a tool-using agent through retrieved context," not a gadget ID they have to memorize. The corpus indexes every gadget on four internal axes plus two external taxonomies.
The four Austa axes are orthogonal questions about an attack:
| Axis | Question it answers | Example values |
|---|---|---|
| Delivery surface | Where does the payload enter? | user turn, retrieved context, tool output, document, memory entry |
| Obfuscation | How is it hidden from filters? | none, encoding, homoglyph, zero-width, language switch |
| Payload goal | What is it trying to achieve? | leak system prompt, exfiltrate data, hijack a tool call, induce refusal bypass |
| Target capability | What must the target be able to do? | chat only, RAG retrieval, tool calling, multi-turn memory, browser actions |
The target-capability axis is what keeps a campaign honest. There is no point firing a tool-hijack gadget at a target with no tools, and the harness already tells the engine what the target can do. Gadgets whose required capability the target lacks are filtered out before generation, so the budget goes to attacks that could actually land.
On top of the internal axes, every gadget carries its mapping to the two industry taxonomies: the OWASP Top 10 for LLM Applications and the OWASP Agentic Security Initiative list. We covered the agentic list in the OWASP Top 10 for AI Agents walkthrough. The mapping is many-to-one in both directions. A single indirect-injection gadget maps to LLM01 (prompt injection) and to ASI02 (context injection); a tool-hijack gadget reaches ASI03 (unbounded tool execution) and ASI05 (agentic data exfiltration). Carrying both mappings is what lets a finding roll up into a per-OWASP scorecard later without re-classifying anything by hand.
A walk through the gadget families
The corpus is organized into families, each a cluster of gadgets that share a delivery mechanism. Six of them carry most of the weight.
Direct injection
The payload arrives in the user turn and tries to override the system prompt directly: instruction overrides, role-play framings, "ignore previous instructions," fake system messages. The knobs are the framing (authority claim, fictional context, urgency), the override marker, and the goal. This family is the floor, the thing every target should already survive, which is exactly why it belongs in the corpus as a regression baseline.
Indirect / RAG injection
The payload rides in through a data surface the agent reads later: a retrieved document, a fetched page, an email, a memory entry. The defining knob is the carrier, where in the data the instruction hides, paired with persistence (one document, split across several, or written into memory for a later turn). This family is where ground-truth canaries matter most, because "did the injected instruction execute" is far cleaner to prove than "did the model say something it should not have."
Encoding smuggling
The forbidden content is carried in a representation the safety layer does not inspect, then the model is asked to decode and act on it: base64, ROT, homoglyph substitution, zero-width-character splitting, language switching. The knobs are the encoding, the nesting depth, and the decode prompt. We treat each encoding as a seeded pure transform so the same knob setting always produces the same bytes. The mechanics get a full treatment in encoding smuggling in prompt injection.
Tool hijack
For tool-using agents: steer the model into calling a privileged tool with attacker-chosen arguments, or into chaining a read tool to a send tool. The knobs are the target tool, the argument shape, and the pretext that justifies the call. Success here is judged on the tool call itself (recorded by the harness without real side effects), not on the prose around it, which makes it one of the most deterministic families to score.
Instruction / prompt leakage
Get the model to reveal its system prompt, hidden tool definitions, or developer instructions. The knobs are the elicitation framing (debugging request, translation request, "repeat the text above"), the format coercion, and whether the ask is split across turns. A planted canary string in the system prompt turns this from a judgment call into a string match.
Multi-turn escalation
Some attacks only work after the conversation has been steered. This family encodes the shape of a dialogue rather than a single message: build rapport or establish a frame across early turns, then pivot to the real ask. The knobs parameterize the ladder, how many rungs, how steep, what frame. The orchestrator in Part 5 drives the turns; the gadget supplies the plan. The pattern is detailed in the multi-turn prompt injection pattern.
Curation, dedup, and versioning
A corpus that anyone can append to rots quickly. Two operators write the same direct-injection gadget with different wording, a public benchmark gets imported twice, and within a quarter the engine is wasting budget firing near-duplicates. The corpus has three disciplines to stop that.
Curation with provenance. Every gadget records where it came from: an internal authoring session, a published technique, or a public benchmark set. The engine ingests public corpora as gadgets rather than as strings, so a flat list of finished attacks becomes a populated skeleton with its knobs back-fitted. The seed material here is real: the harmful-behavior sets in JailbreakBench and HarmBench become payload-goal entries in the goal library, and the probe families in garak map onto gadget families. Provenance is not decoration. When a benchmark updates, every gadget derived from it is traceable.
Deduplication on the gadget, not the string. Because the unit of storage is the parametric template, two authors who wrote the same mechanism collapse to one gadget the moment their skeletons and knob types match, even if every example string differs. Dedup runs on a structural fingerprint of the skeleton plus the knob signature, so the corpus holds one canonical indirect-RAG gadget, not forty variations of it. Concrete near-duplicate attempts that the generator produces at runtime are deduplicated separately, downstream, when findings are scored.
Content-addressed versioning. Every gadget is versioned, and a campaign records the exact corpus version it ran against (a content hash over the full set of gadgets and the goal library). This is the corpus half of the engine's determinism guarantee. The deterministic transforms are seeded pure functions and every exchange is recorded, so a finding replays exactly; pinning the corpus version means the same campaign re-run against the same target draws from precisely the same gadgets, with the same knob ranges, in the same order. When you bump a gadget to version 4 because you added a carrier value, last month's campaign still points at version 3 and still reproduces. A campaign is a tuple of (corpus version, seed, budget, target), and that tuple is enough to rebuild the run.
The corpus is the difference between "we have a list of attacks" and "we have an expandable, classified, reproducible search space." A gadget is small to author and large to run. That ratio is what lets the engine keep finding new failures against a target that already survived last quarter's strings.
The Austa engine series
- Architecture overview
- The target harness
- The attack corpus and taxonomy
- The adversarial generator
- The orchestrator and multi-turn campaigns
- The judge
- Canaries and deterministic ground truth
- Scoring, severity, and regression