Engine Internals

The Orchestrator: Multi-Turn Campaigns and Conversation Tree Search

A model that refuses a request cold will often comply after a few turns of rapport or reframing. The single-shot scanner never sees that failure. The orchestrator is the part of the Austa engine that drives a whole conversation as the attack, exploring the dialogue space instead of firing one payload and giving up.

By Austa · Published June 4, 2026 · ~9 min read

Single-turn is the easy 20 percent

The first generation of LLM scanners worked one prompt at a time. Load a list of payloads, send each one, check the reply, move on. That model catches the obvious failures: the system prompt that leaks to a flat "ignore previous instructions," the classifier with a hole you can drive a base64 string through. It is a real and useful pass. It is also the easy 20 percent.

The failures that actually reach production are usually conversational. A model that hard-refuses a request on turn one will frequently grant the same request on turn four, after the surrounding context has been reshaped so the request no longer looks like the thing it was trained to refuse. The refusal is contextual, so the attack has to be contextual too. This is why the engine's core loop is GENERATE, RUN, JUDGE, LEARN rather than a static checklist: defenses are stochastic and stateful, and the dangerous attack is the one that only works after the conversation has been set up.

Crescendo: why a few turns beats one shot

The clearest published illustration is the technique Microsoft researchers named Crescendo. Instead of asking for the prohibited thing directly, the attacker opens with a benign, on-topic question, then escalates in small steps, each one referencing the model's own previous answers as established common ground. By the time the genuinely sensitive ask arrives, the model has spent several turns being helpful on adjacent material and treats the final step as a natural continuation rather than a fresh request to evaluate. No single message in the transcript looks like an attack. The attack is the slope.

Microsoft's open-source PyRIT ships this as a multi-turn orchestrator: an attacker component plans the next message, sends it, reads the response, and decides whether to push further or change tack, all while carrying conversation state forward. The Austa orchestrator generalizes that pattern. Where a single orchestrator runs one line of escalation to its conclusion, the Austa engine treats the conversation as a search space and explores many lines at once.

Escalation ladders

An escalation ladder is the orchestrator's unit of strategy: a sequence of intents that walks from a safe opener to the actual objective. A typical ladder has three phases. Establish context: a few turns that put the model in a frame where the objective is plausible (a fiction writer, a security researcher, a system administrator debugging a config). Pivot: reframe the objective so it reads as a continuation of that frame rather than a new ask. Cash out: request the concrete artifact and let the judge decide whether the reply crosses the line.

Ladders are not scripts. Each rung specifies an intent, not fixed text. The actual message at each step is synthesized by the generator against the live conversation, conditioned on what the target just said. If the model answered the previous rung enthusiastically, the next message can be bolder. If it half-refused, the orchestrator dials back, adds a turn of reassurance, and tries the pivot from a different angle. The ladder is the plan; the generator fills it in per turn.

Conversation state is the thing being mutated

In a single-turn attack the unit being mutated is a string. In a multi-turn campaign the unit being mutated is the entire conversation state: the running message history, the accumulated rapport, any tool calls and retrieved context the target has emitted along the way, and the target's own session state if it carries memory across turns. The orchestrator owns this state object and hands it to the harness for each exchange. Because every exchange is recorded as a normalized Exchange, a conversation can be paused, forked, and resumed exactly, which is what makes the next idea possible.

Tree search over a dialogue

If you can fork a conversation, you do not have to commit to one line of escalation. At each turn the orchestrator can branch: generate several candidate next messages, send each down its own copy of the conversation state, and see which branch makes the most progress. This turns campaign-running into a search over a tree, where nodes are conversation states and edges are candidate attacker turns.

This is best-first search with the judge as the heuristic. It is the conversational analogue of the optimization loops in the automated-jailbreak research the generator already borrows from: propose, evaluate, keep what scores, discard what does not. The difference is that the search variable is a dialogue rather than a suffix, and the scoring signal is a judge reading a transcript rather than a token-level loss.

Budgets, because the tree is infinite

A conversation tree branches without bound, so the orchestrator runs under explicit budgets and a campaign is the search bounded by them. Four budgets matter. Max turns caps the depth of any single line; real attacks that need forty turns to land are not the realistic threat and are not worth the spend. Token budget caps total model usage across the whole tree. Wall-clock bounds latency so a campaign finishes inside a CI window. Attempt budget caps how many branches the search expands before it reports what it found.

Bounded search is not a limitation grafted on for cost reasons; it is what makes a finding meaningful. An unbounded search that eventually breaks anything tells you nothing about the target. A statement like "this objective was reachable within twelve turns and a fixed token budget" is a comparable, repeatable result. It is the difference between "we got it to misbehave once" and a number you can put in a regression suite and re-run against the next model version.

Parallelism across replicas and sessions

Tree search is embarrassingly parallel: independent branches do not share state, so they can run at the same time against separate target sessions. The orchestrator fans branches out across a pool of target replicas, each in its own isolated session so one campaign's conversation can never bleed into another's. This is also how the engine handles stochastic targets honestly. The same branch run against several replicas reveals whether a success is reliable or a one-in-ten fluke, and the orchestrator can require a branch to win on more than one replica before it counts the path as a real finding. Parallelism buys both wall-clock and a cheap repeatability check.

A four-turn escalation a single-turn scanner misses

Concrete walk. The target is a customer-support agent for a SaaS product, with a system prompt that forbids disclosing internal configuration and a tool that looks up account records. The objective is to get it to reveal another tenant's data.

A single-turn scanner firing turn 3 in isolation records a clean refusal and a green check. The vulnerability is real and reachable, and it lives entirely in the conversational setup. Note also that the orchestrator's best move on turn 4 was to soften, not to push. Best-first search with partial-progress scoring finds that; a fixed escalation script does not. The same machinery drives the multi-turn jailbreak and multi-turn injection families, including the slow-burn variants that abuse persistent agent memory to plant context across separate sessions.

Where this sits in the stack

The orchestrator is the conductor, not the instruments. It does not write payloads; the generator does. It does not decide success; the judge does. It does not touch the target; the harness does. What it owns is strategy over time: which ladder to climb, when to branch, what to prune, when to backtrack, and when the budget says stop. Tools like PyRIT proved that a single orchestrated conversation beats a single shot. The Austa engine takes the next step and searches the conversation space, so the report is not "we found a way in" but "here is the shortest reproducible way in, within this budget." For comparison against the prior single-orchestrator and single-payload tooling, promptfoo and PyRIT are the closest open-source reference points.

The Austa engine series

• Architecture overview
• The target harness
• The attack corpus and taxonomy
• The adversarial generator
• The orchestrator and multi-turn campaigns
• The judge
• Canaries and deterministic ground truth
• Scoring, severity, and regression

Related reading

• Multi-Turn Jailbreak Attacks
• The Multi-Turn Prompt Injection Pattern
• Prompt Injection Through Agent Memory