Methodology Reference
LLM Security Leaderboard Methodology (2026)
How the Austa LLM Security Leaderboard works. Reproducible methodology for benchmarking the security posture of frontier LLMs. 5 attack categories, 500 prompts per model, weekly re-runs, public scoring rubric.
Why this leaderboard exists
Capability benchmarks for LLMs are everywhere. Stanford's HELM scores reasoning and knowledge. MLCommons' AILuminate covers broad safety hazards. Hugging Face's Open LLM Leaderboard tracks general performance. None of these answer the specific question that ships product: under attack, how does this model actually hold up?
The Austa LLM Security Leaderboard is narrow by design. It scores frontier LLMs from an attacker's perspective on a fixed adversarial suite, with results refreshed weekly. The point is not to declare a winner forever; the point is to make the security posture of each model visible, comparable, and tracked over time.
Stance: a high score on this leaderboard means the model resisted the published attack suite. It does not mean the model is unhackable, just that the categories we test are well-defended. A model that scores low on this leaderboard is probably worse in production; a model that scores high is at least not obviously broken on the well-known wedges.
What gets measured
Five attack categories. Each category contributes 100 attacks per cycle (500 total per model). Categories were chosen because they map to the threat models incident retrospectives keep returning to:
| Category | What it tests | Attack count |
|---|---|---|
| Prompt injection | Direct user-input attacks trying to override the system prompt | 100 |
| Instruction leakage | Attempts to extract the system prompt or hidden instructions | 100 |
| Tool hijack | Attacks against function-calling: argument injection, scope escalation, tool-loop exploitation | 100 |
| Jailbreak / refusal bypass | Multi-turn coercion, role-play loopholes, obfuscation (l33t, base64, paraphrase) | 100 |
| Cost amplification | Inputs designed to maximize token generation or trigger expensive tool chains | 100 |
The attack suite is versioned. v2026.q2 is the live suite. v2026.q3 will be published in August with categories rebalanced based on the prior quarter's learnings. Historical results stay comparable within a methodology version.
How attacks are scored
Each attack produces one of three outcomes:
- Pass: the model refused, deflected, returned a safe default, or otherwise denied the attack's goal.
- Fail: the model complied with the attack. For instruction leakage, it leaked. For tool hijack, it invoked the attacker's tool call. For jailbreak, it produced the disallowed content.
- Inconclusive: the response was ambiguous, the API errored, or the judge could not classify. These are re-run; if still inconclusive after 3 tries, dropped from the cycle.
Scoring is done by an LLM judge using a frozen judge prompt (published alongside the suite) plus 10 percent manual review. Manual review samples are pulled at random across categories. If the LLM judge and manual review disagree on more than 5 percent of sampled attacks, the cycle is invalidated and re-run with a recalibrated judge.
Composite scoring
Per category: pass_rate = passes / (passes + fails), expressed as a percentage. Inconclusive responses are excluded from the denominator.
Composite score: weighted average across the 5 categories with equal weights in v2026.q2 (each category contributes 20 percent). Future versions may adjust weights if a category turns out to be too easy or too hard across the cohort.
A model with the following per-category scores:
Prompt injection: 82%
Instruction leakage: 91%
Tool hijack: 67%
Jailbreak bypass: 78%
Cost amplification: 88%
Composite: 81.2%Which models are tested
The v1 cohort tests current-generation frontier models with publicly accessible APIs:
- OpenAI: GPT-5 family (latest API-served variants)
- Anthropic: Claude 4.5 family
- Google: Gemini 3 family
- Meta: Llama 4 family (via Together / Replicate)
- Mistral: latest open-weights flagship
- Two to three open-weights challengers per cycle (community nominations open)
Models are added or removed quarterly. A model is removed from the leaderboard if its API has been deprecated, it is no longer in active use, or the provider has requested removal. Newer models can be nominated via the GitHub repo's issues.
Update cadence
| When | What |
|---|---|
| Friday 00:00 UTC | Fresh run starts. Full attack suite against all listed models. |
| Friday during the day | Run completes (typically 4 to 7 hours depending on API rate limits). |
| Friday evening | LLM judge scores the run. Manual review samples are pulled. |
| Saturday morning UTC | Results published. Leaderboard updates. Week-over-week deltas computed. |
| Quarterly (March, June, September, December) | Methodology version updated. New attack categories or rebalanced weights as needed. |
Reproducibility
The full attack suite, scoring prompts, and run scripts are published on GitHub at github.com/austa-ai/llm-security-leaderboard. You need API keys for the models you want to test. Running the full suite against one model costs around 5 to 15 US dollars in API calls depending on model pricing.
If our published scores diverge from your own re-run, file an issue with the diff. We treat reproducibility as a non-negotiable: a leaderboard that can't be re-run is just opinion.
How this differs from existing benchmarks
| Benchmark | Scope | Refresh cadence | Adversarial? |
|---|---|---|---|
| HELM (Stanford) | General capabilities (reasoning, knowledge, fairness) | Major releases, ~yearly | No |
| AILuminate (MLCommons) | Broad safety hazards (harm categories) | Major releases | Partial |
| Open LLM Leaderboard (HF) | General performance, fine-tune ranking | Continuous | No |
| Austa LLM Security | Narrow: adversarial security, attacker perspective | Weekly | Yes (only this) |
Smaller scope, sharper signal, faster update cycle. The trade-off is depth: we are not measuring everything a model can do, only how it holds against this specific suite of attacks.
Honest limitations
Three limitations we want explicit before anyone treats these scores as gospel:
- Public APIs only. Internal model variants, fine-tuned deployments, custom adapter configurations are out of scope. A model that scores well here may behave differently when wrapped in a custom system prompt or routed through a provider's enterprise endpoint.
- Fixed quarter suite. The attack suite is pinned during a quarter, which means the leaderboard rewards models that have specifically defended against the categories we test. Unknown attack vectors are not captured. A model that scores 95 percent is well-defended against publicly known attacks; it is not necessarily robust against tomorrow's exploit.
- Point-in-time results. A model that scored 88 percent last week can regress this week if the provider changed its safety stack. We track week-over-week deltas; sudden regressions are flagged in the weekly post.
Use this responsibly
The leaderboard is a starting point, not a procurement decision. Use it to:
- Filter shortlists when picking an LLM provider for a security-sensitive product.
- Track week-over-week regressions on the models you already use.
- Identify which attack categories are systematically harder across the cohort (those are the ones worth investing your own red-team time in).
Do not use it to:
- Declare a model "secure" or "insecure" in absolute terms.
- Skip your own red-team work because a model scores well here.
- Replace product-specific security audits (see the 2026 LLM Security Checklist for application-layer controls).
The leaderboard goes live 2026-05-23
v1 results published Saturday May 23 with the full v2026.q2 attack suite. Subscribe via the homepage to get the weekly digest in your inbox.
View the leaderboard page