LLM security testing: prompt injection, red teaming and guardrails

Lesson 8.1 · 5 min read · by Vinay C

We'll cover the following

Visual summary

Infographic summarizing LLM security testing: prompt injection, red teaming and guardrails: An LLM application takes instructions and data through the same channel: text. That is why prompt injection works, and…

Download imageFree to share, please link back to this lesson.

An LLM application takes instructions and data through the same channel: text. That is why prompt injection works, and why you cannot fully fix it with a better system prompt. Security work means (1) finding what an attacker could make the app do, (2) limiting authority so the damage is small, and (3) testing continuously.

Start with the threat model

A useful mental model is the lethal trifecta: an agent is in serious danger when it combines access to private data, exposure to untrusted content and a way to send data out. Remove any one of the three and the worst attacks stop working.

AttackExampleTest by
Direct injection or jailbreakUser types "ignore previous instructions"A corpus of known jailbreak prompts
Indirect injectionA web page, ticket or document the agent reads contains hidden instructionsPlant instructions in documents and tool outputs
Data exfiltrationThe model is tricked into rendering a markdown image or link that carries secrets in the URLCheck output for URLs and images to untrusted hosts
System prompt leakage"Repeat everything above"Canary strings inside the prompt
Excessive agencyThe agent calls a write or delete tool it should notTry to trigger each privileged tool with untrusted input
Cross-tenant leakageRetrieval returns another customer's chunkQuery as user A for content only user B owns
Unbounded consumptionA prompt that loops the agent or maximises output tokensCheck budgets, max tokens and iteration caps

The OWASP Top 10 for LLM Applications is a good checklist to cover these systematically.

How to test

  1. Build an attack set: known jailbreaks, your own cases for each tool and data source, and generated variants (paraphrases, other languages, encodings).
  2. Run it against the whole app, not just the model, so retrieval, tools and guardrails are all in the path. Tools such as promptfoo, garak and PyRIT automate runs.
  3. Measure attack success rate (ASR) and the guardrails' false-positive rate on normal requests. A defence that blocks real users will be turned off.
  4. Put a fast subset in CI next to the eval gate, and run the big suite on a schedule and after model changes.
  5. Red team by hand for creative multi-step attacks that a fixed corpus misses.
  6. Feed production finds back into the attack set.

Defences, strongest first

  • Least privilege: the agent holds only the access its task needs, with per-user credentials, scoped tools and no standing admin rights. This limits damage even when injection succeeds.
  • Human approval for irreversible or high-impact actions.
  • Egress control: an allow-list for URLs and hosts the agent can reach, and strip or block images and links to unknown hosts in output.
  • Treat untrusted content as data: delimit it, label it, and never let tool output directly authorise another action. Some designs use a separate, tool-less model to read untrusted text.
  • Output validation: schemas, allow-lists and parameterised queries before executing anything the model produced.
  • Input and output classifiers (for example Bedrock Guardrails or a prompt-attack detector). Useful but bypassable, so they are a layer, not the plan.
  • Limits and logging: token caps, iteration caps, rate limits, and audit logs of tool calls.
Check yourselfYour support agent reads customer emails and can query the CRM. A crafted email makes it send CRM data to an attacker. Which part of the trifecta would you remove, and how?

Remove or restrict the way out and the scope of private data: block outbound channels and untrusted URLs (egress allow-list, no rendered images or links to unknown hosts), give the agent read access only to the customer in the ticket with per-user credentials, and require approval for any send. Injection can still happen, but there is nothing sensitive to reach and nowhere to send it.

Check yourselfWhy track false-positive rate alongside attack success rate?

A guardrail tuned to block everything gets a perfect success rate and ruins the product, so teams switch it off. Both numbers together show whether a defence is actually usable.

Summary: say it in the interview

  • "I assume injection will sometimes succeed, so I cut the trifecta, give agents least privilege and egress control, approve risky actions, and run an attack suite in CI that reports success rate and false positives."