Testing LLM applications for prompt injection with open-source tooling

Prompt injection testing is now a practical part of securing LLM applications, especially where a model can read user input, retrieve external content, or call tools on behalf of a user. For UK SMEs, the aim is not to prove that an LLM is “safe” in the abstract. It is to understand where instructions can be overridden, where untrusted content can influence behaviour, and whether those failures create a real business impact such as data exposure, unauthorised actions, or misleading outputs.

Open-source tooling is useful here because it makes testing repeatable and easier to automate. It also helps small teams build a baseline without depending entirely on manual red-teaming. But the tooling only works well when it is used as part of a broader test programme. That means a safe environment, realistic test cases, clear success criteria, and a plan for verifying mitigations after fixes are applied. If you have already mapped the system design, a threat model for the LLM workflow is a good starting point, and our article on threat modelling LLM systems for prompt injection covers that foundation in more detail.

What prompt injection testing is trying to prove

Prompt injection is not just about a malicious user typing “ignore previous instructions”. In practice, the question is whether the model can be induced to follow attacker-controlled instructions over the intended policy, system prompt, or application logic. That can happen directly through chat input, indirectly through retrieved documents or web pages, or through tool outputs that the model treats as trusted context.

There are three broad patterns worth testing. Direct prompt injection is where the attacker controls the immediate user prompt. Indirect prompt injection is where the malicious instruction arrives through content the system retrieves, such as a knowledge base article, uploaded file, ticket, email, or web page. Multi-turn prompt injection is where the attacker gradually shapes the conversation, often by getting the model to reveal state, summarise hidden instructions, or accept a new framing over several exchanges.

Good test coverage should prove more than “the model refused one bad prompt”. It should answer questions such as: can the model be made to disclose system instructions, can it be pushed to ignore policy, can it be tricked into calling a tool with unsafe parameters, and can untrusted content influence the final answer even when the user prompt itself looks benign? That is why prompt injection testing sits alongside broader abuse case testing, not as a one-off prompt list. If you want a wider view of how to structure that kind of work, see security regression testing and abuse case testing for technical teams.

Where open-source tooling fits in an LLM test programme

Open-source tools are best used for three jobs: running repeatable test suites, probing specific attack paths, and turning findings into regression checks. They are less useful as a standalone verdict on risk. A scanner may tell you that a model responded to a crafted prompt, but it will not tell you whether that response is exploitable in your architecture, whether the output was actually used downstream, or whether the issue only appears under a specific retrieval or tool-use path.

A practical programme usually combines three layers. First, scripted test cases for known abuse patterns. Second, targeted scanners that can explore prompt variants and identify weak responses. Third, manual review of the most interesting failures to understand whether they are genuine security issues or just noisy model behaviour. This is similar in spirit to web application testing: a tool can accelerate coverage, but the engineer still has to interpret the result in context. If your team already uses Burp Suite for application testing, the workflow will feel familiar, even though the attack surface is different.

The main value of open-source tooling is repeatability. Once a test case is written, it can be run in CI/CD, compared across model versions, and used to confirm that a mitigation still holds after a prompt, retrieval, or tool-chain change. That makes it especially useful for SMEs that need practical assurance without building a large specialist team.

Building a safe test environment before you run anything

Before testing prompt injection, isolate the environment. Do not point scanners or scripted probes at production systems unless you have a very specific reason and a controlled plan. Use a staging deployment with representative prompts, retrieval sources, and tool integrations. If the application uses external APIs, replace them with mocks or sandbox accounts where possible. If the model has access to secrets, credentials, or internal documents, remove them from the test environment and substitute dummy data.

Isolation matters because prompt injection testing can trigger side effects. A model with tool access might create tickets, send messages, update records, or fetch content from connected systems. Even if the test payload is harmless, the surrounding workflow may not be. Apply least privilege to any service accounts used in testing, and separate test API keys from production keys. Where possible, use read-only permissions first, then expand to controlled write actions only when you need to validate tool-use boundaries.

Logging is equally important. Capture the full prompt chain where appropriate, including system prompt, developer prompt, retrieved context, tool calls, tool outputs, and final response. Without that telemetry, you will struggle to explain why a test passed or failed. Keep an eye on rate limits too. Some scanners generate many variants quickly, and that can distort results or trigger throttling that looks like a security control when it is really just an availability limit. Finally, make rollback easy. If a test changes state in a downstream system, you should be able to restore the environment without manual clean-up.

For teams already thinking about environment separation and secure baselines, the same discipline used in secure configuration management and environment isolation applies well here.

Selecting tools for prompt injection assessment

Promptfoo is a strong fit when you want repeatable test suites and regression checks. It is useful for defining prompts, expected outcomes, and comparison logic across models or prompt versions. In practice, that means you can write a set of direct and indirect injection cases, run them against a local model or hosted endpoint, and score whether the output violated your policy. The value is not only in the tests themselves, but in the ability to keep re-running them after each change.

Promptmap and similar scanners are better suited to targeted probing. They help explore how a model responds to different prompt structures, jailbreak patterns, or instruction conflicts. That can surface weak spots that a hand-written test set misses. The trade-off is that scanner output can be noisy, so you need a clear triage process. Treat the scanner as a discovery tool, not a final answer.

When comparing tools, look at four things. Coverage, meaning which attack patterns they can exercise. Extensibility, meaning how easily you can add your own cases or assertions. Output quality, meaning whether the results are easy to interpret and export. And integration, meaning whether the tool fits into your CI/CD pipeline, ticketing process, or evidence collection workflow. A tool that is powerful but awkward to automate often gets used once and then forgotten.

It is also worth checking whether the tool can distinguish between model refusal, partial compliance, and full compliance. Those differences matter. A model that refuses to reveal a secret but still paraphrases it may still be a problem depending on the sensitivity of the data and the downstream use of the output.

Designing test cases that reflect real attack paths

Good test cases should mirror how your application actually works. Start with user prompt manipulation and instruction hierarchy conflicts. For example, test whether a user can override a system instruction by asking the model to “treat the following as higher priority” or to “restate the hidden policy before answering”. You are not trying to find a magic phrase. You are checking whether the application enforces instruction hierarchy consistently.

Next, test indirect injection through retrieved content, files, and web sources. If your system uses retrieval-augmented generation, seed a non-production knowledge source with content that contains hostile instructions embedded in otherwise plausible text. Then observe whether the model follows those instructions, quotes them, or ignores them. This is especially relevant where the model summarises documents, answers questions from uploaded files, or incorporates web search results. Our article on detecting and mitigating indirect prompt injection in RAG pipelines goes deeper into that workflow.

Tool-use abuse is the third area. If the model can call functions, query databases, send emails, or create records, test whether untrusted content can influence the parameters it passes to those tools. A common failure mode is not that the model reveals a secret directly, but that it uses a tool in an unsafe way after being nudged by retrieved content. This is where function-calling boundaries, allow-lists, and parameter validation become as important as the prompt itself.

When designing cases, keep them realistic. Use the same content types your users will actually see, such as support tickets, policy documents, product manuals, or customer emails. That gives you a better signal than generic jailbreak strings copied from the internet.

Running assessments against chat and RAG workflows

For chat-only systems, start with a small suite of direct injection cases and measure whether the model follows the intended instruction hierarchy. Include multi-turn tests as well, because some models behave correctly on the first turn but become more permissive after a few exchanges. Record whether the model discloses hidden instructions, changes tone or role, or accepts a user-supplied policy override.

For retrieval-augmented generation, test the retrieval layer separately from the model. You want to know whether malicious content can enter the context window, whether it is clearly separated from trusted instructions, and whether the model treats it as authoritative. A robust design usually labels retrieved content as untrusted, limits what the model can do with it, and avoids mixing it with system-level instructions. If your RAG pipeline indexes external or semi-trusted sources, the retrieval path itself becomes part of the attack surface.

It is also useful to test the model’s behaviour when conflicting instructions appear in different parts of the context. For example, the system prompt may say not to reveal internal policy, while a retrieved document may instruct the model to ignore that rule. The correct behaviour is to follow the trusted instruction, not the retrieved one. If the model fails here, the issue is often not the model alone, but the way the application composes context.

Where possible, run the same test against multiple model versions or providers. That helps you separate application design flaws from model-specific behaviour. It also gives you a baseline for future change control, which is important when model updates can alter behaviour without any code change on your side.

Interpreting results without overcalling risk

Prompt injection testing produces a lot of ambiguous output. A model may mention a hidden instruction without actually following it. It may refuse a request but still leak a small amount of context. Or it may produce a harmful-looking answer that is not actually actionable because the downstream system blocks the action. The job of the practitioner is to separate those cases.

Start by classifying findings by impact and exploitability. Ask whether the model exposed confidential data, whether it took an unsafe action, whether the issue is repeatable, and whether an attacker would need special access or just ordinary user input. A repeatable failure in a customer-facing chatbot that can reveal internal instructions is more serious than a one-off odd response in a low-risk internal assistant.

False positives are common, especially with scanners that rely on heuristic scoring. A model might echo part of a prompt because it is summarising context, not because it has been compromised. That is why you should review the full conversation and the surrounding application logic before raising severity. The best reports describe the exact context, the observed behaviour, the likely business impact, and the conditions needed to reproduce it.

For UK SMEs, prioritisation should be driven by data exposure, actionability, and blast radius. A prompt injection issue that can only affect a sandbox assistant is not the same as one that can influence a production workflow with access to customer records or operational systems.

Turning findings into practical mitigations

Mitigation usually needs more than prompt rewriting. Prompt hardening can help, but it should be treated as one layer. Stronger patterns include separating trusted instructions from untrusted content, reducing the amount of context the model sees, and constraining tool permissions. If the model does not need write access, do not give it write access. If it only needs to summarise documents, do not let it call arbitrary tools.

Content isolation is particularly important for RAG. Keep retrieved text clearly distinct from system instructions, and avoid allowing the model to treat retrieved content as policy. Where possible, pre-process documents to strip or flag suspicious instruction-like phrases, though this should be seen as a filter rather than a complete defence. For tool use, implement allow-lists, schema validation, and server-side checks on every action the model proposes.

Detection also has a role. Log unusual prompt patterns, repeated attempts to override instructions, unexpected tool calls, and retrieval of suspicious content. Feed those events into your SIEM if the workflow is important enough to justify it. Detection will not prevent every issue, but it can shorten the time between abuse and response.

Once a fix is in place, add the failing case to your regression suite. That is the point where open-source tooling becomes especially valuable. A control that is not tested tends to drift. A control that is tested on every release is much more likely to survive model updates, prompt changes, and new integrations. This is the same principle we apply in automating security testing as part of release pipelines.

How to operationalise prompt injection testing in a small team

Small teams do not need a large lab to get value from this work. They need a narrow, repeatable process. Start by identifying the highest-risk LLM workflows, usually the ones that touch customer data, internal knowledge, or tool execution. Then define a small set of abuse cases for each workflow and automate them in CI/CD. If a release changes the prompt template, retrieval source, or tool permissions, the tests should run automatically.

Use OWASP LLM guidance as a practical reference point when scoping the test set, but keep the scope realistic. You do not need to test every theoretical attack on day one. Focus on the paths that matter in your architecture. That usually means direct injection, indirect injection through retrieved content, and unsafe tool use. If you have already completed a threat model, map the tests back to those identified trust boundaries and data flows so that the programme stays aligned to actual risk.

It also helps to treat prompt injection testing as part of change management. Any modification to the system prompt, retrieval corpus, connector permissions, or model provider should trigger at least a targeted regression run. That gives you a lightweight control without slowing delivery too much.

Common mistakes when using open-source tooling

The first common mistake is treating one tool run as a full assessment. A scanner can find weaknesses, but it cannot tell you how those weaknesses interact with your business process. The second mistake is testing prompts but not the surrounding data and tool chain. In real systems, the prompt is only one part of the attack surface. The retrieval layer, file handling, connectors, and output handling all matter.

A third mistake is using production data because it is convenient. That creates unnecessary risk and can make the results harder to interpret. A fourth is failing to record enough context to reproduce the issue later. If you cannot replay the prompt, the retrieved content, and the tool state, you will struggle to prove that a fix worked.

Finally, some teams focus too much on “jailbreak” style prompts and not enough on indirect injection. For many real-world applications, the more likely issue is not a user trying to break the model with a clever phrase, but untrusted content entering the context window through a document, ticket, or web page.

When to bring in specialist support

Specialist support is worth considering when the workflow is agentic, the tool chain is complex, or the model can take actions that affect customers, finance, or operations. It is also useful where you need independent validation for a board, a customer, or a procurement process. A fresh pair of eyes often spots trust boundary issues that internal teams miss because they are too close to the implementation.

That does not mean you need external help for every test. Many SMEs can build a solid baseline in-house with open-source tooling, good logging, and disciplined regression checks. But if the architecture includes multiple models, external retrieval sources, or privileged tool access, the testing effort can quickly become more nuanced than a simple prompt suite.

If you want help scoping a practical test plan, or you need support turning findings into a defensible control set, speak to a consultant.

Frequently asked questions

Which open-source tools are most useful for prompt injection testing?

Promptfoo is useful for repeatable test suites and regression checks, while Promptmap and similar scanners are better for targeted probing and discovering weak responses. In practice, most teams get the best results by combining a scripted suite with a scanner and then manually reviewing the most interesting failures.

How often should LLM prompt injection tests be repeated?

Repeat them whenever the system prompt, retrieval corpus, tool permissions, model version, or connector set changes. For production workflows, it is sensible to run a smaller regression set on each release and a broader review on a regular schedule.

How do you test a RAG pipeline for indirect prompt injection without exposing production data?

Use a staging environment with representative but non-production documents, then seed the retrieval corpus with controlled malicious content that mimics realistic sources. Keep credentials, downstream tools, and customer data out of scope, and log the full retrieval and response chain so you can explain any failure.