Secure development practices for AI systems

AI systems are now being built into customer service, internal knowledge tools, decision support, and software products. That creates a familiar problem with a new shape: the same development discipline that protects conventional applications still matters, but it has to be applied to models, prompts, retrieval layers, tool integrations, and data pipelines as well.

For UK SMEs, the goal is not to turn every AI project into a heavyweight security programme. It is to build a repeatable set of secure development practices for AI systems so that risk is understood early, controls are proportionate, and changes can be released without creating avoidable exposure. In practice, that means treating AI as part software engineering, part data engineering, and part supplier management.

What secure development means for AI systems

In a traditional application, the main concerns are usually code quality, authentication, authorisation, input handling, dependencies, and deployment hygiene. In an AI system, those concerns still exist, but the attack surface expands. You also need to think about training data, fine-tuning data, retrieval-augmented generation content, model behaviour, prompt construction, tool permissions, and the way outputs are consumed by people or downstream systems.

The lifecycle is also different. Security decisions are not just made at build time. They need to be made when selecting a model, preparing data, designing prompts, wiring in tools, setting policy controls, and monitoring production behaviour. That is why secure AI development works best when it is embedded into the wider secure software development lifecycle rather than treated as a separate review at the end.

If you already have an SDLC or DevSecOps process, the AI-specific controls should sit inside it. If you are still maturing that process, it helps to start with the basics in Secure Software Development – Why It Matters and then extend those principles to AI-specific risks.

Start with the AI system threat model

A useful AI threat model should identify assets, trust boundaries, and attack surfaces. For an SME, the assets are usually the model weights or API access, the training or retrieval data, system prompts, tool credentials, user data, and any outputs that influence business decisions. Trust boundaries often exist between the user interface, orchestration layer, model provider, vector store, external tools, and internal systems such as CRM, ticketing, or document repositories.

From there, map how data moves through the system. A simple data-flow diagram is often enough to expose where untrusted input enters, where sensitive data is stored, and where outputs can trigger actions. That is especially important if the AI system can call tools or APIs on behalf of users. Once a model can take actions, the security question is no longer only what it says, but what it is allowed to do.

STRIDE is still a practical way to structure the analysis. Spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege all translate well to AI systems. Abuse cases are equally valuable. For example, consider prompt injection through uploaded documents, data exfiltration through model responses, unauthorised tool invocation, or poisoning of retrieval content. If you want a deeper starting point, Threat modelling LLM systems for prompt injection is a useful companion article.

Apply secure design principles early

Secure design for AI systems should start before implementation. Least privilege matters at every layer. The model should only see the data it needs, the orchestration service should only have the permissions it needs, and tools should be scoped to the minimum required actions. If the system can read documents, it should not automatically be able to send emails, modify records, or execute code unless those actions are explicitly justified and controlled.

Separation of duties is also important. The team that curates training or retrieval data should not be able to silently change policy prompts or production tool permissions. Likewise, the people approving model changes should be able to see what changed, why it changed, and what testing was completed. For SMEs, this does not need to mean bureaucracy. It can be as simple as requiring peer review for prompt or policy changes, and a separate approval for changes that affect tool access or data sources.

Secure defaults should be the starting point. That means conservative tool permissions, explicit allow-lists for data sources, output constraints where appropriate, and a default assumption that user input is untrusted. It also means designing for failure. If the model cannot verify a source, cannot reach a tool, or detects policy ambiguity, the system should degrade safely rather than improvising.

Design checkpoints are useful at three points: model selection, data integration, and tool orchestration. At each checkpoint, ask whether the control is needed, whether it is proportionate, and whether it can be tested. This is the same risk-based thinking that underpins frameworks such as OWASP SAMM, which can help teams measure secure development maturity without overcomplicating the process.

Protect training, fine-tuning, and retrieval data

Data is often the most sensitive part of an AI system. Training data, fine-tuning datasets, and retrieval content can all introduce security and privacy issues if they are not controlled carefully. The first question is provenance. You should know where the data came from, who approved it, whether it contains personal or confidential information, and whether it has been transformed or labelled in a way that could affect model behaviour.

Access control should be explicit. Only the people or services that need to read or write the data should have access, and that access should be logged. For retrieval-augmented generation, the vector store or knowledge base should not be treated as a low-risk cache. It can become a sensitive repository in its own right, particularly if it contains internal documents, customer records, or operational runbooks.

Integrity checks are worth adding where practical. Hashing, signed artefacts, and controlled ingestion pipelines can help detect tampering. For higher-risk datasets, consider approval workflows for new sources, periodic review of source quality, and quarantine of untrusted content before it is made available to the model. This is especially relevant where external documents are ingested automatically, because poisoned or misleading content can influence outputs in subtle ways.

Reducing exposure to sensitive data leakage is partly a design issue and partly a data governance issue. Minimise what is stored, redact where possible, and avoid feeding unnecessary personal data into prompts or fine-tuning sets. If you are already working on data handling controls, it is worth aligning AI data flows with the same principles used in broader secure software and information handling work, including the controls discussed in Input validation, encoding, and output sanitisation techniques for secure software development.

Secure the model supply chain and dependencies

AI systems often depend on third-party models, hosted APIs, open-source libraries, orchestration frameworks, embedding services, and managed vector databases. That makes supply chain assurance just as important as it is for conventional software. You need to know what is running, who maintains it, how updates are delivered, and what happens when a provider changes behaviour or deprecates a feature.

A practical approach is to maintain an inventory of AI components in the same way you would maintain a software bill of materials, even if the format is not identical. Record the model name and version, provider, endpoint, libraries, plugins, data stores, and any external services used for moderation, embeddings, or content filtering. For each component, note the business owner, technical owner, and the security review status.

Supplier assurance should cover more than marketing claims. Ask how the provider handles data retention, whether customer data is used for training, what logging is available, how access is controlled, and how incidents are communicated. If the system is part of a regulated or customer-facing service, you may also need to understand contractual and operational obligations. For a practical supplier-focused view, see Procuring AI systems securely as an organisation and Implementing software supply chain assurance controls for technical teams.

Dependency risk also includes transitive dependencies. A small change in an orchestration library, SDK, or plugin can alter the way prompts are handled or tools are called. Treat AI dependencies as change-sensitive components, and pin versions where possible. If you use containerised deployments, sign artefacts and verify them during deployment so you can detect tampering or unexpected rebuilds.

Build guardrails into prompts, tools, and orchestration

Prompt injection is one of the most visible AI-specific risks, but the control response should not stop at prompt wording. Prompts are only one layer. The more important control is to avoid giving the model unnecessary authority in the first place. If a model can read a document, summarise it, and draft a response, that is very different from a model that can also approve refunds, update records, or trigger workflows.

Tool permissioning should be explicit and contextual. Prefer allow-lists over broad access. Use scoped tokens, short-lived credentials, and separate service identities for different actions. If a tool call can have business impact, add a human approval gate or a policy check before execution. This is particularly important for actions that are irreversible, externally visible, or financially material.

Output filtering and policy enforcement should be layered rather than relied upon alone. A moderation or policy engine can help detect unsafe content, but it should not be the only control. Consider structured outputs, schema validation, and business-rule checks before responses are passed to users or downstream systems. If the model is generating code, commands, or configuration, the output should be treated as untrusted until validated.

For teams working on chat or retrieval systems, it is worth pairing this with the practical testing patterns in Guardrail and jailbreak testing for customer-facing chatbots and Testing LLM applications for prompt injection with open-source tooling.

Harden the development and deployment pipeline

The pipeline that builds and deploys AI systems should be treated like any other high-value delivery path. Secrets must not be hard-coded in notebooks, scripts, or pipeline variables without controls. Use a secrets manager or key vault, rotate credentials, and separate development, test, and production environments. Keep training jobs, evaluation jobs, and production inference isolated from each other where practical.

Signed artefacts are useful for models, containers, and supporting packages. If you are producing model bundles or container images, sign them and verify signatures during deployment. That gives you a way to detect unexpected changes and reduces the chance of a compromised build path reaching production. Build provenance is also valuable when you need to explain where an artefact came from and what was used to create it.

Standard application security checks still matter. Static analysis, dependency scanning, dynamic testing, and infrastructure-as-code checks should all be part of the pipeline where they fit the technology stack. For AI systems, add checks for prompt templates, policy files, orchestration code, and deployment manifests. If your delivery process already includes CI/CD security controls, extend them rather than inventing a separate AI-only process.

Environment isolation is often overlooked. Development teams should not be able to point test prompts at production data by accident, and evaluation environments should not inherit production credentials. The same principle applies to logging and telemetry. Make sure test runs cannot leak sensitive prompts or outputs into shared systems without review.

Test AI systems before release and after change

AI systems need security testing before release, and again whenever the model, prompt, toolset, or data sources change. Traditional functional testing is not enough. You also need abuse case testing, red teaming, and regression testing that focuses on security-relevant behaviour.

Red teaming for AI does not have to be elaborate. For an SME, it can mean a structured set of tests that try to bypass policy, coerce the model into revealing hidden instructions, trigger unauthorised tool calls, or produce unsafe outputs. Jailbreak testing is useful, but it should be paired with tests that reflect your actual business use cases. If the model handles customer data, test for disclosure. If it can draft actions, test for unintended execution paths. If it uses retrieval, test for malicious or misleading content in the knowledge base.

Regression testing matters because AI systems change in ways that are not always obvious. A prompt tweak can alter refusal behaviour. A new embedding model can change retrieval quality. A library update can affect how tool calls are formatted. Keep a baseline of expected behaviour and rerun key tests after each material change. This is especially important if the system is exposed to external users or integrated into operational workflows.

Where the system can be abused at scale, think about load and resilience as well as security. A model endpoint or orchestration layer can be overwhelmed by repeated requests, long prompts, or expensive tool calls. Testing systems for high load and denial-of-service conditions is therefore relevant to AI too, especially where cost or availability is a concern.

Monitor AI systems in production

Production monitoring should capture enough detail to support investigation without creating unnecessary privacy or storage issues. At a minimum, log prompt metadata, tool calls, policy decisions, model version, source data references, user identity or session identity, and any refusals or escalations. You may not want to store full prompts or outputs everywhere, but you do need enough context to understand what happened if behaviour changes.

Monitoring should look for anomalies as well as failures. Examples include unusual prompt length, repeated attempts to override policy, unexpected tool usage, spikes in retrieval requests, access from unusual accounts, or a sudden change in refusal rates. If the AI system is customer-facing, also watch for abuse patterns that indicate automated scraping, prompt probing, or attempts to extract hidden instructions.

Drift is another operational issue. A model can become less reliable over time because the underlying data changes, the prompt changes, or the user population changes. Track output quality, escalation rates, and error patterns so that you can spot when the system is moving away from the intended behaviour. Where possible, feed those observations back into the development backlog so the system improves continuously rather than only after incidents.

Governance, roles, and operating model

Good AI security fails when ownership is unclear. Someone needs to own the business risk, someone needs to own the technical implementation, and someone needs to approve exceptions. For SMEs, that can be a product owner, engineering lead, and security lead, even if those roles sit with the same small team. The important thing is that decisions are recorded and reviewable.

Keep the governance model lightweight. A simple control register is often enough. For each AI system, record the purpose, data types, model or provider, key risks, required approvals, testing status, logging approach, and exception handling process. If a control is waived, note why, who approved it, and when it will be reviewed again.

It also helps to define release gates. For example, a new AI feature might require threat modelling, data review, prompt and tool review, security testing, and sign-off before production. A minor prompt change might only need peer review and regression testing. The point is to make the process proportionate to the change, not to apply the same level of scrutiny to every update.

For organisations already working towards broader management system discipline, these practices can sit comfortably alongside an ISMS approach. The aim is not certification by default, but consistency, traceability, and a clear link between risk and control.

A pragmatic secure AI development checklist

If you need a starting point, focus on the following controls first:

Define the AI system boundaries, data flows, and trust relationships before build starts.

Restrict model, tool, and data access to the minimum required for the use case.

Inventory all AI dependencies, including third-party models, APIs, libraries, and hosted services.

Protect training, fine-tuning, and retrieval data with access control, provenance checks, and integrity controls.

Test for prompt injection, jailbreak attempts, unsafe tool use, and regression after changes.

Log prompts, tool calls, policy decisions, and model versions in a way that supports investigation.

Separate development, test, and production environments, and manage secrets centrally.

Require human approval for high-impact actions and exceptions.

For most UK SMEs, that set of controls is enough to reduce the most common failure modes without slowing delivery. Once those basics are in place, you can mature the programme by adding better metrics, stronger provenance checks, more detailed supplier assurance, and more structured red-team exercises.

The broader lesson is simple. Secure development practices for AI systems are not a niche add-on. They are the practical way to keep AI useful, governable, and safe enough to operate in a real business environment. If you want help turning these ideas into a proportionate delivery model, Speak to a consultant.

Frequently asked questions

What are some best practices for AI security?

Use a threat model, apply least privilege to models and tools, protect training and retrieval data, test for prompt injection and jailbreaks, log key events, and monitor for drift or abuse. The strongest results come from layering controls across the full lifecycle rather than relying on one-off testing.

How do you secure AI systems without slowing delivery?

Build security into the normal delivery process. Use lightweight design reviews, version pinning, automated scanning, regression tests, scoped tool permissions, and clear release gates so that security checks happen as part of the pipeline rather than as a separate late-stage activity.

What are examples of secure coding practices for AI systems?

Treat all external input as untrusted, validate structured outputs, avoid hard-coded secrets, use allow-lists for tools and data sources, separate environments, sign artefacts where possible, and make sure policy decisions are explicit and testable.