Testing systems for high load and denial-of-service conditions

Why high-load and denial-of-service testing belongs in security architecture

Availability is a security property. If a service cannot respond when customers, staff, or partners need it, the business impact can be immediate: lost revenue, missed service levels, support overload, and reputational damage. For UK SMEs, that matters whether the pressure comes from a genuine traffic spike, a software defect, a supplier outage, or a hostile denial-of-service attempt.

Testing systems for high load and denial-of-service conditions helps you find weak points before they become incidents. It is not only about proving that a website can survive a burst of traffic. It is about understanding how the whole service behaves when resources are constrained, dependencies slow down, queues fill, or one component fails and the rest of the stack has to absorb the impact.

In security architecture terms, this sits alongside threat modelling, capacity planning, and resilience design. A good threat model should consider availability threats as well as confidentiality and integrity threats. In practice, that means asking questions such as: what happens if requests double, if a downstream API becomes slow, or if a reverse proxy starts rejecting connections? Those are design questions, not just operational ones.

For technical teams, the value is in turning vague assumptions into measurable thresholds. For example, you may decide that a customer-facing API should maintain a p95 latency below 500 ms at expected peak load, return no more than 1% 5xx responses during a controlled stress test, and recover to normal performance within five minutes after load drops. Those numbers will vary by service, but the principle is the same: define what acceptable looks like before you test.

Define the service boundaries before you test

Before running any load or resilience test, define the service boundary. That means identifying which user journeys matter most, which components are in scope, and which dependencies can become bottlenecks. A login flow, checkout process, file upload, or internal case management screen may all have very different performance profiles and business consequences.

Start by mapping the critical path. For a typical web application, that might include the browser, CDN, WAF, load balancer, application tier, cache, database, message queue, and any third-party APIs. If the service depends on identity, payment, or document processing platforms, include those too. If you do not model them, you may end up testing only the application code while ignoring the real constraint.

It is also worth identifying shared bottlenecks. A single database cluster, NAT gateway, connection pool, or message broker can limit multiple services at once. In SMEs, these shared components are often the first thing to saturate because they are convenient to centralise. That is fine if the design is intentional, but it should be visible and tested.

Success criteria should be specific and measurable. Useful examples include:

• Maximum acceptable response time for key transactions
• Permitted error rate during peak and stress conditions
• Maximum queue depth before work is dropped or delayed
• Recovery time after load returns to normal
• Whether degraded service is acceptable for non-critical features

Do not rely on a single “system up” metric. A service can be technically available while still being unusable because requests are timing out, sessions are failing, or background jobs are lagging by hours.

Build a test model that reflects real-world pressure

Different test types reveal different failure modes. Load testing checks how the system behaves at expected or forecast traffic levels. Stress testing pushes beyond normal capacity to find the breaking point. Spike testing applies sudden bursts to see whether the service can absorb abrupt demand. Soak or endurance testing runs for long periods to expose memory leaks, resource exhaustion, and slow degradation.

For denial-of-service resilience, you usually need a combination of these. A service may cope with steady load but fail under a sudden spike because autoscaling is too slow or connection pools are too small. Another may survive the spike but degrade over time because threads, file handles, or memory are not released cleanly.

Include dependency failure in the model. Real incidents rarely involve just one component behaving badly. A slow database, a throttled third-party API, or a delayed queue consumer can create backpressure that ripples through the stack. Test what happens when a dependency becomes slow rather than fully unavailable, because that is often the more realistic failure mode.

Rate limiting should also be part of the model. If your architecture uses per-user, per-IP, or per-token limits, test how those limits behave under load. Check whether the service returns clear errors, whether legitimate users are blocked too aggressively, and whether the limit enforcement itself becomes a bottleneck.

Queue saturation is another common issue. If requests are accepted faster than they can be processed, queues grow until memory, disk, or worker capacity is exhausted. At that point, the service may fail in a way that is harder to recover from than if it had rejected excess traffic earlier. Good design often prefers controlled rejection over uncontrolled collapse.

What to measure during testing

Good testing depends on good telemetry. You need application, infrastructure, and network signals together, otherwise you will know that something is wrong without knowing where the pressure is building.

At application level, track request rate, response time percentiles, error rates, timeouts, retries, and business transaction success. At infrastructure level, watch CPU, memory, disk I/O, file descriptors, container restarts, autoscaling events, and host-level network throughput. At service level, monitor connection pool usage, thread pool saturation, queue depth, cache hit rate, and database lock contention.

For network and edge controls, include TLS termination metrics, load balancer health, WAF blocks, CDN cache hit ratio, and any rate-limiting counters. If you use mutual TLS or API gateways, check whether certificate validation or gateway policy evaluation becomes a hidden choke point.

Saturation indicators are particularly important. A system often gives warning signs before it fails completely. Examples include:

• Thread pools reaching maximum occupancy
• Connection pools queuing requests
• Memory pressure causing garbage collection pauses
• Queue depth increasing faster than it drains
• Retry storms amplifying load on a failing dependency

Those signals are useful because they tell you where to intervene. If the application is CPU-bound, code optimisation or scaling may help. If the database is the limit, you may need query tuning, caching, or read/write separation. If the edge is the limit, you may need better rate limiting or content delivery controls.

Practical tooling and implementation approaches

There is no single best tool for this work. The right choice depends on your stack, the realism you need, and how much automation you want in your pipeline. Common options include k6, JMeter, Gatling, and Locust. Cloud-native load generators can also be useful when you need distributed traffic from multiple regions or when you want to emulate internet-scale access patterns.

For API-heavy services, k6 is often a good fit because it is scriptable, integrates well with CI pipelines, and produces clear metrics. JMeter remains useful when you need a broad protocol mix or a GUI for quick experimentation. Gatling is strong for high-throughput scenarios and reproducible test scripts. Locust is flexible when Python-based test logic is preferred.

Whatever tool you use, instrument the system before the test. That means enabling metrics, logs, traces, and synthetic checks in advance. If you only add observability after the failure, you will miss the most useful evidence. OpenTelemetry is a practical way to standardise traces and metrics across services, especially where multiple teams or languages are involved.

A simple implementation pattern is:

• Define a baseline workload from production-like traffic
• Run a load test at expected peak plus headroom
• Increase traffic gradually until saturation appears
• Run a spike test to simulate abrupt demand
• Run a soak test for memory and resource leaks
• Repeat after each meaningful architecture change

For containerised systems, test the behaviour of HPA or other autoscaling mechanisms, but do not assume scaling will save you. Autoscaling is only effective if the scale-out time is shorter than the time it takes for the service to degrade. If the queue or connection pool fills in seconds but the new instances take minutes to become useful, the architecture still needs adjustment.

Design controls that improve behaviour under pressure

Testing is only useful if it leads to better design. Several controls can improve resilience under load and during denial-of-service conditions.

Caching reduces repeated work and protects downstream dependencies. It is especially valuable for read-heavy workloads, static assets, and expensive computed responses. Be careful to define cache invalidation and cache expiry clearly, because stale data can create its own business risk.

Backpressure helps a service slow down gracefully when downstream components are under strain. Instead of accepting unlimited work, the system signals that it is busy and asks clients to retry later. This is often better than allowing queues to grow until the service collapses. Circuit breakers serve a similar purpose by stopping repeated calls to a failing dependency and giving it time to recover.

Graceful degradation is another useful pattern. If a non-critical feature cannot be served, the system should still support the core business function. For example, a dashboard may omit live analytics while still allowing order processing. That kind of design keeps the business operating even when some features are unavailable.

At the edge, WAF rules, CDN controls, and rate limiting can reduce noise and absorb some classes of abuse. These controls are not a substitute for application resilience, but they can buy time and reduce pressure on origin systems. If you use them, test the false positive rate as well as the blocking effectiveness, because over-aggressive rules can harm legitimate users during busy periods.

Network and platform design also matter. Segmentation, separate worker pools, and isolation between critical and non-critical workloads can stop one busy service from starving another. In cloud environments, consider whether shared quotas, regional limits, or account-level throttles could create a single point of failure.

How to test denial-of-service resilience safely

Denial-of-service testing should be controlled, authorised, and carefully scoped. The goal is to understand resilience, not to cause avoidable disruption. In most cases, the safest approach is to run tests in a dedicated non-production environment that mirrors production closely enough to be meaningful.

If you must test production, use a tightly controlled window, a limited traffic profile, and explicit rollback criteria. Coordinate with operations, support, hosting providers, and any third-party suppliers whose services may be touched by the test. Make sure everyone knows what normal looks like, what thresholds will trigger a stop, and who has authority to halt the test.

Be careful with shared services. A test against one application can still affect a shared database, identity provider, or outbound network path used by other systems. That is one reason why environment isolation matters. If you cannot isolate properly, reduce the scope of the test and treat the remaining uncertainty as a risk to be managed.

It is also worth distinguishing between internal resilience testing and external abuse simulation. You do not need to reproduce a real attack pattern to learn something useful. Often, a controlled increase in concurrent requests, connection churn, or malformed but safe traffic is enough to reveal the architectural limit. Keep the test defensive and avoid any approach that would cross into exploit instruction territory.

Interpreting results and turning them into architecture changes

The most useful output from testing is not a chart. It is a decision about what to change. Start by separating application defects from infrastructure constraints and third-party limits. A slow endpoint may be caused by inefficient code, but it may also be the result of database contention, DNS latency, or a supplier API that cannot keep up.

Prioritise findings by user impact, exploitability, and implementation effort. A bottleneck that affects login or checkout deserves more attention than one that only affects a low-value internal report. Similarly, a weakness that can be triggered by a small amount of traffic or a single unauthenticated endpoint is more urgent than one that only appears under extreme and unlikely load.

Common remediation actions include query optimisation, cache introduction, connection pool tuning, worker separation, queue redesign, autoscaling adjustments, and edge rate limiting. Sometimes the right answer is to reduce functionality under pressure rather than to keep every feature available. That is a valid architectural choice if it protects the core service.

Document what you learned. Record the test conditions, the observed thresholds, the failure modes, and the assumptions that were proven wrong. This becomes part of your architecture knowledge base and helps future changes avoid repeating the same mistake.

Embedding the test into release and governance processes

High-load and denial-of-service testing should not be a one-off exercise. It works best when it becomes part of release management, architecture review, and ongoing service ownership. If a change affects request paths, caching, authentication, queueing, or scaling, it may also affect resilience under pressure.

Practical governance patterns include:

• Adding load and resilience checks to change approval for high-risk services
• Re-testing after major code, infrastructure, or supplier changes
• Recording assumptions about traffic levels and dependency behaviour
• Setting clear thresholds for when a service needs redesign rather than tuning
• Reviewing test results alongside incident trends and capacity reports

For SMEs, this does not need to be heavy-handed. A lightweight but consistent process is usually enough. The key is to make resilience visible and repeatable so that it is not forgotten until a busy period or an outage exposes the gap.

If you are building or reviewing a service architecture, testing under load is one of the most practical ways to validate whether the design will hold up in the real world. It gives you evidence, not assumptions, and it helps you make proportionate decisions about where to invest in caching, scaling, isolation, and graceful degradation.

If you would like help shaping a practical testing approach for your environment, including how it fits into wider security architecture and risk management, speak to a consultant.