In the previous post, we inspected calls to OpenAI APIs triggered within Langchain and LlamaIndex by using OpenTelemetry auto-instrumentation. The spans shown in Jaeger UI were nice to see, but were missing rich information that is expected from a proper instrumentation approach. In this post, we will explore how to enrich spans with additional information using manual instrumentation.

Manual instrumentation

OpenTelemetry provides means to add additional attributes to spans. The OpenTelemetry standard defines two rules:

1. Keys must be non-null string values
2. Values must be a non-null string, boolean, floating point value, integer, or an array of these values

Additionally, most commonly used fields follow naming conventions and are referred to as semantic attributes.

Note: Beware of adding fields that may contain PII information to span context. Unless you guarantee that all systems processing the telemetry drop stored data after a fixed period of time (e.g. 30 days), you may run into challenges related to privacy regulation, such as GDPR and its ‘Right to be forgotten’.

Adding instrumentation to own code

Instrumenting own code is as simple as shown in the code below. It starts a new span called function_name with an attribute arg with value 42.

Any spans that are added using auto-instrumentation to functions called by function_name will automatically become its child spans.

1
2
3
4
5
6
7

from opentelemetry import trace
(...)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("function_name") as span:
    arg = 42
    span.set_attribute("arg", arg)
    result = function_name(arg)

Alternatively, one can use the provided decorator, which results in simpler code in case it’s not necessary to capture any attributes in the spans.

1
2
3

@tracer.start_as_current_span("foobar")
def foobar(arg):
    result = foo_bar(arg)

Adding instrumentation to Langchain’s LLM Chains

Langchain offers Custom Callback Handlers as means to execute additional functions in well-defined stages of the chains. To collect statistics on the prompts and token usage from the LLM calls, we can add spans in the on_llm_start and on_llm_end calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        with tracer.start_as_current_span("on_llm_start") as span:
            prompts_len += sum([len(prompt) for prompt in prompts])
            span.set_attribute("num_processed_prompts", len(prompts))
            span.set_attribute("prompts_len", prompts_len)

def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
        with tracer.start_as_current_span("on_llm_end") as span:
            # example output: {'completion_tokens': 14, 'prompt_tokens': 71, 'total_tokens': 85}
            token_usage = response.llm_output["token_usage"]
            for k, v in token_usage.items():
                span.set_attribute(k, v)

Adding instrumentation for OpenAI Embeddings in LlamaIndex

LlamaIndex does not provide callback mechanisms for its embeddings functions. Instead, we can to extend the OpenAIEmbedding class, include instrumentation code in the overridden methods, and pass an instance of this class to the relevant methods of the library. In the added spans we collect the text lengths as span attributes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

class InstrumentingOpenAIEmbedding(OpenAIEmbedding):
    def __init__(
        self,
        mode: str = OpenAIEmbeddingMode.TEXT_SEARCH_MODE,
        model: str = OpenAIEmbeddingModelType.TEXT_EMBED_ADA_002,
        deployment_name: Optional[str] = None,
        **kwargs: Any,
    ) -> None:
        """Init params."""
        super().__init__(**kwargs)
        self.mode = OpenAIEmbeddingMode(mode)
        self.model = OpenAIEmbeddingModelType(model)
        self.deployment_name = deployment_name

    def _get_query_embedding(self, query: str) -> List[float]:
        with tracer.start_as_current_span("_get_query_embedding") as span:
            span.set_attribute("query_length", len(query))
            return super()._get_query_embedding(query)

    def _get_text_embedding(self, text: str) -> List[float]:
        with tracer.start_as_current_span("_get_text_embedding") as span:
            span.set_attribute("text_length", len(text))
            return super()._get_text_embedding(text)

    async def _aget_text_embedding(self, text: str) -> List[float]:
        with tracer.start_as_current_span("_aget_text_embedding") as span:
            span.set_attribute("text_length", len(text))
            embeddings = await super()._aget_text_embedding(text)
            return embeddings

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        with tracer.start_as_current_span("_get_text_embeddings") as span:
            span.set_attribute("texts_len", sum([len(txt) for txt in texts]))
            return super()._get_text_embeddings(texts)

    async def _aget_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        with tracer.start_as_current_span("_aget_text_embeddings") as span:
            span.set_attribute("texts_len", sum([len(txt) for txt in texts]))
            embeddings = await super()._aget_text_embeddings(texts)
            return embeddings

The obvious downside of the approach is that the code needs to be kept in sync with the extended base class, which results in increased maintenance effort in case of library upgrades.

Inspecting the spans

Running and using the code mentioned earlier produces two traces. First, the embedding span with the added attribute texts_len:

Next, the embedding traces and on_llm_start and on_llm_end traces with the captured query_length and token usage attributes:

Writing an Instrumentor for OpenAI Embeddings in LlamaIndex

Extending classes can be cumbersome and an unnecessary maintenance overhead. The built-in instrumentation offered by many of the OpenTelemetry instrumentation packages for Python offer inspiration for a different approach of instrumentation using function wrappers.

Following the example of the Redis instrumentation library, we use the convenient wrapt package to write a simple wrapper function for three methods in the OpenAIEmbedding class. The wrapper _traced calculates the length of the passed string(s) depending on the function’s argument type (str or List[str]).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

from wrapt import wrap_function_wrapper

from llama_index.embeddings.openai import OpenAIEmbedding

from opentelemetry import trace
from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
from opentelemetry.instrumentation.utils import unwrap
from opentelemetry.trace import SpanKind, Tracer, get_tracer

def _instrument(
    tracer: Tracer
):
    def _traced(func, instance, args, kwargs):
        with tracer.start_as_current_span(
            "get_embedding", kind=SpanKind.CLIENT
        ) as span:
            if span.is_recording():
                if len(args) > 0 and args[0]:
                    if isinstance(args[0], list):
                        span.set_attribute("text_length", sum(len(e) for e in args[0]))
                    else:
                        span.set_attribute("text_length", len(args[0]))
         
            response = func(*args, **kwargs)
            return response

    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding.get_query_embedding", _traced)
    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding.get_text_embedding", _traced)
    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding._get_text_embeddings", _traced)

class OpenAIEmbeddingInstrumentor(BaseInstrumentor):
    def instrumentation_dependencies(self) -> Collection[str]:
        return ("llama-index ~= 0.4.32",)

    def _instrument(self, **kwargs):
        """Instruments llama-index module"""
        tracer_provider = kwargs.get("tracer_provider")
        tracer = get_tracer(__name__, "custom-tracer-version", tracer_provider)
        _instrument(tracer)

    def _uninstrument(self, **kwargs):
        unwrap(OpenAIEmbedding, "get_query_embedding")
        unwrap(OpenAIEmbedding, "get_text_embedding")
        unwrap(OpenAIEmbedding, "_get_text_embeddings")

To ensure the instrumentor is actually used, it needs to be initialized with OpenAIEmbeddingInstrumentor().instrument() before the first library calls are initiated. The resulting traces generated by the instrumentor code are as follows:

Summary

We explored adding additional context to spans by adding instrumentation in three different ways: (1) manual instrumentation of individual function calls, (2) extending classes to override methods with ones that include tracing code, (3) instrumenting library code using function wrappers. When to use which approach is highly contextual and depends on the use case at hand. Approach 1 is best used for one’s own code, approach 3 for instrumenting libraries, and approach 2 when a high degree of control over instrumentation is required.

It’s important to be careful and not overdo instrumentation and rely on the provided instrumentation packages whenever applicable. When considering adding manual instrumentation, it’s important to balance the benefits of additional detail with the potential complexity it may introduce. Note that in production deployments, tracing data is often sampled to deal with high data volume and keep the tracing cost footprint in check and this affects the accuracy of the collected data.

References

• OpenTelemetry for Python: Adding manual instrumentation to automatic instrumentation
• OpenTelemetry for Python: Tracing
• OpenTelemetry: Trace Semantic Conventions
• OpenTelemetry: What to instrument
• OpenTelemetry: Python instrumentation libraries
• OpenTelemetry: Redis Instrumentation library

The “Hello World” of LLMs is the Q&A Knowledge Base use case where the app creates embeddings for source documents and feeds a vector storage. Next, for each query it calculates the embedding, fetches top N documents from the vector store and uses the question and knowledge snippets to prompt the GPT-3/4 language models. Some vector stores are 3rd party stores (Pinecone, Weaviate), others in-memory (FAISS, GPTSimpleVectorIndex).

With the sheer amount of these and other demos and applications using popular libraries like langchain and llama-index, the question arises how to analyze new demos to quickly understand what APIs are being called and in which order? How frequently and with which latency? Checking code is tedious and provides only half the answers. Proper Telemetry tooling will help us discover the API endpoints called by the applications.

Automatic Instrumentation for Python applications

OpenTelemetry provides Automatic Instrumentation for Python that comes to the rescue here. Using the Python agent that is attached to the application, it dynamically injects bytecode to capture telemetry from popular libraries and frameworks. The Langchain, OpenAI, and LlamaIndex libraries use Python’s requests under the hood, so we’ll need to make sure to use opentelemetry-instrumentation-requests explicitly.

To visualize the telemetry, we need to setup some tools first. There is a convenient demo that ships with a docker-compose to boot up the collector and a few more tools. We ignore the two chatty demo apps that run in the background, though those are helpful to check if the telemetry setup works correctly.

$ git clone git@github.com:open-telemetry/opentelemetry-collector-contrib.git
$ cd opentelemetry-collector-contrib/examples/demo
$ docker-compose up -d     
[+] Running 7/7
 ⠿ Network demo_default                Created      0.0s
 ⠿ Container prometheus                Started      0.6s
 ⠿ Container demo-jaeger-all-in-one-1  Started      0.5s
 ⠿ Container demo-zipkin-all-in-one-1  Started      0.4s
 ⠿ Container demo-otel-collector-1     Started      0.7s
 ⠿ Container demo-demo-server-1        Started      0.9s
 ⠿ Container demo-demo-client-1        Started      1.1s

The collector runs on port 4317, so any other error message than a timeout/connection error, means that the application is running:

$ curl localhost:4317
curl: (1) Received HTTP/0.9 when not allowed

With this setup, in addition to the open telemetry collector, we also get Jaeger UI running under http://0.0.0.0:16686/ which will help in visualizing the calls.

The next step is to run the code with the auto-instrumentation agent. For this we need a few packages to be added as part of the project setup. In the post, I use poetry to manage python projects, but you can install the packages with pip as well. First, we add the instrumentation for the Python requests library and the telemetry exporter.

$ poetry add opentelemetry-instrumentation-requests
$ poetry add opentelemetry-exporter-otlp

Afterwards, we add the agent via:

$ poetry add opentelemetry-distro

and start the application via the agent (see reference) and keep a text logfile:

$ poetry run opentelemetry-instrument --traces_exporter console,otlp \
--metrics_exporter console \
--service_name llm-playground \
--exporter_otlp_endpoint 0.0.0.0:4317 \
python main.py | tee output.log

If the metrics collector on port 4317 is not running correctly, the app will log an error:

WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting traces, retrying in 1s.

In case of SSL handshake issues (or similar ones)

E0423 17:04:25.197068000 6150713344 ssl_transport_security.cc:1420]    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

one can instruct the exporter with an environment variable to ignore SSL errors:

$ export OTEL_EXPORTER_OTLP_INSECURE=true

If this does not help to establish connectivity, try increasing the verbosity of gRPC logging to find the error.

$ export GRPC_VERBOSITY=debug
$ export GRPC_TRACE=http,call_error,connectivity_state

As configured in “traces_exporter”, in addition to the OTLP endpoint, the spans are also written to the console. In the demo I’m running, the code uses gradio to create a simple UI. I was surprised to see four calls made by gradio even if launched without any public sharing: launch(share=False).

$ cat output.log | grep http.url
"http.url": "<https://checkip.amazonaws.com/>",
"http.url": "<https://api.gradio.app/gradio-messaging/en>",
"http.url": "<https://api.gradio.app/pkg-version>",
"http.url": "<http://127.0.0.1:7860/startup-events>",
"http.url": "<http://127.0.0.1:7860/>",
"http.url": "<https://api.gradio.app/gradio-initiated-analytics/>",
"http.url": "<https://api.gradio.app/gradio-launched-analytics/>",
"http.url": "<https://api.gradio.app/gradio-launched-telemetry/>",

According to the docs, there is a parameter analytics_enabled and a GRADIO_ANALYTICS_ENABLED environment variable. The description is really convoluted though “default: None; If None, will use environment variable or default to True”…

Back in the telemetry data, we also find the expected OpenAI requests:

"http.url": "<https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings>",
"http.url": "<https://api.openai.com/v1/completions>",

The collected telemetry is helpful to understand the amount and latency of the requests over time. Let’s filter in Jaeger UI (http://0.0.0.0:16686/) for one of the URLs using the tag http.url=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings:

We see 20 calls in the last hour ranging from 280 to 883 ms. Each call can be expanded to check for a bit more details:

Sadly, the UIs of Jaeger and Zipkin are unexpectedly basic and don’t support wildcard searches, so one still needs to tee the console output to a file to quickly search for all the called APIs.

Summary

For more rich telemetry information with custom metadata, the calls to OpenAI (and others) would need to be instrumented manually. While Langchain comes with Callbacks that provide access to the API call results, at time of writing LlamaIndex does not have a similar mechanism for OpenAI embeddings.

The automatic instrumentation is an easy way to inspect calls made by Python applications. The early-stage libraries built in the LLM context tend to have lots of defaults that magically make calls to OpenAI or call lots of 3rd party APIs and externalize its vector data. Even supposedly open-source applications for self-hosting proxies to OpenAI, still rely on other 3rd party services. OpenTelemetry provides an easy way to verify external calls in a sandbox environment.

If you’re interested in getting more out of OpenTelemetry, check out the follow-up post, that gets into details of manual instrumentation with OpenTelemetry for Langchain and LlamaIndex.

References

• OpenTelemetry Python Automatic Instrumentation
• OpenTelemetry Exporter Configuration Options (environment variables)
• Building AI Chatbot with ChatGPT API and LlamaIndex: Building a Chatbot
• Dependencies section for poetry:

1
2
3
4
5
6
7
8

[tool.poetry.dependencies]
python = "^3.11"
openai = "^0.27.2"
llama-index = "^0.4.32"
gradio = "^3.22.1"
opentelemetry-instrumentation-requests = "^0.38b0"
opentelemetry-distro = "^0.38b0"
opentelemetry-exporter-otlp = "^1.17.0"

Emergency procedures used in incident response are aimed at stabilizing the system in a degraded state. When used properly, they result in faster incident response and become a foundation for further resiliency improvements in your system. In the post we’ll also explore how emergency procedures differ from runbooks.

Runbooks

Imagine you get paged in the middle of the night. The situation you encounter looks familiar to you and you’re sure you or your colleague had dealt with it before. However, as seconds ago your were still in deep sleep, you simply can’t recall what to do. Wouldn’t it be great if you had something to help your memory? This is what runbooks are for.

Typically linked in the description of the alert that paged you, runbooks describe routine procedures aimed at restoring regular service of your application. Runbooks list the exact steps to take. It’s helpful to name a runbook with a short summary of the executed procedure as title (e.g. “scale up application”, “retrigger batch job”). Other types of runbooks document migrations, failovers, DB upgrades to help retain know-how within your teams as this type of work is typically executed rather infrequently. Lastly, there are also runbooks aimed at helping with triaging unknown failure situations. When followed, they help inspect typical metrics of the system across layers and components in search for the culprit of the observed issues. A good example here is the sequence of commands used in Brendan Gregg’s Linux performance analysis in 60 seconds.

Emergency procedures

Let’s consider a different scenario: when paged, you observe that the system is overloaded and users experience increased latencies and error rates. You can’t scale up the system due to your dependencies (or storages) taking too long to scale up. To restore service you need to reduce load by 25% as soon as possible. How will you proceed? Will you disable feature A or B? Will you degrade service for user/country X or Y? Do you even have means to do so?

Enter emergency procedures. Unlike for typical runbooks, the goal of an emergency procedure is to bring the system into a degraded, yet stable state. Such state needs to be acceptable to users and stakeholders while trading off availability over customer experience.

Structure

Emergency procedures have defined trigger conditions and impact, both from the business and operational side. It’s important that they’re agreed with business owners ahead of time and thus do not require active approval during the incident response. Impact can be expressed in customer behavior, description how a feature will be working when the procedure has been carried out, or expressed in the change to the system’s load, for example.

Here an example for a food delivery application:

• title: reduce search radius to 400m
• trigger: increased latency or error rate for search queries
• business impact: as all search queries will be limited to a max. 400m radius, customers will see 10–20% less search results, leading to a drop in conversion rate
• operational impact: load on the datastore load will be reduced by 20% within 2 minutes of activating the feature toggle
• steps: …

It’s important that the on-call team regularly practices the emergency procedures. This will verify the correctness of the to be executed steps and operational implications. Additionally, it ensures that the team (and stakeholders) are familiar with the degraded state of the system, which would be rarely observed otherwise. It’s highly recommended to include stakeholder contacts in the procedures in order to keep them informed about the interventions taken during incident response.

Designing for resilience

Systems need to be explicitly designed for supporting emergency procedures. Be it through runtime toggles that allow controlling certain features (e.g. on/off switches, enabling less expensive processing using cached values), or infrastructure mechanisms (e.g. short-circuiting processing for certain user groups or request types). This also requires annotating incoming requests with sufficient metadata to be able to apply differing treatment per feature, traffic origin, etc.

Here a few example degradations that can be introduced to a system with the system property outlined in curly braces:

• enforce serving data from a cache instead to reduce load on the datastore (data freshness)
• always serve the first page of a result set to reduce load on the datastore (data completeness)
• limit retrieved records to N reducing the working dataset of the DB (data completeness defined by amount, distance, or time)
• switch HD video to SD (degrade quality to save bandwidth) or serve images instead (reduce load by preventing auto-play)
• drop traffic from unauthenticated users (user coverage)
• pause all asynchronous batch jobs (feature completeness, data freshness)
• process only critical requests (feature degradation, data completeness)

Automation

It’s certainly advisable to automate frequently used emergency procedures over time by building them into the system as part of your resiliency patterns (fallbacks, retries on error with adjusted input, etc.). Manual execution of the emergency procedure ensures that a human assesses the situation before proceeding with the procedure, which helps harden the defined preconditions. A few manual executions enable you to evaluate if automation is really of value when compared with than other product features planned. It’s important to factor in on-call health into the prioritization. At times of unexpectedly high growth when system availability is a concern, automation is just necessary to cope with overload scenarios efficiently.

Conclusion

Defining emergency procedures requires taking a different view on your system — one where some features are explicitly switched into a degraded mode, thus enabling the overall system to get healthy. The thought exercise of imagining such a degraded, yet usable state is time well spent and highly recommended throughout the design or production readiness stage for your applications. By building in the necessary failure handling mechanisms into the software, or designing it in a way that naturally accommodates the failure states, incident mitigation becomes simpler and less tedious. Having the procedures at hand, you will thank yourself next time you’re on-call in the middle of the night (or day).

Operating software in production offers great insights into software quality. Learning from production incidents is key to improving existing software and learning how to design reliable applications. The Production Readiness Review is an established practice in Site Reliability Engineering aiming at applying past post-incident experience and findings into the software development process. This post provides an overview of a few common themes and pitfalls I’ve experienced being surfaced during production readiness and post-incident reviews.

Using defaults is just asking for trouble

Every framework, http client library or server, connection pool, database, and operating system assume defaults for configuration settings. Aside from risking publicly exposing sensitive data, default settings are often impacting the application’s performance and reliability. There are important settings that must be revisited before deploying an application into a production environment.

Overall, the most commonly missed defaults that impact reliability are timeouts: http client timeouts (connection, read timeout), DNS cache timeouts, and database connection pool and statement timeouts. Framework authors bear great responsibility when setting default values, but often fail to make the developer’s life easy. For example, Java’s DNS cache can be infinite, Apache HttpClient’s v4 RequestConfig.Builder uses “-1” as default timeouts with its documentation stating “A negative value is interpreted as undefined (system default)” and the newer v5 uses 3 minutes instead, whereas .NET’s default timeout is 100 seconds. Timeouts need to be set with care, so it’s important to understand the meaning of the different configuration possibilities. For a great overview on the http request lifecycle and timeouts, check out guide to Go net/http timeouts.

Misconfigured reliability patterns

Retries

Retrying a failed http request is one of the simplest reliability patterns to implement. When done right, retries have an exponentially increasing wait time between attempts and leverage jitter to prevent retry storms leading to the thundering herd problem. However, setting the timeouts to lower values than those that the dependency uses internally, will lead to subsequent retries that pile up and overload the dependency with work that is useless (given that the client won’t wait for the computation result it initially requested). Timeout values must therefore be carefully aligned with service providers, ideally based on their SLOs.

Circuit breakers

Circuit breakers enable the application to fail early in case of an overloaded or faulty dependency and serve a degraded experience via fallbacks. Dropping requests that would fail to be processed before its clients time out is also helping the dependency to recover from failure due to the load reduction. The additional time can be used by the service provider to stabilise the system (e.g. through provisioning of additional instances) and thus regain the ability to serve the required load.

Like every reliability pattern, it needs to be properly configured to function well. Configuring an execution timeout that is too high, will lead to a situation where the circuit breaker never opens, thus keeping the load on the dependency when its performance degrades and making recovery more difficult. Correctly configuring the circuit breaker requires careful planning based on peak load and p99 latencies. The original Hystrix documentation contains detailed guidance for this. Note that the default execution timeout is 1 second. Too bad that official tutorials for frameworks (e.g. Spring) fail to even mention the word “timeout” and do not link to the appropriate documentation.

Circuit breakers require proper isolation

Even if timeouts are configured properly, the circuit breaker may not function as intended keeping the business use case in mind as the degradation will be too broad. Imagine an application A that is fetching a risk score for shipping addresses by calling system B. Because countries may be served by a different risk scoring provider, B will have multiple connectors (one for each provider) and will internally hold a logic defining which provider to choose based on the received address. Service A has a circuit breaker for calls to B. The failure rate of B however, will depend on the failure rate of the connected providers and the distribution of the requests across the providers. In such situation, a failure of a single provider can cause the circuit breaker to open preventing calls to be routed to the remaining providers thus degrading responses for all calls.

In the example below, B receives 300 rps and calls Provider 1 with 200 rps. When Provider 1 becomes unavailable, more than 50% of requests from A to B fail, causing the circuit breaker to open (following the default configuration of the popular Hystrix library) whereas requests routed to other providers would have been processed correctly.

A potential solution for this type of insufficient isolation requires a custom strategy for counting the failure rate per provider or the creation of distinct circuit breakers per country within service A. Both solutions provide different types of isolation. Note that the former requires exposing additional information through the APIs (provider) whereas the latter is purely steered through knowledge that the caller already has based on the processed addresses (country). The right question to ask about this example is — why doesn’t B have circuit breakers to the individual providers? While these would be great to have, it’s often practically impossible, because B is a black box (e.g. a 3rd party service, a monolith that’s hard to adjust, …) and cannot be easily adjusted.

Mixing synchronous and asynchronous workloads

Let’s imagine a service that has a spike in the p99 latencies every x minutes. Sounds familiar? Frequently, it’s a log rotation demon running on the machine where the gzip operation is eating up resources, but more often than not, it is the service itself that causes such spike. It can be a scheduled job that is fetching and processing information, for example periodically refreshing a cache or cleaning up old entries in the DB. If not designed for carefully, such asynchronous execution will impact the p99 latency of the service for synchronous calls.

If you really need to mix such workloads within one application, ensure at least that the http, database connection pools, and thread pools are properly isolated from one another. Otherwise, a long-running async task will impact the synchronous workloads and worst case prevent those from being processed at all.

Missing protection from overload situations

Rate limiting as means to protect from incoming request load

A safe strategy for preventing overload of a service is applying a rate limit to incoming requests. Rate limits are set based on the resources, which impact the scaling ability of the application. These may be driven by its dependencies (e.g. database, 3rd party API) or just costs. Rate limiting can be applied within the application itself or outside, for example in API gateways or ingress controllers. A stricter version of rate limiting is load shedding through request rejection to signal overload situations. To implement load shedding, aside from the technical capabilities to execute this operation, it is helpful to understand per client the business impact of rejecting requests completely as this allows for easy selection of which clients to block first until the service is stabilised.

Rate limits ensure also that clients are forced to negotiate a limit increase with the service provider making scaling needs and capacity planning an explicit conversation. Lastly, rate limiting uncovers and helps dealing with rogue clients of the service. Imagine a service hosting static configuration, which can be cached for a long period of time (e.g. 4 hours). This service should process a request load that is dependent on the number of clients this service has. Load is therefore expected to fluctuate only due to its clients scaling up or down to accommodate incoming load. This service’s load is not expected to follow the traffic patterns of its clients. If it does, it means that such clients are not caching retrieved data correctly and just retrieve it while processing incoming requests.

Protection from unexpected request execution

While rate limits provide external protection, services must implement internal protection as well. As discussed before, services should define SLOs, on which clients will base their timeout configuration. However, the service itself must be designed to honour this SLO across all operations. This is achieved using timeouts on various levels starting from the persistence layer (e.g. via database statement timeouts) through http connection and request timeouts, up to TCP keepalives and timeouts and similar operating system settings. Pure compute operations can leverage execution budgets where a maximum execution time is defined, after which the computation will be aborted. If done correctly, it will prevent overload in cases where the calculation time is unexpectedly influenced by the processed data, for example a regex causing catastrophic backtracking. As past incidents have shown, this is important even for calculation executed in the background, because even though results are not used for the response itself, the calculation consumes CPU cycles and will overload the application anyway.

Lack of control of the application

Having the ability to control the applications inner workings is very helpful during incident response. Though it should not be required during normal operations of the service where one relies on reliability patterns, it comes in extremely handy when mitigating incidents. This can be a capability to pause batch jobs, re-trigger processing, or disabling expensive computation in favor of a degraded, but simpler one. Too often, similar changes require a code change and a deployment of the service instead of a simple switch in a feature flag system or via a management API endpoint. The longer the execution of the CI/CD pipeline in the absence of such controls, the worse the ability to quickly react during an incident.

Insufficient visibility

Aside from monitoring the four golden signals (latency, traffic, errors, saturation), further metrics help to gain quick understanding of production incidents. This starts with collecting metrics on connection pools, the rate of incoming requests per client, outgoing requests per dependency, or duration of batch job execution, etc. It can also include metrics very specific to the service itself — for example, if a service offering a batch API is collecting statistics on the batch size, such data can be used to verify whether clients use its API effectively or break it with unexpectedly big batches. Plotting response times per batch size, will provide insights into the processing times and drive discussions on SLOs for the service. Detailed service instrumentation using standard formats, like OpenTelemetry allows to drill down even further and find causes of incidents across the service call chain as well as identify areas that can be optimised (e.g. through parallelisation of calls) further improving performance and stabilising systems.

Summary

This post provided an overview of common service design flaws and pitfalls that impact reliability. I can only encourage you to check the out the references provided, especially the SRE book, which is a great starting point to dive deep into reliability engineering. Further, if you learned something from production issues, please share your failure stories, so that others can learn from your findings.