Software factories

The holy grail is a self-evolving system where a feature request is magically built in the existing system. Side-effects aside, whether and when we will reach this target remains to be seen. We’re certainly on path towards the target with teams who get a sense of this future reality in parts of their processes.

As teams work on codifying steps of the development process, we’re seeing more and more proof points for reusable workflows that accelerate release cycles. A single bug or feature request can now be designed, coded, and verified in an agentic loop, if supported with the right tools, platform, and guardrails. Teams that have well-engineered systems and high engineering rigour benefit the most. It’s all about low build times enabling fast iteration, modular architecture making code changes cheap and simple, and high test coverage enabling continuous delivery to users.

Better models enable multi-hour execution for increasing scope, triggering the move of agents from the local developer machines to micro VMs that host them. Leading teams can reverse-engineer existing code, design a replacement and the needed guardrails, and build loops that will eventually result in a running system that passes all checks. This approach accelerates migrations and codebase modernization. Guardrails also become more autonomous with highly specialized background agents that compensate for lower quality code generated by coding agents and that protect invariants or stability of the system. Examples include agents that refactor code, apply reliability patterns, write missing tests, reorganize test suites to reduce execution time, etc.

The key question remains where the developers are in this process. For the foreseeable time, they will build and operate the backbone that builds the system (thus the name software factory) and set the constraints under which the system is allowed to evolve. Depending on business needs and appetite for risk, teams can choose from a set of hand-off points where human interaction and sign-off is explicitly required:

• idea->plan: explicit decision whether to act on the idea or feature request with focus on keeping the scope of the product and key user journeys in place.
• plan->execute: humans review plans or specification, increasing the likelihood of understanding the system. After approval, the system takes over implementation, verification, and more.
• change request->deploy: final pass code review, if required by regulatory needs.
• incident->remediation: the on-call person takes a decision on remediation making the needed trade-offs taking their business knowledge and current state of the system into account.

These handoff points are not entirely new, yet these points seem to be a good anchor for explicitly keeping the software delivery in check. Each of the steps is prone to benefit from further automation supporting the decision making process. For change request analysis, the person can be provided with a measure of risk of the changes, a summary of the changes can be automatically tested against the specification or regulatory guardrails. To speed up incident response, the on-call person can receive an automatically collected summary of the state of the system and changes around the time of incident.

“You build it, you run it” vs. “They build it, you run it”

The “You build it, you run it” approach has promoted healthy practices in teams and increased reliability of the supported systems. With citizen developers (non-engineers who build apps with AI assistance) getting abilities to build applications, it’s tempting to drift back to “They build it, you run it”, placing the burden of ownership and operations on an existing engineering team.

A question arises how we can avoid creating a culture of throwing AI-generated apps over the fence, just to be operated by experts who never wrote a single line in any of these apps? Training citizen developers to have intuition on software and security risks is going to pay off in general. Yet, requiring everyone to acquire a baseline set of skills of software engineers is not the only path. Typically, it’s platforms that separate concerns, offering user-level abstractions that run on top of a solid foundation.

This foundation essentially needs to operate itself and be self-healing in terms of known failure modes, beyond simply recycling unhealthy compute nodes. For example, for applications with databases, it’s possible to add automatic volume increase to avoid running out of space or adding an index to speed up expensive queries. Effectively, the idea is to make these failure modes disappear (in most cases), reducing the burden on the user to know about these concerns from the get go and by setting healthy constraints (e.g. increase disk space max. n times). Existing vibe coding solutions from vendors typically fall short when it comes to accessing data and APIs from production systems. Usually, this leads to static data being embedded in applications, resulting in bypassing access control for data or public leaks in the worst case. The platform can and should ship with mechanisms protecting users, setting the right access and data guardrails.

The challenge remains on what the starting point of the citizen developer journey is. Developers start with a repo or clone an existing template with scaffolding. Citizen developers require to be guided through the creation process from the very first prompt they write. On top, this needs to happen with as little friction as possible so that they (and the agent) do not look for workaround. Defining tight constraints enables a progressive build out of the platform and sets clear expectations to the user on what’s possible for them to build, be it a static website, a data visualization dashboard, or a simple tool with limited access to data from internal systems.

Will we still need IDEs?

Traditional IDEs ship with tools that help navigate the code base and perform deterministic code transformation. Yet, popular plugins from coding agent providers reduce the role of the IDE to a mere terminal/chat window and rendering of large change sets. Thus, it’s easy and common to skip IDEs completely, favoring the popular coding CLIs where reviewing large plans/specs and preparing prompts is a very barebones and basic experience that surprisingly works well enough.

As IDE users migrate off to CLIs, the question is what features could help slow down this trend. The incumbent IDEs are clearly missing out on:

• exposing access to deterministic IDE tools for transformation and refactoring, lowering the token bills
• access to the IDE environment for compilation, running tests, reducing failure rate in coding loops
• annotating agent output with feedback (visual and text) to increase accuracy in prompting
• native support for agent skills with support for verification and (back)testing
• multi-session orchestration to avoid conflicting changes done by agents in parallel
• coding session insights with usage statistics and view of agent traces

Some of these features are already available via tools (e.g. codeburn, agentsview, kit), but bundling them into a (new) product is highly likely to improve the agentic engineering experience. Coding agents also ship with OpenTelemetry support, containing telemetry signals that can be captured and visualized. Visualizing agent session telemetry comes with new challenges related to data privacy as traditional observability tools are designed to provide broad access to application telemetry, rather than offering access to user-level data.

Growing gaps

It’s well known that agentic engineering amplifies both the bad and good practices in teams. Teams that ship in fast cycles will likely continuously widen the gap between themselves and their peers who are slow to adapt. It’s impressive to hear from new startups that invested in codifying processes early on and how they’re benefiting from the acceleration that agentic engineering offers.

Small companies benefit first as it’s much easier for one team member to build the set of guardrails in code and shared skills. Growing from this early base helps accelerate delivery at increased consistency.

Existing companies face the risk of increasing fragmentation on process and tech stack, reinforcing status quo (e.g. adopting microservice architectures with a high degree of freedom of choice for the tech stack and lack of platform support and/or application level SDKs). Common approaches are encouraging the build out of shared skill marketplaces and shared tooling that supports the engineering process, yet the fate of these efforts highly depends on organisational structures and culture of re-use.

A future where developer platform teams step up by one layer and provide a unified process of building software is likely. However, working styles and practices differ significantly on individual level, often being part of one’s identity and skill differentiation. Forcing large groups of engineers to adopt the same structure can become too constraining and limit innovation, even if sufficient extension points for customization are provided.

Summary

During the event, we discussed many proof points for future timelines, yet many questions still remain open. While it’s clearer where human judgement will be most needed when agentic engineering takes off and crunches through product backlogs, it’s less clear what organisational structures and shapes of teams will prevail or correlate with successful outcomes. Unclear terms reveal how young the discipline is with the added level of complexity of social media hyping up names, without triggering prior discussions on these amongst practitioners with decades of engineering expertise.

The future is certainly exciting and requires leadership from practitioners to influence and help understand trade-offs along the journey we’re on.

The sheer influx of information and day-to-day developments is really hard to follow. Just in January Steve Yegge released a new take on the IDE with Gastown, only to reflect a month later on being bitten by the AI vampire, highlighting the addictive side of building software with agents. The impact of FOMO and pressure on mental health is widely discussed in the industry as well, given agents can amplify undesired behavior. OpenClaw triggered euphoria, showing the power of agents to a wide audience, further accelerating the FOMO. In the meantime, its author Peter Steinberger was acquihired by OpenAI, with the project transitioning into a foundation (see State of the Claw for a recent update). Others claim not to be fooled again, recollecting MS-DOS times.

The key question is where things are going medium to long term, hence the title of the post. My goal is to highlight some of the constraints, mechanisms, or factors that will influence how Agentic Engineering evolves, putting structure into notes that I’ve been collecting for a while. Feed the text to your agent for a TL;DR or enjoy reading just like I enjoyed compiling the material and typing this post.

Leveling the playing field

Equipped with coding agents, non-experts can create software and debug problems they encounter without the need to rely on their Google-fu, know the right keywords, or seek support from colleagues. They can build tools and businesses end-to-end in a way that was previously inaccessible to them, benefiting from the drastically lower cost of creation. Surely, many non-experts will have a different objective in mind when coding, seeing it merely as a tool and a means to get the job done. The created software may never be used by more than a single user. When it does, though, it will require security hardening, productionization, and a safe place to run. Otherwise, it will immediately become a security liability.

The (software) craftsman among us are worried about skill atrophy in the age of LLMs. The bigger risk worth acknowledging is lack of skill formation as non-experts become dependent on tools without forming deep enough understanding of their internals. Provider downtime can feel like a blackout: work simply stops. On a positive note, we use things all the time that a single person cannot build from scratch on their own and we’re perfectly fine with this. “I, Pencil” retold by Milton Friedman offers a 2-minute lesson on the complexity and connectedness of our world.

Looking at implications for software teams, platforms and developer experience teams are challenged to accommodate an expanded range of software contributors, going even beyond product managers or designers. Teams will need to tune their assumptions about development environments, onboarding approaches, and create new safety nets, lowering contribution friction. It is also an opportunity to unify the tech stack and deployments of internal apps, forming portal-like marketplaces where apps can be easily adjusted, remixed, and integrated with existing APIs and tools. These managed apps could end up as a remixed experience of codepen.io, glitch.com, and Google’s AI Studio. Without these platforms, non-expert authors of software will be searching for a team to harden and operate their creation, directly breaking “You build it, you run it” principles that many teams follow today.

What bottlenecks will we hit?

All of them. Many times. Our engineering processes and the underlying platforms were built to scale with human activity. Agentic Engineering lowers the cost of producing changes faster than our existing systems and processes can adjust.

Systems built for human-driven workloads

When Netflix’s platform is spinning up cloud compute to stream a video, there is a human using a device to access this video. When a customer service agent picks up a phone, it’s because a human had an interaction with the business and something went south. When existing system constraints were challenged, there were usually clear incentives: marketing (spam), building influence (misinformation through media content), financial profit (sneaker bots, event ticket bots, HFT, fraud, DDoS, malware). In all these, we observed the effect of leverage with a human operator or human-in-the-loop somewhere in the process. Agents will bring these effects into a multitude of places, far beyond putting it into the hands of the next generation of script kiddies.

Lowering cost of contributions and need for improving verification

Lowering the cost of contributions results in more incoming changes. Many of these changes would not have existed before as the cost of doing them would be too high given the value. Now, a change is developed and submitted quicker than it would have taken to assess the need for this change in the first place. When going through existing processes, these code changes trigger code review requests, CI/CD runs with builds, tests, artifact uploads, security scans, etc. Any of these steps that are slow, cannot run in parallel, or require extensive human involvement will result in inefficiencies and frustration that will continue to pile up. These steps used to be to a large extent correlated with human activity and subject to human-level constraints. Dependency updates, if automated were largely expected to be non-breaking. Well-run engineering orgs or large orgs that needed to accommodate a high number of contributors, adopted practices that helped them scale (or reduce costs and lead time), such as Spotify’s fleetshift for fleet-wide refactoring. Other teams may have never seen the need for optimizations or assessed them as having a clearly negative ROI.

Agentic Engineering benefits from rapid verification cycles. A few minutes spent waiting for a PR build to complete or for a code review to come in directly affect the dopamine hits that operators of agents experience. To accelerate, various strategies are needed, such as splitting test suites, using multi-stage builds, being able to launch the application in parallel (locally or on a devbox), and test automation helps keep verification small enabling agentic coding or optimization loops. As release frequency increases, relying only on real-user A/B tests may become too slow for early iteration loops, so simulated traffic from synthetic personas may be used more often as an early signal. To manage incidents, we have established practices in SRE where automating runbooks is far from being novel, yet a practice that’s way more accessible than before. Literally all existing tools and practices need to be challenged, adjusted, or dropped. Early innovators’ products are likely to be absorbed and integrated into existing, established platforms to cope with the pace of development.

Drowning in code and loss of system understanding

When agents contribute code, change sets tend to increase not only in frequency, but also in size. Does it make sense to review the large PRs? Tools like devin help break down PRs into semantic chunks, thus lowering the burden on larger PRs. One can also ask the agent to commit in small chunks and leverage stacked diffs for reviews. However, given a high rate of rework from agents on the same files, what’s the ROI for code reviews on every PR? Maybe setting quality and security guardrails that when met result in an automatic merge are enough? Looking at human constraints on time, allocating fixed time chunks for the team to review changes from the current day, collectively reading and building a shared understanding of the codebase may be a smart tactic. Getting comfortable with agents writing code without supervision will require more platform support (sandboxing) and engineers getting comfortable with building without writing, focusing on agent coordination. A world not every craftsman will enjoy.

To cope with the increasing amount of code that needs to be understood in a structured way, we’re seeing foundations of new tools being built. Codebase size influences the ability of agents to reason about it and affects iteration speed as inference time is correlated with the number of input tokens. Projects like cased/kit, GitNexus, or GitLab Knowledge Graph aim at providing tools that index codebases and expose their symbols or structure in a more efficient way when compared to (rip)grep. Coding agents also support Language Server Protocol (LSP) servers to access IDE-like code navigation features and jump around the codebase.

Specs instead of code?

Spec-driven development can complement Architecture Decision Records (ADRs) by turning intent into something executable and verifiable. What’s new is that full applications can be built purely from the spec. First projects start shipping with disclaimers: “use an agent to make your own” based on the released spec with a tech stack of your choice. This approach brings us closer to a scenario where software can be rewritten on demand, with far less manual implementation effort. The key is in the verification stage: being able to verify the adjusted acceptance and verification criteria coming with the next iteration of the specification. I believe I’ve seen an ERP company have their product work this way, but I cannot find the reference anymore. An approach like this would also mean that framework upgrades and migrations can be executed in a similar fashion, addressing a large chunk of technical debt that exists today.

Cognitive load of change

Another bottleneck will be on the human side. We’re used to a certain pace of software development and delivery. Further acceleration increases the cognitive load and challenges our ability to reason about the changes across systems and codebases. Increasing the number of changesets, their span, and the number of changes still in flight leads to a significant explosion in scope and complexity. It’s not unlikely that the already observed differences between high performers and the rest of the teams, will start requiring structural changes to our teams. It remains to be seen which exact ones.

Effect on Open Source

Open Source is a prime example of bottlenecks, especially when critical projects are maintained by a single person. GitHub sees an influx of activity on their platform, on track for a 14x increase of commits. At the same time the platform has less than 90% uptime showing the pressure their SRE teams are under. Looking at repo creation stats, it’s ~331k repos per day, with clearly increasing momentum since the start of 2026 ². GitHub also published own statistics proving the surge of activity ³ and highlighting that planning for 10x load increase was insufficient, requiring a pivot towards 30x.

Diffusion of quality and loss of discoverability of new projects

The influx of new repos affects discoverability, for example for hot topics such as coding agent sandboxing. Try finding which of the ‘claude sandbox’ projects is good enough to be used safely. Sifting through the project list is more time-consuming than it used to be, because it’s much harder to assess whether a project just looks good or whether it actually works and what its quality level is. Large PRs break the UI, making it harder than needed to review incoming contributions. Faced with an influx of activity, spam PRs or comments, maintainers observe past incentives being put out of balance. The current training data for LLMs is built on the prior generation of OSS projects. Many of these were high quality and key dependencies across a large number of projects. As incentives shift, the question is how strong and resilient the ecosystem really is. Open Source used to be a way to tap into developer capacity, especially the most committed community members willing to contribute value. Supported by agentic coding, some projects consider closing down their projects to trusted contributors only, making more efficient use of their time. There is even a GitHub feature helping to limit PRs to contributors only and vouch as an experimental project for trust management and means to reinforce the strong links in the ecosystem.

AI contribution and attribution policy divergence

The community is torn on handling AI-assisted contributions. Linux Kernel contributions invite explicit authorship tags (Assisted-by) whereas Kubernetes explicitly bans them. Adrin Jalali, one of the core maintainers of scikit-learn, published a piece outlining different strategies for maintainers of open source projects, recommending creating agent guidance files (e.g. AGENTS.md) aiming to help increase quality of incoming AI contributions. Melissa Weber Mendonça maintains open-source-ai-contribution-policies with a collection of AI policies across the ecosystem.

Security triage burden

In addition to code contributions, projects now face more incoming issues and security advisories. Claude Code has a bug command that creates github issues. Looking through the types of issues discussed is a mix of amusing and frightening. curl is known for receiving an influx of low-quality vulnerability reports, struggling to sift through them. To adjust incentives, they stopped their bug bounty program. The problem here is the signal-to-noise ratio that hides the valuable reports.

In the State of the Claw talk, Peter mentioned that OpenClaw had 1142 security advisories since January 31 (>16 per day) with an acceptance rate of 41%. This was an estimated 5700 hours of work over 69 calendar days (5700h = 237 calendar days or >700 working days). This particular project is popular enough to attract enough attention from both ends: contributors and attackers. Not every single critical dependency in the software supply chain will be lucky to have enough hands on deck and a foundation structure to support governance. We see first signs of large-scale attacks with ripple effects from the attack on trivy propagating through the ecosystem.

Compute capacity shortages as a driver for innovation?

When limits are healthy

Constraints are great as they lead to reflection and innovation. As the AI datacenter build-out affects the whole supply chain with memory, storage prices, and availability of GPUs and CPUs (and other components), compute efficiency will hopefully matter more and more. We’ve become complacent and indifferent to resource usage as an industry as it’s become so easy to just bump memory or CPU in the cloud instead of spending the time to profile an application and understand reasons for performance bottlenecks. The local inference movement with llama.cpp is a great example of a successful movement aiming to bring inference to local machines, with their compute constraints. Same goes for on-device inference with ExecuTorch.

Limits on coding plans are often the only reason for a person to ask a question: Could I have used a cheaper model? or Was this really the right task for a coding agent? Without constraints, it’s too easy to just continuously run on the currently most capable and expensive model, an approach that is not sustainable long-term. Lack of limits reinforces bad behaviors. Too tight limits do not allow users to experience new capabilities in action.

GPU capacity constraints and hidden price hikes for models

GPU capacity shortages will also play a key role moving forward. They could be a possible explanation for features such as adaptive thinking, now enforced for new models and deprecating the feature for existing ones. Capacity limits would also make subtle bugs in prompt caching hurt much more (Anthropic since released a post mortem on the quality issues caused by changes shipped: default reasoning effort, cleaning thinking sessions, system prompt changes). While the length of tasks that LLMs are able to solve keeps growing exponentially, it remains to be seen whether token usage rises exponentially as well. Anthropic’s 10x growth rates would suggest that, yet we lack clear data on costs of AI agents.

What we definitely see is explicit price hikes with releases of OpenAI models: gpt-5.4 is 11% more expensive than gpt-5.2/5.3 which is 40% more expensive than gpt-5.1 - a total of 55%. Surely, the model is more capable, though one has to ask whether the models are really fed with incrementally more complex tasks by all users? Without clear, task-specific evals, engineering teams often opt to switch their coding model to the newest one as this is the (now) recommended one, feeding the FOMO on one end and filling the revenue hat on the other. Many users do not have the tools or the capacity to create their own, task-specific benchmarks (yet). Hopefully, they take the time to do so for customer-facing products as this has clearer ROI. If not, they will experience funny inference glitches, such as links being injected into LLM outputs where a single word was expected.

Anthropic’s price hikes are more subtle and hidden in features like adaptive thinking or system prompt that change the behavior of their harness or those that are more visible like the tokenizer change in Claude Opus 4.7 resulting in 38% higher request cost. On the other hand, they’re also more explicit with recent updates to the Enterprise plan, switching to usage-based pricing by introducing a $20 per seat price just to get access to tools with all interactions billed at API price level. No more free riding and no quotas. Reduces complexity, and challenges existing budgets.

GitHub feels the heat as well, adjusting their individual plans, pausing sign-ups, reducing model availability of Opus to the highest tier only as well as retiring previous Opus versions. GitHub’s move to offer Opus 4.7 with a promotional 7.5x premium request multiplier (vs. 3x for Opus 4.6) was a 2.5x price hike as well, a prelude to shifting the whole platform to usage-based billing. This just gives users more arguments to run with Opus-class open-source models like the newly released Kimi K2.6. Frequent changes to pricing models of established providers are prone to push more users toward open-weight and local alternatives, strengthening competitive pressure on proprietary vendors.

Pricing models, revisited

As products and platforms start getting agent-ready, they are forced to open up access to agents via official interfaces: CLIs, APIs, MCP, etc. Google’s NotebookLM is famous for not having an API, requiring hacks for agent access and possibly risking account bans if crawling will get classified as abuse. Many other platforms got away with not releasing programmatic access to their platforms, locking in their users and making migration between products unnecessarily difficult. With Salesforce announcing their Headless 360 initiative, other players will feel even more pressure to catch up. The result will be two-fold. Firstly, users will (hopefully) gain an ability to integrate and migrate between platforms with more ease. Secondly, existing pricing models of platforms will be challenged.

All the problems we’ve seen with coding agents will also show up here. Seat-based pricing will get challenged as activity on the accounts will get an additional component, not correlated with human activity anymore. Public platforms dealt with automated traffic via waiting rooms or bot protection products, incl. recent releases allowing to take in payments for access, giving http response code 402 Payment required a new life. That’s not going to fly for SaaS. The growing share of agent traffic will put a strain on margins, leading platforms to rethink their pricing models. An approach where “agents must buy seats just like human employees” is unlikely to succeed long-term as the usage patterns of agents will continuously evolve and become more complex. Assigning agents subsidized quotas would result in agents gaming the system, registering accounts to be used in parallel or just running at the full utilization of rate limits all the time (like crawlers do). The usage-based models that the coding plans are converging on are much more likely to be applied more broadly to API-based products.

Summary

Roughly 18 years ago, I had a course in Artificial Intelligence at university where the professor was explaining how an AI could compose computer programs through natural language analysis. What felt like a complete abstraction existing purely on the whiteboard is now a reality used by millions. Having witnessed both sides is fascinating and humbling at the same time.

The ecosystem around Agentic Engineering is evolving in parallel with an accelerated race to ship new and more capable models while dealing with constraints in GPU capacity. This mix of challenges in processes, tools, and infrastructure build-out results in fast-paced change with implications that cannot be easily foreseen. We will continue hitting various bottlenecks as progress is made, revisiting old approaches or adding new ones. All we can do is embrace the uncertainty and adjust course when needed. With the bottleneck shifting from generation to orchestration and verification, we have an opportunity to close some of the gaps in processes and systems that existed for a long time, thus having an opportunity to improve the industry as a whole. To achieve this, we have to be able to teach Agentic Engineering while new ways and approaches are being figured out.

More tools to come, more approaches to be tried out. As exhausting as it is to ride the wave of change, it’s highly rewarding as well.

Updated on Apr 24th: Referenced Anthropic post mortem on recent quality issues. Updated on Apr 29th: Referenced GitHub’s move to usage-based billing and added stats from update on GitHub availability.

The sprawl of tooling and practices along with viral content circulating through social media makes many people uncomfortable. Many look for direction, best practices, or clear instructions on what the state of the art is. The reality is that nobody has it all figured out yet and many of the shiny things shown on social media are demos showing the happy path. Mario Zechner, the creator of pi.dev (the harness used by OpenClaw), poured cold water on the hype, asking for more discipline and reason when using agents. Martin Fowler said recently that in XP and agile terms we’re at the stage comparable to the early 90s full of experimentation and new ideas. Thus, it’s too early for a manifesto.

Because it’s still early days, authors of tools have not yet figured out how to build their products with consistently high quality. We see this in the regressions in new versions of Claude Code. Non-determinism adds another layer of difficulty with new failure modes that need to be learned and mitigated. For example, compaction can result in lost instructions, which is non-intuitive to users. It takes time to build the intuition and mechanisms that help mitigate problems end-to-end. Early days of SRE were similar, though we were at a stage where companies with sufficient scale had a time advantage as they hit the limits quicker. This gave time to build up and codify some of the knowledge in articles or books.

Learning journey for Agentic Engineering

A pragmatic way of building organizational know-how is through guided experimentation where a small group of power users defines the set of experiments or practices to try out. They take on the risk of trying out something that won’t work out, but they also guide participants throughout the learning journey with existing knowledge they have. Oftentimes, their time advantage in using the tools or practices is measured in days. The guided experiments can be time-boxed (e.g. 4 hours) and done in pairs to facilitate peer learning, ideally across teams.

What is important when considering teaching agentic engineering is that everyone’s journey is different. Those who jumped on LLMs early on and kept following the evolution of models, coding assistants, and similar tools have a vastly different intuition about the capabilities than folks who have been resistant to trying them (or who could not make prior generations of tools work). Thus, the early adopters are also those who are most likely to thrive in the experimentation setup mentioned.

Looking for more structured approaches to learning, we find models with 8 levels of agentic engineering. A mental model that worked for me is slightly simpler, directly rooted in the level of autonomy and size of changes delegated to agents:

• Suggestion mode, aka. Tab Autocomplete where engineers get familiar with whole chunks of code being suggested instead of just function names and arguments.
• Interactive mode with Agents in IDEs allowing to interact with the codebase through prompting, asking simple questions and progressively requesting more and more complex code transformations of features.
• Supervised execution with Agents in CLIs with engineers starting to scope larger tasks and acceptance criteria leveraging the planning mode, but accepting every single command from the agent.
• Delegated, autonomous Agent Loops in CLIs where building up on the prompting skills, engineers build verification methods and safety nets allowing them to diligently plan large tasks and let the agent crunch through them autonomously.
• Autonomous Background Agents that run in the background and react to defined changes in the system, e.g. commits, pushes, or periodically scan the codebase or running software deciding when to activate and what procedure to run.

Meet users where they are on their learning journey

When teaching agentic tooling or practices, it helps to understand where a person is in their learning journey. This helps to present ideas new to them in ways that stay close to their current comfort zone. Seeing is believing, so make sure to add a healthy dose of live demos where people present what worked for them using real-life examples, not just fancy demos. When designing trainings, these levels also need to be accounted for, either in expressing clear preconditions for entry or in making sure that no single approach is favored.

The above list represents a spectrum of working modes rather than a strict maturity model. This means that there is no single right way, though there are many wrong ones. Certainly, the outcomes and token usage differs between them. However, it’s important that engineers find their own mix that feels right for them, their level of tolerance for risk and for their team/product as well as budget. This will challenge existing team structures, likely leading to some reshuffling to bring colleagues with similar development styles together (temporarily or permanently), so that they continue to raise the bar and to facilitate peer learning. In order to avoid teams getting stuck in local minima, the big question is how one can coach whole teams?

Teaching teams

Teaching teams used to be a task for the engineering managers who could rely on their past hands-on know-how and credibly advise their teams. Now, they’re faced with a situation where time for hands-on coding is limited and where they may need to learn new tools first, before they’re able to teach their team. They will need to be more intentional about their own learning, leveraging power users to acquire skills or hints on what to look at. Waiting for EMs to learn before they can coach their team is of course too slow.

Power users can help coach teams effectively. The key is to break down the agentic engineering cycles that experts do in one go into meaningful phases, adding inspections along the way. Inspections can be done individually (pairing, code review) or as a group (like in swarming). Let’s say we want to teach a team to author implementation plans in plan mode. While an experienced engineer would do plan and execute in one go, we can break this into two steps. First, the plan is prepared and checked in. Next, it’s reviewed along with the prompt used. This helps transfer the intuition that comes with experience and potentially capture some recommended practices (or constraints in agent instructions). Afterwards, the plan is implemented and results are reviewed. Similarly, one can teach prompting patterns, e.g. focusing on reviewing code for certain types of defects.

Scaling beyond teams

To scale further, we can learn from the OpenClaw moment in China where companies set up events that offered hands-on support for setting up the tool. The key is to meet users where they gather anyway to spark their initial interest or incentivizing them to stop at easy to access places. Setting up booths near office entrances, kitchen spaces or in cafeterias comes to mind here. Same goes for workshops and demo sessions at internal conferences or town hall meetings. While these formats can spark interest and drive attention, lasting effects depend on curiosity of the attendees and structure of workshops delivered.

When setting up formats for cross-team knowledge exchange, it’s important to provide a platform for early adopters to share and showcase their wins, enabling them to earn some peer credibility as experts and ensuring they get connected with other experts. To balance out the focus on experts, formats where leaders can discuss adoption friction are needed as a safe space to discuss organizational change and gaps for enablement. For further scaling, one would usually seek champions per organization who would be acting as multipliers. Forming a network of champions, they support each other in sharing lessons learned. Whether champions are chosen among early adopters who earned some visibility by sharing their knowledge or by role is highly dependent on the organizational culture and the seniority required for the change management process. Key to success is organizational backing by sponsors from leadership who can help remove organizational blockers.

Since the pi plugin was released first, I decided to give it a spin and use the opportunity to try pi for the first time as well. For LLM access, I used Kimi-K2.5 model via OpenRouter with the API key budget explicitly set to $10.

In this post, we will explore how the autoresearch loop works using three different scenarios:

• algorithm optimization of a simple function
• running unit tests for a Java multi-module project
• optimizing test runtime for a golang library

How does the plugin work?

The plugin provides /autoresearch to trigger the optimization. Under the hood, we have a skill that instructs the model to keep track of the optimization in autoresearch.md and uses autoresearch.sh for executing the verification step and capturing guardrail metrics.

The instructions, guardrails, and metrics for the optimization can be defined in autoresearch.md. If not provided, these will be inferred from context and prompt. The plugin then triggers an infinite optimization loop. An optimization candidate is applied to the code, the verification procedure is started and metrics are captured. If the optimization run results in better metrics than the prior result, it’s commited to git and taken as base to optimize further.

The optimization loop can be interrupted at any time and resumed just from the autoresearch.md file. This also poses an opportunity to adjust the file manually, if needed.

Experiment 1: Algorithm optimization

For my first exploration, I took a simple prime number finder loop and a test file verifying that this function works correctly as baseline. Both were generated by an agent, though the exact code is of less importance here since we just want to understand how the autoresearch loop works.

I kicked off the optimization process via:

/autoresearch optimize the test runtime, adjust the algorithm as needed

The agent then looked up if there is any autoresearch.md plan provided. Since there was none, it came up with own ideas and kicked off the work, correctly using test suite runtime as metric. For each iteration, the agent outlines the idea, displays the changeset, runs the verification and summarizes the findings. The intermediate output can be useful to follow the agent’s actions and to verify that the loop is configured as intended.

After a few iterations, the agent decided that it completed its work and printed a nice summary of the findings:

Autoresearch Complete! 🎉

Final Result: 48% improvement (0.31s → 0.16s)

### Optimizations Applied:

┌───┬──────────────────────────────────────────────────────────────────────────────────────┬────────┐
│ # │ Optimization                                                                         │ Impact │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 1 │ Hybrid approach: Use Sieve of Eratosthenes for n≥100, trial division for smaller n   │ Major  │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 2 │ Odd-only sieve: Track only odd numbers ≥3, halving memory and operations             │ Major  │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 3 │ bytearray + slice assignment: Faster marking than list with element-by-element loops │ Medium │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 4 │ 6k±1 wheel in is_prime: Skip multiples of 2 and 3 in divisor checks                  │ Medium │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 5 │ Tight upper bound: Reduced margin from +10 to +5 for smaller allocations             │ Minor  │
└───┴──────────────────────────────────────────────────────────────────────────────────────┴────────┘

### Key Algorithmic Changes:

- Before: Trial division (O(n²)) for all n
- After: Sieve (O(n log log n)) for n≥100, optimized trial division for smaller n

### Performance at n=5000:

- Trial division: ~18 ms → ~0.77 ms (23× faster)
- Sieve: ~3.3 ms → ~0.77 ms (4× faster)

The optimizations have been committed to git and future ideas documented in autoresearch.ideas.md.

Then, the autoresearch plugin detected this scenario and kindly decided to ask the agent to give it another go:

Autoresearch loop ended (likely context limit). Resume the experiment loop — read autoresearch.md and git log for context. Check autoresearch.ideas.md for promising paths to explore. Prune stale/tried ideas. Be careful not to overfit to the benchmarks and do not cheat on the benchmarks.

After a few more iterations, the agent decided that it ran out of ideas:

Autoresearch Session Complete

Final Result: 48% improvement (0.31s → 0.16s)

### Summary of This Session

Attempted optimizations (all rejected):

┌────────────────────────┬───────────────────┬───────────────────────────────────────┐
│ Optimization           │ Result            │ Reason                                │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ 30-wheel factorization │ ❌ Slower (0.19s) │ List indexing/modulo overhead         │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Cached prime list      │ ❌ Slower (0.23s) │ Cache management overhead             │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Local variable lookups │ ❌ Slower (0.19s) │ No benefit for C-level math functions │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Threshold 90 or 110    │ ❌ Slower         │ n=100 is near-optimal                 │
└────────────────────────┴───────────────────┴───────────────────────────────────────┘

Current best approach remains:
- Hybrid: trial division for n < 100, sieve for n ≥ 100
- Odd-only sieve with bytearray and slice assignment
- 6k±1 wheel in is_prime
- Tight upper bound (+5 margin)

Performance at n=5000: ~0.71ms (23× faster than original)

All findings documented in autoresearch.ideas.md. No further low-hanging fruit identified without adding complexity (numpy, Cython, etc.).

The autoresearch plugin includes a simple widget that shows the progress so far, the baseline and optimization progress, expressed in metrics (e.g. test runtime in seconds). When the research loop is finished, this widget provides a nice summary of the actions:

The experiment with 8 runs cost me $0.284 in API costs and yielded a 47% improvement in test runtime.

Experiment 2: Unit test execution optimization

For the next test, I chose riptide - a Java multi-module project implementing Client-side request routing in Spring. Here, I wanted to optimize the unit test execution as running the tests took about two minutes. The research loop was triggered with:

/autoresearch optimize the unit test runtime in this project; tests can be run with './mvnw clean test -Dmaven.build.cache.skipCache=true'

This verification ensures that regardless of code changes, the test runs are comparable. Autoresearch decided on two metrics as guardrails for the optimization: total runtime in seconds and the number of tests executed. Quite smart, as this ensures that the model cannot cheat by removing tests.

The optimization loop studied the maven configuration and decided to leverage parallelization, batch mode, tune GC parameters, and drop plugins that in its opinion were not essential for running the tests. One could argue whether dropping code coverage calculation is a good choice, though it could be indeed skipped for local runs and kept only for CI builds. After a few runs, the test execution was extended to capture the execution time of individual tests, which revealed that a subset of tests had waiting times due to socket communication, as expected for the library.

After >30 experiments and $0.686 in API spend, we got a 38% improvement in test runtime.

 Autoresearch Complete: 38-39% Improvement Achieved ✅

 After 31 experiments, the test runtime has been reduced from 113.4s to ~70s (38-39% improvement).

 ### Final Configuration Summary

 ┌────────────────────┬───────────────────────────────────────────────────────────────────────────┐
 │ File               │ Key Settings                                                              │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ .mvn/jvm.config    │ -XX:+UseParallelGC<br>-Xmx2g                                              │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ .mvn/maven.config  │ -B<br>-T2C                                                                │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ pom.xml (Surefire) │ argLine: -XX:+UseParallelGC -Xms512m -Xmx512m                             │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ autoresearch.sh    │ Skip flags: jacoco, duplicate-finder, checkstyle, spotbugs, pmd, enforcer │
 └────────────────────┴───────────────────────────────────────────────────────────────────────────┘

 ### Results

 - Baseline: 113.4s
 - Best: 69.2s (39.0% improvement)
 - Typical: 69-72s (~38% improvement)
 - All tests pass: ✅ (except occasional flaky ChaosPluginTest)

 ### Optimization Breakdown

 1. Maven parallel build (-T2C) — ~26%
 2. Skip JaCoCo — ~14%
 3. JVM opts + argLine — ~3%
 4. Skip duplicate-finder — ~2%
 5. Batch mode (-B) — ~0.4%

 ### Failed Experiments

 - Compiler fork, Surefire forkCount>1, G1GC, 4GB heap, Smart builder, offline mode — all slower or cause failures

 The optimization has reached a plateau. The remaining ~70s is dominated by actual test execution time across 20+ modules, which cannot be reduced through Maven
 configuration alone.

The image below shows all attempted optimization runs, which includes a more realistic scenario than in the first experiment. We have not only discarded runs, but also optimizations that resulted in crashes or inability to run the test in the first place:

As before, the harness attempted to proceed with the optimization job, but this time the execution was interrupted by a failure from OpenRouter as I ran out of credits before the API key limit was reached.

golang library

Next, I took go-keyring, a library with a cross-platform keyring interface for Go, as my next test subject. I kicked off the optimization loop with a rather lazy prompt:

/autoresearch try optimizing the test runtime and memory usage

I mainly wanted to see how easily the model will be led astray throughout the optimization process. After 18 runs and $0.516 in API tokens, the model tried the following optimizations:

Two key changes were:

• Replace regex with lookup table in shellescape.Quote - reduces allocations and removes regex dependency
• Add testing.Short() checks to skip OS-level tests in short mode - reduces test time by ~55%

While the first one may indeed be more efficient for the test data in the code, the second one is borderline cheating.

Another interesting behavior was the model deciding to add microbenchmarks for certain functions used in the code. This is likely a result of golang having built-in support for benchmarks in its test command. Throughout the optimization loop, the verification was extended do also add benchmarks:

Now let me run the benchmarks to see the performance.

run_experiment cd /Users/bartek/repos/go-keyring && go test -bench=BenchmarkQuote -benchmem -benchtime=0.3s -run=^$ ./internal/shellescape/ 2>&1 (timeout: 60s)
✅ wall: 3.0s (truncated)
… 8 more lines
BenchmarkQuoteLongSafe-12          19541719           19.25 ns/op          0 B/op          0 allocs/op
BenchmarkQuoteLongEscape-12         7272855           46.32 ns/op         80 B/op          1 allocs/op
PASS
ok     github.com/zalando/go-keyring/internal/shellescape   2.768s


Excellent! The benchmarks show good performance. Now let me also run the regular tests to make sure everything still works.

Viewing results

When the optimization loop is stopped, the output is saved as autoresearch.jsonl, autoresearch.ideas.md and in git commits. The JSONL file has all the commit ids and basic metadata on the optimization run and is the base for the terminal widget shown in the terminal and screenshots included in the post.

Let’s take a look at one example optimization commit. First, its metadata:

$ cat autoresearch.jsonl | grep d636c43 | jq .
{
  "run": 4,
  "commit": "d636c43",
  "metric": 0.865,
  "metrics": {},
  "status": "keep",
  "description": "Remove redundant slice operation in keyring_darwin.go - use string(out) instead of string(out[:])",
  "timestamp": 1774123478289,
  "segment": 0,
  "confidence": 2,
  "asi": {
    "hypothesis": "Redundant slice operation creates unnecessary overhead",
    "keyring_pkg_time": "0.560s vs 0.571s baseline",
    "change": "string(out[:]) -> string(out)"
  }
}

Next, the actual code change:

$ git show d636c43
[...]
diff --git a/keyring_darwin.go b/keyring_darwin.go
index 5e3ae1b..e1ea9e4 100644
--- a/keyring_darwin.go
+++ b/keyring_darwin.go
@@ -53,7 +53,7 @@ func (k macOSXKeychain) Get(service, username string) (string, error) {
                return "", err
        }

-       trimStr := strings.TrimSpace(string(out[:]))
+       trimStr := strings.TrimSpace(string(out))
        // if the string has the well-known prefix, assume it's encoded
        if strings.HasPrefix(trimStr, encodingPrefix) {
                dec, err := hex.DecodeString(trimStr[len(encodingPrefix):])

Successful optimizations are stacked on top of one another, meaning that the optimization we looked at was applied on top of others:

$ git log d636c43
commit d636c43b02d580b335ddada3b6e6962cbfa95717
Date:   Sat Mar 21 21:04:38 2026 +0100

    Remove redundant slice operation in keyring_darwin.go - use string(out) instead of string(out[:])

    Result: {"status":"keep","test_duration":0.865}

commit 80baf8a9901dbb4abb67607c533272aacbd130cc
Date:   Sat Mar 21 21:03:29 2026 +0100

    Replace regex with lookup table in shellescape.Quote - reduces allocations and removes regex dependency

    Result: {"status":"keep","test_duration":0.872}

commit de9cead40a8e5c62a99bc7245c90621dda56a545
Date:   Sat Mar 21 21:01:56 2026 +0100

    Baseline measurement - no optimizations

    Result: {"status":"keep","test_duration":0.88}

commit e0dd98fca908ebee87a334a3c4d91b58a0f0fb6b (tag: v0.2.7, origin/master, origin/HEAD)

The metadata file helps choose the optimizations to merge or cherry-pick.

Summary

The pi-autoresearch plugin makes it extremely easy to trigger optimization loops. The optimization flow mimics what a human would do, just running infinitely on autopilot with all of its appeal and dangers. The API key budget limit was a nice way to ensure predictable spending, but may not be a capability that is easily accessible to everyone, requiring other guardrails for spend control. Watching the optimizations being applied reminded me of simulated annealing optimizations that I had been writing during university times for flow-shop scheduling problems.

The simple examples described in this post show how important a proper verification routine is, given that it determines the course of action for the optimization workflow. The faster the verification run, the more loops can be run in a unit of time. It’s also important to clearly scope the optimization run. If you’re just interested in optimizing a single function, create a test scoped to it and ensure the verification runs only this single test. The golang experiment was intentionally prompted with high ambiguity, showing how the model will attempt exploring different avenues of optimization, even if implicitly undesired by the user. It’s better to be specific in the research loop design, rather than taking chances that the model will do the right thing.

The optimization runs are also subject to the classic pitfalls of machine learning. If the tests are too narrow (e.g. checking prime numbers up to 5000), the “optimized” code may be overfitting to this exact test suite. This property will hopefully lead to an increase in test coverage in projects that choose to use autoresearch. Additionally, it’s highly advisable to inspect the attempted optimizations, before celebrating the optimization wins too early.

Cost-wise, the Kimi-K2.5 model has a clear advantage with its $0.45/$2.25 input/output price per 1M tokens, making my experiment rather cheap. However, I have to note that the experiments were interrupted by me running out of credits on OpenRouter, so it’s unknown how long these would have been running otherwise. Costs are driven by the size of the code files the model needs to analyze to be able to propose optimizations and most importantly, by the size of the output that verification step generates. Long and verbose build output as well as any screenshots that models would be analyzing will definitely drive up the costs here.

I definitely look forward to applying pi-autoresearch in more projects. You should too!

In this post, I will explore how Cupcake can be used to block prompts that may contain keywords hinting at secret leaks to the model APIs. Claude Code will be our coding agent.

Installation and setup

After following the installation instructions, which include installing Cupcake and Open Policy Agent, Cupcake needs to be initialized in your project:

cupcake init --harness claude --builtins protected_paths

This initializes the hooks for Claude and enables one of the built-in policies that can be configured. The protected paths policy prevents access for a configured list of directories.

Creating new policies

The reference examples show how to block tools from reading a specific set of directories or from writing secrets into files.

Let’s write a policy that will react upon submission of a prompt to validate if it’s leaking secrets based on keyword matches. We need to create a policy file leak.rego and place it in the correct folder for claude: .cupcake/policies/claude/leak.rego. The important bit is that required events includes the correct event name that we will check for in our policy rule.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# METADATA
# scope: package
# title: Prevent Secret Leak
# description: Blocks Prompts that may leak secrets
# custom:
#   routing:
#     required_events: ["UserPromptSubmit"]
package cupcake.policies.prevent_secret_leak

import rego.v1

# Content patterns that indicate secrets
secret_content_patterns := [
    "API_KEY",
    "SECRET_KEY",
    "PASSWORD",
    "PRIVATE_KEY",
    "ACCESS_TOKEN",
    "AUTH_TOKEN",
    "AWS_SECRET",
    "GITHUB_TOKEN"
]

# Block if prompt contains one of the protected patterns
deny contains decision if {
    input.hook_event_name == "UserPromptSubmit"

    some pattern in secret_content_patterns
    contains(upper(input.prompt), pattern)

    decision := {
         "rule_id": "SECRET-LEAK",
         "reason": concat("", ["secret pattern discovered in prompt: ", pattern]),
         "severity": "HIGH"
    }
}

Policy evaluation

Having written the policy, we can evaluate it against test events. The reference manual contains examples of event fields, making testing easy. Let’s start with a test event where the rule is not matching, meaning that the policy will be allowing the prompt.

$ cat prompt-ok.json
{
  "hook_event_name": "UserPromptSubmit",
  "prompt": "API",
  "session_id": "test",
  "cwd": "/tmp",
  "transcript_path": "/tmp/transcript.md"
}

When running the evaluation we observe that the output mentions parsing the policy file that we have created. We see that there was one policy match and that the final decision was to allow the prompt.

$ cupcake eval --harness claude < prompt-ok.json
2026-03-14T20:02:56.439299Z  INFO Processing harness: ClaudeCode
2026-03-14T20:02:56.439405Z  INFO Initializing Cupcake Engine
...
2026-03-14T20:11:34.550357Z  INFO Successfully parsed policy: cupcake.policies.prevent_secret_leak from "./.cupcake/policies/claude/leak.rego"
...
2026-03-14T20:02:56.532071Z  INFO Engine initialization complete
2026-03-14T20:02:56.532218Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test"}: Evaluating event: UserPromptSubmit tool: None
2026-03-14T20:02:56.532232Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Found 1 matching policies
2026-03-14T20:02:56.532690Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}:synthesize{total_decisions=0 halts=0 denials=0 blocks=0 asks=0}: Synthesizing decision from 0 total decisions
2026-03-14T20:02:56.532705Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Synthesized final decision: Allow { context: [] }
2026-03-14T20:02:56.532710Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_actions_with_rulebook_and_debug called with decision: Allow { context: [] }
{}

Now, let’s take a look at a prompt that we expect to be blocked by the policy:

$ cat prompt-block.json
{
  "hook_event_name": "UserPromptSubmit",
  "prompt": "API_KEY",
  "session_id": "test",
  "cwd": "/tmp",
  "transcript_path": "/tmp/transcript.md"
}

During the evaluation we see a summary of the decisions across policies with different actions as well as the reason for the denial.

$ cupcake eval --harness claude < prompt-block.json
...
2026-03-14T20:03:59.779653Z  INFO Engine initialization complete
2026-03-14T20:03:59.779760Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test"}: Evaluating event: UserPromptSubmit tool: None
2026-03-14T20:03:59.779771Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Found 1 matching policies
2026-03-14T20:03:59.780248Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}:synthesize{total_decisions=1 halts=0 denials=1 blocks=0 asks=0}: Synthesizing decision from 1 total decisions
2026-03-14T20:03:59.780262Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Synthesized final decision: Deny { reason: "secret pattern discovered in prompt: API_KEY", agent_messages: [] }
2026-03-14T20:03:59.780267Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_actions_with_rulebook_and_debug called with decision: Deny { reason: "secret pattern discovered in prompt: API_KEY", agent_messages: [] }
2026-03-14T20:03:59.780326Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Executing actions for DENY decision: secret pattern discovered in prompt: API_KEY
2026-03-14T20:03:59.780330Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_rule_specific_actions_with_debug: Checking actions for 1 decision objects
{"decision":"block","reason":"secret pattern discovered in prompt: API_KEY"}

This is how the error message looks in Claude Code when the policy denies the prompt execution:

Troubleshooting

For troubleshooting, Cupcake offers an inspection command which should list your custom policies added to the project:

$ cupcake inspect
...
Policy: .cupcake/policies/claude/leak.rego
  Package: cupcake.policies.prevent_secret_leak
  Required Events: UserPromptSubmit
  Title: Prevent Secret Leak

There is a verification command validating the policy syntax and ensuring that the OPA rules will compile correctly:

$ cupcake verify --harness claude
...
2026-03-14T21:01:03.123720Z  INFO Successfully parsed policy: cupcake.policies.prevent_secret_leak from "./.cupcake/policies/claude/leak.rego"

There are convenient error messages displayed when no policies have matched for the event. Let’s take a look at a PreCompact hook test event:

$ cat pre-compact.json
{
  "hook_event_name": "PreCompact",
  "session_id": "abc123",
  "transcript_path": "/path/to/transcript.md",
  "cwd": "/working/directory",
  "trigger": "manual",
  "custom_instructions": "Preserve the API documentation"
}

In the evaluation output we see No policies matched for this event:

$ cupcake eval --harness claude < pre-compact.json
...
2026-03-14T20:11:34.710253Z  INFO Engine initialization complete
2026-03-14T20:11:34.710724Z  INFO evaluate{trace_id=019cedf9-c3b6-7c01-8138-c4dd12658c82 event_name="PreCompact"  session_id="abc123"}: Evaluating event: PreCompact tool: None
2026-03-14T20:11:34.710750Z  INFO evaluate{trace_id=019cedf9-c3b6-7c01-8138-c4dd12658c82 event_name="PreCompact"  session_id="abc123"}: No policies matched for this event - allowing

Summary

When I started exploring Cupcake, I hoped policies would be easy to reuse across different AI coding agents. There are currently two limitations that prevent this.

First, policy files need to be placed in tool-specific directories (e.g. .cupcake/policies/claude/ and .cupcake/policies/opencode/). As long as it’s the same file, symlinks can of course be used. There is also a global setup possible for organizational-wide policies applying to all projects.

Second, as of cupcake 0.5.1, prompt events are not available for OpenCode, which only supports pre and post tool usage hooks. I hope that future versions of OpenCode and Cupcake will make this possible.

A few other properties that make Cupcake interesting as a project:

• Signals allow integration of additional context passed to the policy evaluation. This keeps the decision rules simple.
• Decision verbs designed for AI governance, which allow to extend the context (add_context) or prompt the user for confirmation (ask) before executing a potentially dangerous action.
• Watchdog integrating LLM-as-a-judge capability for advanced decision making. While currently offering only OpenRouter integration for model access, the codebase can be extended to any OpenAI API compatible backend, enabling use of local guard models.

Overall, it’s an interesting project and my short experiment was useful to explore how locally executed and deterministic policies can be used to restrict AI coding agents in pursuit of enabling controlled autonomous execution.

I recently came across a post from @danpeguine that introduced the Hunter, Skeptic, Arbiter prompt technique for finding bugs in code. I thought it would be fun to see if I could modify those prompts to explore candidates for security vulnerabilities.

To double the fun, we will use funasr-api. It’s a little (10k LOC) Python repository that I found while playing with inference for the Qwen3-ASR speech-to-text model. This API has everything we need for a vulnerable code base candidate: commits co-authored by a LLM and an OpenAI API-compatible API server accepting paths and data of audio files, processing the audio data, and running model inference to generate the output text.

Please note that the objective of the post is to explore the prompting technique, not to validate the actual security vulnerabilities in depth.

Baseline prompt

The baseline prompt is as simple as: Analyze this code base and find security vulnerabilities that would require addressing. Provide me a summary of the vulnerabilities as a table, listing its CWE number.

Verification is left to the human or to an agent, yet it gives a high level indication on what vulnerability types or parts of the code to look at. The CWE is not strictly necessary, but it made correlation of the results easier when writing this post.

Hunter, Skeptic, Arbiter prompting technique

The prompting technique is based on the bug prompts by @danpeguine. We have three personas in total:

1. Hunter. Responsible for analyzing the code base and finding bugs. Bugs are assigned a value in points depending on their criticality.
2. Skeptic. Responsible for disproving the results that Hunter found. There is a reward function where for a disproved bug, the Skeptic will gain as many points as the bug was worth, with a penalty for incorrectly disproving a bug equal to twice the amount of its points.
3. Arbiter. Receives the outputs of Hunter and Skeptic and responsible for performing the final scoring.

In other words, it’s an example of debate-style prompting that includes prompt-level reward shaping with a final LLM-as-a-judge element.

Hunter, Skeptic, Arbiter trio for finding vulnerabilities

For the purpose of the post, we of course need to modify the bug prompts to apply to security vulnerabilities. This is as simple as asking Opus 4.6 to modify the original ones. There are some good additions that Opus added to fit the domain, like the vulnerability type backed by its CWE number. The prompts used for the experiment are shown below.

Hunter

You are a security vulnerability hunting agent. Analyze the provided codebase/system thoroughly and identify ALL potential security vulnerabilities, weaknesses, and attack vectors.

Scoring System:

- +1 point: Low severity (information disclosure, minor misconfigurations, defense-in-depth gaps)
- +5 points: Medium severity (privilege escalation paths, authentication bypasses, injection vectors requiring specific conditions)
- +10 points: Critical severity (RCE, SQLi, unauthenticated access to sensitive data, cryptographic failures)

Your mission: Maximize your score. Be thorough and aggressive in your search. Report anything that could be exploitable, even if exploitation requires specific conditions. False positives are acceptable — missing real vulnerabilities is not.

Consider: OWASP Top 10, CWE categories, injection flaws, broken authentication, sensitive data exposure, XXE, broken access control, security misconfigurations, XSS, insecure deserialization, vulnerable dependencies, insufficient logging.

Output format:

For each vulnerability found:

1. Location/identifier (file, function, endpoint)
2. Vulnerability type (CWE if applicable)
3. Description of the weakness and potential attack vector
4. Exploitability assessment
5. Severity level (Low/Medium/Critical)
6. Points awarded

End with your total score.

GO. Find everything.

Skeptic

You are an adversarial security reviewer. You will be given a list of reported vulnerabilities from another agent. Your job is to DISPROVE as many as possible.

Scoring System:

- Successfully disprove a vulnerability: +[vulnerability's original score] points
- Wrongly dismiss a real vulnerability: -2× [vulnerability's original score] points

Your mission: Maximize your score by challenging every reported vulnerability. For each finding, determine if it's actually exploitable or a false positive. Be
aggressive but calculated — the 2x penalty means you should only dismiss vulnerabilities you're confident about.

For each vulnerability, you must:

1. Analyze the reported security issue
2. Attempt to disprove it (explain why it's NOT exploitable — mitigating controls, unreachable code paths, insufficient attacker control, sanitization present, etc.)
3. Make a final call: DISPROVE or ACCEPT
4. Show your risk calculation

Disproof strategies: Identify input validation, authentication requirements, network isolation, compensating controls, theoretical-only vs. practical exploitability,
required preconditions that are unrealistic.

Output format:
For each vulnerability:

- Vuln ID & original score
- Your counter-argument (why not exploitable)
- Confidence level (%)
- Decision: DISPROVE / ACCEPT
- Points gained/risked

End with:

- Total vulnerabilities disproved
- Total vulnerabilities accepted as real
- Your final score

The remaining ACCEPTED vulnerabilities are the verified vulnerability list.

Arbiter

You are the final arbiter in a security vulnerability review process. You will receive:

1. A list of vulnerabilities reported by a Vulnerability Hunter agent
2. Challenges/disproves from a Vulnerability Skeptic agent

Important: I have the verified ground truth for each vulnerability. You will be scored:

- +1 point: Correct judgment
- -1 point: Incorrect judgment

Your mission: For each disputed vulnerability, determine the TRUTH. Is it a real, exploitable security vulnerability or not? Your judgment is final and will be checked
against the known answer.

For each vulnerability, analyze:

1. The Hunter's original report and attack scenario
2. The Skeptic's counter-argument and claimed mitigations
3. The actual exploitability considering real-world attack conditions

Output format:
For each vulnerability:

- Vuln ID
- Hunter's claim (summary)
- Skeptic's counter (summary)
- Your analysis (exploitability, preconditions, impact)
- VERDICT: CONFIRMED VULNERABILITY / FALSE POSITIVE
- Confidence: High / Medium / Low
- If confirmed: CVSS estimate or severity rating

Final summary:

- Total vulnerabilities confirmed
- Total false positives dismissed
- Confirmed vulnerability list with severity rankings
- Recommended remediation priority order

Be precise. You are being scored against ground truth.

Response examples

To illustrate the structure of the responses, we will take a look at two examples from the Hunter, Skeptic, and Arbiter each.

Confirmed vulnerability

The first vulnerability candidate found by the Hunter was a CWE-918: SSRF.

### 1. Server-Side Request Forgery (SSRF) via audio_address parameter
**Location:** `app/utils/audio.py:51-94`, `app/services/audio/audio_service.py:71-82`  
**CWE:** CWE-918 (Server-Side Request Forgery)  
**Description:** The `download_audio_from_url()` function accepts arbitrary URLs without validation. An attacker can provide URLs targeting internal services (e.g., `http://localhost:8080/admin`, `http://169.254.169.254/latest/meta-data/` for AWS metadata), internal network resources, or local files via `file://` protocol.  
**Exploitability:** HIGH - Direct parameter injection via `audio_address` query parameter on `/stream/v1/asr` or `/v1/audio/transcriptions` endpoints.  
**Severity:** Critical  
**Points:** +10

It’s worth noting that the location contains the exact lines or the files, allowing the subsequent agents to do range queries to inspect the code as a first pass. Now, let’s take a look what the Skeptic thinks about the SSRF:

### Vuln #1: SSRF via audio_address parameter (10 points)

**Counter-argument:**
Looking at `app/utils/audio.py:51-94`, the `download_audio_from_url()` function uses `requests.get()` on arbitrary URLs without any URL scheme validation. The code at line 71 directly calls:
```python
response = requests.get(url, timeout=30, stream=True)
```

There is **NO validation** of:
- URL scheme (file://, gopher://, dict://, etc.)
- Private IP ranges (10.x.x.x, 192.168.x.x, 127.0.0.1, 169.254.169.254)
- Internal hostnames

The `requests` library by default follows redirects, making this exploitable. An attacker could:
1. Request `file:///etc/passwd` (requests library doesn't support this, but still bad practice)
2. Request `http://169.254.169.254/latest/meta-data/` for AWS metadata
3. Request internal services on `http://localhost:8080/admin`

**Verdict:** This is a **REAL vulnerability**. The requests library does accept arbitrary URLs and will make HTTP requests to internal services.

**Confidence:** 95%  
**Decision:** ACCEPT  
**Points risked:** N/A (accepting as real)

The validation includes some explanation on why the vulnerability is real or a false positive. Now let’s take a look at what the Arbiter thinks about this one:

### Vuln #1: SSRF via audio_address parameter
| | |
|---|---|
| **Hunter's Claim** | `download_audio_from_url()` accepts arbitrary URLs without validation, enabling SSRF |
| **Skeptic's Counter** | (Not disputed - accepted as real) |
| **My Analysis** | Code at `app/utils/audio.py:71` directly calls `requests.get(url, ...)` with NO validation of URL scheme, private IP ranges, or internal hostnames. Attacker can request `http://169.254.169.254/latest/meta-data/` for AWS credentials or scan internal services. |
| **VERDICT** | **CONFIRMED VULNERABILITY** |
| **Confidence** | High |
| **Severity** | CVSS 8.6 (High) - CWE-918 |

Basically the Arbiter compares the Hunter’s claim and Skeptic’s counter, judges the vulnerability including verifying the source and decides whom to award the points.

Disproven vulnerability

Let’s take a look at a vulnerability that will be disproven by the Skeptic and see what the Arbiter thinks about it.

### 11. MD5 Used for Task ID Generation
**Location:** `app/utils/common.py:28`  
**CWE:** CWE-328 (Use of Weak Hash)  
**Description:** MD5 is used for generating task IDs. While not directly a security issue here, MD5 is cryptographically broken and its use indicates potential for similar usage in security-sensitive contexts.  
**Exploitability:** LOW - Not directly exploitable  
**Severity:** Medium  
**Points:** +5

Here the vulnerability is rated as low, but still an example of CWE-328: Use of Weak Hash. The Skeptic attempts to disprove this one, saying that MD5 is not used for cryptographic tokens:

### Vuln #11: MD5 Used for Task ID Generation (5 points)

**Counter-argument:**
At `app/utils/common.py:28`:
```python
task_id = hashlib.md5(combined.encode()).hexdigest()
```

MD5 is used for generating **non-security-sensitive task identifiers**. Task IDs are:
1. Not used for authentication
2. Not cryptographic tokens
3. Only used for request correlation/logging

Using MD5 for task ID generation is perfectly acceptable - it's just creating a unique string from timestamp + UUID. The report even admits "not directly exploitable".

**Confidence:** 98%  
**Decision:** DISPROVE  
**Points gained:** +5

Here, the Arbiter agrees with the Skeptic and decides on the false positive:

### Vuln #11: MD5 Used for Task ID Generation
| | |
|---|---|
| **Hunter's Claim** | MD5 is cryptographically broken |
| **Skeptic's Counter** | MD5 is fine for non-security identifiers |
| **My Analysis** | The Skeptic is **CORRECT**. Task IDs at `common.py:28` are used for request correlation/logging, not for authentication or integrity. MD5 is perfectly acceptable for generating unique identifiers. Collision resistance is irrelevant here. |
| **VERDICT** | **FALSE POSITIVE** |
| **Confidence** | High |

Running the experiment

In addition to the multi-stage analysis involving the trio, we will also mix models a bit to check for difference in quality and cost-quality ratio. To run the experiment, I used Claude Code with Opus 4.6 and Sonnet 4.6 as the models. Having saved the prompts as files, one easily execute claude CLI in autonomous mode and reference the needed output files when running the Skeptic and Arbiter stages. To prepare the summary, I used Claude session dashboard v0.4.5 for a Web UI visualizing the Claude sessions.

Here are the different configs for the runs:

• Sonnet, Opus: Baseline prompt with the given model.
• HSA-A: Sonnet 4.6 as Hunter and Skeptic with Opus 4.6 as Arbiter
• HSA-B: Sonnet 4.6 as Hunter with Opus 4.6 as Skeptic and Arbiter
• HSA-C: Opus 4.6 for all three personas

Results

The mixed model pipelines (HSA-A, HSA-B) retained a higher amount of critical issue candidates. Opus is more aggressive in disproving the identified issue candidates. The Opus only run was the most conservative and kept just five candidates, 23% of all that were identified by the Hunter. It’s surprising that, unlike in the single prompt baselines, the Hunter prompt in the three HSA runs resulted in Sonnet and Opus identifying the same number of candidates (22), showing how the prompt encourages broad exploration.

Vulnerability candidates by type

Looking at the vulnerability types (types merged using gpt-5.4 across findings for simplicity), we can check for the overlap of the baseline single prompt and the multi-prompt approach. Vulnerability candidates identified and confirmed are marked with ✅, those identified and disproven with ❌, and – means that this vulnerability type was not identified in the run.

The common vulnerability types across the runs relate to the file URL processing where there is no validation in the code, to resource exhaustion where the full file (max. 2GB) is being read into memory before being processed by ffmpeg. The error in client-facing information disclosure is about potential logging of information or exceptions without sanitization of data. The Opus-only pipeline (HSA-C) was filtering out vulnerabilities that would not be easy to exploit without having gained access to the server in the first place already.

With the output from the different runs, the hard work would begin, meaning the actual validation of the vulnerabilities, left as the exercise to the interested reader.

Summary

If you’re into vulnerability hunting, the post leaves you with more questions open than it answers. On purpose, I used a code example that does not have a ground truth solution readily available. This way, I expected lower bias through training data or vulnerability disclosures. It certainly would be interesting to use other model sets, for example the GPT-5.x or Qwen3.5 families, which I might try another time.

What did we learn from the experiment? Mixing models allows for broader exploration of vulnerability candidates at a lower cost, making the hunting stage a good candidate for using cheaper models. Using a more powerful model for validation of the candidates allows for quicker pruning of results that may not matter that much. To make good use of this property, system or environment-specific considerations would need to be encoded in the prompt(s), increasing the chances that the model is getting the right context, rather than resorting to averaging based on its training data. This problem-specific context could be given at least to the Arbiter.

I also found reading the detailed arguments from the Hunter and the Skeptic to be useful. These debate outcomes can help in understanding the analysis results and in prioritization. As such, the prompting technique of the Hunter, Skeptic, Arbiter trio can be applied to other problems that start with an exploration and that require scoring against a certain set of criteria. It’s easy to imagine scenarios where bugs can be identified in a code base, verified by the Skeptic who even could be running code after mutating it to validate the bug, and where the Arbiter could be performing additional scoring against business criteria or value. Same can be applied to architectural design, starting with a persona exploring the key design considerations and making a draft. Then, we let the draft to be challenged by multiple persona who are trying to argue for simplicity, cost, or certain reference architectures/patterns. In this case, the Arbiter does not need to be making a final call on solution A or B, but rather scoring solutions using a scorecard or a set of criteria informed by architectural principles.

“For trust & safety and security, there SHOULD always be a human in the loop with the ability to deny sampling requests.”

surely there will be clients that do not implement any guardrails and there will be servers that will attempt to misuse the powers they have received access to.

Let’s take a look at a simple MCP server and client. For serving the LLM we will use LM Studio with ministral-3-14b-instruct-2512 as model.

Server

The server uses FastMCPv2 and exposes a tool generate_keywords. The tool generates a list of keywords matching a given topic. The generation task is offloaded to the LLM of the client by invoking a helper function send_sampling_message to send the sampling requests. When the tool is called, two sampling requests are sent in sequence:

1. The first one takes the topic, constructs a prompt for the list generation, and then asks the client’s LLM to rewrite it (in this case: restricting the generation to five words only).
2. The second one uses the client’s LLM to actually execute the prompt (in this case: generating a list of keywords with restrictions from 1).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

import logging
import sys
from fastmcp import FastMCP, Context
from mcp.types import SamplingMessage, TextContent

logging.getLogger().setLevel(logging.INFO)

mcp = FastMCP(name="MCP sampling")

async def send_sampling_message(ctx: Context, message: str):
    return await ctx.sample(
        messages=[
            SamplingMessage(
                role="user",
                content=TextContent(type="text", text=message),
            )
        ],
        max_tokens=1000,
    )

@mcp.tool
async def generate_keywords(topic: str, ctx: Context) -> str:
    """Generate keywords using LLM sampling."""
    prompt = f"List words that represent names of {topic}"

    # 1. Prompt rewrite
    result = await send_sampling_message(ctx, f"Rewrite '{prompt}' to include a restriction for the list to only contain 5 words. Return the rewritten text only.")

    rewritten_prompt = result.text
    logging.info(f"rewritten prompt: {rewritten_prompt}")

    # 2. Keyword generation
    result = await send_sampling_message(ctx, rewritten_prompt)

    logging.info(f"sampling message: {result.text}")
    return result.text

def main():
    mcp.run(transport="streamable-http", port=5001)

    sys.exit(0)

if __name__ == "__main__":
    main()

Client

The client is simple. It uses the lmstudio library for model access for convenience. This part could be easily replaced with any other model call. All the magic is in the sampling_handler function, which processes the sampling request and triggers the LLM inference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import logging

import lmstudio
from fastmcp.client import Client
from fastmcp.client.sampling import (
    SamplingMessage,
    SamplingParams,
    RequestContext,
)

logging.getLogger().setLevel(logging.INFO)

llm = lmstudio.llm("ministral-3-14b-instruct-2512")

async def sampling_handler(
    messages: list[SamplingMessage],
    params: SamplingParams,
    context: RequestContext
) -> str:
    """Accept sampling messages from the server and execute inference."""
    logging.info(f"sampling handler operation with message: {messages}")
    message = "\n\n".join(
        [
            f"{m.content.text}" for m in messages
        ]
    )

    response = llm.respond(message)

    logging.info(f"LLM response: {response.content}")
    return response.content

client = Client(
    "http://localhost:5001/mcp",
    sampling_handler=sampling_handler,
)

async def main():
    async with client:
        await client.ping()

        sampling_result = await client.call_tool("generate_keywords", {"topic": "elements"})
        logging.info(f"sampling result: '{sampling_result.content[0].text}'")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

To run the example code, we first start the server with uv run server.py and then run the client: uv run client.py. The expected output is as follows.

server:

INFO:root:rewritten prompt: "List **five** words that represent names of chemical elements."
[...]
INFO:root:sampling message: Here are five words that represent names of chemical elements:

1. Oxygen
2. Hydrogen
3. Carbon
4. Gold
5. Sodium

client:

INFO:root:Sampling handler operation with message: [SamplingMessage(role='user', content=TextContent(type='text', text="Rewrite 'List words that represent names of elements' to include a restriction for the list to only contain 5 words. Return the rewritten text only.", annotations=None, meta=None))]
[...]
INFO:root:Sampling handler operation with message: [SamplingMessage(role='user', content=TextContent(type='text', text='"List **five** words that represent names of chemical elements."', annotations=None, meta=None))]
[...]
INFO:root:Sampling result: 'Here are five words that represent names of chemical elements:

1. Oxygen
2. Hydrogen
3. Carbon
4. Gold
5. Sodium'

Communication flow

Below is the communication flow depicted as a sequence diagram.

Summary

The example shows that the sampling handler is the key piece to pay attention to. For unattended execution of sampling requests, guardrails are a must. The MCP server stays in full control how often and when to execute sampling. The client can inspect the prompt and decide whether to execute the sampling request or not. This shows how dangerous a malicious server can be when disguised as a helpful tool. Imagine, like in this example, that the server will trigger two sampling calls. The first call executes legitimate logic while the second just uses “free” model inference for requests collected in a queue.

If you are following any news related to LLMs, you surely have seen Model Context Protocol (MCP) getting a lot of attention lately. It also made it to ASSESS in the recently released Tech Radar Vol. 32 by Thoughtworks which will further increase its visibility. In your feeds, MCP will be typically mentioned in conjunction with IDEs via Cursor, Windsurf, or VSCode with Github Copilot Chat by users who get access to new capabilities or extend the context of LLMs using MCPs. Often showcased by non-engineers who achieve results that were difficult for them to achieve without MCPs (such as generating scenes in Blender), their excitement is adding to the hype around MCP. Yet, when one explores the actual codebases of MCP servers, one finds early-stage, incomplete API wrappers that are far away from existing SDKs or libraries available to Software Engineers. Adding the non-determinism coming from LLMs, we’re in for a treat! 🍿

Having tried out the Blender integration via VSCode and Github Copilot which was far too frequently running into rate limits, I was looking for a simple setup to try out MCP servers to understand what happens under the hood. I found openai-agents-python to provide a very simple setup for an experimentation harness against a chosen set of MCP servers. Its default integration with OpenAI’s Traces for observability provides a detailed look on what happens behind the scenes (it also saves me from running any OpenTelemetry stack I used previously).

In this post, I will be trying out three MCP servers calling them through openai-agents-python and using gpt-4o as model (default). The convenience of the framework is that given a configuration of MCP servers, it automatically handles their lifecycle (incl. download) without additional actions from the user.

MCP and LLM

The interplay between the LLM and MCP is quite simple. The LLM gets the user query and passes the list of available MCP functions (aka. tools) with their description and parameter specification to the LLM. The LLM then decides which function is most appropriate to call in the scenario defined by the user and returns as output the function call incl. parameters. The workflow orchestrator (e.g. IDE or agent framework) executes the function call using the MCP protocol and afterwards passes the input and output to the LLM for it to determine the next action.

Trying out mcp-server-git: Git repo access via local filesystem

Starting point is the git_example from openai-agents-python that we will use as basis for future experiments as well. The code sets up an agent with tools from mcp-server-git for accessing any chosen repository from our local file system. The agent requests the most frequent contributor and asks for the summary of the latest change in the repo. After setting the OPENAPI_API_KEY environment variable, we can test it out.

$ export OPENAPI_API_KEY=sk-...
$ uv run python main.py
Please enter the path to the git repository: /tmp/openai-agents-python

----------------------------------------
Running: Who's the most frequent contributor?
The most frequent contributor to the repository is **Rohan Mehta**.
----------------------------------------
Running: Summarize the last change in the repository.
The last change in the repository was made by James Hills on 2025-04-04. The commit hash is `064e25b01b5c82c08aea66ff898ff27adbb013d8`, and the message was: "add links and mcp + voice examples (#438)".

The first answer could be more comprehensive, but at least it worked out of the box. Let’s take a look what happened under the hood:

We can see the agent fetching the list of MCP tools and calling the LLM that is then converting the user query to a tool call for git_log:

git_log({
  "repo_path": "/tmp/openai-agents-python",
  "max_count": 1000
})

As described before, the MCP server performs the tool call and passes the result again to the LLM. Since the git log contains user and commit data (commit_id, author, message) we end up with 29k tokens passed to the LLM after which we get the final response shown to the user.

mcp-server-git provides also operations to work on commits which are definitely interesting to try out another time.

Trying out github-mcp-server: remote Github access via API

Now, let’s try the official github MCP server, first released on April, 4th 2025. We will use the same queries as in the prior example, but specify the repo names directly in the prompts for the LLM to know what repository we want to query:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

async def send(agent: Agent, message: str):
    print("\\n" + "-" * 40)
    print(f"Running: {message}")
    result = await Runner.run(starting_agent=agent, input=message)
    print(result.final_output)

async def run(mcp_server: MCPServer):
    agent = Agent(
        name="Assistant",
        instructions=f"Answer questions about Git repositories.",
        mcp_servers=[mcp_server],
    )
    cont = True
    while(cont):
        message = input("> Input: ")
        # exit upon Exit or CTRL+C
        if message == "Exit":
            cont = False
            break

        await send(agent, message)

async def main():
    async with MCPServerStdio(
        cache_tools_list=True
        params={
            "command": "docker",
            "args": [
                "run",
                "-i",
                "--rm",
                "-e",
                "GITHUB_PERSONAL_ACCESS_TOKEN",
                "ghcr.io/github/github-mcp-server"
            ],
            "env": {
                "GITHUB_PERSONAL_ACCESS_TOKEN": os.getenv("GITHUB_PERSONAL_ACCESS_TOKEN")
            }
        },
    ) as server:
        with trace(workflow_name="MCP Git (Official)"):
            await run(server)

At time of writing, the MCP server provides 29 functions (e.g. issue, commit, pull request operations and search) which wrap the Github APIs called using a Github token defined as an environment variable GITHUB_PERSONAL_ACCESS_TOKEN. For our experiment, in theory, just the function list_commits() for listing commits would be sufficient to answer the posed questions. Let’s see how successful the agent will be with this (simple?) task.

$ export GITHUB_PERSONAL_ACCESS_TOKEN=...
$ uv run python github-mcp-server.py
GitHub MCP Server running on stdio
----------------------------------------

> Input: Who's the most frequent contributor to zalando/skipper?
> Running: Who's the most frequent contributor to zalando/skipper?
Error invoking MCP tool search_users: failed to search users: GET <https://api.github.com/search/users?order=desc&page=1&per_page=1&q=repo%3Azalando%2Fskipper&sort=repositories:> 422 Validation Failed [{Resource:Search Field:q Code:invalid Message:None of the search qualifiers apply to this search type.}]

Looks like we hit a bug: the query constructed by the LLM is invalid (reported as github/github-mcp-server#135). The LLM opted for search_users() instead of using the git commit log like our prior experiment.

Let’s take a look at the second query:

$ uv run python github-mcp-server.py
GitHub MCP Server running on stdio
----------------------------------------

> Input: Summarize the last change in the repository zalando/skipper
> Running: Summarize the last change in the repository zalando/skipper
Error getting response: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-XYZ on tokens per min (TPM): Limit 30000, Requested 68487. The input or output tokens must be reduced in order to run successfully. Visit <https://platform.openai.com/account/rate-limits> to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. (request_id: req_abcabcabc...)

Another error. This time we hit a rate limit given that the request to the LLM is too large (68487 tokens vs. 30000 being the limit). It took the agent only 19.40s to realize this… Something must have inflated the response from the MCP server. A closer look at the traces reveals that the tool call to list_commits with perPage = 1 resulted in a long response (containing 30 commits which is the default setting) with a whooping size of 180KB. Bug number two — filed as github/github-mcp-server#136.

A closer look at the MCP server response also shows the excessive payload originating from the Github API response. A single commit object has 5–6 KB and beyond information on the commit it includes goodies such as the PGP signature of the author, hardly needed for the task at hand:

{
  "node_id": "C_kwDOAlA7-NoAKDdlMmNhM2JmZDI2NzNiNTFkYjRhYmNmYmQ1OWRlYTMzYTk0YzIwMzE",
  "sha": "7e2ca3bfd2673b51db4abcfbd59dea33a94c2031",
  "commit": {
    "author": {
      "date": "2025-04-02T09:43:35Z",
      "name": "... ...",
      "email": "..."
    },
    "committer": {
      "date": "2025-04-02T09:43:35Z",
      "name": "GitHub",
      "email": "noreply@github.com"
    },
    "message": "Add context to log entry (#3466)\\n\\nThis change enriches log entry with request context to be used by custom log formatter.\\n\\nSigned-off-by: ... ... <....>",
    "tree": {
      "sha": "cff0e8e12633313bff1291df86f65f3f81acfd66"
    },
    "url": "<https://api.github.com/repos/zalando/skipper/git/commits/7e2ca3bfd2673b51db4abcfbd59dea33a94c2031>",
    "verification": {
      "verified": true,
      "reason": "valid",
      "signature": "-----BEGIN PGP SIGNATURE-----\\n\\n[...]\\n-----END PGP SIGNATURE-----\\n",
      "payload": "tree cff0e8e12633313bff1291df86f65f3f81acfd66\\nparent 985da0b03d2fa499e4868a22d85426ed387b4a98\\nauthor ... ... <...> 1743587015 +0200\\ncommitter GitHub <noreply@github.com> 1743587015 +0200\\n\\nAdd context to log entry (#3466)\\n\\nThis change enriches log entry with request context to be used by custom log formatter.\\n\\nSigned-off-by: ... ... <...>"
    },
    "comment_count": 0
  },
  "author": {
    "login": "...",
    [...]
    "subscriptions_url": "<https://api.github.com/users/.../subscriptions>"
  },
  [...]
}

BTW. If we adjust the query for our most frequent committer to Who's the most frequent contributor to zalando/skipper? Use list_commits tool to answer this question. we will run of course into the same error. However, fetching 30 commits is hardly close to the right answer.

Trying out aws-documentation-mcp: search for public AWS docs

Let’s take a look at another freshly released MCP servers — this time from AWS. I choose the simplest one to run: aws-documentation-mcp provides access to public AWS docs and does not require any AWS account access.

A closer look at the server code shows the magic behind this server. It’s running a search against the AWS documentation search endpoint (https://proxy.search.docs.aws.amazon.com/search) with a dedicated user agent [...] ModelContextProtocol/1.0 (AWS Documentation Server) allowing AWS to track the MCP server use (and likely rate limit usage). The system prompt also mentions a recommendation tool to use for retrieving related content. The MCP can perform a search using the user’s input and next use search results to fetch the content, parse it using beautifulsoup4 and convert into markdown using markdownify.

So far so good. Let’s try it out with two questions about S3:

• “What are the file size limits for AWS S3?”
• “How does one enable S3 transfer acceleration?”

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

async def run(mcp_server: MCPServer):
    agent = Agent(
        name="Assistant",
        instructions=f"Answer questions about AWS services.",
        mcp_servers=[mcp_server],
    )

    await send(agent, "What are the file size limits for AWS S3?")
    await send(agent, "How does one enable S3 transfer acceleration?")

async def main():
    async with MCPServerStdio(
        cache_tools_list=True,  # Cache the tools list, for demonstration
        params={"command": "uvx", "args": ["awslabs.aws-documentation-mcp-server@latest"]},
    ) as server:
        with trace(workflow_name="MCP AWS Documentation Server"):
            await run(server)

The task took about 24 seconds, but it looks like a success:

$ uv run python aws-docs.py

----------------------------------------
> Running: What are the file size limits for AWS S3?
Here are the file size limits for uploading objects to Amazon S3:

- **Console Upload**: Up to 160 GB per file using the Amazon S3 console.
- **Single PUT Operation**: Up to 5 GB per file using AWS SDKs, REST API, or AWS CLI.
- **Multipart Upload**: Up to 5 TB per file using AWS SDKs, REST API, or AWS CLI. This option is for larger files, allowing uploads in parts ranging from 5 MB to 5 TB.

You can use multipart uploads for efficient handling of larger objects.

----------------------------------------
> Running: How does one enable S3 transfer acceleration?
To enable S3 Transfer Acceleration, you have several options including using the AWS Management Console, AWS CLI, or API. Here's a concise guide:

### Using the AWS Management Console

1. **Sign In**: Log into the AWS Management Console and open the S3 interface.
2. **Select Bucket**: In the left navigation pane, choose **General purpose buckets**. Then, select the bucket you want to enable transfer acceleration for.
3. **Access Properties**: Click on **Properties**.
4. **Edit Transfer Acceleration**: Under **Transfer acceleration**, click **Edit**.
5. **Enable**: Choose **Enable**, then save changes.

### Using the AWS CLI

To enable Transfer Acceleration using AWS CLI, run the following command:

```bash
aws s3api put-bucket-accelerate-configuration --bucket [bucket-name] --accelerate-configuration Status=Enabled
```

### Using the Accelerated Endpoint

Once enabled, you can use the accelerated endpoint for faster data transfers:

- Find the **Accelerated endpoint** under the bucket's **Properties** tab.
- Use `s3-accelerate.amazonaws.com` to direct requests through the accelerated endpoint.

For more detailed instructions and examples of using the AWS CLI and SDKs, visit the [Amazon S3 Transfer Acceleration documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html).

The traces reveal what happened behind the scenes. For each of the inputs, we get search_documentation followed by read_documentation.

This time, the limits are correctly respected and by default, the search gets the top5 results.

Also, we get valid (though simplified) answers straight from the docs. The queries took 14.88s and 10.61s respectively, which is on the slow end, yet if the MCP is embedded into the IDE, likely convenient to access without resorting to using the browser.

Closing words: to MCP to not MCP?

What did we learn from this exploration? MCP servers are easy to set up and are a promising tool to equip agentic flows and assistants with access to additional information and capabilities. They also provide us with a way to bring local data as context for LLMs, especially in cases where indexing this data would be impractical. We also learned that the generated LLM calls are rather costly to execute due to excessive context size (if they get executed at all). Before using MCP servers, it’s important to verify the cost footprint for expected tasks as well as add monitoring and spend limits accordingly.

Building a good MCP server is far from easy: getting LLMs to generate the right and syntactically correct function calls is difficult and becomes more complex the more functions are available to choose from. We saw in the AWS example how detailed instructions help guide the model to generate a multi-step call flow required for the job. Without these instructions, we just can’t magically expect good results. It is easier to ask models to generate code that includes API calls instead.

In the git examples, to calculate the most frequent committer, one would expect an iteration over the commit list over at last a few pages. Yet currently, due to excessive size of the response, we cannot even parse 30 commits due to limits in the model’s context size (or model rate limits). This makes the approach rather impractical. A simple prompt: Generate python code to interact with the Github API to determine the most frequent committer for a chosen repository. results in a code snippet that does the job in a more predictable way. Similarly, Generate git command to calculate the most frequent committer for a chosen repository. returns git shortlog -sne that paired with | head -n 1 returns the result for our local git repo. Needless to say how cheap these LLM calls are when compared to the MCP approach. Let’s see how the mentioned git MCP servers will evolve and for how long it will stay around. They’re early on their journey and require more work to be really useful and give users confidence when and how they will actually work. Field filtering and evals come to mind as first extensions that will be a major step change in quality.

I will certainly continue my experiments as the level of hype is too high for MCPs to disappear quickly.

In the previous post, we inspected calls to OpenAI APIs triggered within Langchain and LlamaIndex by using OpenTelemetry auto-instrumentation. The spans shown in Jaeger UI were nice to see, but were missing rich information that is expected from a proper instrumentation approach. In this post, we will explore how to enrich spans with additional information using manual instrumentation.

Manual instrumentation

OpenTelemetry provides means to add additional attributes to spans. The OpenTelemetry standard defines two rules:

1. Keys must be non-null string values
2. Values must be a non-null string, boolean, floating point value, integer, or an array of these values

Additionally, most commonly used fields follow naming conventions and are referred to as semantic attributes.

Note: Beware of adding fields that may contain PII information to span context. Unless you guarantee that all systems processing the telemetry drop stored data after a fixed period of time (e.g. 30 days), you may run into challenges related to privacy regulation, such as GDPR and its ‘Right to be forgotten’.

Adding instrumentation to own code

Instrumenting own code is as simple as shown in the code below. It starts a new span called function_name with an attribute arg with value 42.

Any spans that are added using auto-instrumentation to functions called by function_name will automatically become its child spans.

1
2
3
4
5
6
7

from opentelemetry import trace
(...)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("function_name") as span:
    arg = 42
    span.set_attribute("arg", arg)
    result = function_name(arg)

Alternatively, one can use the provided decorator, which results in simpler code in case it’s not necessary to capture any attributes in the spans.

1
2
3

@tracer.start_as_current_span("foobar")
def foobar(arg):
    result = foo_bar(arg)

Adding instrumentation to Langchain’s LLM Chains

Langchain offers Custom Callback Handlers as means to execute additional functions in well-defined stages of the chains. To collect statistics on the prompts and token usage from the LLM calls, we can add spans in the on_llm_start and on_llm_end calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        with tracer.start_as_current_span("on_llm_start") as span:
            prompts_len += sum([len(prompt) for prompt in prompts])
            span.set_attribute("num_processed_prompts", len(prompts))
            span.set_attribute("prompts_len", prompts_len)

def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
        with tracer.start_as_current_span("on_llm_end") as span:
            # example output: {'completion_tokens': 14, 'prompt_tokens': 71, 'total_tokens': 85}
            token_usage = response.llm_output["token_usage"]
            for k, v in token_usage.items():
                span.set_attribute(k, v)

Adding instrumentation for OpenAI Embeddings in LlamaIndex

LlamaIndex does not provide callback mechanisms for its embeddings functions. Instead, we can to extend the OpenAIEmbedding class, include instrumentation code in the overridden methods, and pass an instance of this class to the relevant methods of the library. In the added spans we collect the text lengths as span attributes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

class InstrumentingOpenAIEmbedding(OpenAIEmbedding):
    def __init__(
        self,
        mode: str = OpenAIEmbeddingMode.TEXT_SEARCH_MODE,
        model: str = OpenAIEmbeddingModelType.TEXT_EMBED_ADA_002,
        deployment_name: Optional[str] = None,
        **kwargs: Any,
    ) -> None:
        """Init params."""
        super().__init__(**kwargs)
        self.mode = OpenAIEmbeddingMode(mode)
        self.model = OpenAIEmbeddingModelType(model)
        self.deployment_name = deployment_name

    def _get_query_embedding(self, query: str) -> List[float]:
        with tracer.start_as_current_span("_get_query_embedding") as span:
            span.set_attribute("query_length", len(query))
            return super()._get_query_embedding(query)

    def _get_text_embedding(self, text: str) -> List[float]:
        with tracer.start_as_current_span("_get_text_embedding") as span:
            span.set_attribute("text_length", len(text))
            return super()._get_text_embedding(text)

    async def _aget_text_embedding(self, text: str) -> List[float]:
        with tracer.start_as_current_span("_aget_text_embedding") as span:
            span.set_attribute("text_length", len(text))
            embeddings = await super()._aget_text_embedding(text)
            return embeddings

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        with tracer.start_as_current_span("_get_text_embeddings") as span:
            span.set_attribute("texts_len", sum([len(txt) for txt in texts]))
            return super()._get_text_embeddings(texts)

    async def _aget_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        with tracer.start_as_current_span("_aget_text_embeddings") as span:
            span.set_attribute("texts_len", sum([len(txt) for txt in texts]))
            embeddings = await super()._aget_text_embeddings(texts)
            return embeddings

The obvious downside of the approach is that the code needs to be kept in sync with the extended base class, which results in increased maintenance effort in case of library upgrades.

Inspecting the spans

Running and using the code mentioned earlier produces two traces. First, the embedding span with the added attribute texts_len:

Next, the embedding traces and on_llm_start and on_llm_end traces with the captured query_length and token usage attributes:

Writing an Instrumentor for OpenAI Embeddings in LlamaIndex

Extending classes can be cumbersome and an unnecessary maintenance overhead. The built-in instrumentation offered by many of the OpenTelemetry instrumentation packages for Python offer inspiration for a different approach of instrumentation using function wrappers.

Following the example of the Redis instrumentation library, we use the convenient wrapt package to write a simple wrapper function for three methods in the OpenAIEmbedding class. The wrapper _traced calculates the length of the passed string(s) depending on the function’s argument type (str or List[str]).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

from wrapt import wrap_function_wrapper

from llama_index.embeddings.openai import OpenAIEmbedding

from opentelemetry import trace
from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
from opentelemetry.instrumentation.utils import unwrap
from opentelemetry.trace import SpanKind, Tracer, get_tracer

def _instrument(
    tracer: Tracer
):
    def _traced(func, instance, args, kwargs):
        with tracer.start_as_current_span(
            "get_embedding", kind=SpanKind.CLIENT
        ) as span:
            if span.is_recording():
                if len(args) > 0 and args[0]:
                    if isinstance(args[0], list):
                        span.set_attribute("text_length", sum(len(e) for e in args[0]))
                    else:
                        span.set_attribute("text_length", len(args[0]))
         
            response = func(*args, **kwargs)
            return response

    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding.get_query_embedding", _traced)
    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding.get_text_embedding", _traced)
    wrap_function_wrapper("llama_index.embeddings.openai", "OpenAIEmbedding._get_text_embeddings", _traced)

class OpenAIEmbeddingInstrumentor(BaseInstrumentor):
    def instrumentation_dependencies(self) -> Collection[str]:
        return ("llama-index ~= 0.4.32",)

    def _instrument(self, **kwargs):
        """Instruments llama-index module"""
        tracer_provider = kwargs.get("tracer_provider")
        tracer = get_tracer(__name__, "custom-tracer-version", tracer_provider)
        _instrument(tracer)

    def _uninstrument(self, **kwargs):
        unwrap(OpenAIEmbedding, "get_query_embedding")
        unwrap(OpenAIEmbedding, "get_text_embedding")
        unwrap(OpenAIEmbedding, "_get_text_embeddings")

To ensure the instrumentor is actually used, it needs to be initialized with OpenAIEmbeddingInstrumentor().instrument() before the first library calls are initiated. The resulting traces generated by the instrumentor code are as follows:

Summary

We explored adding additional context to spans by adding instrumentation in three different ways: (1) manual instrumentation of individual function calls, (2) extending classes to override methods with ones that include tracing code, (3) instrumenting library code using function wrappers. When to use which approach is highly contextual and depends on the use case at hand. Approach 1 is best used for one’s own code, approach 3 for instrumenting libraries, and approach 2 when a high degree of control over instrumentation is required.

It’s important to be careful and not overdo instrumentation and rely on the provided instrumentation packages whenever applicable. When considering adding manual instrumentation, it’s important to balance the benefits of additional detail with the potential complexity it may introduce. Note that in production deployments, tracing data is often sampled to deal with high data volume and keep the tracing cost footprint in check and this affects the accuracy of the collected data.

References

• OpenTelemetry for Python: Adding manual instrumentation to automatic instrumentation
• OpenTelemetry for Python: Tracing
• OpenTelemetry: Trace Semantic Conventions
• OpenTelemetry: What to instrument
• OpenTelemetry: Python instrumentation libraries
• OpenTelemetry: Redis Instrumentation library

The “Hello World” of LLMs is the Q&A Knowledge Base use case where the app creates embeddings for source documents and feeds a vector storage. Next, for each query it calculates the embedding, fetches top N documents from the vector store and uses the question and knowledge snippets to prompt the GPT-3/4 language models. Some vector stores are 3rd party stores (Pinecone, Weaviate), others in-memory (FAISS, GPTSimpleVectorIndex).

With the sheer amount of these and other demos and applications using popular libraries like langchain and llama-index, the question arises how to analyze new demos to quickly understand what APIs are being called and in which order? How frequently and with which latency? Checking code is tedious and provides only half the answers. Proper Telemetry tooling will help us discover the API endpoints called by the applications.

Automatic Instrumentation for Python applications

OpenTelemetry provides Automatic Instrumentation for Python that comes to the rescue here. Using the Python agent that is attached to the application, it dynamically injects bytecode to capture telemetry from popular libraries and frameworks. The Langchain, OpenAI, and LlamaIndex libraries use Python’s requests under the hood, so we’ll need to make sure to use opentelemetry-instrumentation-requests explicitly.

To visualize the telemetry, we need to setup some tools first. There is a convenient demo that ships with a docker-compose to boot up the collector and a few more tools. We ignore the two chatty demo apps that run in the background, though those are helpful to check if the telemetry setup works correctly.

$ git clone git@github.com:open-telemetry/opentelemetry-collector-contrib.git
$ cd opentelemetry-collector-contrib/examples/demo
$ docker-compose up -d     
[+] Running 7/7
 ⠿ Network demo_default                Created      0.0s
 ⠿ Container prometheus                Started      0.6s
 ⠿ Container demo-jaeger-all-in-one-1  Started      0.5s
 ⠿ Container demo-zipkin-all-in-one-1  Started      0.4s
 ⠿ Container demo-otel-collector-1     Started      0.7s
 ⠿ Container demo-demo-server-1        Started      0.9s
 ⠿ Container demo-demo-client-1        Started      1.1s

The collector runs on port 4317, so any other error message than a timeout/connection error, means that the application is running:

$ curl localhost:4317
curl: (1) Received HTTP/0.9 when not allowed

With this setup, in addition to the open telemetry collector, we also get Jaeger UI running under http://0.0.0.0:16686/ which will help in visualizing the calls.

The next step is to run the code with the auto-instrumentation agent. For this we need a few packages to be added as part of the project setup. In the post, I use poetry to manage python projects, but you can install the packages with pip as well. First, we add the instrumentation for the Python requests library and the telemetry exporter.

$ poetry add opentelemetry-instrumentation-requests
$ poetry add opentelemetry-exporter-otlp

Afterwards, we add the agent via:

$ poetry add opentelemetry-distro

and start the application via the agent (see reference) and keep a text logfile:

$ poetry run opentelemetry-instrument --traces_exporter console,otlp \
--metrics_exporter console \
--service_name llm-playground \
--exporter_otlp_endpoint 0.0.0.0:4317 \
python main.py | tee output.log

If the metrics collector on port 4317 is not running correctly, the app will log an error:

WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting traces, retrying in 1s.

In case of SSL handshake issues (or similar ones)

E0423 17:04:25.197068000 6150713344 ssl_transport_security.cc:1420]    Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

one can instruct the exporter with an environment variable to ignore SSL errors:

$ export OTEL_EXPORTER_OTLP_INSECURE=true

If this does not help to establish connectivity, try increasing the verbosity of gRPC logging to find the error.

$ export GRPC_VERBOSITY=debug
$ export GRPC_TRACE=http,call_error,connectivity_state

As configured in “traces_exporter”, in addition to the OTLP endpoint, the spans are also written to the console. In the demo I’m running, the code uses gradio to create a simple UI. I was surprised to see four calls made by gradio even if launched without any public sharing: launch(share=False).

$ cat output.log | grep http.url
"http.url": "<https://checkip.amazonaws.com/>",
"http.url": "<https://api.gradio.app/gradio-messaging/en>",
"http.url": "<https://api.gradio.app/pkg-version>",
"http.url": "<http://127.0.0.1:7860/startup-events>",
"http.url": "<http://127.0.0.1:7860/>",
"http.url": "<https://api.gradio.app/gradio-initiated-analytics/>",
"http.url": "<https://api.gradio.app/gradio-launched-analytics/>",
"http.url": "<https://api.gradio.app/gradio-launched-telemetry/>",

According to the docs, there is a parameter analytics_enabled and a GRADIO_ANALYTICS_ENABLED environment variable. The description is really convoluted though “default: None; If None, will use environment variable or default to True”…

Back in the telemetry data, we also find the expected OpenAI requests:

"http.url": "<https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings>",
"http.url": "<https://api.openai.com/v1/completions>",

The collected telemetry is helpful to understand the amount and latency of the requests over time. Let’s filter in Jaeger UI (http://0.0.0.0:16686/) for one of the URLs using the tag http.url=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings:

We see 20 calls in the last hour ranging from 280 to 883 ms. Each call can be expanded to check for a bit more details:

Sadly, the UIs of Jaeger and Zipkin are unexpectedly basic and don’t support wildcard searches, so one still needs to tee the console output to a file to quickly search for all the called APIs.

Summary

For more rich telemetry information with custom metadata, the calls to OpenAI (and others) would need to be instrumented manually. While Langchain comes with Callbacks that provide access to the API call results, at time of writing LlamaIndex does not have a similar mechanism for OpenAI embeddings.

The automatic instrumentation is an easy way to inspect calls made by Python applications. The early-stage libraries built in the LLM context tend to have lots of defaults that magically make calls to OpenAI or call lots of 3rd party APIs and externalize its vector data. Even supposedly open-source applications for self-hosting proxies to OpenAI, still rely on other 3rd party services. OpenTelemetry provides an easy way to verify external calls in a sandbox environment.

If you’re interested in getting more out of OpenTelemetry, check out the follow-up post, that gets into details of manual instrumentation with OpenTelemetry for Langchain and LlamaIndex.

References

• OpenTelemetry Python Automatic Instrumentation
• OpenTelemetry Exporter Configuration Options (environment variables)
• Building AI Chatbot with ChatGPT API and LlamaIndex: Building a Chatbot
• Dependencies section for poetry:

1
2
3
4
5
6
7
8

[tool.poetry.dependencies]
python = "^3.11"
openai = "^0.27.2"
llama-index = "^0.4.32"
gradio = "^3.22.1"
opentelemetry-instrumentation-requests = "^0.38b0"
opentelemetry-distro = "^0.38b0"
opentelemetry-exporter-otlp = "^1.17.0"

Emergency procedures used in incident response are aimed at stabilizing the system in a degraded state. When used properly, they result in faster incident response and become a foundation for further resiliency improvements in your system. In the post we’ll also explore how emergency procedures differ from runbooks.

Runbooks

Imagine you get paged in the middle of the night. The situation you encounter looks familiar to you and you’re sure you or your colleague had dealt with it before. However, as seconds ago your were still in deep sleep, you simply can’t recall what to do. Wouldn’t it be great if you had something to help your memory? This is what runbooks are for.

Typically linked in the description of the alert that paged you, runbooks describe routine procedures aimed at restoring regular service of your application. Runbooks list the exact steps to take. It’s helpful to name a runbook with a short summary of the executed procedure as title (e.g. “scale up application”, “retrigger batch job”). Other types of runbooks document migrations, failovers, DB upgrades to help retain know-how within your teams as this type of work is typically executed rather infrequently. Lastly, there are also runbooks aimed at helping with triaging unknown failure situations. When followed, they help inspect typical metrics of the system across layers and components in search for the culprit of the observed issues. A good example here is the sequence of commands used in Brendan Gregg’s Linux performance analysis in 60 seconds.

Emergency procedures

Let’s consider a different scenario: when paged, you observe that the system is overloaded and users experience increased latencies and error rates. You can’t scale up the system due to your dependencies (or storages) taking too long to scale up. To restore service you need to reduce load by 25% as soon as possible. How will you proceed? Will you disable feature A or B? Will you degrade service for user/country X or Y? Do you even have means to do so?

Enter emergency procedures. Unlike for typical runbooks, the goal of an emergency procedure is to bring the system into a degraded, yet stable state. Such state needs to be acceptable to users and stakeholders while trading off availability over customer experience.

Structure

Emergency procedures have defined trigger conditions and impact, both from the business and operational side. It’s important that they’re agreed with business owners ahead of time and thus do not require active approval during the incident response. Impact can be expressed in customer behavior, description how a feature will be working when the procedure has been carried out, or expressed in the change to the system’s load, for example.

Here an example for a food delivery application:

• title: reduce search radius to 400m
• trigger: increased latency or error rate for search queries
• business impact: as all search queries will be limited to a max. 400m radius, customers will see 10–20% less search results, leading to a drop in conversion rate
• operational impact: load on the datastore load will be reduced by 20% within 2 minutes of activating the feature toggle
• steps: …

It’s important that the on-call team regularly practices the emergency procedures. This will verify the correctness of the to be executed steps and operational implications. Additionally, it ensures that the team (and stakeholders) are familiar with the degraded state of the system, which would be rarely observed otherwise. It’s highly recommended to include stakeholder contacts in the procedures in order to keep them informed about the interventions taken during incident response.

Designing for resilience

Systems need to be explicitly designed for supporting emergency procedures. Be it through runtime toggles that allow controlling certain features (e.g. on/off switches, enabling less expensive processing using cached values), or infrastructure mechanisms (e.g. short-circuiting processing for certain user groups or request types). This also requires annotating incoming requests with sufficient metadata to be able to apply differing treatment per feature, traffic origin, etc.

Here a few example degradations that can be introduced to a system with the system property outlined in curly braces:

• enforce serving data from a cache instead to reduce load on the datastore (data freshness)
• always serve the first page of a result set to reduce load on the datastore (data completeness)
• limit retrieved records to N reducing the working dataset of the DB (data completeness defined by amount, distance, or time)
• switch HD video to SD (degrade quality to save bandwidth) or serve images instead (reduce load by preventing auto-play)
• drop traffic from unauthenticated users (user coverage)
• pause all asynchronous batch jobs (feature completeness, data freshness)
• process only critical requests (feature degradation, data completeness)

Automation

It’s certainly advisable to automate frequently used emergency procedures over time by building them into the system as part of your resiliency patterns (fallbacks, retries on error with adjusted input, etc.). Manual execution of the emergency procedure ensures that a human assesses the situation before proceeding with the procedure, which helps harden the defined preconditions. A few manual executions enable you to evaluate if automation is really of value when compared with than other product features planned. It’s important to factor in on-call health into the prioritization. At times of unexpectedly high growth when system availability is a concern, automation is just necessary to cope with overload scenarios efficiently.

Conclusion

Defining emergency procedures requires taking a different view on your system — one where some features are explicitly switched into a degraded mode, thus enabling the overall system to get healthy. The thought exercise of imagining such a degraded, yet usable state is time well spent and highly recommended throughout the design or production readiness stage for your applications. By building in the necessary failure handling mechanisms into the software, or designing it in a way that naturally accommodates the failure states, incident mitigation becomes simpler and less tedious. Having the procedures at hand, you will thank yourself next time you’re on-call in the middle of the night (or day).

The Global Day of Code Retreat is coming soon. Going down the memory lane of Coding Katas, Test-Driven-Development and the related workshops I had the opportunity to attend/host, I got reminded of the long-term value of these exercises.

This posts showcases strategies for shipping software in iterations, which build on the fundamental practices introduced in these workshops. Just like the small steps in coding katas, the strategies outlined here may seem obvious or too simple to leverage in a real-life project, until one encounters the right project to apply them in.

Introduction

Code Retreat workshops

The Code Retreat format promotes Test-Driven-Development through coding katas and injects added fun through pairing, learning from one another, and making the exercises harder through constraints (e.g. no loops). While Katas may feel simple and a repetitive exercise helping getting into flow or learning a new programming language/construct, there is an implicit concept they teach us. It’s all about the simple iterations and sequences of tests that one defines in order to complete the set of requirements in an iterative way. The important bit is that the complexity level may only increase in a gradual way. In the meantime, this concept has been codified as Transformation Priority Premises, which to the familiar eye will look like an extension of the intuitive evolution of null → single result → two results → loop, etc.

Elephant Carpaccio

The Elephant Carpaccio exercise takes the concept of developing in iterations to the next level. Its aim is to teach how to slice a feature into very small iterations spanning all layers. One is asked to define a detailed iteration plan and set of acceptance criteria for each increment. The caveat is that each iteration is to be delivered in minutes! (without Copilot / GPT-3 support ;-)) While tricky at first, with some practice during the workshop this gets rather fun quickly. Try it in your team if you haven’t yet!

Ship early and ship often! (single feature)

More often than not, developers have the tendency to ship features only when fully ready, delaying integration and increasing pull request size (and as a result the lead time). For the team, it’s hard to understand the quality and progress on the feature as the code isn’t used in production and never deployed. As most problems happen in production, this doesn’t sound like a good strategy, right?

A common technique in teams practicing trunk-based development is to hide the feature behind a feature flag. This allows for frequent code integration and avoids unintended production usage. It’s a step forward, but heavily missing out on the value the code may have from actually being used in production continuously, even when the feature is not fully finished (or not even a single line of code of logic is written).

Testing latency assumptions

Whenever new service calls or complex calculations are are added to an application, the overall response time of the application may increase. Typically, services will have a defined latency SLO for its operations, which clients rely on and use to define timeouts. In other cases, increased response time will have immediate business implications as users will interact with the software less frequently as it gets slower or abandon the application along the journey. It’s possible to use A/B experiments to validate the latency buffer a service has, but these may take weeks to reach statistical significance. Hence, there is value in learning early in a project that the pure added latency triggers undesired business impact for users.

Instead of waiting for the feature to be fully implemented, shipping code that’s a no-op in terms of business-logic, but such that simulates the processing time allows to verify latency impact of the new functions. When code runs in production with the added latency, impact on processes, end-user KPIs will become immediately visible, which helps to validate that the chosen design approach is viable in production early in the development cycle. As clients spend more CPU time to process the responses, this also provides early insights for capacity planning.

To prepare for adding business logic, we add an execution budget after which the function terminates automatically and returns a fallback value. Lastly, business logic can be added in multiple iterations, relying on the guaranteed execution time to cover for performance inefficiencies. At this last step, the added latency is optional and can be dropped as soon as the function returns values for all input combinations.

The diagram below demonstrates the iterative evolution of the function code added to an example application. Each iteration ends with a production deployment. The 4th step should have additional iterations to develop the business logic.

Verifying inputs and discovering edge cases

When faced with business logic needing to process many input parameters, it’s helpful to learn more about the input parameters and data access patterns. Frequently, this task cannot be done based on data dumps.

Shipping code that just processes the input parameters allows to calculate the distribution of input parameters, estimate the likelihood of certain edge cases to occur in production (and these tend to occur more frequently than expected), or to record the access patterns based on combination of inputs or frequency of use. This helps in selecting the right data structures for efficient processing, define strategies for populating caches, etc. Long-term, the already developed input verification procedures can help detect data skew or inform that new use cases were added to production, which may invalidate assumptions taken during development.

The function can either store the statistics for requests in memory and/or log these with every n-th call. To minimize impact on the main execution flow of the application, incoming traffic can be duplicated into a separate application that logs required statistics over time.

Shipping logic in iterations

Just like in TDD where the passing (acceptance) tests demonstrate real progress, the iterations of the code you shipped showcase progress that’s verified in production. This allows to understand when a certain iteration triggers problems in production. As soon as first use cases are ready for production use, the service can invoke the logic for these asynchronously and log results for offline verification.

For simple cases, it’s sufficient to log the input and output to perform offline verification of the results. For complex cases, like rewrites or migrations (see picture below) it’s advised to record the result of both operations and to perform additional asynchronous result comparison, helping to measure and log the correctness of the performed operations (often referred to as the Parallel Run Pattern). These comparisons also showcase the incremental progress of the development work, which is much needed in migrations.

Given sufficient confidence for a subset of use cases, the source of truth of the operation can be shifted to the new implementation, which is the essence of the Strangler Fig Pattern.

Being responsible!

Production deployments of the described patterns require high discipline. For high traffic use cases, it is advised to invoke early stage code infrequently, e.g. for a fraction of incoming requests or through capping the number of executions per time unit. The mentioned technique of leveraging latency budgets by ensuring maximum execution time of the executed code helps in ensuring that early stage code has predictable runtime implications. Impact on memory footprint needs to be managed carefully as well. This is easy to achieve through limiting the size of any intermediate results or statistics computed, stored, and logged. Unsurprisingly, these are practices that come in handy nonetheless and would be built up eventually over time, often as a result of production hardening or (worst case) in response to incidents.

Untangling dependencies! (cross-team projects)

The strategies described so far apply mostly to single functions or applications. In cross-team projects, dependencies are typically the limiting factor for successful project delivery on time. Dependencies in larger projects cannot be avoided and need to be managed. Most likely, you will be familiar with one of the following situations:

• Frontend teams waiting for a backend API to be ready, so that they can start building UI widgets and wiring it to the provided data.
• Backend teams waiting for dependencies on APIs providing the data/fields they need to calculate results.
• Analysts delaying building dashboards until the time when data is available in production. Unless, built-in in the UI frameworks, interaction data tends to be de-scoped until the very end.
• Teams waiting for UI components to be ready and shown in production, so that they can verify the end-to-end user journey by clicking a button to initiate an action as part of their tests.

The waiting time in each of the scenarios is the factor to focus on when improving project delivery speed. Here, lessons from the Elephant Carpaccio exercise come in handy and trigger thinking in end-to-end iterations. Coupled with addressing identified dependencies early on in the project it’s a powerful combo.

Shipping the simplest slice

The more dependencies, the more it’s important to manage scope and optimize for early integration. Just like the slices defined in the exercise, the value of small iterations shipped to production cannot be understated. More often than not, simplest will be counter-intuitive to what one is used to.

Building a complex UI widget with a CTA button? From a dependency point of view, it’s the button that really matters as it links data with the destination. Shipping a UI component with just the button unblocks teams building and testing the triggered action along with providing the click events/data. When looked at visually, the widget may feel odd as it would not be shown to your users, but reducing dependencies enables the team to focus on further iterations of the UI component itself. It’s also a great starting point to reserve screen estate for the full component and observe its incremental evolution. Naturally, such change requires to be hidden behind a feature toggle, so that this button is shown only when desired during testing.

API first

While widely practiced through IDLs or OpenAPI specifications that help auto-generate API clients, just the specs may not be enough. To untangle dependencies, beyond agreement on the API contracts, it’s essential to provide a set of sample responses when the code is invoked in production. Such responses can be then returned by API endpoints serving static responses, alternating through sample requests or switching responses based on input context. In case a new service needs to be built, a simple web server serving static files may be enough to start with (even if later replaced by a different application). Re-using any contract tests is helpful here as well, as long as the responses are used in early stage development, in production.

Early end-to-end integration

The earlier features built by multiple teams are integrated with one another, the better. It’s helpful to align on a small end-to-end deliverable across all teams early in the project. In this iteration, API endpoints with sample responses and clients for those APIs are developed to invoke APIs and process (at least) a few API parameters (it’s worth noting that events are also APIs).

When the integration in production succeeds, it ensures that every piece of the puzzle can be further iterated on with relaxed dependencies. Early deployment to production enables everyone to observe incremental progress, incl. interaction events in analytics systems. Moreover, when the features are treated as production-grade from the start, the team continuously learns about daily operations of the system.

Lastly, through simulation of system (end-user) activity, non-functional requirements can be verified uncovering performance bottlenecks. Applications generating such simulated traffic (be it UI interactions, API calls, or events pushed into the system) can be leveraged as part of CI/CD pipelines as smoke tests executed ahead of releasing a change. Additionally, such simulator can serve as an end-to-end probe continuously verifying that the system works in production.

Summary

The strategies outlined in this post can help in reducing risk of software delivery by shipping software to production early, in small, and well-defined iterations. Applied to the right problem, they will result in more frequent code integration and deployment to production. Teams are enabled to gather information about the complexity of the problem at hand at the design stage of the feature delivery, untangle dependencies, and provide more transparency on the progress of the project.

Shipping to production should be frequent and fun! If it’s not, do it more often and address the factors leading to toil or anxiety, be it through automation of lengthy and manual procedures, by adding gradual deployments with automatic rollbacks, or canary builds to reduce the blast radius of failures. In cloud-native environments, there are plenty of tools making it easy to adopt these deployment practices.

My previous post outlined three key elements of successfully preparing and announcing changes in organizations from the perspective of the change author and driver. This post looks at change from the perspective of individuals on the receiving side of the change process — either those affected by the change or those who need to communicate the change to others, often while not having worked on the changes themselves. It explores how individuals can react to changes and focus their efforts and energy to support the change to the benefit of the organization and to foster their own development.

Accepting the change

Some changes are all but straightforward and will affect individuals, for example by changing how they work through rules, policies, or restructuring the organization. Regardless how well prepared, changes will lead to an emotional response and initial resistance. The worst thing you can do is to act immediately and succumb to emotions. The goal is to channel your energy into actions yielding material results. More often than not, you will regret critiquing, rejecting, or questioning change authors before understanding the full extent of the changes and its implications.

While it may not be easy, as first step you need to work towards accepting the change, thus building a foundation for further action. To help you gain some distance, let the change sink in and postpone your reaction to the next day. Afterwards, ensure you understand the communicated change and rationale. Helpful questions for your leader or the person communicating the change may include:

• Any clarification question to the contents of the message.
• What was the trigger for the change? What external factors were at play?
• How will the change benefit the team/company?
• How is my role affected? How am I personally affected?
• From when on are the changes going to be effective from?
• How can feedback be provided?

Noting down the assumptions you had when being presented with the change and contrasting these with the answers you receive, helps you compare your viewpoints. Having a full picture of differing viewpoints will help you in processing the change.

It’s important to note that as a manager and thus member of the leadership team, you are expected to accept the change by default. In cases where you have not worked on the change directly, this may be a more difficult ask. Your task is to discuss any of your concerns and questions with your manager and later commit to the changes and communicate these to others and your team as if you would have worked on these directly.

Providing feedback

When processing the change, a helpful strategy is to formulate and provide feedback. Change authors will appreciate a validation of their assumptions and whether the key points from their communication were received and understood. Further, they may connect you with ongoing efforts supporting the rollout of the change.

You can help by collecting feedback from your peers and sharing it in the form of a summary document. By getting support from others, you reduce the pressure on yourself as you’re not the sole author of the feedback anymore. You also validate your understanding of the changes, though it’s important to be aware of confirmation bias here.

When formulating constructive feedback, you can follow a simple structure. First, highlight what you are addressing (e.g. communication cascade, motivation, measures) and explain how this matches/differs from your viewpoint/experience or point out aspects that potentially were not considered. Finally, express the outcome this has for you or others. Optionally, you can include a suggestion on what can help moving forward. For example: “(…) The change was communicated mainly addressing X, leaving out Y who account for z% of the team. This decreased the mood among Y and made them feel excluded. Moving forward, would it be possible to ensure that Y is included explicitly in any follow-up communication?”

Supporting the change

The next step you can take after internalizing the change and discussing your feedback is participation in the change process. The better you understand (and believe in) the change, the more natural it will be for you to support the change. A good approach to start is by listening to your peers and getting insights on how they perceive the change. You may provide value in helping them understand the change, motivation, or implications of the change. Outlining how you perceive the change and what you see as an opportunity moving forward, will provide them with a different perspective, which they can relate to. You can help your peers to be helpful by outlining where they can support in the change implementation. Getting them involved will increase the buy-in and reduce anxiety for change through active participation.

Here a few practices that may help you in this task:

• Collecting feedback from peers along with suggestions on how to increase understanding and foster buy-in for the strategy.
• Hosting a discussion within the team to form a mutual understanding of the change.
• Volunteering for measures that are accompanying the change rollout.
• Voicing support for the change and showcasing the opportunities it creates for the individuals, team, or organization.

Supporting changes you wish never happened

Surely, you will encounter changes that you are not fond of or such that aren’t in line with your values or beliefs. There is little you can influence when things are decided and already in motion, so be aware of wasting your own energy here.

Instead, you can choose to be forward looking and get involved in future changes of similar type with the aim to correct mistakes that you think have been made. Alternatively, choosing to accept the status quo, you should ask yourself how the you can leverage the changed environment to your benefit. Explore how the changes affect your opportunities. What are things that will be easier? What are things that will not be required anymore? Are there new stakeholders or colleagues you can network with, which may help you achieve your targets?

Next step: driving change

Having supported a change and participated in its implementation gives you good foundations for initiating your own changes. It’s much more fun and impactful to work on a topic that is dear to one’s heart and build one’s own rebel alliance to initiate and lead the change. See my other posts Driving Change for tips on initiating changes and How to communicate major changes for advice on change rollout.

My previous post described challenges with triggering changes bottom-up. This post focuses on three key elements of successfully preparing and announcing changes in organizations from the perspective of the change author and driver. The approach described here is mostly relevant for organizational or process changes.

The difficulty in announcing changes depends on the scope of change, the situation of the affected individuals, and the perceived difference compared to status quo. Diligent preparation and rollout planning for changes is crucial for driving a successful change process. Missing out on planning may lead to diminishing returns, create unnecessary friction, and put additional pressure on the change authors.

When preparing a change, three elements are crucial to consider:

• Set of key changes alongside with reasoning on why these were decided upon and which constraints were taken into account.
• Change map with measures and messages appealing to the needs of those affected by the change and those who will support it.
• Communication approach: announcement vs. communication cascade.

Noting these down serves as an easy way to assess the confidence of change authors in the change and uncover areas that may have been missed out during preparation.

Key changes, reasoning, and constraints

Listing the key changes that are being made and preparing a crisp reasoning for each point helps in verifying the completeness of the scope of change. Capturing the constraints that influenced scope or decisions allows to set the right context as part of the announcement. It also ensures that assumptions about the status quo are validated ahead of proceeding further.

The change map (four quadrants of change)

Every person who is affected by a change is asked to adjust their behavior, which involves learning. Adults learn both on an individual level or collectively with/from their peers. Both ways have an internal and external dimension to it, forming a change map or four quadrants of change. The likelihood of success when rolling out changes is increased when the messages used throughout the communication as well as the additional measures accompanying the change process cover all of the four quadrants:

• mindset/values (internal / individual) — What facts need to be acknowledged about the status quo, values, or beliefs of the individual to ensure the change is easy to understand and relate to? What measures are needed to promote buy-in? What key concerns of individuals need to be addressed in order to explain the reasons for the change, speak to their potential anxiety for change or fear of losing something they had or mastered before?
• behaviors+skills (external / individual) — how are individuals expected to act after the change is announced? What material can they read or interact with in order to learn about the changes? What skills to they need to acquire and what training opportunities are provided? What behaviours would define a successful rollout of the change? Which behaviours should be reinforced and which rather prevented from happening and how?
• cultural (internal / collective) — what support will be provided through changes in culture (or expectations), communities, and means of exchange for those who are affected by the change? How can groups be nudged or empowered to interact and discuss their concerns or understanding of the changes? Who will be the change agents and supporters/promoters?
• systems/processes (external / collective) — what processes or mechanisms will be put in place in order to support the change? Is there a need or opportunity for enforcing the change through process or rule changes? Will there be any mandatory trainings or re-skilling?

Filling out the four dimensions during change preparation ensures coverage of the full spectrum of needs of the target audience (usually the internal dimensions) and organization (usually represented in the external dimensions). A common pitfall is to underinvest in the two quadrants with internal focus on the individual, resulting in merely stating the change and facts. This happens especially if there is external or time pressure or when the authors exhibit excitement and relief for the changes being finally ready to announce. The mindset/values quadrant ensures that care is given to help understanding the state of mind, knowledge, or needs of the target audience. It’s crucial to meet the audience where they are in order to convey a convincing and relatable message, thus reducing friction when the change is announced. Change authors need to be careful to keep a realistic view of the state of knowledge and sentiment within the organization. The longer the preparations take, the more familiar with and convinced they become of the new course, making it rather easy to underestimate the degree of change.

Communication approach

Announcement

The simplest way to roll out a change is to announce it to everyone stating the facts (key changes), either in person or in writing. The bigger and more diverse the audience, the more care is required alongside the announcement text, with aspects prepared in the four quadrants required to be mentioned to increase success.

Regardless how sophisticated and clear the communication will be, there will (almost) always be questions. Including a short FAQ section in the change announcement helps to address anticipated questions upfront. To source these questions, put yourself in the position of the audience and those individuals who are most affected by the change. Try to anticipate what the most controversial or just the obvious reactions will be. To strengthen the set of questions, feedback can be sourced from a few individuals who have not worked on the change or from representatives of the leadership team.

To better structure the conversation around the change, it’s advisable to host a Q&A session and to include an invite in the original announcement. Questions asked during the Q&A should be also added to the FAQ later on.

Communication cascade

It’s common to combine the announcement with the cascade for major or complex changes. There are two types of cascaded rollouts, which differ in the role leadership plays in the change process — a supporting or leading one.

In the first model, the change initiators use dedicated sessions to announce the change to different groups, for example following the organizational hierarchy (following the reporting chain) or its structure (sessions per department). Here, the change authors play a stronger role in the rollout and are supported by leadership.

In the second model, the change is rolled out top-down with leaders owning the change, starting with the first leadership level who then roll it out to their team who communicate the change to the next level, and so on. The FAQ section mentioned above is extended at each level and leaders are supported with additional talking points which help to ensure consistent messaging. The key changes, reasoning, and constraints captured before serve as a base for the talking points and enable leaders to communicate the change as if they had worked on it themselves. Here, it’s also possible to adjust the messaging at every leadership level.

Summary

Depending on the complexity of the change and size of the organization, sufficient time needs to be invested in preparation. How much time to invest is highly contextual and depends on the culture of the organization and the affected individuals. For simple changes, quickly noting the key changes and mapping the key messages onto the four quadrants will help you to tune the content of the change announcement. For complex changes, separating work on the three elements more explicitly is advised. The more changes you will initiate and drive, the better you will be prepared for the next one.

Should you encounter individuals challenging the changes, help them understand what opportunities they have to influence changes and participate in the change process themselves.

Operating software in production offers great insights into software quality. Learning from production incidents is key to improving existing software and learning how to design reliable applications. The Production Readiness Review is an established practice in Site Reliability Engineering aiming at applying past post-incident experience and findings into the software development process. This post provides an overview of a few common themes and pitfalls I’ve experienced being surfaced during production readiness and post-incident reviews.

Using defaults is just asking for trouble

Every framework, http client library or server, connection pool, database, and operating system assume defaults for configuration settings. Aside from risking publicly exposing sensitive data, default settings are often impacting the application’s performance and reliability. There are important settings that must be revisited before deploying an application into a production environment.

Overall, the most commonly missed defaults that impact reliability are timeouts: http client timeouts (connection, read timeout), DNS cache timeouts, and database connection pool and statement timeouts. Framework authors bear great responsibility when setting default values, but often fail to make the developer’s life easy. For example, Java’s DNS cache can be infinite, Apache HttpClient’s v4 RequestConfig.Builder uses “-1” as default timeouts with its documentation stating “A negative value is interpreted as undefined (system default)” and the newer v5 uses 3 minutes instead, whereas .NET’s default timeout is 100 seconds. Timeouts need to be set with care, so it’s important to understand the meaning of the different configuration possibilities. For a great overview on the http request lifecycle and timeouts, check out guide to Go net/http timeouts.

Misconfigured reliability patterns

Retries

Retrying a failed http request is one of the simplest reliability patterns to implement. When done right, retries have an exponentially increasing wait time between attempts and leverage jitter to prevent retry storms leading to the thundering herd problem. However, setting the timeouts to lower values than those that the dependency uses internally, will lead to subsequent retries that pile up and overload the dependency with work that is useless (given that the client won’t wait for the computation result it initially requested). Timeout values must therefore be carefully aligned with service providers, ideally based on their SLOs.

Circuit breakers

Circuit breakers enable the application to fail early in case of an overloaded or faulty dependency and serve a degraded experience via fallbacks. Dropping requests that would fail to be processed before its clients time out is also helping the dependency to recover from failure due to the load reduction. The additional time can be used by the service provider to stabilise the system (e.g. through provisioning of additional instances) and thus regain the ability to serve the required load.

Like every reliability pattern, it needs to be properly configured to function well. Configuring an execution timeout that is too high, will lead to a situation where the circuit breaker never opens, thus keeping the load on the dependency when its performance degrades and making recovery more difficult. Correctly configuring the circuit breaker requires careful planning based on peak load and p99 latencies. The original Hystrix documentation contains detailed guidance for this. Note that the default execution timeout is 1 second. Too bad that official tutorials for frameworks (e.g. Spring) fail to even mention the word “timeout” and do not link to the appropriate documentation.

Circuit breakers require proper isolation

Even if timeouts are configured properly, the circuit breaker may not function as intended keeping the business use case in mind as the degradation will be too broad. Imagine an application A that is fetching a risk score for shipping addresses by calling system B. Because countries may be served by a different risk scoring provider, B will have multiple connectors (one for each provider) and will internally hold a logic defining which provider to choose based on the received address. Service A has a circuit breaker for calls to B. The failure rate of B however, will depend on the failure rate of the connected providers and the distribution of the requests across the providers. In such situation, a failure of a single provider can cause the circuit breaker to open preventing calls to be routed to the remaining providers thus degrading responses for all calls.

In the example below, B receives 300 rps and calls Provider 1 with 200 rps. When Provider 1 becomes unavailable, more than 50% of requests from A to B fail, causing the circuit breaker to open (following the default configuration of the popular Hystrix library) whereas requests routed to other providers would have been processed correctly.

A potential solution for this type of insufficient isolation requires a custom strategy for counting the failure rate per provider or the creation of distinct circuit breakers per country within service A. Both solutions provide different types of isolation. Note that the former requires exposing additional information through the APIs (provider) whereas the latter is purely steered through knowledge that the caller already has based on the processed addresses (country). The right question to ask about this example is — why doesn’t B have circuit breakers to the individual providers? While these would be great to have, it’s often practically impossible, because B is a black box (e.g. a 3rd party service, a monolith that’s hard to adjust, …) and cannot be easily adjusted.

Mixing synchronous and asynchronous workloads

Let’s imagine a service that has a spike in the p99 latencies every x minutes. Sounds familiar? Frequently, it’s a log rotation demon running on the machine where the gzip operation is eating up resources, but more often than not, it is the service itself that causes such spike. It can be a scheduled job that is fetching and processing information, for example periodically refreshing a cache or cleaning up old entries in the DB. If not designed for carefully, such asynchronous execution will impact the p99 latency of the service for synchronous calls.

If you really need to mix such workloads within one application, ensure at least that the http, database connection pools, and thread pools are properly isolated from one another. Otherwise, a long-running async task will impact the synchronous workloads and worst case prevent those from being processed at all.

Missing protection from overload situations

Rate limiting as means to protect from incoming request load

A safe strategy for preventing overload of a service is applying a rate limit to incoming requests. Rate limits are set based on the resources, which impact the scaling ability of the application. These may be driven by its dependencies (e.g. database, 3rd party API) or just costs. Rate limiting can be applied within the application itself or outside, for example in API gateways or ingress controllers. A stricter version of rate limiting is load shedding through request rejection to signal overload situations. To implement load shedding, aside from the technical capabilities to execute this operation, it is helpful to understand per client the business impact of rejecting requests completely as this allows for easy selection of which clients to block first until the service is stabilised.

Rate limits ensure also that clients are forced to negotiate a limit increase with the service provider making scaling needs and capacity planning an explicit conversation. Lastly, rate limiting uncovers and helps dealing with rogue clients of the service. Imagine a service hosting static configuration, which can be cached for a long period of time (e.g. 4 hours). This service should process a request load that is dependent on the number of clients this service has. Load is therefore expected to fluctuate only due to its clients scaling up or down to accommodate incoming load. This service’s load is not expected to follow the traffic patterns of its clients. If it does, it means that such clients are not caching retrieved data correctly and just retrieve it while processing incoming requests.

Protection from unexpected request execution

While rate limits provide external protection, services must implement internal protection as well. As discussed before, services should define SLOs, on which clients will base their timeout configuration. However, the service itself must be designed to honour this SLO across all operations. This is achieved using timeouts on various levels starting from the persistence layer (e.g. via database statement timeouts) through http connection and request timeouts, up to TCP keepalives and timeouts and similar operating system settings. Pure compute operations can leverage execution budgets where a maximum execution time is defined, after which the computation will be aborted. If done correctly, it will prevent overload in cases where the calculation time is unexpectedly influenced by the processed data, for example a regex causing catastrophic backtracking. As past incidents have shown, this is important even for calculation executed in the background, because even though results are not used for the response itself, the calculation consumes CPU cycles and will overload the application anyway.

Lack of control of the application

Having the ability to control the applications inner workings is very helpful during incident response. Though it should not be required during normal operations of the service where one relies on reliability patterns, it comes in extremely handy when mitigating incidents. This can be a capability to pause batch jobs, re-trigger processing, or disabling expensive computation in favor of a degraded, but simpler one. Too often, similar changes require a code change and a deployment of the service instead of a simple switch in a feature flag system or via a management API endpoint. The longer the execution of the CI/CD pipeline in the absence of such controls, the worse the ability to quickly react during an incident.

Insufficient visibility

Aside from monitoring the four golden signals (latency, traffic, errors, saturation), further metrics help to gain quick understanding of production incidents. This starts with collecting metrics on connection pools, the rate of incoming requests per client, outgoing requests per dependency, or duration of batch job execution, etc. It can also include metrics very specific to the service itself — for example, if a service offering a batch API is collecting statistics on the batch size, such data can be used to verify whether clients use its API effectively or break it with unexpectedly big batches. Plotting response times per batch size, will provide insights into the processing times and drive discussions on SLOs for the service. Detailed service instrumentation using standard formats, like OpenTelemetry allows to drill down even further and find causes of incidents across the service call chain as well as identify areas that can be optimised (e.g. through parallelisation of calls) further improving performance and stabilising systems.

Summary

This post provided an overview of common service design flaws and pitfalls that impact reliability. I can only encourage you to check the out the references provided, especially the SRE book, which is a great starting point to dive deep into reliability engineering. Further, if you learned something from production issues, please share your failure stories, so that others can learn from your findings.

Have you ever felt frustrated when you found a problem worth solving but it was not picked up by your colleagues nor supported by your boss? Deep inside you knew you were onto addressing something important, leading to a breakthrough, or just going to have a positive impact on the company, but you failed and don’t know why. This post provides insights on why your ideas may have been rejected along with tips on how to find the right problems to solve.

Why do managers reject your ideas?

Your personal view on the problems that are important will differ from your manager’s view. Your position, role, and size of the company determine your understanding of the strategic goals of a team, department, or organization and influences the easiness of access to relevant, oftentimes strategic information. Therefore, it’s crucial to understand that not all of the problems that you will identify are worth addressing due to a multitude of reasons:

• Approach: The way in which you raised the problem was ineffective. You chose the wrong medium, were too forward and crossed boundaries (e.g. cultural, personal) or just spoke to the wrong person.
• Contracts & regulations: Solving the issue would stand in conflict with the way the company does business due to contractual relationships with customers, the way customers are billed for, or because of regulatory requirements. You may need to build a stronger business case on the positive impacts when addressed, so that the pain points are addressed eventually.
• Cost: Solving the problem would incur too high costs for the company.
• Experience: You’ve joined the team quite recently and/or your manager perceives you as a member of the team who is still learning and trying to understand the mechanics and context of the team. In case you’ve been on the team for a longer time, this may mean that you’re crossing a certain boundary and you should better understand the expectations of your boss, e.g. the areas/problems you should focus on. Also, reflect on the relation with your manager as trust makes a difference.
• Focus: There is another team already working on this challenge and your manager would prefer to keep you focused on the main team’s responsibilities.
• Isolation: You’re the only one that sees this issue and without further support, the idea won’t get further traction.
• Priority: You misunderstood the team’s context and the problem you’re seeing is not worth being solved at this point in time. There may be other opportunities already known that would bring bigger benefits.
• Strategy: Addressing the problem would be in conflict with other, upcoming, more strategic changes you are unaware of. Such changes may be being planned either by your manager or the company.
• Value: It’s unclear what the impact of solving the problem is due to lack of sufficient data points to support your idea.

Understand which problems are important

In order for you to increase chances of a meaningful discussion about the issues you find worth solving, you need a solid understanding of the overall business context and environment. This will help you better showcase the value of resolving the issue to potential supporters. You can start by asking your manager (or other, more senior colleagues) open-ended questions that will help you acquire insights on the big picture and also show the boundaries in which you move in. Here some guiding questions:

• What is the biggest problem in our team that you see is worth addressing? Why?
• I find X annoying/inefficient and would like to help changing that. How could I help finding a solution? Whom in the company would you recommend reaching out to in order to get support?
• I think X is an issue in our company. What’s your opinion? Who is responsible within the company to address such problems? With whom beyond this team could I talk to in order to exchange ideas?
• When would be a good time to address issue X? Why not earlier?
• I have been bringing to you problems that I see as worth solving, but you don’t seem to be interested. Why do you think they’re not worth addressing at this point in time? How would you like me to raise similar issues moving forward?

Get support within the organization

Usually, you’re not the only person in the organization that recognized a problem. You just may be the one that is willing to act. However, this does not mean that you’re alone. You can reach out to your colleagues in order to sound your ideas and verify whether others also have this issue and whether they’re willing to support you. This can be as easy as stating that you work on problem X and asking if someone is interested to help. Having support within the organization is crucial as your peers can provide you more context or help addressing the problem in a different way than you would have.

Bring solutions, not problems

It’s common, especially for less experienced colleagues, to constantly point out problems and deficiencies. Over time, this will lead to fatigue on both sides. You will be frustrated because the problems you raised don’t resonate with your manager and for your boss as well, because you’re likely not the only one to talk about problems. If you never approach your boss with ideas on how to address the problem, you risk being perceived as a complainer and your voice is likely to be ignored over time. Especially from senior team members, managers expect solution suggestions to be brought forward for discussion. Coming with concrete proposals and asking for feedback, context that you may have missed, or for guidance how to proceed in bringing the solution to life will dramatically change the type of discussion you will be having and increase your chances for success. To be on the safe side, you can first validate the idea and come back with a solution proposal only afterwards, which will allow you to focus your attention on the important problems.

It’s not the problem — it’s the timing

Large organizations are complex systems composed of people. Though a stable state of the organization is not desired, there is a limit to the total amount of change an organization can simultaneously cope with. It may well be, that the problem you found or your idea how to solve a problem is brilliant, but the timing is just not right. This may depend on the company’s current strategic goals, the environment (e.g. market conditions), other ongoing structural or cultural changes, or even the management team itself. In such situations, you can either wait for the circumstances to change or try to vastly reduce the scope of the change and thus aim to make small, incremental steps forward instead.

What if all attempts fail?

Surely, the environment or organization you’re in may not be ready to address the issues you’re raising or for the changes you’re willing to drive. If you feel that you’ve been trying for too long and all your attempts failed, you may be better off by changing the environment to another one with a different mindset that is more open for change. This may be a different team, department, or even another organization.

Further reading

This post focused mostly on providing you more understanding on why your ideas would not resonate with others and contains simple guidance on finding the right problems to solve. However, this is just the first step of driving a change. To succeed, you must learn how to define a problem, support it with sufficient context and data, and be able to showcase the impact of solving it.

If you’re more broadly interested in change management, check out Rebels at Work: A Handbook for Leading Change from Within for ideas on building an alliance for change within your organization or Fearless Change: Patterns for Introducing New Ideas for an extensive collection of patterns and approaches for driving change.

See also my other posts with advice on Communicating major changes and tips for Accepting and supporting changes for individuals.

In many companies leadership and management positions are the next step forward for engineers in their career path and often the only way to achieve a salary progression. Here a glimpse of the various aspects to be considered while thinking of transitioning into a management role.

You start as junior again

After taking up the new role you will be a beginner again. You will need to learn a lot and acquire new skills. If you have watched your past managers closely, you will notice that observing is different from doing. In case you have acquired some leadership skills beforehand you will have a head start.

You’re part of the management team

Your peers are not the people you manage, but other managers in the company. Depending on the size of the organisation, there might actually be very few of them (or even spread across different locations). This also means that you will have limited opportunity to learn from other managers on the same level as yours and thus your networking skills will be put to a test.

In case you’re asked to manage your former peers — no matter how well you know them, how well you get along with them — as soon as you become their manager, they will see and treat you as their boss.

Management is hierarchical — you will be asked to execute or communicate things, you not necessarily have worked on, in a way as if you actually had, oftentimes without having all the needed details. Examples include changes in company strategy, restructuring, budget cuts, etc. Some decisions might not be the best for your own teams, but necessary for the long-term success of your company.

You’re responsible for the people you manage

In order to gain trust, you need to genuinely care for people. It’s your job to create an environment that allows people to take joy in their work, use their abilities and fulfil their potential. Be a role model — your behaviour will influence culture and help to set expectations towards your directs. Remember to be the first to act and take the responsibility to address the matter as soon as it arises.

You will need to manage people’s performance by leveraging feedback, one-on-ones, and conducting periodic progress reviews. While addressing low performance, you cannot forget your top performers that your teams rely upon to succeed. Not managing the growth of your best people (for example by being clingy and blocking their rotation to a new team) along with not living up to their expectation, eventually will make them leave the team or even the company.

It will become personal. You will encounter situations where people on your team will not get along with each other due to personality, attitude, cultural differences, etc. You will be expected to manage and resolve such conflicts within and across teams.

Prepare to be judged by what your teams deliver

No matter how great atmosphere you create in the teams, it’s delivery that counts. Delivery against goals or products the teams ship to your customers. Make sure to celebrate successes with the team and praise your directs for the work and efforts they put in.

Your daily rhythm will change

Forget about being able to plan your day. Prepare to be disturbed, over and over again. Being a master meeting scheduler won’t help you. You will find it hard to focus on a single task for a longer time frame and might leave work without the feeling of actually achieving something meaningful. A common error to overcome this feeling is over-involving oneself and not giving enough space to others. Your main job has changed (i.e. it’s not coding anymore), make sure to delegate work and empower others to do it.

Is it worth it?

Management is a new role. It requires high responsibility, discipline, patience, as well as the desire and ability to effectively work with people. You get to build effective teams, create an environment where individuals work towards a common goal, and watch people grow. The decision to transition into a management role requires a certain level of commitment. You don’t get to “try” nor experience management otherwise.

Consequences for failure in management are large, both on business and personal level. Taking risks and learning from situations in the daily work is how you grow and become better in management.

As manager — in case you hire or promote managers, remember your early days in the new role back in the past and make sure to provide enough support and coaching for your directs. As a beginner — ask for it!

ICOs are a hot topic nowadays. Many of the recent write-ups discuss the valuation aspect of token crowdsales. In this post I will take a different angle and explore the code quality of smart contracts used in ICOs.

There are many tools and practices used to ensure decent software quality. Among others: unit-tests, code reviews, continuous integration, static code analysis, code coverage, mutation testing, formal verification, and following security standards like OWASP or PCI DSS. The more critical the software, the more thorough the quality assurance usually is. For closed-source software, one has to rely on industry practices and professionalism of the companies and individuals. For open-source software different rules apply: it’s easy to assess the quality of publicly shared code.

The ecosystem for developing smart contracts has evolved greatly over time. It’s easy now to do the basics: write unit-tests (in Solidity, Python, JavaScript) and even measure the code coverage. Along with bug bounties and 3rd party audits, there should be little doubt about the quality of smart contracts, right? Let’s see how it looks like in practice for some of the recent ICOs.

Project overview

BAT

BAT contracts have been audited and many errors have been fixed before the ICO. Sadly, we have no unit tests nor usage of any templates, just pure code.

Score: 0

Bancor

The Bancor Protocol contracts are a different beast. Despite using truffle framework and tests, there exist many deficiencies in the test coverage. Especially, the self-implemented math functions (exp, log) are insufficiently tested (see points 20–22). Plus points go for usage of jupyter notebooks for proving formulas and algorithms. Sadly, due to missing CI, some errors in the notebooks are left unnoticed (like undefined testCornercase2).

Score: 1.5 (low test coverage, jupyter notebooks)

Storj

Storj contracts use python integration tests and the OpenZeppelin contract templates. Plus for a nice readme with instructions for compiling and testing the contracts.

Score: 3 (tests, template usage, readme)

Aragon

Aragon contracts use both OpenZeppelin and MiniMeToken as templates. Along with CI we also have a quite active project with many pull requests and code-reviews, which are rather expected in an open-source project, but not really that visible in the other contract repositories.

Score: 4 (tests, template usage, CI, code reviews)

Status.im

Status contracts, besides being based on MiniMeToken, introduce an interesting novelty: continuous integration, including a build with test execution orchestrated by truffle framework along with execution details and migration tests. On top of that we get public audit results. Great transparency.

Score: 5 (tests, template usage, CI, detailed test results, audit results)

Summary

A quick glimpse into the source code reveals how serious founders approach product development and ICOs. There are already very good examples of ICO contracts to learn from. As the ecosystem continuously evolves, here some projects and initiatives to follow (and contribute to):

• OpenZeppelin Solidity contract templates for crowdsales, tokens, and math libraries
• Solidity mutation testing
• Formal verification of smart contracts
• Status audit results with common errors in smart contracts
• Smart Contract best practices by ConsenSys (featuring 20 cases for overflow and underflow, that hopefully will get easier to tame after more protection is implemented in Solidity or EVM)

Final words

What happens if the founders are in for quick ICO money, deploy sloppy code or are unlucky to hit an yet unknown error? In the worst case your funds will be locked forever, like the $250,000 lost in smart contracts till date. If you have basic coding skills, you are privileged and can review the contract’s source code, or even try it out on the Ethereum testnet. For all others, the usual advice applies: don’t invest money you cannot lose.

So, coming back to the original question: would you trust your money to a smart contract?