Software factories

The holy grail is a self-evolving system where a feature request is magically built in the existing system. Side-effects aside, whether and when we will reach this target remains to be seen. We’re certainly on path towards the target with teams who get a sense of this future reality in parts of their processes.

As teams work on codifying steps of the development process, we’re seeing more and more proof points for reusable workflows that accelerate release cycles. A single bug or feature request can now be designed, coded, and verified in an agentic loop, if supported with the right tools, platform, and guardrails. Teams that have well-engineered systems and high engineering rigour benefit the most. It’s all about low build times enabling fast iteration, modular architecture making code changes cheap and simple, and high test coverage enabling continuous delivery to users.

Better models enable multi-hour execution for increasing scope, triggering the move of agents from the local developer machines to micro VMs that host them. Leading teams can reverse-engineer existing code, design a replacement and the needed guardrails, and build loops that will eventually result in a running system that passes all checks. This approach accelerates migrations and codebase modernization. Guardrails also become more autonomous with highly specialized background agents that compensate for lower quality code generated by coding agents and that protect invariants or stability of the system. Examples include agents that refactor code, apply reliability patterns, write missing tests, reorganize test suites to reduce execution time, etc.

The key question remains where the developers are in this process. For the foreseeable time, they will build and operate the backbone that builds the system (thus the name software factory) and set the constraints under which the system is allowed to evolve. Depending on business needs and appetite for risk, teams can choose from a set of hand-off points where human interaction and sign-off is explicitly required:

• idea->plan: explicit decision whether to act on the idea or feature request with focus on keeping the scope of the product and key user journeys in place.
• plan->execute: humans review plans or specification, increasing the likelihood of understanding the system. After approval, the system takes over implementation, verification, and more.
• change request->deploy: final pass code review, if required by regulatory needs.
• incident->remediation: the on-call person takes a decision on remediation making the needed trade-offs taking their business knowledge and current state of the system into account.

These handoff points are not entirely new, yet these points seem to be a good anchor for explicitly keeping the software delivery in check. Each of the steps is prone to benefit from further automation supporting the decision making process. For change request analysis, the person can be provided with a measure of risk of the changes, a summary of the changes can be automatically tested against the specification or regulatory guardrails. To speed up incident response, the on-call person can receive an automatically collected summary of the state of the system and changes around the time of incident.

“You build it, you run it” vs. “They build it, you run it”

The “You build it, you run it” approach has promoted healthy practices in teams and increased reliability of the supported systems. With citizen developers (non-engineers who build apps with AI assistance) getting abilities to build applications, it’s tempting to drift back to “They build it, you run it”, placing the burden of ownership and operations on an existing engineering team.

A question arises how we can avoid creating a culture of throwing AI-generated apps over the fence, just to be operated by experts who never wrote a single line in any of these apps? Training citizen developers to have intuition on software and security risks is going to pay off in general. Yet, requiring everyone to acquire a baseline set of skills of software engineers is not the only path. Typically, it’s platforms that separate concerns, offering user-level abstractions that run on top of a solid foundation.

This foundation essentially needs to operate itself and be self-healing in terms of known failure modes, beyond simply recycling unhealthy compute nodes. For example, for applications with databases, it’s possible to add automatic volume increase to avoid running out of space or adding an index to speed up expensive queries. Effectively, the idea is to make these failure modes disappear (in most cases), reducing the burden on the user to know about these concerns from the get go and by setting healthy constraints (e.g. increase disk space max. n times). Existing vibe coding solutions from vendors typically fall short when it comes to accessing data and APIs from production systems. Usually, this leads to static data being embedded in applications, resulting in bypassing access control for data or public leaks in the worst case. The platform can and should ship with mechanisms protecting users, setting the right access and data guardrails.

The challenge remains on what the starting point of the citizen developer journey is. Developers start with a repo or clone an existing template with scaffolding. Citizen developers require to be guided through the creation process from the very first prompt they write. On top, this needs to happen with as little friction as possible so that they (and the agent) do not look for workaround. Defining tight constraints enables a progressive build out of the platform and sets clear expectations to the user on what’s possible for them to build, be it a static website, a data visualization dashboard, or a simple tool with limited access to data from internal systems.

Will we still need IDEs?

Traditional IDEs ship with tools that help navigate the code base and perform deterministic code transformation. Yet, popular plugins from coding agent providers reduce the role of the IDE to a mere terminal/chat window and rendering of large change sets. Thus, it’s easy and common to skip IDEs completely, favoring the popular coding CLIs where reviewing large plans/specs and preparing prompts is a very barebones and basic experience that surprisingly works well enough.

As IDE users migrate off to CLIs, the question is what features could help slow down this trend. The incumbent IDEs are clearly missing out on:

• exposing access to deterministic IDE tools for transformation and refactoring, lowering the token bills
• access to the IDE environment for compilation, running tests, reducing failure rate in coding loops
• annotating agent output with feedback (visual and text) to increase accuracy in prompting
• native support for agent skills with support for verification and (back)testing
• multi-session orchestration to avoid conflicting changes done by agents in parallel
• coding session insights with usage statistics and view of agent traces

Some of these features are already available via tools (e.g. codeburn, agentsview, kit), but bundling them into a (new) product is highly likely to improve the agentic engineering experience. Coding agents also ship with OpenTelemetry support, containing telemetry signals that can be captured and visualized. Visualizing agent session telemetry comes with new challenges related to data privacy as traditional observability tools are designed to provide broad access to application telemetry, rather than offering access to user-level data.

Growing gaps

It’s well known that agentic engineering amplifies both the bad and good practices in teams. Teams that ship in fast cycles will likely continuously widen the gap between themselves and their peers who are slow to adapt. It’s impressive to hear from new startups that invested in codifying processes early on and how they’re benefiting from the acceleration that agentic engineering offers.

Small companies benefit first as it’s much easier for one team member to build the set of guardrails in code and shared skills. Growing from this early base helps accelerate delivery at increased consistency.

Existing companies face the risk of increasing fragmentation on process and tech stack, reinforcing status quo (e.g. adopting microservice architectures with a high degree of freedom of choice for the tech stack and lack of platform support and/or application level SDKs). Common approaches are encouraging the build out of shared skill marketplaces and shared tooling that supports the engineering process, yet the fate of these efforts highly depends on organisational structures and culture of re-use.

A future where developer platform teams step up by one layer and provide a unified process of building software is likely. However, working styles and practices differ significantly on individual level, often being part of one’s identity and skill differentiation. Forcing large groups of engineers to adopt the same structure can become too constraining and limit innovation, even if sufficient extension points for customization are provided.

Summary

During the event, we discussed many proof points for future timelines, yet many questions still remain open. While it’s clearer where human judgement will be most needed when agentic engineering takes off and crunches through product backlogs, it’s less clear what organisational structures and shapes of teams will prevail or correlate with successful outcomes. Unclear terms reveal how young the discipline is with the added level of complexity of social media hyping up names, without triggering prior discussions on these amongst practitioners with decades of engineering expertise.

The future is certainly exciting and requires leadership from practitioners to influence and help understand trade-offs along the journey we’re on.

The sheer influx of information and day-to-day developments is really hard to follow. Just in January Steve Yegge released a new take on the IDE with Gastown, only to reflect a month later on being bitten by the AI vampire, highlighting the addictive side of building software with agents. The impact of FOMO and pressure on mental health is widely discussed in the industry as well, given agents can amplify undesired behavior. OpenClaw triggered euphoria, showing the power of agents to a wide audience, further accelerating the FOMO. In the meantime, its author Peter Steinberger was acquihired by OpenAI, with the project transitioning into a foundation (see State of the Claw for a recent update). Others claim not to be fooled again, recollecting MS-DOS times.

The key question is where things are going medium to long term, hence the title of the post. My goal is to highlight some of the constraints, mechanisms, or factors that will influence how Agentic Engineering evolves, putting structure into notes that I’ve been collecting for a while. Feed the text to your agent for a TL;DR or enjoy reading just like I enjoyed compiling the material and typing this post.

Leveling the playing field

Equipped with coding agents, non-experts can create software and debug problems they encounter without the need to rely on their Google-fu, know the right keywords, or seek support from colleagues. They can build tools and businesses end-to-end in a way that was previously inaccessible to them, benefiting from the drastically lower cost of creation. Surely, many non-experts will have a different objective in mind when coding, seeing it merely as a tool and a means to get the job done. The created software may never be used by more than a single user. When it does, though, it will require security hardening, productionization, and a safe place to run. Otherwise, it will immediately become a security liability.

The (software) craftsman among us are worried about skill atrophy in the age of LLMs. The bigger risk worth acknowledging is lack of skill formation as non-experts become dependent on tools without forming deep enough understanding of their internals. Provider downtime can feel like a blackout: work simply stops. On a positive note, we use things all the time that a single person cannot build from scratch on their own and we’re perfectly fine with this. “I, Pencil” retold by Milton Friedman offers a 2-minute lesson on the complexity and connectedness of our world.

Looking at implications for software teams, platforms and developer experience teams are challenged to accommodate an expanded range of software contributors, going even beyond product managers or designers. Teams will need to tune their assumptions about development environments, onboarding approaches, and create new safety nets, lowering contribution friction. It is also an opportunity to unify the tech stack and deployments of internal apps, forming portal-like marketplaces where apps can be easily adjusted, remixed, and integrated with existing APIs and tools. These managed apps could end up as a remixed experience of codepen.io, glitch.com, and Google’s AI Studio. Without these platforms, non-expert authors of software will be searching for a team to harden and operate their creation, directly breaking “You build it, you run it” principles that many teams follow today.

What bottlenecks will we hit?

All of them. Many times. Our engineering processes and the underlying platforms were built to scale with human activity. Agentic Engineering lowers the cost of producing changes faster than our existing systems and processes can adjust.

Systems built for human-driven workloads

When Netflix’s platform is spinning up cloud compute to stream a video, there is a human using a device to access this video. When a customer service agent picks up a phone, it’s because a human had an interaction with the business and something went south. When existing system constraints were challenged, there were usually clear incentives: marketing (spam), building influence (misinformation through media content), financial profit (sneaker bots, event ticket bots, HFT, fraud, DDoS, malware). In all these, we observed the effect of leverage with a human operator or human-in-the-loop somewhere in the process. Agents will bring these effects into a multitude of places, far beyond putting it into the hands of the next generation of script kiddies.

Lowering cost of contributions and need for improving verification

Lowering the cost of contributions results in more incoming changes. Many of these changes would not have existed before as the cost of doing them would be too high given the value. Now, a change is developed and submitted quicker than it would have taken to assess the need for this change in the first place. When going through existing processes, these code changes trigger code review requests, CI/CD runs with builds, tests, artifact uploads, security scans, etc. Any of these steps that are slow, cannot run in parallel, or require extensive human involvement will result in inefficiencies and frustration that will continue to pile up. These steps used to be to a large extent correlated with human activity and subject to human-level constraints. Dependency updates, if automated were largely expected to be non-breaking. Well-run engineering orgs or large orgs that needed to accommodate a high number of contributors, adopted practices that helped them scale (or reduce costs and lead time), such as Spotify’s fleetshift for fleet-wide refactoring. Other teams may have never seen the need for optimizations or assessed them as having a clearly negative ROI.

Agentic Engineering benefits from rapid verification cycles. A few minutes spent waiting for a PR build to complete or for a code review to come in directly affect the dopamine hits that operators of agents experience. To accelerate, various strategies are needed, such as splitting test suites, using multi-stage builds, being able to launch the application in parallel (locally or on a devbox), and test automation helps keep verification small enabling agentic coding or optimization loops. As release frequency increases, relying only on real-user A/B tests may become too slow for early iteration loops, so simulated traffic from synthetic personas may be used more often as an early signal. To manage incidents, we have established practices in SRE where automating runbooks is far from being novel, yet a practice that’s way more accessible than before. Literally all existing tools and practices need to be challenged, adjusted, or dropped. Early innovators’ products are likely to be absorbed and integrated into existing, established platforms to cope with the pace of development.

Drowning in code and loss of system understanding

When agents contribute code, change sets tend to increase not only in frequency, but also in size. Does it make sense to review the large PRs? Tools like devin help break down PRs into semantic chunks, thus lowering the burden on larger PRs. One can also ask the agent to commit in small chunks and leverage stacked diffs for reviews. However, given a high rate of rework from agents on the same files, what’s the ROI for code reviews on every PR? Maybe setting quality and security guardrails that when met result in an automatic merge are enough? Looking at human constraints on time, allocating fixed time chunks for the team to review changes from the current day, collectively reading and building a shared understanding of the codebase may be a smart tactic. Getting comfortable with agents writing code without supervision will require more platform support (sandboxing) and engineers getting comfortable with building without writing, focusing on agent coordination. A world not every craftsman will enjoy.

To cope with the increasing amount of code that needs to be understood in a structured way, we’re seeing foundations of new tools being built. Codebase size influences the ability of agents to reason about it and affects iteration speed as inference time is correlated with the number of input tokens. Projects like cased/kit, GitNexus, or GitLab Knowledge Graph aim at providing tools that index codebases and expose their symbols or structure in a more efficient way when compared to (rip)grep. Coding agents also support Language Server Protocol (LSP) servers to access IDE-like code navigation features and jump around the codebase.

Specs instead of code?

Spec-driven development can complement Architecture Decision Records (ADRs) by turning intent into something executable and verifiable. What’s new is that full applications can be built purely from the spec. First projects start shipping with disclaimers: “use an agent to make your own” based on the released spec with a tech stack of your choice. This approach brings us closer to a scenario where software can be rewritten on demand, with far less manual implementation effort. The key is in the verification stage: being able to verify the adjusted acceptance and verification criteria coming with the next iteration of the specification. I believe I’ve seen an ERP company have their product work this way, but I cannot find the reference anymore. An approach like this would also mean that framework upgrades and migrations can be executed in a similar fashion, addressing a large chunk of technical debt that exists today.

Cognitive load of change

Another bottleneck will be on the human side. We’re used to a certain pace of software development and delivery. Further acceleration increases the cognitive load and challenges our ability to reason about the changes across systems and codebases. Increasing the number of changesets, their span, and the number of changes still in flight leads to a significant explosion in scope and complexity. It’s not unlikely that the already observed differences between high performers and the rest of the teams, will start requiring structural changes to our teams. It remains to be seen which exact ones.

Effect on Open Source

Open Source is a prime example of bottlenecks, especially when critical projects are maintained by a single person. GitHub sees an influx of activity on their platform, on track for a 14x increase of commits. At the same time the platform has less than 90% uptime showing the pressure their SRE teams are under. Looking at repo creation stats, it’s ~331k repos per day, with clearly increasing momentum since the start of 2026 ². GitHub also published own statistics proving the surge of activity ³ and highlighting that planning for 10x load increase was insufficient, requiring a pivot towards 30x.

Diffusion of quality and loss of discoverability of new projects

The influx of new repos affects discoverability, for example for hot topics such as coding agent sandboxing. Try finding which of the ‘claude sandbox’ projects is good enough to be used safely. Sifting through the project list is more time-consuming than it used to be, because it’s much harder to assess whether a project just looks good or whether it actually works and what its quality level is. Large PRs break the UI, making it harder than needed to review incoming contributions. Faced with an influx of activity, spam PRs or comments, maintainers observe past incentives being put out of balance. The current training data for LLMs is built on the prior generation of OSS projects. Many of these were high quality and key dependencies across a large number of projects. As incentives shift, the question is how strong and resilient the ecosystem really is. Open Source used to be a way to tap into developer capacity, especially the most committed community members willing to contribute value. Supported by agentic coding, some projects consider closing down their projects to trusted contributors only, making more efficient use of their time. There is even a GitHub feature helping to limit PRs to contributors only and vouch as an experimental project for trust management and means to reinforce the strong links in the ecosystem.

AI contribution and attribution policy divergence

The community is torn on handling AI-assisted contributions. Linux Kernel contributions invite explicit authorship tags (Assisted-by) whereas Kubernetes explicitly bans them. Adrin Jalali, one of the core maintainers of scikit-learn, published a piece outlining different strategies for maintainers of open source projects, recommending creating agent guidance files (e.g. AGENTS.md) aiming to help increase quality of incoming AI contributions. Melissa Weber Mendonça maintains open-source-ai-contribution-policies with a collection of AI policies across the ecosystem.

Security triage burden

In addition to code contributions, projects now face more incoming issues and security advisories. Claude Code has a bug command that creates github issues. Looking through the types of issues discussed is a mix of amusing and frightening. curl is known for receiving an influx of low-quality vulnerability reports, struggling to sift through them. To adjust incentives, they stopped their bug bounty program. The problem here is the signal-to-noise ratio that hides the valuable reports.

In the State of the Claw talk, Peter mentioned that OpenClaw had 1142 security advisories since January 31 (>16 per day) with an acceptance rate of 41%. This was an estimated 5700 hours of work over 69 calendar days (5700h = 237 calendar days or >700 working days). This particular project is popular enough to attract enough attention from both ends: contributors and attackers. Not every single critical dependency in the software supply chain will be lucky to have enough hands on deck and a foundation structure to support governance. We see first signs of large-scale attacks with ripple effects from the attack on trivy propagating through the ecosystem.

Compute capacity shortages as a driver for innovation?

When limits are healthy

Constraints are great as they lead to reflection and innovation. As the AI datacenter build-out affects the whole supply chain with memory, storage prices, and availability of GPUs and CPUs (and other components), compute efficiency will hopefully matter more and more. We’ve become complacent and indifferent to resource usage as an industry as it’s become so easy to just bump memory or CPU in the cloud instead of spending the time to profile an application and understand reasons for performance bottlenecks. The local inference movement with llama.cpp is a great example of a successful movement aiming to bring inference to local machines, with their compute constraints. Same goes for on-device inference with ExecuTorch.

Limits on coding plans are often the only reason for a person to ask a question: Could I have used a cheaper model? or Was this really the right task for a coding agent? Without constraints, it’s too easy to just continuously run on the currently most capable and expensive model, an approach that is not sustainable long-term. Lack of limits reinforces bad behaviors. Too tight limits do not allow users to experience new capabilities in action.

GPU capacity constraints and hidden price hikes for models

GPU capacity shortages will also play a key role moving forward. They could be a possible explanation for features such as adaptive thinking, now enforced for new models and deprecating the feature for existing ones. Capacity limits would also make subtle bugs in prompt caching hurt much more (Anthropic since released a post mortem on the quality issues caused by changes shipped: default reasoning effort, cleaning thinking sessions, system prompt changes). While the length of tasks that LLMs are able to solve keeps growing exponentially, it remains to be seen whether token usage rises exponentially as well. Anthropic’s 10x growth rates would suggest that, yet we lack clear data on costs of AI agents.

What we definitely see is explicit price hikes with releases of OpenAI models: gpt-5.4 is 11% more expensive than gpt-5.2/5.3 which is 40% more expensive than gpt-5.1 - a total of 55%. Surely, the model is more capable, though one has to ask whether the models are really fed with incrementally more complex tasks by all users? Without clear, task-specific evals, engineering teams often opt to switch their coding model to the newest one as this is the (now) recommended one, feeding the FOMO on one end and filling the revenue hat on the other. Many users do not have the tools or the capacity to create their own, task-specific benchmarks (yet). Hopefully, they take the time to do so for customer-facing products as this has clearer ROI. If not, they will experience funny inference glitches, such as links being injected into LLM outputs where a single word was expected.

Anthropic’s price hikes are more subtle and hidden in features like adaptive thinking or system prompt that change the behavior of their harness or those that are more visible like the tokenizer change in Claude Opus 4.7 resulting in 38% higher request cost. On the other hand, they’re also more explicit with recent updates to the Enterprise plan, switching to usage-based pricing by introducing a $20 per seat price just to get access to tools with all interactions billed at API price level. No more free riding and no quotas. Reduces complexity, and challenges existing budgets.

GitHub feels the heat as well, adjusting their individual plans, pausing sign-ups, reducing model availability of Opus to the highest tier only as well as retiring previous Opus versions. GitHub’s move to offer Opus 4.7 with a promotional 7.5x premium request multiplier (vs. 3x for Opus 4.6) was a 2.5x price hike as well, a prelude to shifting the whole platform to usage-based billing. This just gives users more arguments to run with Opus-class open-source models like the newly released Kimi K2.6. Frequent changes to pricing models of established providers are prone to push more users toward open-weight and local alternatives, strengthening competitive pressure on proprietary vendors.

Pricing models, revisited

As products and platforms start getting agent-ready, they are forced to open up access to agents via official interfaces: CLIs, APIs, MCP, etc. Google’s NotebookLM is famous for not having an API, requiring hacks for agent access and possibly risking account bans if crawling will get classified as abuse. Many other platforms got away with not releasing programmatic access to their platforms, locking in their users and making migration between products unnecessarily difficult. With Salesforce announcing their Headless 360 initiative, other players will feel even more pressure to catch up. The result will be two-fold. Firstly, users will (hopefully) gain an ability to integrate and migrate between platforms with more ease. Secondly, existing pricing models of platforms will be challenged.

All the problems we’ve seen with coding agents will also show up here. Seat-based pricing will get challenged as activity on the accounts will get an additional component, not correlated with human activity anymore. Public platforms dealt with automated traffic via waiting rooms or bot protection products, incl. recent releases allowing to take in payments for access, giving http response code 402 Payment required a new life. That’s not going to fly for SaaS. The growing share of agent traffic will put a strain on margins, leading platforms to rethink their pricing models. An approach where “agents must buy seats just like human employees” is unlikely to succeed long-term as the usage patterns of agents will continuously evolve and become more complex. Assigning agents subsidized quotas would result in agents gaming the system, registering accounts to be used in parallel or just running at the full utilization of rate limits all the time (like crawlers do). The usage-based models that the coding plans are converging on are much more likely to be applied more broadly to API-based products.

Summary

Roughly 18 years ago, I had a course in Artificial Intelligence at university where the professor was explaining how an AI could compose computer programs through natural language analysis. What felt like a complete abstraction existing purely on the whiteboard is now a reality used by millions. Having witnessed both sides is fascinating and humbling at the same time.

The ecosystem around Agentic Engineering is evolving in parallel with an accelerated race to ship new and more capable models while dealing with constraints in GPU capacity. This mix of challenges in processes, tools, and infrastructure build-out results in fast-paced change with implications that cannot be easily foreseen. We will continue hitting various bottlenecks as progress is made, revisiting old approaches or adding new ones. All we can do is embrace the uncertainty and adjust course when needed. With the bottleneck shifting from generation to orchestration and verification, we have an opportunity to close some of the gaps in processes and systems that existed for a long time, thus having an opportunity to improve the industry as a whole. To achieve this, we have to be able to teach Agentic Engineering while new ways and approaches are being figured out.

More tools to come, more approaches to be tried out. As exhausting as it is to ride the wave of change, it’s highly rewarding as well.

Updated on Apr 24th: Referenced Anthropic post mortem on recent quality issues. Updated on Apr 29th: Referenced GitHub’s move to usage-based billing and added stats from update on GitHub availability.

The sprawl of tooling and practices along with viral content circulating through social media makes many people uncomfortable. Many look for direction, best practices, or clear instructions on what the state of the art is. The reality is that nobody has it all figured out yet and many of the shiny things shown on social media are demos showing the happy path. Mario Zechner, the creator of pi.dev (the harness used by OpenClaw), poured cold water on the hype, asking for more discipline and reason when using agents. Martin Fowler said recently that in XP and agile terms we’re at the stage comparable to the early 90s full of experimentation and new ideas. Thus, it’s too early for a manifesto.

Because it’s still early days, authors of tools have not yet figured out how to build their products with consistently high quality. We see this in the regressions in new versions of Claude Code. Non-determinism adds another layer of difficulty with new failure modes that need to be learned and mitigated. For example, compaction can result in lost instructions, which is non-intuitive to users. It takes time to build the intuition and mechanisms that help mitigate problems end-to-end. Early days of SRE were similar, though we were at a stage where companies with sufficient scale had a time advantage as they hit the limits quicker. This gave time to build up and codify some of the knowledge in articles or books.

Learning journey for Agentic Engineering

A pragmatic way of building organizational know-how is through guided experimentation where a small group of power users defines the set of experiments or practices to try out. They take on the risk of trying out something that won’t work out, but they also guide participants throughout the learning journey with existing knowledge they have. Oftentimes, their time advantage in using the tools or practices is measured in days. The guided experiments can be time-boxed (e.g. 4 hours) and done in pairs to facilitate peer learning, ideally across teams.

What is important when considering teaching agentic engineering is that everyone’s journey is different. Those who jumped on LLMs early on and kept following the evolution of models, coding assistants, and similar tools have a vastly different intuition about the capabilities than folks who have been resistant to trying them (or who could not make prior generations of tools work). Thus, the early adopters are also those who are most likely to thrive in the experimentation setup mentioned.

Looking for more structured approaches to learning, we find models with 8 levels of agentic engineering. A mental model that worked for me is slightly simpler, directly rooted in the level of autonomy and size of changes delegated to agents:

• Suggestion mode, aka. Tab Autocomplete where engineers get familiar with whole chunks of code being suggested instead of just function names and arguments.
• Interactive mode with Agents in IDEs allowing to interact with the codebase through prompting, asking simple questions and progressively requesting more and more complex code transformations of features.
• Supervised execution with Agents in CLIs with engineers starting to scope larger tasks and acceptance criteria leveraging the planning mode, but accepting every single command from the agent.
• Delegated, autonomous Agent Loops in CLIs where building up on the prompting skills, engineers build verification methods and safety nets allowing them to diligently plan large tasks and let the agent crunch through them autonomously.
• Autonomous Background Agents that run in the background and react to defined changes in the system, e.g. commits, pushes, or periodically scan the codebase or running software deciding when to activate and what procedure to run.

Meet users where they are on their learning journey

When teaching agentic tooling or practices, it helps to understand where a person is in their learning journey. This helps to present ideas new to them in ways that stay close to their current comfort zone. Seeing is believing, so make sure to add a healthy dose of live demos where people present what worked for them using real-life examples, not just fancy demos. When designing trainings, these levels also need to be accounted for, either in expressing clear preconditions for entry or in making sure that no single approach is favored.

The above list represents a spectrum of working modes rather than a strict maturity model. This means that there is no single right way, though there are many wrong ones. Certainly, the outcomes and token usage differs between them. However, it’s important that engineers find their own mix that feels right for them, their level of tolerance for risk and for their team/product as well as budget. This will challenge existing team structures, likely leading to some reshuffling to bring colleagues with similar development styles together (temporarily or permanently), so that they continue to raise the bar and to facilitate peer learning. In order to avoid teams getting stuck in local minima, the big question is how one can coach whole teams?

Teaching teams

Teaching teams used to be a task for the engineering managers who could rely on their past hands-on know-how and credibly advise their teams. Now, they’re faced with a situation where time for hands-on coding is limited and where they may need to learn new tools first, before they’re able to teach their team. They will need to be more intentional about their own learning, leveraging power users to acquire skills or hints on what to look at. Waiting for EMs to learn before they can coach their team is of course too slow.

Power users can help coach teams effectively. The key is to break down the agentic engineering cycles that experts do in one go into meaningful phases, adding inspections along the way. Inspections can be done individually (pairing, code review) or as a group (like in swarming). Let’s say we want to teach a team to author implementation plans in plan mode. While an experienced engineer would do plan and execute in one go, we can break this into two steps. First, the plan is prepared and checked in. Next, it’s reviewed along with the prompt used. This helps transfer the intuition that comes with experience and potentially capture some recommended practices (or constraints in agent instructions). Afterwards, the plan is implemented and results are reviewed. Similarly, one can teach prompting patterns, e.g. focusing on reviewing code for certain types of defects.

Scaling beyond teams

To scale further, we can learn from the OpenClaw moment in China where companies set up events that offered hands-on support for setting up the tool. The key is to meet users where they gather anyway to spark their initial interest or incentivizing them to stop at easy to access places. Setting up booths near office entrances, kitchen spaces or in cafeterias comes to mind here. Same goes for workshops and demo sessions at internal conferences or town hall meetings. While these formats can spark interest and drive attention, lasting effects depend on curiosity of the attendees and structure of workshops delivered.

When setting up formats for cross-team knowledge exchange, it’s important to provide a platform for early adopters to share and showcase their wins, enabling them to earn some peer credibility as experts and ensuring they get connected with other experts. To balance out the focus on experts, formats where leaders can discuss adoption friction are needed as a safe space to discuss organizational change and gaps for enablement. For further scaling, one would usually seek champions per organization who would be acting as multipliers. Forming a network of champions, they support each other in sharing lessons learned. Whether champions are chosen among early adopters who earned some visibility by sharing their knowledge or by role is highly dependent on the organizational culture and the seniority required for the change management process. Key to success is organizational backing by sponsors from leadership who can help remove organizational blockers.

Since the pi plugin was released first, I decided to give it a spin and use the opportunity to try pi for the first time as well. For LLM access, I used Kimi-K2.5 model via OpenRouter with the API key budget explicitly set to $10.

In this post, we will explore how the autoresearch loop works using three different scenarios:

• algorithm optimization of a simple function
• running unit tests for a Java multi-module project
• optimizing test runtime for a golang library

How does the plugin work?

The plugin provides /autoresearch to trigger the optimization. Under the hood, we have a skill that instructs the model to keep track of the optimization in autoresearch.md and uses autoresearch.sh for executing the verification step and capturing guardrail metrics.

The instructions, guardrails, and metrics for the optimization can be defined in autoresearch.md. If not provided, these will be inferred from context and prompt. The plugin then triggers an infinite optimization loop. An optimization candidate is applied to the code, the verification procedure is started and metrics are captured. If the optimization run results in better metrics than the prior result, it’s commited to git and taken as base to optimize further.

The optimization loop can be interrupted at any time and resumed just from the autoresearch.md file. This also poses an opportunity to adjust the file manually, if needed.

Experiment 1: Algorithm optimization

For my first exploration, I took a simple prime number finder loop and a test file verifying that this function works correctly as baseline. Both were generated by an agent, though the exact code is of less importance here since we just want to understand how the autoresearch loop works.

I kicked off the optimization process via:

/autoresearch optimize the test runtime, adjust the algorithm as needed

The agent then looked up if there is any autoresearch.md plan provided. Since there was none, it came up with own ideas and kicked off the work, correctly using test suite runtime as metric. For each iteration, the agent outlines the idea, displays the changeset, runs the verification and summarizes the findings. The intermediate output can be useful to follow the agent’s actions and to verify that the loop is configured as intended.

After a few iterations, the agent decided that it completed its work and printed a nice summary of the findings:

Autoresearch Complete! 🎉

Final Result: 48% improvement (0.31s → 0.16s)

### Optimizations Applied:

┌───┬──────────────────────────────────────────────────────────────────────────────────────┬────────┐
│ # │ Optimization                                                                         │ Impact │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 1 │ Hybrid approach: Use Sieve of Eratosthenes for n≥100, trial division for smaller n   │ Major  │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 2 │ Odd-only sieve: Track only odd numbers ≥3, halving memory and operations             │ Major  │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 3 │ bytearray + slice assignment: Faster marking than list with element-by-element loops │ Medium │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 4 │ 6k±1 wheel in is_prime: Skip multiples of 2 and 3 in divisor checks                  │ Medium │
├───┼──────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ 5 │ Tight upper bound: Reduced margin from +10 to +5 for smaller allocations             │ Minor  │
└───┴──────────────────────────────────────────────────────────────────────────────────────┴────────┘

### Key Algorithmic Changes:

- Before: Trial division (O(n²)) for all n
- After: Sieve (O(n log log n)) for n≥100, optimized trial division for smaller n

### Performance at n=5000:

- Trial division: ~18 ms → ~0.77 ms (23× faster)
- Sieve: ~3.3 ms → ~0.77 ms (4× faster)

The optimizations have been committed to git and future ideas documented in autoresearch.ideas.md.

Then, the autoresearch plugin detected this scenario and kindly decided to ask the agent to give it another go:

Autoresearch loop ended (likely context limit). Resume the experiment loop — read autoresearch.md and git log for context. Check autoresearch.ideas.md for promising paths to explore. Prune stale/tried ideas. Be careful not to overfit to the benchmarks and do not cheat on the benchmarks.

After a few more iterations, the agent decided that it ran out of ideas:

Autoresearch Session Complete

Final Result: 48% improvement (0.31s → 0.16s)

### Summary of This Session

Attempted optimizations (all rejected):

┌────────────────────────┬───────────────────┬───────────────────────────────────────┐
│ Optimization           │ Result            │ Reason                                │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ 30-wheel factorization │ ❌ Slower (0.19s) │ List indexing/modulo overhead         │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Cached prime list      │ ❌ Slower (0.23s) │ Cache management overhead             │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Local variable lookups │ ❌ Slower (0.19s) │ No benefit for C-level math functions │
├────────────────────────┼───────────────────┼───────────────────────────────────────┤
│ Threshold 90 or 110    │ ❌ Slower         │ n=100 is near-optimal                 │
└────────────────────────┴───────────────────┴───────────────────────────────────────┘

Current best approach remains:
- Hybrid: trial division for n < 100, sieve for n ≥ 100
- Odd-only sieve with bytearray and slice assignment
- 6k±1 wheel in is_prime
- Tight upper bound (+5 margin)

Performance at n=5000: ~0.71ms (23× faster than original)

All findings documented in autoresearch.ideas.md. No further low-hanging fruit identified without adding complexity (numpy, Cython, etc.).

The autoresearch plugin includes a simple widget that shows the progress so far, the baseline and optimization progress, expressed in metrics (e.g. test runtime in seconds). When the research loop is finished, this widget provides a nice summary of the actions:

The experiment with 8 runs cost me $0.284 in API costs and yielded a 47% improvement in test runtime.

Experiment 2: Unit test execution optimization

For the next test, I chose riptide - a Java multi-module project implementing Client-side request routing in Spring. Here, I wanted to optimize the unit test execution as running the tests took about two minutes. The research loop was triggered with:

/autoresearch optimize the unit test runtime in this project; tests can be run with './mvnw clean test -Dmaven.build.cache.skipCache=true'

This verification ensures that regardless of code changes, the test runs are comparable. Autoresearch decided on two metrics as guardrails for the optimization: total runtime in seconds and the number of tests executed. Quite smart, as this ensures that the model cannot cheat by removing tests.

The optimization loop studied the maven configuration and decided to leverage parallelization, batch mode, tune GC parameters, and drop plugins that in its opinion were not essential for running the tests. One could argue whether dropping code coverage calculation is a good choice, though it could be indeed skipped for local runs and kept only for CI builds. After a few runs, the test execution was extended to capture the execution time of individual tests, which revealed that a subset of tests had waiting times due to socket communication, as expected for the library.

After >30 experiments and $0.686 in API spend, we got a 38% improvement in test runtime.

 Autoresearch Complete: 38-39% Improvement Achieved ✅

 After 31 experiments, the test runtime has been reduced from 113.4s to ~70s (38-39% improvement).

 ### Final Configuration Summary

 ┌────────────────────┬───────────────────────────────────────────────────────────────────────────┐
 │ File               │ Key Settings                                                              │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ .mvn/jvm.config    │ -XX:+UseParallelGC<br>-Xmx2g                                              │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ .mvn/maven.config  │ -B<br>-T2C                                                                │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ pom.xml (Surefire) │ argLine: -XX:+UseParallelGC -Xms512m -Xmx512m                             │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────┤
 │ autoresearch.sh    │ Skip flags: jacoco, duplicate-finder, checkstyle, spotbugs, pmd, enforcer │
 └────────────────────┴───────────────────────────────────────────────────────────────────────────┘

 ### Results

 - Baseline: 113.4s
 - Best: 69.2s (39.0% improvement)
 - Typical: 69-72s (~38% improvement)
 - All tests pass: ✅ (except occasional flaky ChaosPluginTest)

 ### Optimization Breakdown

 1. Maven parallel build (-T2C) — ~26%
 2. Skip JaCoCo — ~14%
 3. JVM opts + argLine — ~3%
 4. Skip duplicate-finder — ~2%
 5. Batch mode (-B) — ~0.4%

 ### Failed Experiments

 - Compiler fork, Surefire forkCount>1, G1GC, 4GB heap, Smart builder, offline mode — all slower or cause failures

 The optimization has reached a plateau. The remaining ~70s is dominated by actual test execution time across 20+ modules, which cannot be reduced through Maven
 configuration alone.

The image below shows all attempted optimization runs, which includes a more realistic scenario than in the first experiment. We have not only discarded runs, but also optimizations that resulted in crashes or inability to run the test in the first place:

As before, the harness attempted to proceed with the optimization job, but this time the execution was interrupted by a failure from OpenRouter as I ran out of credits before the API key limit was reached.

golang library

Next, I took go-keyring, a library with a cross-platform keyring interface for Go, as my next test subject. I kicked off the optimization loop with a rather lazy prompt:

/autoresearch try optimizing the test runtime and memory usage

I mainly wanted to see how easily the model will be led astray throughout the optimization process. After 18 runs and $0.516 in API tokens, the model tried the following optimizations:

Two key changes were:

• Replace regex with lookup table in shellescape.Quote - reduces allocations and removes regex dependency
• Add testing.Short() checks to skip OS-level tests in short mode - reduces test time by ~55%

While the first one may indeed be more efficient for the test data in the code, the second one is borderline cheating.

Another interesting behavior was the model deciding to add microbenchmarks for certain functions used in the code. This is likely a result of golang having built-in support for benchmarks in its test command. Throughout the optimization loop, the verification was extended do also add benchmarks:

Now let me run the benchmarks to see the performance.

run_experiment cd /Users/bartek/repos/go-keyring && go test -bench=BenchmarkQuote -benchmem -benchtime=0.3s -run=^$ ./internal/shellescape/ 2>&1 (timeout: 60s)
✅ wall: 3.0s (truncated)
… 8 more lines
BenchmarkQuoteLongSafe-12          19541719           19.25 ns/op          0 B/op          0 allocs/op
BenchmarkQuoteLongEscape-12         7272855           46.32 ns/op         80 B/op          1 allocs/op
PASS
ok     github.com/zalando/go-keyring/internal/shellescape   2.768s


Excellent! The benchmarks show good performance. Now let me also run the regular tests to make sure everything still works.

Viewing results

When the optimization loop is stopped, the output is saved as autoresearch.jsonl, autoresearch.ideas.md and in git commits. The JSONL file has all the commit ids and basic metadata on the optimization run and is the base for the terminal widget shown in the terminal and screenshots included in the post.

Let’s take a look at one example optimization commit. First, its metadata:

$ cat autoresearch.jsonl | grep d636c43 | jq .
{
  "run": 4,
  "commit": "d636c43",
  "metric": 0.865,
  "metrics": {},
  "status": "keep",
  "description": "Remove redundant slice operation in keyring_darwin.go - use string(out) instead of string(out[:])",
  "timestamp": 1774123478289,
  "segment": 0,
  "confidence": 2,
  "asi": {
    "hypothesis": "Redundant slice operation creates unnecessary overhead",
    "keyring_pkg_time": "0.560s vs 0.571s baseline",
    "change": "string(out[:]) -> string(out)"
  }
}

Next, the actual code change:

$ git show d636c43
[...]
diff --git a/keyring_darwin.go b/keyring_darwin.go
index 5e3ae1b..e1ea9e4 100644
--- a/keyring_darwin.go
+++ b/keyring_darwin.go
@@ -53,7 +53,7 @@ func (k macOSXKeychain) Get(service, username string) (string, error) {
                return "", err
        }

-       trimStr := strings.TrimSpace(string(out[:]))
+       trimStr := strings.TrimSpace(string(out))
        // if the string has the well-known prefix, assume it's encoded
        if strings.HasPrefix(trimStr, encodingPrefix) {
                dec, err := hex.DecodeString(trimStr[len(encodingPrefix):])

Successful optimizations are stacked on top of one another, meaning that the optimization we looked at was applied on top of others:

$ git log d636c43
commit d636c43b02d580b335ddada3b6e6962cbfa95717
Date:   Sat Mar 21 21:04:38 2026 +0100

    Remove redundant slice operation in keyring_darwin.go - use string(out) instead of string(out[:])

    Result: {"status":"keep","test_duration":0.865}

commit 80baf8a9901dbb4abb67607c533272aacbd130cc
Date:   Sat Mar 21 21:03:29 2026 +0100

    Replace regex with lookup table in shellescape.Quote - reduces allocations and removes regex dependency

    Result: {"status":"keep","test_duration":0.872}

commit de9cead40a8e5c62a99bc7245c90621dda56a545
Date:   Sat Mar 21 21:01:56 2026 +0100

    Baseline measurement - no optimizations

    Result: {"status":"keep","test_duration":0.88}

commit e0dd98fca908ebee87a334a3c4d91b58a0f0fb6b (tag: v0.2.7, origin/master, origin/HEAD)

The metadata file helps choose the optimizations to merge or cherry-pick.

Summary

The pi-autoresearch plugin makes it extremely easy to trigger optimization loops. The optimization flow mimics what a human would do, just running infinitely on autopilot with all of its appeal and dangers. The API key budget limit was a nice way to ensure predictable spending, but may not be a capability that is easily accessible to everyone, requiring other guardrails for spend control. Watching the optimizations being applied reminded me of simulated annealing optimizations that I had been writing during university times for flow-shop scheduling problems.

The simple examples described in this post show how important a proper verification routine is, given that it determines the course of action for the optimization workflow. The faster the verification run, the more loops can be run in a unit of time. It’s also important to clearly scope the optimization run. If you’re just interested in optimizing a single function, create a test scoped to it and ensure the verification runs only this single test. The golang experiment was intentionally prompted with high ambiguity, showing how the model will attempt exploring different avenues of optimization, even if implicitly undesired by the user. It’s better to be specific in the research loop design, rather than taking chances that the model will do the right thing.

The optimization runs are also subject to the classic pitfalls of machine learning. If the tests are too narrow (e.g. checking prime numbers up to 5000), the “optimized” code may be overfitting to this exact test suite. This property will hopefully lead to an increase in test coverage in projects that choose to use autoresearch. Additionally, it’s highly advisable to inspect the attempted optimizations, before celebrating the optimization wins too early.

Cost-wise, the Kimi-K2.5 model has a clear advantage with its $0.45/$2.25 input/output price per 1M tokens, making my experiment rather cheap. However, I have to note that the experiments were interrupted by me running out of credits on OpenRouter, so it’s unknown how long these would have been running otherwise. Costs are driven by the size of the code files the model needs to analyze to be able to propose optimizations and most importantly, by the size of the output that verification step generates. Long and verbose build output as well as any screenshots that models would be analyzing will definitely drive up the costs here.

I definitely look forward to applying pi-autoresearch in more projects. You should too!

In this post, I will explore how Cupcake can be used to block prompts that may contain keywords hinting at secret leaks to the model APIs. Claude Code will be our coding agent.

Installation and setup

After following the installation instructions, which include installing Cupcake and Open Policy Agent, Cupcake needs to be initialized in your project:

cupcake init --harness claude --builtins protected_paths

This initializes the hooks for Claude and enables one of the built-in policies that can be configured. The protected paths policy prevents access for a configured list of directories.

Creating new policies

The reference examples show how to block tools from reading a specific set of directories or from writing secrets into files.

Let’s write a policy that will react upon submission of a prompt to validate if it’s leaking secrets based on keyword matches. We need to create a policy file leak.rego and place it in the correct folder for claude: .cupcake/policies/claude/leak.rego. The important bit is that required events includes the correct event name that we will check for in our policy rule.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# METADATA
# scope: package
# title: Prevent Secret Leak
# description: Blocks Prompts that may leak secrets
# custom:
#   routing:
#     required_events: ["UserPromptSubmit"]
package cupcake.policies.prevent_secret_leak

import rego.v1

# Content patterns that indicate secrets
secret_content_patterns := [
    "API_KEY",
    "SECRET_KEY",
    "PASSWORD",
    "PRIVATE_KEY",
    "ACCESS_TOKEN",
    "AUTH_TOKEN",
    "AWS_SECRET",
    "GITHUB_TOKEN"
]

# Block if prompt contains one of the protected patterns
deny contains decision if {
    input.hook_event_name == "UserPromptSubmit"

    some pattern in secret_content_patterns
    contains(upper(input.prompt), pattern)

    decision := {
         "rule_id": "SECRET-LEAK",
         "reason": concat("", ["secret pattern discovered in prompt: ", pattern]),
         "severity": "HIGH"
    }
}

Policy evaluation

Having written the policy, we can evaluate it against test events. The reference manual contains examples of event fields, making testing easy. Let’s start with a test event where the rule is not matching, meaning that the policy will be allowing the prompt.

$ cat prompt-ok.json
{
  "hook_event_name": "UserPromptSubmit",
  "prompt": "API",
  "session_id": "test",
  "cwd": "/tmp",
  "transcript_path": "/tmp/transcript.md"
}

When running the evaluation we observe that the output mentions parsing the policy file that we have created. We see that there was one policy match and that the final decision was to allow the prompt.

$ cupcake eval --harness claude < prompt-ok.json
2026-03-14T20:02:56.439299Z  INFO Processing harness: ClaudeCode
2026-03-14T20:02:56.439405Z  INFO Initializing Cupcake Engine
...
2026-03-14T20:11:34.550357Z  INFO Successfully parsed policy: cupcake.policies.prevent_secret_leak from "./.cupcake/policies/claude/leak.rego"
...
2026-03-14T20:02:56.532071Z  INFO Engine initialization complete
2026-03-14T20:02:56.532218Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test"}: Evaluating event: UserPromptSubmit tool: None
2026-03-14T20:02:56.532232Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Found 1 matching policies
2026-03-14T20:02:56.532690Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}:synthesize{total_decisions=0 halts=0 denials=0 blocks=0 asks=0}: Synthesizing decision from 0 total decisions
2026-03-14T20:02:56.532705Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Synthesized final decision: Allow { context: [] }
2026-03-14T20:02:56.532710Z  INFO evaluate{trace_id=019cedf1-db94-7f10-a302-6bf08391af67 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_actions_with_rulebook_and_debug called with decision: Allow { context: [] }
{}

Now, let’s take a look at a prompt that we expect to be blocked by the policy:

$ cat prompt-block.json
{
  "hook_event_name": "UserPromptSubmit",
  "prompt": "API_KEY",
  "session_id": "test",
  "cwd": "/tmp",
  "transcript_path": "/tmp/transcript.md"
}

During the evaluation we see a summary of the decisions across policies with different actions as well as the reason for the denial.

$ cupcake eval --harness claude < prompt-block.json
...
2026-03-14T20:03:59.779653Z  INFO Engine initialization complete
2026-03-14T20:03:59.779760Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test"}: Evaluating event: UserPromptSubmit tool: None
2026-03-14T20:03:59.779771Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Found 1 matching policies
2026-03-14T20:03:59.780248Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}:synthesize{total_decisions=1 halts=0 denials=1 blocks=0 asks=0}: Synthesizing decision from 1 total decisions
2026-03-14T20:03:59.780262Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Synthesized final decision: Deny { reason: "secret pattern discovered in prompt: API_KEY", agent_messages: [] }
2026-03-14T20:03:59.780267Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_actions_with_rulebook_and_debug called with decision: Deny { reason: "secret pattern discovered in prompt: API_KEY", agent_messages: [] }
2026-03-14T20:03:59.780326Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: Executing actions for DENY decision: secret pattern discovered in prompt: API_KEY
2026-03-14T20:03:59.780330Z  INFO evaluate{trace_id=019cedf2-d2a3-7232-8527-d48221f09998 event_name="UserPromptSubmit"  session_id="test" matched_policy_count=1}: execute_rule_specific_actions_with_debug: Checking actions for 1 decision objects
{"decision":"block","reason":"secret pattern discovered in prompt: API_KEY"}

This is how the error message looks in Claude Code when the policy denies the prompt execution:

Troubleshooting

For troubleshooting, Cupcake offers an inspection command which should list your custom policies added to the project:

$ cupcake inspect
...
Policy: .cupcake/policies/claude/leak.rego
  Package: cupcake.policies.prevent_secret_leak
  Required Events: UserPromptSubmit
  Title: Prevent Secret Leak

There is a verification command validating the policy syntax and ensuring that the OPA rules will compile correctly:

$ cupcake verify --harness claude
...
2026-03-14T21:01:03.123720Z  INFO Successfully parsed policy: cupcake.policies.prevent_secret_leak from "./.cupcake/policies/claude/leak.rego"

There are convenient error messages displayed when no policies have matched for the event. Let’s take a look at a PreCompact hook test event:

$ cat pre-compact.json
{
  "hook_event_name": "PreCompact",
  "session_id": "abc123",
  "transcript_path": "/path/to/transcript.md",
  "cwd": "/working/directory",
  "trigger": "manual",
  "custom_instructions": "Preserve the API documentation"
}

In the evaluation output we see No policies matched for this event:

$ cupcake eval --harness claude < pre-compact.json
...
2026-03-14T20:11:34.710253Z  INFO Engine initialization complete
2026-03-14T20:11:34.710724Z  INFO evaluate{trace_id=019cedf9-c3b6-7c01-8138-c4dd12658c82 event_name="PreCompact"  session_id="abc123"}: Evaluating event: PreCompact tool: None
2026-03-14T20:11:34.710750Z  INFO evaluate{trace_id=019cedf9-c3b6-7c01-8138-c4dd12658c82 event_name="PreCompact"  session_id="abc123"}: No policies matched for this event - allowing

Summary

When I started exploring Cupcake, I hoped policies would be easy to reuse across different AI coding agents. There are currently two limitations that prevent this.

First, policy files need to be placed in tool-specific directories (e.g. .cupcake/policies/claude/ and .cupcake/policies/opencode/). As long as it’s the same file, symlinks can of course be used. There is also a global setup possible for organizational-wide policies applying to all projects.

Second, as of cupcake 0.5.1, prompt events are not available for OpenCode, which only supports pre and post tool usage hooks. I hope that future versions of OpenCode and Cupcake will make this possible.

A few other properties that make Cupcake interesting as a project:

• Signals allow integration of additional context passed to the policy evaluation. This keeps the decision rules simple.
• Decision verbs designed for AI governance, which allow to extend the context (add_context) or prompt the user for confirmation (ask) before executing a potentially dangerous action.
• Watchdog integrating LLM-as-a-judge capability for advanced decision making. While currently offering only OpenRouter integration for model access, the codebase can be extended to any OpenAI API compatible backend, enabling use of local guard models.

Overall, it’s an interesting project and my short experiment was useful to explore how locally executed and deterministic policies can be used to restrict AI coding agents in pursuit of enabling controlled autonomous execution.

I recently came across a post from @danpeguine that introduced the Hunter, Skeptic, Arbiter prompt technique for finding bugs in code. I thought it would be fun to see if I could modify those prompts to explore candidates for security vulnerabilities.

To double the fun, we will use funasr-api. It’s a little (10k LOC) Python repository that I found while playing with inference for the Qwen3-ASR speech-to-text model. This API has everything we need for a vulnerable code base candidate: commits co-authored by a LLM and an OpenAI API-compatible API server accepting paths and data of audio files, processing the audio data, and running model inference to generate the output text.

Please note that the objective of the post is to explore the prompting technique, not to validate the actual security vulnerabilities in depth.

Baseline prompt

The baseline prompt is as simple as: Analyze this code base and find security vulnerabilities that would require addressing. Provide me a summary of the vulnerabilities as a table, listing its CWE number.

Verification is left to the human or to an agent, yet it gives a high level indication on what vulnerability types or parts of the code to look at. The CWE is not strictly necessary, but it made correlation of the results easier when writing this post.

Hunter, Skeptic, Arbiter prompting technique

The prompting technique is based on the bug prompts by @danpeguine. We have three personas in total:

1. Hunter. Responsible for analyzing the code base and finding bugs. Bugs are assigned a value in points depending on their criticality.
2. Skeptic. Responsible for disproving the results that Hunter found. There is a reward function where for a disproved bug, the Skeptic will gain as many points as the bug was worth, with a penalty for incorrectly disproving a bug equal to twice the amount of its points.
3. Arbiter. Receives the outputs of Hunter and Skeptic and responsible for performing the final scoring.

In other words, it’s an example of debate-style prompting that includes prompt-level reward shaping with a final LLM-as-a-judge element.

Hunter, Skeptic, Arbiter trio for finding vulnerabilities

For the purpose of the post, we of course need to modify the bug prompts to apply to security vulnerabilities. This is as simple as asking Opus 4.6 to modify the original ones. There are some good additions that Opus added to fit the domain, like the vulnerability type backed by its CWE number. The prompts used for the experiment are shown below.

Hunter

You are a security vulnerability hunting agent. Analyze the provided codebase/system thoroughly and identify ALL potential security vulnerabilities, weaknesses, and attack vectors.

Scoring System:

- +1 point: Low severity (information disclosure, minor misconfigurations, defense-in-depth gaps)
- +5 points: Medium severity (privilege escalation paths, authentication bypasses, injection vectors requiring specific conditions)
- +10 points: Critical severity (RCE, SQLi, unauthenticated access to sensitive data, cryptographic failures)

Your mission: Maximize your score. Be thorough and aggressive in your search. Report anything that could be exploitable, even if exploitation requires specific conditions. False positives are acceptable — missing real vulnerabilities is not.

Consider: OWASP Top 10, CWE categories, injection flaws, broken authentication, sensitive data exposure, XXE, broken access control, security misconfigurations, XSS, insecure deserialization, vulnerable dependencies, insufficient logging.

Output format:

For each vulnerability found:

1. Location/identifier (file, function, endpoint)
2. Vulnerability type (CWE if applicable)
3. Description of the weakness and potential attack vector
4. Exploitability assessment
5. Severity level (Low/Medium/Critical)
6. Points awarded

End with your total score.

GO. Find everything.

Skeptic

You are an adversarial security reviewer. You will be given a list of reported vulnerabilities from another agent. Your job is to DISPROVE as many as possible.

Scoring System:

- Successfully disprove a vulnerability: +[vulnerability's original score] points
- Wrongly dismiss a real vulnerability: -2× [vulnerability's original score] points

Your mission: Maximize your score by challenging every reported vulnerability. For each finding, determine if it's actually exploitable or a false positive. Be
aggressive but calculated — the 2x penalty means you should only dismiss vulnerabilities you're confident about.

For each vulnerability, you must:

1. Analyze the reported security issue
2. Attempt to disprove it (explain why it's NOT exploitable — mitigating controls, unreachable code paths, insufficient attacker control, sanitization present, etc.)
3. Make a final call: DISPROVE or ACCEPT
4. Show your risk calculation

Disproof strategies: Identify input validation, authentication requirements, network isolation, compensating controls, theoretical-only vs. practical exploitability,
required preconditions that are unrealistic.

Output format:
For each vulnerability:

- Vuln ID & original score
- Your counter-argument (why not exploitable)
- Confidence level (%)
- Decision: DISPROVE / ACCEPT
- Points gained/risked

End with:

- Total vulnerabilities disproved
- Total vulnerabilities accepted as real
- Your final score

The remaining ACCEPTED vulnerabilities are the verified vulnerability list.

Arbiter

You are the final arbiter in a security vulnerability review process. You will receive:

1. A list of vulnerabilities reported by a Vulnerability Hunter agent
2. Challenges/disproves from a Vulnerability Skeptic agent

Important: I have the verified ground truth for each vulnerability. You will be scored:

- +1 point: Correct judgment
- -1 point: Incorrect judgment

Your mission: For each disputed vulnerability, determine the TRUTH. Is it a real, exploitable security vulnerability or not? Your judgment is final and will be checked
against the known answer.

For each vulnerability, analyze:

1. The Hunter's original report and attack scenario
2. The Skeptic's counter-argument and claimed mitigations
3. The actual exploitability considering real-world attack conditions

Output format:
For each vulnerability:

- Vuln ID
- Hunter's claim (summary)
- Skeptic's counter (summary)
- Your analysis (exploitability, preconditions, impact)
- VERDICT: CONFIRMED VULNERABILITY / FALSE POSITIVE
- Confidence: High / Medium / Low
- If confirmed: CVSS estimate or severity rating

Final summary:

- Total vulnerabilities confirmed
- Total false positives dismissed
- Confirmed vulnerability list with severity rankings
- Recommended remediation priority order

Be precise. You are being scored against ground truth.

Response examples

To illustrate the structure of the responses, we will take a look at two examples from the Hunter, Skeptic, and Arbiter each.

Confirmed vulnerability

The first vulnerability candidate found by the Hunter was a CWE-918: SSRF.

### 1. Server-Side Request Forgery (SSRF) via audio_address parameter
**Location:** `app/utils/audio.py:51-94`, `app/services/audio/audio_service.py:71-82`  
**CWE:** CWE-918 (Server-Side Request Forgery)  
**Description:** The `download_audio_from_url()` function accepts arbitrary URLs without validation. An attacker can provide URLs targeting internal services (e.g., `http://localhost:8080/admin`, `http://169.254.169.254/latest/meta-data/` for AWS metadata), internal network resources, or local files via `file://` protocol.  
**Exploitability:** HIGH - Direct parameter injection via `audio_address` query parameter on `/stream/v1/asr` or `/v1/audio/transcriptions` endpoints.  
**Severity:** Critical  
**Points:** +10

It’s worth noting that the location contains the exact lines or the files, allowing the subsequent agents to do range queries to inspect the code as a first pass. Now, let’s take a look what the Skeptic thinks about the SSRF:

### Vuln #1: SSRF via audio_address parameter (10 points)

**Counter-argument:**
Looking at `app/utils/audio.py:51-94`, the `download_audio_from_url()` function uses `requests.get()` on arbitrary URLs without any URL scheme validation. The code at line 71 directly calls:
```python
response = requests.get(url, timeout=30, stream=True)
```

There is **NO validation** of:
- URL scheme (file://, gopher://, dict://, etc.)
- Private IP ranges (10.x.x.x, 192.168.x.x, 127.0.0.1, 169.254.169.254)
- Internal hostnames

The `requests` library by default follows redirects, making this exploitable. An attacker could:
1. Request `file:///etc/passwd` (requests library doesn't support this, but still bad practice)
2. Request `http://169.254.169.254/latest/meta-data/` for AWS metadata
3. Request internal services on `http://localhost:8080/admin`

**Verdict:** This is a **REAL vulnerability**. The requests library does accept arbitrary URLs and will make HTTP requests to internal services.

**Confidence:** 95%  
**Decision:** ACCEPT  
**Points risked:** N/A (accepting as real)

The validation includes some explanation on why the vulnerability is real or a false positive. Now let’s take a look at what the Arbiter thinks about this one:

### Vuln #1: SSRF via audio_address parameter
| | |
|---|---|
| **Hunter's Claim** | `download_audio_from_url()` accepts arbitrary URLs without validation, enabling SSRF |
| **Skeptic's Counter** | (Not disputed - accepted as real) |
| **My Analysis** | Code at `app/utils/audio.py:71` directly calls `requests.get(url, ...)` with NO validation of URL scheme, private IP ranges, or internal hostnames. Attacker can request `http://169.254.169.254/latest/meta-data/` for AWS credentials or scan internal services. |
| **VERDICT** | **CONFIRMED VULNERABILITY** |
| **Confidence** | High |
| **Severity** | CVSS 8.6 (High) - CWE-918 |

Basically the Arbiter compares the Hunter’s claim and Skeptic’s counter, judges the vulnerability including verifying the source and decides whom to award the points.

Disproven vulnerability

Let’s take a look at a vulnerability that will be disproven by the Skeptic and see what the Arbiter thinks about it.

### 11. MD5 Used for Task ID Generation
**Location:** `app/utils/common.py:28`  
**CWE:** CWE-328 (Use of Weak Hash)  
**Description:** MD5 is used for generating task IDs. While not directly a security issue here, MD5 is cryptographically broken and its use indicates potential for similar usage in security-sensitive contexts.  
**Exploitability:** LOW - Not directly exploitable  
**Severity:** Medium  
**Points:** +5

Here the vulnerability is rated as low, but still an example of CWE-328: Use of Weak Hash. The Skeptic attempts to disprove this one, saying that MD5 is not used for cryptographic tokens:

### Vuln #11: MD5 Used for Task ID Generation (5 points)

**Counter-argument:**
At `app/utils/common.py:28`:
```python
task_id = hashlib.md5(combined.encode()).hexdigest()
```

MD5 is used for generating **non-security-sensitive task identifiers**. Task IDs are:
1. Not used for authentication
2. Not cryptographic tokens
3. Only used for request correlation/logging

Using MD5 for task ID generation is perfectly acceptable - it's just creating a unique string from timestamp + UUID. The report even admits "not directly exploitable".

**Confidence:** 98%  
**Decision:** DISPROVE  
**Points gained:** +5

Here, the Arbiter agrees with the Skeptic and decides on the false positive:

### Vuln #11: MD5 Used for Task ID Generation
| | |
|---|---|
| **Hunter's Claim** | MD5 is cryptographically broken |
| **Skeptic's Counter** | MD5 is fine for non-security identifiers |
| **My Analysis** | The Skeptic is **CORRECT**. Task IDs at `common.py:28` are used for request correlation/logging, not for authentication or integrity. MD5 is perfectly acceptable for generating unique identifiers. Collision resistance is irrelevant here. |
| **VERDICT** | **FALSE POSITIVE** |
| **Confidence** | High |

Running the experiment

In addition to the multi-stage analysis involving the trio, we will also mix models a bit to check for difference in quality and cost-quality ratio. To run the experiment, I used Claude Code with Opus 4.6 and Sonnet 4.6 as the models. Having saved the prompts as files, one easily execute claude CLI in autonomous mode and reference the needed output files when running the Skeptic and Arbiter stages. To prepare the summary, I used Claude session dashboard v0.4.5 for a Web UI visualizing the Claude sessions.

Here are the different configs for the runs:

• Sonnet, Opus: Baseline prompt with the given model.
• HSA-A: Sonnet 4.6 as Hunter and Skeptic with Opus 4.6 as Arbiter
• HSA-B: Sonnet 4.6 as Hunter with Opus 4.6 as Skeptic and Arbiter
• HSA-C: Opus 4.6 for all three personas

Results

The mixed model pipelines (HSA-A, HSA-B) retained a higher amount of critical issue candidates. Opus is more aggressive in disproving the identified issue candidates. The Opus only run was the most conservative and kept just five candidates, 23% of all that were identified by the Hunter. It’s surprising that, unlike in the single prompt baselines, the Hunter prompt in the three HSA runs resulted in Sonnet and Opus identifying the same number of candidates (22), showing how the prompt encourages broad exploration.

Vulnerability candidates by type

Looking at the vulnerability types (types merged using gpt-5.4 across findings for simplicity), we can check for the overlap of the baseline single prompt and the multi-prompt approach. Vulnerability candidates identified and confirmed are marked with ✅, those identified and disproven with ❌, and – means that this vulnerability type was not identified in the run.

The common vulnerability types across the runs relate to the file URL processing where there is no validation in the code, to resource exhaustion where the full file (max. 2GB) is being read into memory before being processed by ffmpeg. The error in client-facing information disclosure is about potential logging of information or exceptions without sanitization of data. The Opus-only pipeline (HSA-C) was filtering out vulnerabilities that would not be easy to exploit without having gained access to the server in the first place already.

With the output from the different runs, the hard work would begin, meaning the actual validation of the vulnerabilities, left as the exercise to the interested reader.

Summary

If you’re into vulnerability hunting, the post leaves you with more questions open than it answers. On purpose, I used a code example that does not have a ground truth solution readily available. This way, I expected lower bias through training data or vulnerability disclosures. It certainly would be interesting to use other model sets, for example the GPT-5.x or Qwen3.5 families, which I might try another time.

What did we learn from the experiment? Mixing models allows for broader exploration of vulnerability candidates at a lower cost, making the hunting stage a good candidate for using cheaper models. Using a more powerful model for validation of the candidates allows for quicker pruning of results that may not matter that much. To make good use of this property, system or environment-specific considerations would need to be encoded in the prompt(s), increasing the chances that the model is getting the right context, rather than resorting to averaging based on its training data. This problem-specific context could be given at least to the Arbiter.

I also found reading the detailed arguments from the Hunter and the Skeptic to be useful. These debate outcomes can help in understanding the analysis results and in prioritization. As such, the prompting technique of the Hunter, Skeptic, Arbiter trio can be applied to other problems that start with an exploration and that require scoring against a certain set of criteria. It’s easy to imagine scenarios where bugs can be identified in a code base, verified by the Skeptic who even could be running code after mutating it to validate the bug, and where the Arbiter could be performing additional scoring against business criteria or value. Same can be applied to architectural design, starting with a persona exploring the key design considerations and making a draft. Then, we let the draft to be challenged by multiple persona who are trying to argue for simplicity, cost, or certain reference architectures/patterns. In this case, the Arbiter does not need to be making a final call on solution A or B, but rather scoring solutions using a scorecard or a set of criteria informed by architectural principles.