aiDevCon: Exploring the building blocks of Software Factories

I went to aiDevCon NYC with a key question that I was looking for insights on: How many software factories does a large company need? This post is a quick recap of the ideas from a few of the talks that were relevant to this question.

The Software Factory and its parts

The presentations shed light on the factory concept from many angles, typically showing the parts needed to eventually build a fully autonomous, self-improving system:

• set of skills (and related harness config) that standardize processes and workflows,
• issue to PR automation,
• issue to deploy automation,
• the mindset of improving the process over manually handling edge cases,
• platform foundations helping deliver any of the above.

What do teams actually need to build first?

Guy Podjarny from tessl.io promotes the transition from skills to loops to factories and factories scoped to product level. This allows teams to gradually improve their processes and reap the benefits of automation. The tessl.io product can be connected to GitHub repositories and enables automation by orchestrating the issue to PR workflow along with a code review bot. Afterwards, additional workflows can be defined as (maintenance) loops, for example a loop that summarizes PR review feedback and suggests adjustments to code review rules that help address these gaps in a systematic way. During the talk, it was stated as essential that these loops become self-improving: extracting information from observations, optimizing across quality, cost, security dimensions, and keeping skills up to date with model and system evolution.

During one of the workshops, I gave the tessl.io factory a spin. The UI offers an admin panel for linking the factory with the git repo, configuring, and inspecting loop configuration and runs. The CLI helps coding agents inspect the state of the factory and drive further improvements. The demo we worked on was steered by a tutorial.md written for agents, which ensured smooth setup. The workshop anchored on a Rollercoaster Tycoon themed carnival park, deployed on GitHub Pages and playable in the browser. Common feature requests to add new theme park games or modify the UX could be raised as GitHub issues and worked on by the factory. Skills supported these common tasks and their one-off background execution by the factory. I only learned after the workshop that we had been trying out product features that were freshly released just ahead of the conference. Kudos to the team for their execution and great tutorial!

Is the factory concept rediscovering what we already have?

Bryan Finster highlights that the same types of collaboration friction exist in teams as they did years ago when continuous delivery was gaining popularity. Any delivery friction eats up all the gains from faster code generation. It’s important to address bottlenecks to timely delivering work that the team worked on in a given day. Teams that addressed such bottlenecks and embraced continuous delivery practices, have largely already built the prerequisites and/or large pieces of the factory approach and can now focus on selective upgrades. He stresses that the mindset of continuous improvement is essential. If the factory makes a mistake, it’s key to tune the factory (linters, skills, style guides, etc.) instead of just manually fixing the mistake in code. Lastly, his talk provided guidance on clear division of responsibilities: engineers owning definition of done (intent and acceptance criteria), quality controls (defining scope and rules for review agents), and production outcomes (what users experience). Every problem in production must be a feedback loop for hardening the quality controls. Agents are responsible for drafting the spec and plan according to intent, executing on the plan by building code that passes the quality gates, and the deployment. Bryan also shared his current factory setup, the devteam.

Can a factory be trustworthy and long lived?

Daniel Meppiel from Microsoft was promoting an approach where the software factory is versioned. His apmx toolkit helps run agentic software factories from markdown contracts and produces receipts that help verify the apm dependencies and factory definition digests. This helps with traceability, verification, and thanks to apm with reproducibility as each run shares the same set of instructions, skills, prompts, agents, hooks, plugins, MCP servers.

George Fahmy from Vercel was speaking about the pains of writing SDKs to integrate with the different harnesses and the pursuit to have a single Harness API. He also showed afk as an app built using the Harness API that allows to move sessions between local and cloud environments.

Steve Yegge was showing how his environment and working patterns look like when implementation costs are negligible. His $10k/month subscription budget helps him test patterns that will be available to the mainstream in a year or two, provided that token costs continue their downtrend. Near-infinite token budgets still can’t overcome limitations of LLMs around creativity. His agent crews with named personas and specializations trigger often absurd situations where agents refuse work, are anxious and moody, afraid of not doing their job well, rigidly follow some sort of constitution they invented to apply rules, and many more human-like behaviors he captures in the form of comic strips. Still, as Steve emphasizes, the bottleneck is taste and judgement. Agents do not exhibit the level of creativity that humans do and it’s necessary to direct agents towards the creativity levels that they cannot reach themselves.

What do we need to fundamentally rethink to run factories?

Erik Meijer spoke of Futamura Projections (wiki, tl;dr in types) with the 3rd projection being equal to RSI (Recursive self-improvement) now practiced by the leading AI labs. He made bold projections that dependencies can now be dropped easier than it was ever possible. One can just ask for an implementation of certain technical patterns from scratch, rather than depending on libraries that pose supply chain risk. Drawing parallels to the industrial revolution, LLMs can help in partial evaluation of existing frameworks to their simpler, more narrowly scoped versions, generating programs that are specialized on a per use-case level, thus cheap and personalized.

Those who get the factory pieces running uncover new exciting bottlenecks. Reliance on A/B testing is a common one where teams can now build features quicker than they’re able to validate in production with end users. The more features they test in parallel, the longer it takes for the experiments to reach statistical significance. Simulated synthetic personas to the rescue. Patrick Debois was showing emerging patterns and challenges that are in need of solving as we enable concurrent delivery. Agent Kanban Boards help set limits on per-feature work that can be done in parallel; agents can still align on a different work split between themselves. Next comes conflict resolution for agents where conflict is detected and agents agree on means to resolve it, adapting their local plans of work. High amount of work churned by agents also requires merge train handling with speculative merge queues that are composed of sets of changes that can merge without conflicts, maintain original intent, and ensure the tests are green. Patrick also highlighted the need for a ledger with decisions made and automatic summaries of work done and deployed.

The merge train reminded me of speculative execution and branch prediction in CPUs. Thinking further, even the flow-shop scheduling problem where optimal scheduling of tasks is determined comes to mind. It remains to be seen whether the CI/CD stage is the right one to validate the changes or whether we’d be better served by flipping feature flags in production, attempting to automatically find and validate the right combination of flags against the system’s SLO targets.

What’s needed for factories to succeed?

Feeding the context layer

Birgitta Boeckeler spoke about context engines and what these can today handle. Indexing of knowledge spread across different systems, processing snippets without any common structure or categorization, and dealing with outdated or conflicting information using signals helping with conflict resolution are all manageable tasks for the engines of today. Still, human oversight is required for writing down knowledge (or marking it as such), capturing thoughts and decisions, making information available to engines, and monitoring for sources or individual docs that impede conflict resolution. Birgitta also shared data from Unblocked on the most overruled document sources being Wikis and Tickets:

Adjusting to stay on the frontier

Model providers, as presented by Nicolas Carmont Zaragoza from Anthropic, recommend starting from scratch every time a new model is released to ensure that there is no bloat in skills that may be confusing the model or causing inefficiencies. Upon every model launch, four steps are recommended: (1) un-harnessing by cutting stale rules, forced verification and tool-use examples, (2) re-baselining through running evals, sweeping effort levels, tracking cost and latency, (3) re-harnessing by adding only what the model can now use, like a checklist or time budget, (4) shipping the updated setup as long as evals hold.

Great in theory, but much harder to achieve in practice for individual teams even those with ambition to switch between models and providers. Building such baseline evals takes time and effort and real-life coding work is harder to evaluate against than typical SWE benchmarks. I guess the simplest approach would be taking a few prior snapshots of the code, applying the updated harness setup, and verifying how close the output is to the previously known state. This type of repository time travel and automatic evaluations would be a great platform capability to have. Not all teams have resources to invest in such sophisticated setups and settle for the bare minimum. Massimo Re Ferre’ shared that Amazon’s Kiro team invests more than 20% of capacity into building the infrastructure of their team’s software factory.

Investing in the platform layer

Yet, teams need infrastructure to run autonomous agent loops, either acquired from vendors or self-built. In my talk Scaling Agentic Engineering at Zalando, I shared how we enable Zalando teams with infrastructure for interactive and autonomous agent loops:

• risk-based PR scoring assessing readiness to merge in terms of level of risk for deployment,
• agent platform enabling remote agent sessions using kagent and agentgateway,
• Zalando’s OSS agentic identity broker responsible for giving agents safe, scoped, exchangeable, and auditable access to third-party services,
• publishing workflows and hosting of applications helping non-engineers to build and deliver software.

Citizen developer enablement was also framed as developing software for one where Matthias Lübken was promoting a simple software stack based on self-hosted, distributed durable objects (celld.dev). Durable Objects are also used by Codex for their Sites feature. One interesting idea shared in the talk was that while agents can build apps for humans, building apps could also become a form of communication between agents since its computer use and image understanding might be better suited for some use cases than text.

Summary

Coming back to the question How many software factories does a large company need?, the insights from the conference hint at a funnel. The closer to the specification stage, the higher the differentiation will be across teams, products, and stacks. The closer to the deployment and validation stage, the more standardized the factory can be, building and shipping containers of tested and validated features. The following layers emerge:

• models providing baseline intelligence,
• skills encoding the know-how and domain knowledge that the models do not know yet,
• loops as the repeatable workflows that help maintain quality guardrails, protect design or system-level invariants,
• platform foundation with infrastructure that provides the factory templates that can be customized or instantiated by individual teams, provides access to models (gateways), knowledge and data (context or APIs), and other infrastructure that helps run the factory in an efficient way.

As models evolve (and as AI labs push us), the skill layer is bound to be continuously simplified and hollowed out, reducing it to the bare minimum that nudges models in the right way. Loops are where teams wire up the skills, defining triggers, sequencing, thus helping each contributor to follow the desired workflow steps. We’ve seen this in the Kubernetes project with the different manually invoked workflow steps triggering GitHub bots. Decision models come to mind to help provide an even smoother experience. Working examples use tools like GitHub Agentic Workflows for development workflow automation, e.g. running spec-kit based workflows in the Agentic Identity Broker. Workflows, on the other hand, will be eaten up by and replaced by the evolving platform infrastructure.

Platforms will continue to abstract the loops built by individual teams and bundle these into easy-to-use templates or configurable tools, borrowing ideas from apm. Platform teams have a crucial role to play here. If done right, there is a great opportunity to provide leverage to teams and ground them in best practices that cannot be skipped anymore. Just imagine that the generation of IaC manifests is supported by skills provided by the platform team. It’s a perfect candidate for a skill to be centralized. Many of the well-known and already adopted deterministic tools can be used to build reliable factories. Stable CI/CD systems, gradual deployment features with automated traffic switching and monitoring of deployments against guardrail SLOs can all be done in deterministic ways. These proven tools will, however, be more and more challenged in terms of parallelism and required speed of delivery, especially for common workflows like dependency or security updates. Agentic approaches alone won’t be enough to solve the upcoming challenges.

Teams that want to adopt the software factory approach successfully need to continuously adjust their ways of working:

• set foundations with strictly configured linters, quality and architecture guardrails, and automated testing at multiple levels,
• adopt a continuous improvement mindset where the fix goes into the machine (codified as guardrails and workflow definitions) instead of applying one-off fixes,
• move away from interactive sessions for feature delivery, since they undermine automation (fine for exploration),
• move coding sessions to the cloud as a prerequisite for issue-to-PR automation and safe, long-running agent sessions,
• converge on reproducible harness setups across the team with well-defined (e.g. plugin marketplace) or versioned configuration (apmx),
• meticulously document design decisions to ensure consistency in agent decisions, increase trust in the system, and set the groundwork for (formal) verification against intent or domain guardrails,
• embrace regenerative software letting go of any attachment to certain code constructs or language choices, being open to software being rewritten many times.

Aside from the last point, these thoughts are not hugely controversial and often already best practice across many teams. Execution is what differentiates teams. It’s easy to start off with high discipline and drop the ball over time as delivery pressure mounts. Similarly, it’s easy to skip automation as not required yet and miss out on the productivity gains.