A blueprint for a dark factory: the infrastructure behind an AI software factory
- ai
- gcp
- architecture
A dark factory is a factory that runs without people on the floor: the lights can stay off, because nobody needs to see. The software version of that idea is a system where an issue goes in and a reviewed pull request comes out, and eventually a deploy to production, without a developer typing the code.
Most articles about coding agents are about prompts. This one is about the infrastructure around them, because that's where most setups fall short. A coding agent on a laptop, or on a VM that runs all day, doesn't scale, isn't isolated and costs money while it does nothing.
What follows is a blueprint for one: a factory that picks up issues, asks the product owner questions, writes a spec and failing acceptance tests, implements the change and has it reviewed by another agent. Its infrastructure comes down to a deterministic orchestrator that starts short-lived workers, where each worker plays exactly one role. It costs close to nothing when nobody is using it.
The three building blocks
The whole factory rests on three concepts. It pays to name them precisely, because they each map to a different piece of infrastructure.
- Orchestrator: plain code that keeps the state of every issue and starts the next step when something happens.
- Worker: one container image with an entrypoint, which executes a role for one task and then disappears.
- Role: a versioned configuration that defines one kind of agent work: prompt, model, allowed tools, writable paths and output schema.
A task is one execution of one role for one issue, in a fresh worker. "The spec agent" or "the review agent" is shorthand for a role during a task, not a separate service.
Around these three sit two outside systems the factory works with: an issue tracker, where people write issues and answer questions, and a Git platform, with the repository, pull requests and CI. I use Linear and GitHub, so those are the examples in this article, but nothing in the design depends on them. Jira and GitLab fit just as well; I'll come back to how in principle 4.
Principle 1: the orchestrator is not an agent
It's tempting to make the orchestrator an agent too: a "manager" model that decides what happens next. I deliberately didn't. Routing, limits and the kill switch have to be predictable, auditable and impossible to talk out of. None of it needs judgement, so none of it gets a language model.
The orchestrator is a state machine per issue, split into a pure core and a thin shell. The core is one function:
decide(state, event, ctx) -> { state, commands }
It takes the current state of an issue and an event (an issue delegated in the tracker, a CI result from the Git platform, the result of a task) and returns the new state plus a list of commands: start a task, post a comment, set a status. All flow logic lives there and is covered by plain unit tests. The shell around it does the I/O: it stores state, runs commands and talks to the issue tracker, the Git platform and Cloud Run.
This choice also determines the cost. Handling an event takes milliseconds, so the orchestrator can scale to zero between events.
Principle 2: one task per container
Every task gets a fresh container: a clean checkout of the target repo, no state left over from an earlier task, and nothing left behind afterwards. That gives isolation and reproducibility for free. If a task misbehaves, the next one doesn't notice.
The second half of this principle is that a worker never waits. When the spec role has questions for the product owner, it doesn't keep a container running for two days until someone answers. It returns its questions, the task ends, and the orchestrator posts them in the issue tracker. The answer is a new event, which starts a new task with the questions and answers as an input file. I considered resuming the agent's session instead, but starting over is simpler, easier to reproduce, and costs nothing while a human thinks.
In order, a worker does this:
- Load the role configuration.
- Clone the target repo and check out the issue's branch.
- Write the task inputs (the issue, earlier decisions, CI output) as files.
- Run the role's setup commands, such as
npm ci. - Run the role's phases through the Claude Agent SDK.
- Validate the agent's output against the role's JSON schema.
- Check the diff: did the agent only change paths it's allowed to change?
- Commit, push, and open or update the pull request.
- Publish the result to the orchestrator.
Note who does steps 8 and 9: the worker, not the agent. The agent never holds a token for the Git platform. It finishes its work, and the worker's own code decides what gets committed.
Principle 3: a role is configuration, not code
There's one worker image. What makes a task a spec task or a review task is the role configuration it loads. Adding a role means adding a directory with a prompt, a role.json and an output schema, plus one line in Terraform for its Cloud Run Job.
This is the implement role, slightly trimmed:
{
"model": "claude-sonnet-5-5",
"maxTurns": 150,
"maxBudgetUsd": 8,
"phases": [
{ "name": "plan", "prompt": "phase-plan.md", "allowedTools": ["Read", "Glob", "Grep"], "outputFile": "intent/{issue}/plan.md" },
{ "name": "build", "prompt": "phase-build.md" }
],
"allowedTools": ["Read", "Glob", "Grep", "Write", "Edit", "Bash(npm ci)", "Bash(npm test *)", "Bash(npm run lint)", "Bash(npm run build)"],
"disallowedTools": ["WebFetch", "WebSearch", "Bash(git push *)", "Bash(gh *)", "Bash(curl *)"],
"protectedPaths": ["tests/acceptance/", ".factory/", ".github/", ".claude/", "intent/", "CLAUDE.md"],
"subagents": {
"verifier": { "description": "Runs the tests, lint and typecheck, and reports what fails.", "prompt": "verifier.md" }
},
"outputSchema": "output.schema.json"
}
Everything that matters about a role is in that one file: which model, how many turns and dollars it may spend, which tools it may use, which paths it may never touch, and what its answer must look like. The plan phase can only read; the build phase can write. The acceptance tests the spec role wrote are protected, so the implementer can't make them pass by changing them.
The blueprint starts with three roles:
| Role | Starts when | May | Delivers | Model |
|---|---|---|---|---|
| spec | An issue is delegated to the factory, or answered | Read the repo; write the spec and acceptance tests | Spec, failing acceptance tests, questions, a proposed risk class | Opus |
| implement | The final spec run has no more questions | Plan read-only, then edit and run the repo's own commands | Plan, code, a pull request ready for review | Sonnet |
| review | CI is green | Read only | Findings on two axes: standards and spec | Opus |
Two details make this work. First, every role returns structured output that matches a JSON schema. The orchestrator calculates with the result instead of reading prose: a review returns a list of findings, each a fix, a decision for a human or a remark, and the verdict follows from that list. Second, the review runs on a different model than the implementer, without its plan or transcript. The reviewer judges the code and the spec, not the implementer's story about them.
Principle 4: the tracker and the Git platform are adapters
The orchestrator's core doesn't know Linear or GitHub. It knows events such as issue delegated, issue answered, ready for build and CI completed, and commands such as post a question, set a status and start a task. Translating between those and a real system is the job of an adapter at the edge:
- Inbound, a webhook route per system verifies the signature and turns the payload into a factory event.
- Outbound, a tracker port and a pull request port execute the commands against the system's API.
That keeps the choice of tools a configuration choice. It also makes a stub tracker possible: one that keeps issues in Firestore and is driven from the command line. With it you can run the whole factory end to end before you connect a real issue tracker, and the core runs the same on both.
Swapping systems means writing these adapters, and mapping each factory concept onto what the system offers. For the issue tracker:
| Factory concept | Linear | Jira |
|---|---|---|
| Hand an issue to the factory | Delegate it to the agent app | Assign it to a factory user |
| Ask a question, get an answer | A comment in the agent session | A comment with a mention |
| Refinement, Ready for build | Workflow states | Workflow statuses, through transitions |
For the Git platform:
| Factory concept | GitHub | GitLab |
|---|---|---|
| The proposed change | Pull request | Merge request |
| A CI result | check_suite webhook |
Pipeline webhook |
| A token for one task and one repo | GitHub App installation token | Project access token with the narrowest scope and shortest expiry |
| The merge gate as required approver | Branch protection, merge queue | Approval rules, merge trains |
Outside the adapters, only two things depend on the Git platform. The roles protect the CI configuration from the implementer (.github/ on GitHub, .gitlab-ci.yml on GitLab), and the worker image needs the Git platform's CLI or API client to open the pull or merge request. The agent itself never talks to either system, so its prompts and tools stay the same.
The blueprint on Google Cloud
This is the core of the design: which Google Cloud resource does which job, and why it costs (almost) nothing when idle.
| Resource | Job in the factory | Idle cost |
|---|---|---|
Cloud Run service orchestrator |
Receives webhooks and Pub/Sub pushes, runs decide() |
0 to 1 instances, CPU only while handling a request |
Cloud Run Job per role (factory-spec, factory-implement, factory-review) |
One execution per task | Billed only while a task runs |
Pub/Sub topics events, task-results and dead-letter |
The bus between everything | Per message |
| Firestore | Issue state, the outbox, the task slot | Within the free tier |
| Secret Manager | The model API key, the Git platform and tracker credentials, webhook secrets | Within the free tier |
| Cloud Scheduler | A sweep every 10 minutes | One job |
| Artifact Registry | Worker and orchestrator images | Keeps the newest 5 |
The orchestrator is a Cloud Run service with at most one instance, 1 CPU and 512 MiB. One instance is plenty: it only routes events, and a single instance keeps reasoning about concurrency simple. Each role is a Cloud Run Job with 2 CPUs, 4 GiB of memory and a timeout of an hour. That's enough to install dependencies, run a test suite and let the agent work. Cloud Run's own retries are off: a crashed task counts as a failed attempt, and the orchestrator decides what happens next.
What the blueprint deliberately leaves out matters just as much. There's no Kubernetes cluster, no NAT gateway, no VPC connector, no Cloud SQL and no instance that's always on. Each of those would put a fixed monthly bill under a factory that, most of the time, is waiting for a human to answer a question.
In practice, an idle factory costs less than €10 a month. Cloud Run's free monthly allowance and the free tiers of Pub/Sub, Firestore and Secret Manager absorb most of what little it does between issues. The real cost is in the model tokens a task uses, and the role configuration caps those per task.
Everything is in Terraform. A human applies it; CI only checks terraform fmt and validate. New images roll out through the factory's own CI (GitHub Actions, GitLab CI), which authenticates with Workload Identity Federation, so there's no service account key stored in CI. Terraform ignores the image field, so a plan stays clean after a rollout.
Following one issue through the factory
To see how the pieces connect, follow one issue: WEB-1, "show the build version in the footer".
- The product owner hands WEB-1 to the factory in the issue tracker. The tracker sends a webhook to the orchestrator, whose adapter checks the signature, turns it into an issue delegated event and publishes it on the
eventstopic. It answers right away, because it does nothing else yet; Linear, for example, gives a webhook five seconds. - Pub/Sub pushes the event back to the orchestrator. It checks the push's OIDC token and applies the event in one Firestore transaction: the new state of WEB-1, the task slot, the commands
decide()returned and the event ID. - The orchestrator runs the commands. It sets the status to Refinement, mints a Git platform token that's valid for this one repo, and starts an execution of
factory-specwith the task input as a container override. - The worker runs the spec role. It pushes a branch with the spec and failing acceptance tests, opens a draft pull request, and publishes its result on
task-results: three questions for the product owner and a proposed risk class. - The orchestrator validates the result against the spec role's schema and decides the next step: post the questions in the issue tracker and wait.
From there it's the same loop with different events. The answer starts a new spec task. The product owner moves the issue to Ready for build, which starts a final spec run and then the implement role. A CI result for the pushed commit either starts the review, or sends the implementer back with the failing log. Each step is an event in, a decision, and a task out.
Reliability without servers that stay up
A system that scales to zero and starts containers on demand must stay correct when something crashes halfway. There's no long-running process that remembers what it was doing, so the design has to remember for it.
- Outbox. The state change and the commands it causes are written in the same transaction, and then executed in order. If a command fails, the push answers with an error, Pub/Sub delivers the event again, and the orchestrator continues from the failed command. Starting a task is the one command that must never happen twice: before it launches, it checks whether an execution for that task already exists.
- Idempotency. Every event has an ID, and an event that was already applied changes nothing. Pub/Sub doesn't guarantee order, so
decide()also has to handle events that arrive in an unexpected order, such as a blocker that's done before the issue was even delegated. - The sweep. Instead of a watchdog process, Cloud Scheduler publishes a
sweepevent every ten minutes. The orchestrator then looks for tasks whose execution crashed, timed out or never reported back, and counts them as failed. - Dead letters. An event that fails five times goes to a dead-letter topic. Once the cause is fixed, one command puts it back.
- A failure budget per issue. Red CI, a rejected review and a crashed task all spend the same budget. When it's gone, the issue goes to a human. A broken setup, such as an invalid API key or a spend limit, escalates immediately and spends nothing: every retry would fail the same way.
Efficiency: spend tokens and CPU only where they add value
"Doesn't use unnecessary resources" is easy to claim. These are the concrete measures behind it:
- Nothing polls. Every component wakes up on an event and goes back to zero.
- A limited number of task slots. Start with one task at a time. An issue that can't get a slot is parked and woken up when one frees. Add slots per repo only as far as changes can land without getting in each other's way.
- A budget per task.
maxTurnsandmaxBudgetUsdin the role configuration cap what a single task can cost. - Small inputs. A task gets only the files its role needs. After red CI, the implementer gets the last lines of the failed jobs, not the full log.
- Cheap gates before expensive ones. The review, with the most expensive model, only starts once CI is green.
- The right model per role. Sonnet where the volume is, Opus where judgement matters.
- A kill switch. One command pauses the factory: a running task finishes, nothing new starts.
Security: every role its own identity
Without a human reviewing every change, the factory must not be able to get around its own rules. The answer is the same as for any other system: least privilege, per identity.
- One service account per role. A worker can publish only to
task-resultsand read only the model API key. It can't fake a webhook event, and it has no access to the issue tracker at all; only the orchestrator talks to the tracker. - Two identities on the Git platform. An author that can push and open pull requests, and a reviewer that can only read. The orchestrator hands out a token per task, valid for one repo. On GitHub these are two GitHub Apps with one-hour installation tokens; on GitLab, project access tokens with the narrowest scope come closest.
- Rules the agent can't loosen. The worker image contains managed settings for the agent, with a
PreToolUsehook that blocks writes to protected paths. Hooks from the target repo don't run. - A diff check as the backstop. After the agent finishes, the worker compares the checkout with where it started. A change outside the role's paths fails the task, whichever tool made it.
- No secrets for the repo's code. Setup and test commands run the target repo's code, so they get a stripped environment without the API key or the Git platform token.
- A review without the plan. The review role gets a checkout with the implementer's plan removed from the git history. That's a lesson from practice: in non-interactive mode, the agent still ran read-only shell commands like
git showthat weren't on its allowlist. A rule that says "don't read the plan" isn't enough when the plan is onegit showaway. - Content is data. Issue text, comments and code are input, never instructions. The review explicitly checks for changes that have nothing to do with the issue.
From factory to dark factory
Everything so far produces reviewed pull requests, and a person still merges them. That's where to start. Turning off the lights takes three more pieces:
- Risk classes. Every change gets a class from 0 (copy, docs) to 3 (authentication, payments, infrastructure, the factory itself). The spec role proposes one, a gate calculates one from the diff, and the higher of the two wins.
- A factory gate. A bot with a policy, as the required approver on the Git platform (branch protection on GitHub, approval rules on GitLab). It merges the low classes itself, through a merge queue or merge train, and leaves the high classes to a person.
- A canary deploy. Cloud Deploy with automatic rollback. Agents never get access to production; only the pipeline does.
Autonomy should be earned, not assumed. Before the gate merges anything, the AI review runs next to human review, and the factory measures how often they agree. A class only becomes autonomous once the numbers show it can. And scaling back is faster than scaling up: one incident sends a class back to human review.
Build it yourself
If you want to build your own, this is the order I'd recommend:
- Run one role locally with the Claude Agent SDK, on a real repo, with issues you pick by hand.
- Write a handful of evals per role: real tasks with a known good outcome.
- Build the worker image and one Cloud Run Job per role, each with its own service account.
- Add a deterministic orchestrator with Pub/Sub, Firestore, the outbox and the sweep.
- Write the adapters for your issue tracker and Git platform, and connect their webhooks.
- Only then add the merge gate and the deploy.
Conclusion
Three ideas carry the whole design: a deterministic orchestrator, disposable workers, and roles as configuration. Adapters at the edge keep it independent of the tracker and Git platform you happen to use. The low cloud bill isn't an optimization added afterwards; it follows from those three choices. The orchestrator handles an event in milliseconds, a worker exists only while it works, and nothing waits for a human while a container runs.