Every Gate Passed

Last week I was tagged on a pull request that four AI agents had already agreed was good.

The system is called Connie. It maintains Otto, our AI travel agent, through four stages running across different models and agentic frameworks: a judger that scores production conversations and files tickets, a triager that scopes them, a fixer that writes the PR, and a reviewer that checks the change against the ticket and traces. Its backlog has taken 427 tickets since March.

What happened to the traveller

A traveller approved a flight. Before Otto creates a reservation, book_flight runs a risk check with our fraud vendor. It came back declined. Our code set the booking row to FRAUD_DECLINED and raised a plain exception carrying one hardcoded string.

“I’m sorry, but your payment could not be processed - please try another payment method.”

That string is the only thing the language model receives. No error code. No recovery field.

So the model did what models do with a gap. It filled it. With nothing else to go on, Otto treated the decline as a card problem: it asked the traveller to add a different card, though the traveller had one on file which Otto had just listed. When the traveller pushed back, Otto invented a bank-side explanation, a hold on the card, call the number on the back.

Nothing had been presented to any bank. The traveller called anyway, and came back nine hours later to say the bank had no record of any purchase reaching them.

Otto invented a second cause, a billing address mismatch. The traveller updated their address, retried, and got the identical string. Otto’s final answer was to contact Otto support. There is no such route.

About eleven hours, no booking, and a hard arrival constraint missed. A second traveller lost fifteen hours to the same shape the next day. Under our severity rubric the absence of financial loss kept both at medium, a classification that now looks incomplete given the time lost and the fabricated recovery paths.

The boundary no gate represented

Otto’s refusal to say “fraud” to a traveller is deliberate and written down, in baml_src/cars.baml and baml_src/hotels.baml. That is correct policy. The ticket knew it, said so explicitly, and declined to argue with it. Connie’s judger is better at this than I expected.

Here is what nothing in the loop could see. That policy was written as a constraint on what we say to a traveller, and implemented as a constraint on the error channel, which also feeds the model. Nobody wrote down which boundary it governed, because when it was written there was only one reader.

So vagueness meant for a traveller landed on the model too, and the model, unlike the traveller, had no way to know it was being kept in the dark. Every gate could verify that the string was returned correctly and the booking row set correctly. Not one could verify it was aimed at the wrong reader.

The second wall

The ticket prescribed the right fix: type the error so the model can tell what happened, stop telling travellers to contact their issuer, supply a real escalation route.

The fixer implemented it faithfully. That is the problem.

The instruction telling Otto to ask for a different card does not live in the booking executor. It lives in the same policy files that encode the correct rule about never saying “fraud.” One file, two rules, one right and one wrong.

So a correct ticket, faithfully implemented, produced a pull request that edits policy. That is what I could not merge. Not because the change was wrong, I think it is right, but because those files have an owner outside engineering and no stage of the loop had standing to make that call on their behalf.

Notice that the reviewer did exactly the job we had given it. It checked the PR against the ticket and the traces, the PR matched, the ticket was correct. Every check passed and every one was the right check to run. Nobody checked that the diff crossed a boundary needing someone else’s sign-off, because which boundaries need whose approval is also not written anywhere the loop can read.

S, K ⊢ R

Requirements engineering named this thirty years ago. Zave and Jackson’s “Four Dark Corners of Requirements Engineering” (ACM TOSEM, 1997) gives the criterion. You have a specification S, what the machine does, and requirements R, what the customer wants of the world:

“Let K be the relevant domain knowledge, i.e., the set of relevant indicative properties. Then S and K together must be sufficient to guarantee that the requirements are satisfied.”

The implementation and the relevant facts about the world must together entail the desired outcome.

Our harness has S and checks it all day. The missing K was not the policy, which was written down. It was the policy’s scope, and that sentence exists nowhere. It was true by accident in March, when nobody thought of the model as a reader of that channel, and stopped being true without anyone touching the line that made it false.

A friend building design automation at a large AI infrastructure company hit the same shape elsewhere: a network topology design that was not buildable, because the arrangement of the buildings made the intended fiber routing impossible. His harness modeled topology; the facts that invalidated it were facts about buildings.

What we changed in Connie

Running now

The triager extracts assumptions before it scopes. Every ticket now produces an explicit list of what the work takes to be true about the world. For this one it produced five entries.

Three checked out. One came back false in a way anyone could have caught: the fix assumed the remedy text was ours to change unilaterally, and it was not, because that file has an owner outside engineering. That is the second wall, surfaced by a step that costs almost nothing to run.

The fifth entry read: the policy governs only the traveller surface. Source: none found.

That is not a wrong answer. It is an unanswerable one, and it is the most useful thing the step produces. A false assumption is a bug, and bugs get found eventually. An assumption with no source is a missing capability. Nothing in the loop can resolve it, and no amount of re-reading the diff will help.

A premise-review proxy was already sitting in Jira. Of those 427 tickets, 28 were closed Won’t Do. That is not 28 premise failures, since Won’t Do also covers deprioritisation and duplication, and a classification pass is queued. Until that classification is complete we use the count only as a tripwire: zero is not evidence of a healthy loop, it is a prompt to check whether anything is being challenged at all.

Being built

Ticket-level challenge verdicts. Our reviewer can approve or request changes, both scoped to the diff. Neither means this work should not exist, or this should exist, smaller. Four in flight:

• PROCEED: assumptions hold
• NARROW: some fail; here is the subset that survives
• KILL-PREMISE: the item rests on a false assumption; close it and suppress the pattern
• ESCALATE: an assumption is unresolvable; a named human decides

NARROW matters most, because it is the correct verdict here. The traveller-facing string should stay vague. The tool result the model reads should be typed. One boundary changes, the other does not, and a binary gate cannot express that.

Escalation routed by blast radius, not by confidence. We route human attention by reviewer uncertainty, which is backwards: this PR was maximally confident. Once enforcement lands, certain categories will never auto-merge however clean the diff, enforced by path-scoped required reviewers rather than by prompt:

• customer-facing copy in payment, fraud, or cancellation paths
• any diff touching a file whose owner sits outside engineering
• any item where the challenger returned ESCALATE
• any item whose assumption list contains an entry with no source of truth

That last rule is the one I would recommend first to anyone else. An assumption nobody can check is not low-risk. It is unowned, and unowned is the class that hurt us.

Not built, and it is the one that mattered

Authoritative constraint retrieval. Some of those assumptions are only answerable from material outside the diff: policy files, vendor semantics, the compliance wiki. We have not built it. Assumption extraction can now ask the question; nothing can answer it automatically.

My first sketch was to hand the whole policy corpus to the fixer, and I have moved off that, though not because the fixer should be kept ignorant. An implementation agent that does not know the applicable constraint writes the wrong patch. The problem is volume: a compliance corpus contains many clauses that look applicable but are not. Shi et al. provide controlled evidence that irrelevant but topically adjacent context can materially reduce reasoning accuracy. That does not settle the architecture for coding agents, but it is enough to make “put the entire corpus in the prompt” a design to test rather than assume. So the fixer should receive the minimum set of applicable constraints we can retrieve with confidence, while an independent challenger gets broader access and standing to reject or escalate the premise.

Retrieval cannot key only on ticket text, because the ticket will often omit the policy, which is the failure we are trying to catch. Before implementation it has to route from the affected subsystem, the predicted interfaces, dependency relationships and ownership metadata. After implementation the actual changed paths give a second check. A ticket that already knew which policy mattered would not have needed the challenge.

Judger recall is the other gap, and probably the bigger one. We measure precision because false positives are visible. We have never measured what the judger fails to route anywhere at all. Next is an exhaustive human review of a sampled week, compared afterwards against the judger’s routed findings, and I expect not to enjoy the result.

We expect low precision from the challenger at first: many challenges will not uncover a premise defect. We have agreed in advance that this alone is not grounds for removing the control.

What this does not solve

None of it helps with constraints written down correctly and then quietly changed in scope. The scope assumption became false without anyone editing the file that encoded it. The policy file did not change; the system around it did. A new reader entered the dataflow without triggering a review of the assumptions attached to that channel. The relevant diff existed somewhere else, where nothing identified this policy as a dependent constraint.

What we lack is assumption-level dependency tracking: a way to say that this policy relies on a particular set of readers, interfaces and ownership boundaries, and to revisit it when any of those change. A registry of deliberate weirdness helps only if changes to readers, interfaces and dataflow can identify which assumptions they may invalidate. We do not yet have that dependency model. Building one means constraints carry scope and ownership metadata, readers and interfaces are modelled as dependencies, and a change to that graph triggers a review of the assumptions attached to it. Every entry needs provenance and an effective date. Then we would have to find out whether the review catches real invalidations without burying anyone in prompts.

Brooks had this in 1986: tooling eats accidental complexity, and what is left is “deciding precisely what to build.” The line between intent and execution does move, but it moves fastest through constraints we can represent and keep tied to an authoritative source. It slows wherever ownership is unclear, or the relevant facts change somewhere the system cannot see. Which makes “can long-running agents run a company” a question about how much of your K you can externalize, and how fast the rest goes stale.

The part I keep coming back to

I did not catch this because I am a designed control. I caught it because auto-merge is still switched off, and because I happen to know who owns those policy files. One of those is a setting we never got round to changing. The other is a fact about me.

That is a near-miss with a fortuitous recovery, and Tinsley, Dillon and Cronin have shown what organizations do with those: people who watch a bad outcome fail to materialize update toward the system being safe rather than toward having been lucky. Ours would have been recoded that way easily. The PR did not merge, so the process worked.

The process did not work. A configuration flag we had not gotten around to flipping worked.

The other direction is the more uncomfortable half. An earlier ticket in this family was closed as Won’t Do in July. The recorded reason was a negative reaction to how one internal write-up had landed. That ticket covered a different path, so I cannot claim that acting on it would have prevented this. What I can say is narrower, and worse: the close recorded how a finding had landed internally, and stopped there. Whether the finding was true went unexamined.

Premise defects are not something AI agents do to you. Any decision process produces them when nobody is required to state what the work assumes. Our loop produced one. So did we, faster, with a smaller audit trail and no assumption list at all.

Which leaves the actual job. Get K out of my head and into something a machine can read, before somebody flips the flag.