Runs #19–20 closed with the first fully successful nine-step run in this project's history, immediately followed by a real, deterministic bug found in the wreckage: the engine marks a run "completed" before it ever attempts the real GitHub write that turns it into a PR, guaranteeing that write is refused every time. That finding — plus a downstream silent-500 gap and an unrelated unapplied migration — was handed off for a decision on whether and how to fix it. Run #21 launched before that decision was made, on the code as it stood.
What we did to prepare
Nothing new. The provider-configuration overrides that closed out Runs #18–20's open questions were already in place; nothing about the ordering bug or the observability gap found in Run #20 had been touched yet. This run was a plain rerun on unchanged code, mostly to see how far a second full attempt would get before hitting the same wall — or something else entirely.
What happened
Scope, Architecture, Security, and UX all completed in line with every recent run — 43, 56, 54, and 51 seconds respectively, Security and UX running in parallel as usual. The routine approval gate after them was decided, and Builder dispatched.
Builder succeeded — 113 seconds, comfortably inside budget, no truncation. Five of nine steps done, all clean, nothing new.
Then the run stopped. Not failed. Not blocked on a gate. Nothing.
Test, Review, and both remaining Releaser steps stayed in pending — never started, no attempt
recorded — for over eight minutes, checked directly against the database rather than the page
showing it. No execution event fired. No error was logged anywhere. The run's own status stayed
running the entire time, exactly as if progress were still happening, while nothing was.
What we found tracing it
This is a different failure from Run #20's — that one happened at the very end, after every step had genuinely finished. This one happened in the middle, with four steps still waiting, and it wasn't caused by anything AI-provider-related at all.
Progressing a run to completion depends entirely on a client-side loop
(lib/store.ts's advanceRun) that calls the server's /advance endpoint, fetches the fresh
result, and repeats — for as long as the run reports running. That loop has no retry logic and no
error handling of any kind. It runs as a fire-and-forget promise, started once when a run begins or
a gate is decided. If a single call inside it throws for any reason — a dropped connection, a tab
losing focus, a transient server hiccup — the loop dies as a silent, unhandled rejection. Nothing
in the interface reflects that it died. There is no other mechanism, anywhere in this deployment,
that drives a run forward: no scheduled job, no server-side retry, nothing. Once that one loop
stops, the run simply never moves again, and looks — to anyone watching the page — identical to a
run that's still working normally.
The page itself compounded the confusion rather than causing it: the run-detail view fetches a run's state once, on load, and never again on its own. Reloading it doesn't help either, because reloading only re-fetches — it doesn't restart the loop that drives the run forward. The two problems stack: the display doesn't update on its own, and the one thing that would have updated it had already died quietly, with no way to tell from the screen which of those was happening, or that anything was wrong at all.
What the evidence actually supports
This is not the ordering bug from Run #20. That one is guaranteed to happen at the very end of every successful run, after real work has been genuinely completed. This one can happen at any point in any run, before any of the later steps get a chance to run at all — a structurally different, and in some ways more consequential, gap: it doesn't just cost one run's PR, it can silently strand a run that would otherwise have succeeded completely.
The absence of any visible error is not a missing detail — it's the actual defect. A run that stops because something failed and a run that stops because it succeeded look the same from this interface. Distinguishing "still working" from "silently dead" required going around the UI entirely and reading the database directly, twice, several minutes apart, to confirm nothing had changed.
Cancelling was safe and clean — no step was actively running when the decision was made, so
nothing was interrupted mid-flight. The run's own audit trail closed out with a normal, durable
cancelled status, no lingering running row this time.
Where this leaves things
- A second, independent stall mode is now confirmed, distinct from Run #20's end-of-run ordering bug — this one can strand a run mid-flight, with zero visible indication anything went wrong.
- The retry-less client loop and the lack of any server-side backstop are the same underlying gap flagged as far back as this project's early reviews — Run #21 is the first time it was actually caught happening, rather than argued as a risk on paper.
- Three findings from Run #20's investigation and this run's stall were handed off together — the ordering bug, the silent-500 observability gap, and this liveness gap, plus one unrelated migration that was written but never applied to production. What happens with that handoff is the open question heading into whatever run comes next.