Run #9: The Fix Held, and a Different Wall Showed Up

Run #8 closed with Builder's stall reproduced more cleanly than before, but still unexplained: a timeout that fired exactly on schedule on attempt one, and a second attempt that sat at running for eleven minutes with no provider record at all — the same signature as Run #7, just with better surrounding evidence. Before touching Builder again, we went looking for the actual platform-level cause, not another guess at one.

What we did manually, before this run

• Pulled live Vercel runtime logs for the exact invocation that ran Run #8's Builder attempt two. Found a direct, unambiguous line: Vercel Runtime Timeout Error: Task timed out after 300 seconds. Not inferred from the silence — the platform's own log, for the exact deployment and request, confirming the invocation was killed mid-flight before any of our own code — catch block, execution row, step status — ever ran.
• Did the arithmetic against real, observed numbers, not assumed defaults: attempt one's own 60-second timeout firing correctly still left the engine's retry loop dispatching attempt two inside the same request. Context acquisition plus attempt one's dispatch already consumed most of the 300-second ceiling; attempt two never stood a chance of finishing before the platform ended the process.
• Changed the retry's shape, not its budget. A retryable failure now durably returns its step to pending and ends the request, instead of sleeping through backoff and redispatching in-process. The next advance() call — a fresh request — picks the step back up with attempt correctly carried over and its own full, unshared budget. No timeout was shortened, no attempt was added, no maxAttempts was raised.
• Fixed two follow-ups the same investigation surfaced: a role could now correctly select a genuinely relevant file large enough to triple its own token cost on its own — a new, size-specific ceiling excludes it with an explicit, auditable reason instead of either silently including it or silently dropping it. And the context-audit panel showing "0 selected" for two of four roles despite the database holding real rows for all four — root cause: reading context selections back used an unranged query, and Postgres's own 1,000-row default response cap was silently truncating a run whose four roles together persisted over 1,600 rows.
• Had the retry-deferral change specifically reviewed for what a fresh-invocation-per-attempt model could get subtly wrong — cross-invocation attempt accounting, a cancellation landing in the gap between a deferred retry and the next poll, concurrent claims. The review approved it, with three concrete gaps to close first: the pagination fix's sort key wasn't provably unique across ties, and two of the new behaviors — the real loadRun() process boundary, and cancellation arriving mid-defer — weren't yet covered by a test that reproduced them for real rather than by inference. Closed all three before this run, including proving each of the two engine tests failed against the actual pre-fix code before trusting them.

The task

The same request as the last three runs, unchanged on purpose:

"Improve the BinChicken run-detail page by adding a small 'Last updated' timestamp showing when the run data was most recently refreshed. Use the existing run-detail components and styling. Add appropriate tests. Do not change the workflow engine, AI providers, GitHub integration, governance, authentication, or database schema."

What happened

Scope, Architecture, Security, and UX all completed cleanly — Security and UX in parallel, as designed. Security's completion opened an approval gate; once decided, the run resumed on its own.

Then Builder's first attempt ran 279 seconds before failing — "The request to Anthropic exceeded the configured timeout." Almost exactly Run #8's shape: a single attempt consuming nearly the entire 300-second platform ceiling on its own. Under the old code, this is precisely where attempt two would have been dispatched into whatever handful of seconds were left in the same request, and gotten killed mid-flight the way Run #8's did.

It didn't happen that way this time. The durable event log shows exactly the sequence the fix was built to produce: step_failed at 07:33:23, step_retrying logged a second later, the step returned to pending. Sixteen seconds after that, a genuinely separate invocation — a different lease owner, a new execution row — picked the step back up and dispatched attempt two with its own full budget, attempt correctly carried forward as 2, never reset to 1.

Attempt two timed out too — cleanly, this time, not silently. 266 seconds in, the same "exceeded the configured timeout" failure, and with both configured attempts now genuinely exhausted, the step went to a real, classified failed state with a real error message attached. No orphaned running row. No eleven-minute silence. The run itself failed a few seconds later, honestly, for a reason anyone looking at it can read directly.

Both of the other two live fixes did exactly what they were built to do, for real, at real scale. Architecture's context selection excluded one file specifically for exceeding the new cost ceiling — cost_budget_exceeded, visible in the audit trail, not silently dropped and not silently included. And this run alone persisted 2,085 context-selection rows across its four context-acquiring roles — more than double the 1,000-row page size that was hiding Run #8's display bug — with every role still showing its correct, complete, non-zero count: 19, 19, 20, and 40.

What's actually different now, and what isn't

The platform-duration problem is closed. We watched attempt one consume almost the entire 300-second ceiling on its own — the exact precondition that killed Run #8 — and this time it produced a clean failure and a fully-budgeted second attempt instead of a silent kill. That's not a closed-form argument anymore; it happened, on a real run, and the evidence is durable and complete.

What's left is a narrower, different question: why is the Anthropic call itself taking longer than 60 seconds for Builder, twice, in two separate invocations, each with its own uncontended budget? That's not a platform problem — both attempts had all the time they needed and still didn't get a response in time. One thing worth checking, not yet confirmed as a cause: Builder selected 40 files of context this run, roughly double what Architecture, Security, or UX each selected. Whether that volume of input is actually why the call is slow, or whether it's unrelated, isn't established by anything gathered so far — it's a real candidate, not a diagnosis.

Where this leaves things

• The retry-deferral fix is confirmed working under the exact conditions it was built for — not by argument, by a real recurrence of the original failure shape, handled correctly, with a complete durable record at every step.
• The context cost ceiling and the pagination fix are both confirmed working at real scale, not just in the test cases that motivated them — one live exclusion, and a run whose context-selection count comfortably exceeded the row limit that was previously hiding data.
• Builder still cannot complete this task, for a reason that has nothing to do with any of this run's fixes. The next investigation is squarely about the provider call itself — actual latency, actual input size, actual response behavior — not the execution shape around it.
• No run is planned until that's understood. The same discipline that got the last three fixes right: trace the actual failure with actual evidence before deciding what, if anything, to change.