Run #9 confirmed the cross-invocation retry fix worked, and left one honest open question: Builder's Anthropic call kept failing on the application's own 60-second timeout, on both attempts, in two fully separate invocations, each with a full unshared budget. Before running it again, we went and found out why — not by guessing, by reading what Run #9 had already recorded.
What we did manually, before this run
- Read Run #9's own numbers precisely, not the rounded ones. Every failed Build attempt's
provider_usage.duration_ms— which measures only the Anthropic SDK call itself — was recorded as essentially exactly 60,000 milliseconds. The other ~205-217 seconds of each ~270-second attempt, roughly 77-78% of it, happened before the provider was ever dispatched, inside context acquisition. - Checked whether this was Builder-specific, and it wasn't. Architecture, Security, and UX — none of which failed — showed the identical ~206-216-second acquisition cost in the same run. Whatever was slow, it wasn't something about Builder's role or its prompt.
- Corrected our own earlier framing in the process. The Run #9 post read Build's 40 total context-selection rows as roughly double the other roles' 19-20, and treated that as a plausible contributing factor. It wasn't a doubled selection — it was 20 files selected on each of Build's two attempts, 20 + 20 = 40 across both, the exact same count as everyone else. Worth stating plainly rather than letting stand: that hypothesis didn't survive the actual data.
- Traced the real cause via code, then confirmed it against the timestamps. Every single context candidate — around 417 of them per attempt, selected or excluded — triggers its own database write, and every one of those writes independently re-checked whether the default workspace exists before proceeding, an uncached round trip that never needed repeating within one request. ~417 sequential writes, each silently doubled, is a real, ordinary N+1 pattern — not a GitHub problem, not a context quality problem.
- Fixed exactly that, and nothing past it. The workspace check is now resolved once per request and reused. Individual selection writes became one batched, chunked insert instead of hundreds of sequential ones. The application's provider timeout, retry count, and the cross-invocation retry fix itself were all left untouched — there was no live evidence for how long Builder's actual model call needs, and changing that number without evidence is exactly the mistake Run #8 already made once.
- Caught a real regression in the fix's own first draft before it shipped. An early version of the batched write could lose already-accumulated selections if one candidate's fetch failed partway through the batch. Found and fixed before merging, with a before/after test proving the fix doesn't reintroduce it.
The task
The same request as the last four runs, deliberately unchanged, specifically so this run would be comparable to Run #9 rather than a different experiment wearing the same name:
"Improve the BinChicken run-detail page by adding a small 'Last updated' timestamp showing when the run data was most recently refreshed. Use the existing run-detail components and styling. Add appropriate tests. Do not change the workflow engine, AI providers, GitHub integration, governance, authentication, or database schema. The timestamp should appear alongside the existing 'started' timestamp and use the existing timestamp formatting and muted metadata styling."
What happened
Scope, Architecture, Security, and UX all completed cleanly, and clearly faster than Run #9 — Architecture in particular went from 4 minutes 31 seconds to 69 seconds. Security's approval gate opened the same way it did last time.
Builder failed both attempts again — in roughly a quarter of the time. Attempt one ran 73.0
seconds this run against Run #9's 277.1 seconds; attempt two ran 71.3 seconds against 265.2. Both
failures, in both runs, were the identical application_timeout.
The fix's actual target measured cleanly, isolated from everything else. Reading
context_selections timestamps directly: attempt one's first write landed 2.3 seconds after the
step started, its last of 420 rows landed 8.37 seconds later — acquisition done in roughly 9
seconds, down from roughly 217. Attempt two: essentially identical, roughly 9 seconds
against Run #9's roughly 205. Both attempts still selected exactly 20 files, the same as every
other role in this run and the last one — nothing about what Builder was given changed, only how
long assembling it took.
And that's exactly what makes the remaining failure legible instead of murky. With acquisition no longer eating three-quarters of the clock, what's left in a ~72-second attempt is almost entirely the Anthropic call itself, aborting at the application's own 60-second ceiling — visible now with nothing else stacked on top of it to obscure the shape.
What's actually different now, and what isn't
The latency fix worked exactly as intended, and the evidence for it is about as clean as this project's own logs get: two independent measurement methods — execution-duration arithmetic and raw context-selection timestamps — agree to within about a second, on both attempts, that acquisition dropped by roughly 20-25x. That's not a hypothesis anymore.
What it was never going to fix, and didn't: why the Builder role's own Anthropic call needs more than 60 seconds. Nothing about this run's fix touched the provider call, the timeout value, or Builder's prompt. Builder still selected the same 20 files the other roles select, at the same volume — so an oversized-input theory doesn't survive this run's evidence any better than it did Run #9's. What does remain untested: Builder's structured-output schema asks for file diffs and commit messages, a genuinely different and larger shape of response than the review roles' markdown output, and every successful role's own measured Anthropic duration this run and last (roughly 29 to 53 seconds) sits real but meaningfully under 60 seconds — closer to that ceiling than comfortable, on work that isn't Builder's.
Where this leaves things
- The context-acquisition latency problem is closed, confirmed by two independent, closely agreeing measurements against real production data, not argued from the fix's design alone.
- Builder's actual failure is now isolated far more precisely than before — not confounded by ~200 seconds of unrelated database overhead, a genuinely different problem than either of the two ideas already ruled out (platform-duration kill, oversized input).
- The 60-second provider timeout itself is the next thing to actually examine — not by raising it on a guess, but by getting real evidence for how long Builder's call actually needs, the same discipline that caught Run #8's premature 120-second timeout before it shipped.
- No run is planned until that evidence exists. A fix that measurably worked is still a reason to keep going, not a reason to declare the job done.