Back to blog
Aug 30
dogfooding
byok
anthropic
provider-reliability
engineering-log

Run #18: Furthest Yet, and the Same Wall One Step Later

Run #17 closed with the furthest progress this project had ever made — six steps deep, Builder finally succeeding — and one new, different kind of failure at the very next step: Test refusing to dispatch at all, because its role's first-choice provider had no real adapter behind it. Before Run #18, that got a precise, narrow fix.

What we did to prepare

Traced end to end, not guessed at: the Tester role's own definition lists ["openai", "anthropic"] as its preferred providers, in that order. The engine picks a step's provider from step.preferredProviderKind if one is set, falling back to the role's own first preference otherwise — and nothing had ever set an override for the test step specifically, so it fell through to openai, which this deployment has never had a real adapter for. No Execution row was created, no dispatch was attempted — a clean, structural refusal working exactly as designed, just never visible before because no run had lived long enough to reach it.

The fix was the smallest one available, and not a new pattern — the ux step had already needed and received the identical treatment earlier in this project's history. A step-level preferredProviderKind: "anthropic" override on test in the workflow definition, pinning it to the one provider this deployment actually has wired up for real. Nothing about the Tester role's own definition changed, nothing about provider resolution generally changed, nothing about retry policy or validation touched.

What happened

Scope, Architecture, Security, and UX all completed cleanly, no drama — Architecture's call ran 40,727 milliseconds, back in the faster range this project saw before Runs #12–#15's timeout episode, a fourth consecutive unremarkable success since the 120-second timeout shipped.

Build succeeded again — 108,319 milliseconds, the second consecutive clean success since the commit-message fix. Two for two now; that question stays closed.

Review succeeded, same as Run #17.

Then Test succeeded — for the first time in this entire project's history. A real Anthropic call, 49,865 milliseconds, producing a genuine test-plan artifact instead of an instant refusal. The fix worked exactly as intended, on the first real attempt.

Then Approval failed, the exact same way Test had, one step later:

No real provider adapter is configured for preferred kind "gemini"

Same structural signature — no execution row, no dispatch attempted, non-retryable, instant. Not a new kind of problem: the Releaser role's own definition lists ["gemini", "ollama"] as its preferred providers, and this deployment has no real adapter for either one. The test-specific fix was scoped exactly to test — it was never going to help a completely different step built on a completely different role's unconfigured preference.

What the evidence actually supports

This isn't a new kind of failure — it's the second occurrence of a category. Two different roles (Tester, Releaser), two different unconfigured providers (OpenAI, Gemini), the identical underlying shape: a role's declared preference points at a provider nothing real backs, and the step that uses it has no override correcting that. The ux step needed this fix once before Run #17 ever happened; test needed it after Run #17; approval needs it now. That's a pattern, not a coincidence, and release — which shares the exact same Releaser role definition as approval — would almost certainly hit the identical wall if it were ever reached, though it never got the chance this run to prove that one way or the other.

Every fix so far in this whole investigation series has been correct and narrow, and that's exactly why this keeps happening one step at a time. Fixing precisely the step that just failed, and nothing more, has been the right call at every stage — it's what kept the retry-deferral fix from touching Builder's timeout, what kept Builder's timeout fix from touching Architecture's, what kept the commit-message fix from touching validation limits. Applied to this specific category of problem, though, it means each run surfaces exactly one more instance of the same underlying gap instead of finding them all at once.

Where this leaves things

  • Run #18 is the deepest run yet — seven of nine steps succeeded, including two genuine firsts (Test's only-ever success, Build's second).
  • Build and Test's fixes are both confirmed working, not single-occurrence luck.
  • The provider-configuration gap is now understood as a category, not a one-off — worth an audit of every role's preferredProviders against what this deployment actually has real adapters for, rather than discovering and patching the remaining instances one failed run at a time.
  • release is the next open question, sharing Approval's exact role and — on present evidence — its exact risk, unconfirmed only because the run never survived long enough to reach it.