In short: The original pipeline was hardwired to GitHub Models. This post covers how I extracted every LLM backend into its own class with a shared interface, added a relay that automatically falls back to the next backend on connection failure, and what I learned about building resilient AI infrastructure around unreliable upstream APIs.

For months, the pipeline only spoke to one API: GitHub Models. That was fine for development. In production it became a single point of failure. One timeout, one rate-limit spike, one flaky endpoint — and the whole 8-minute pipeline dies at minute 7.

The fix wasn’t just adding a retry loop. It was rethinking the architecture.

The original problem

base_agent.py started at around 200 lines. By the time it supported GitHub Models, Anthropic Claude, Ollama, OpenCode CLI, OpenCode Zen, OpenCode Go, Copilot, and NVIDIA NIM — it was over 900 lines, with eight separate if/elif chains that each duplicated auth logic, header setup, and error handling.

Adding the ninth backend would have meant copying that pattern again. Worse, there was no way to say “try Ollama, fall back to GitHub Models if it’s unreachable” — the choice of backend was baked in at startup.

That’s when I decided to extract the backends properly.

The interface

Every backend now implements a shared interface:

class LLMBackend(ABC):
    @property
    def model(self) -> str: ...

    def call(self, messages: list[dict]) -> str: ...

    def call_with_tools(
        self,
        messages: list[dict],
        tools: list[dict],
    ) -> tuple[str, list]: ...

    def supports_tools(self) -> bool:
        return True

The supports_tools() method matters because not all backends support function calling. OpenCode CLI runs as a subprocess — it doesn’t expose a tool-calling API. Anthropic via the raw SDK needed special handling. Having supports_tools() as an explicit capability means the relay can route tool calls around backends that can’t handle them, instead of failing at runtime.

One file per backend

Each backend lives in agents/backends/:

agents/backends/
    base.py             # LLMBackend ABC + OpenAICompatibleBackend mixin
    github_models.py    # GitHubModelsBackend
    ollama.py           # OllamaBackend (think/stream/preserve_thinking)
    anthropic.py        # AnthropicBackend (Anthropic SDK)
    copilot.py          # CopilotBackend (token refresh via threading.RLock)
    nvidia_nim.py       # NvidiaNimBackend
    opencode.py         # OpenCodeBackend (subprocess)
    opencode_zen.py     # OpenCodeZenBackend
    opencode_go.py      # OpenCodeGoBackend
    fallback.py         # FallbackLLMBackend (the relay)
    factory.py          # create_backend(cfg, github_token)

base_agent.py went from 905 lines to 534.

The relay

FallbackLLMBackend wraps a list of backends:

class FallbackLLMBackend(LLMBackend):
    def __init__(self, backends: list[LLMBackend]):
        self._backends = backends

    def call(self, messages):
        for backend in self._backends:
            try:
                return backend.call(messages)
            except FALLBACK_ERRORS as e:
                print(f"⚠️  {backend.model} unreachable ({type(e).__name__}), trying next…")
        raise RuntimeError("All backends failed")

FALLBACK_ERRORS is a tuple of exception types that indicate a connectivity problem rather than a logic error — connection timeouts, InternalServerError 500, ServiceUnavailableError. Logic errors (bad request, invalid model name) are not caught — they surface immediately so you know something is misconfigured.

In config:

llm:
  model: "ollama/qwen3.6-plus"
  fallbacks:
    - model: "ollama/qwen3.6"
    - model: "gpt-4.1-mini"

If qwen3.6-plus is unreachable, the relay switches to qwen3.6. If that’s also down, it falls through to gpt-4.1-mini in the cloud.

Mixed tool-capability handling

One tricky case: what if your primary backend supports tools and your fallback doesn’t?

The first version of FallbackLLMBackend rejected this with a ValueError: All backends must have same supports_tools() capability. The logic was sound in theory — if you’re relying on tool calling, mixing in a backend that can’t do it causes silent failures.

But in practice, an Ollama backend calling a local model does support tools. Its fallback might be a raw Anthropic SDK call that doesn’t. You still want the relay to try.

The fix: downgrade the mismatch to a warning at construction. For call_with_tools(), the relay silently skips backends where supports_tools() is False, and only raises if no tool-capable backend is available. This keeps the protection where it matters without blocking useful fallback chains.

Fallback config inheritance

An early bug: fallback entries didn’t inherit their parent’s connection settings.

llm:
  model: "ollama/qwen3.6-plus"
  ollama_url: "http://192.168.1.50:11434"
  fallbacks:
    - model: "ollama/qwen3.6"
    # ← inherits ollama_url? Not originally.

Each fallback was constructed from its own entry only. So ollama_url was missing from the fallback — it connected to localhost:11434 (not found) instead of the remote host, and silently failed in a confusing way.

The fix was a deep-merge: before constructing each fallback, the factory merges the parent config (minus fallbacks itself) into the fallback entry. Explicitly set values in the fallback take precedence; everything else inherits from the parent.

What I saw in production

The first real fallback event in a live run:

⚠️  qwen3.6-plus unreachable (InternalServerError 500), falling back to qwen3.6

The pipeline continued without intervention. The fallback model was slower, but the run completed. Without the relay, that 500 would have crashed the pipeline mid-run and lost all checkpointed work up to that point.

The second event, a few runs later:

⚠️  qwen3.6-plus unreachable (InternalServerError 500), falling back to qwen3.6
⚠️  qwen3.6 unreachable (ConnectionError), falling back to gpt-4.1-mini

Both local models were down simultaneously. The pipeline finished on GitHub Models. A three-backend relay with one cloud fallback covers most failure modes.

The Copilot backend

One backend needed special treatment: CopilotBackend. GitHub Copilot access tokens expire and need periodic refresh. The original implementation refreshed the token inline, which caused a race condition when multiple agents ran in parallel threads — two agents could both detect an expiring token and both try to refresh it simultaneously.

The fix was a threading.RLock with double-checked locking:

def _pre_call(self):
    if self._token_expires_soon():
        with self._lock:
            if self._token_expires_soon():   # re-check under lock
                self._refresh_token()

_pre_call() is called at the top of every iteration of the tool loop — not just before the first call — because a multi-turn tool interaction can span several minutes.

The before and after

Before: base_agent.py with eight if/elif chains, 905 lines, no fallover.

After: 10 focused backend classes (50–150 lines each), a relay that handles failover transparently, and base_agent.py down to 534 lines focused on agent logic only.

Adding a new LLM provider now means writing one new backend class that implements three methods. The factory picks it up automatically via a prefix in the model name. No changes to orchestrator, no changes to agents.

Code: github.com/wanleung/ai-dev-team