ADR-0020: Multi-provider model routing — Vertex AI Gemini as second provider

▶ Watch the 1:19 summary — ADR-0020 — Multi-provider model routing — Vertex AI Gemini as second provider, explained

Status: Proposed
Date: 2026-05-17
Decider: Seth Shoultes (with Blair Williams approval per decision-rights matrix; counsel reviews Vertex AI data-handling posture)
Author: Seth Shoultes

Context

ADR-0005 (Anthropic SDK direct, mitigation seam at llm.call) established that all LLM calls flow through a single function llm.call(handle, operation, prompt, ...) whose signature is provider-neutral. The reasoning: cost discipline (max_tokens enforcement, recorded cost) and eval coverage are both easier with one boundary, and a future provider swap becomes a wrapper change rather than a codebase-wide refactor. ADR-0005 considered and deferred multi-provider routing explicitly — “LiteLLM router — rejected for V1: adds a layer for cross-provider routing we don’t need yet; revisit if multi-provider becomes a real ask.”

Three things have changed since ADR-0005 was accepted on May 8 that make the revisit timely.

First, the Anthropic-dependency risk has been confirmed as a live, unaddressed strategic risk in the Phase 1–2 friction-points one-pager. Phase 1 has the mitigation seam; what it lacks is a second provider actually sitting behind it. The seam unused is documentation, not mitigation.

Second, several use cases have emerged where Anthropic is the wrong economic choice. The wrap-up generation in ADR 0019 is the canonical example — it’s structured summarization at session end, not advisory work, and routing it to Claude pays the advisory-model premium for filler-tier work. The same is true for support-ticket triage classification, MEMORY-relevance filtering, sync-delta materiality detection, and any high-volume deterministic classification. Sonnet’s cost on these workloads is unjustified; a cheaper, comparable-quality model is the right answer where one exists.

Third, MemberIntel is already on GCP (ADR-0001, ADR-0009, ADR-0006). Vertex AI Gemini calls from Cloud Run services run same-region, share IAM/CMEK/audit posture with the rest of the stack, and integrate natively with BigQuery (ADR-0006). The operational cost of adopting Vertex is materially lower than adopting any cross-cloud provider.

The decision now is what providers sit behind the mitigation seam, what routing policy decides between them per (tier, operation), and how the entitlement layer (ADR-0001) and model-routing single source of truth (ADR-0002) extend to a multi-provider world without breaking the cost-discipline invariants.

Decision

The mitigation seam established in ADR-0005 gains a second provider: Vertex AI Gemini, routed via the same single source of truth in entitlement/tiers.py (ADR-0002).

Routing policy by operation class (current state in parentheses; “no change” preserves ADR-0002 routing):

• Advisory chat / conversational reasoning — Anthropic Claude (Sonnet for Pro, Haiku for Free). No change.
• Structured summarization — wrap-ups (ADR 0019), daily briefings, BIBLE drafting if ever LLM-generated — Vertex AI Gemini 2.5 Flash.
• High-volume classification and filtering — support triage, MEMORY-relevance filtering, sync-delta materiality, anomaly explanation — Vertex AI Gemini 2.5 Flash-Lite.
• Eval judge — Anthropic Claude. The judge is held distinct from any generation model to avoid same-model bias.
• Fallback when Anthropic is unavailable — Vertex AI Gemini 2.5 Pro, Pro-tier only, with a “degraded mode” banner surfaced to the user. Free tier continues to degrade gracefully to the SPEC’s existing “AI advisor temporarily unavailable” path; we do not silently swap Free users onto a different model.
• Embeddings — pgvector storage unchanged (ADR-0004), embedding generation remains the current provider. Vertex AI Embeddings is reserved as a switchable alternative behind the same wrapper for the case where re-embedding becomes economically right. No live multi-provider routing on embeddings in V1.

The routing decision is encoded in the existing TIERS dict (ADR-0002) as (tier, operation) → (provider, model_id). The dict grows in column count; no new source of truth is introduced. The opaque ModelHandle (ADR-0001) gains a provider field that the wrapper reads to dispatch.

Implementation is a thin in-house provider dispatcher in src/memberintel/llm/, not LiteLLM or another third-party router. ADR-0005’s reasoning for rejecting LiteLLM still holds at two providers: max_tokens enforcement, cost recording, and eval boundary discipline live in our wrapper today, and a routing-only third-party adds an abstraction layer that obscures those invariants. Adding a provider switch case is roughly 50 lines of code; adopting LiteLLM is a dependency with surface area we’d own forever.

This ADR commits to the second provider as an architectural fact. Implementation lands in Phase 2 alongside the wrap-up generation work in ADR 0019.

Alternatives considered

Alternative 1: Status quo — keep ADR-0005 unchanged, mitigation seam unused

Keep the seam as documentation only. Defer the second provider until a future “real ask” forces the issue.

Strongest argument: zero new work, zero new vendor relationship, zero new operational surface. The mitigation seam was designed precisely to allow this deferral; ADR-0005’s whole reasoning is that the option doesn’t cost anything until exercised, so why exercise it before we have to.

Rejected because the “real ask” has now arrived. The friction-points doc identified the Anthropic-dependency risk; ADR 0019 concretely needs Flash-class economics that single-provider Anthropic doesn’t deliver. The cost of waiting for an outage or a price change to force the move is materially worse than spending the implementation cost now, while the team has the context and the eval suite is being built fresh.

Alternative 2: OpenAI as the second provider

Add OpenAI (GPT-4o / GPT-4o-mini) behind the abstraction instead of Vertex AI Gemini.

Strongest argument: OpenAI is the best-known alternative to Anthropic, has the most mature tooling, and is closely benchmark-comparable on most operation classes. Engineering familiarity is high.

Rejected on three grounds. First, cloud locality: OpenAI calls cross-cloud from GCP, paying egress and adding latency. Vertex AI calls stay in-region. Second, counsel-review surface: adding a second cloud provider adds an item to the May 31 engagement (data handling, training opt-out, regional pinning) that we don’t need if we use Vertex. Third, the Gemini 2.5 Flash family currently hits the cost/quality sweet spot for our high-volume non-advisory workloads better than GPT-4o-mini at comparable pricing. OpenAI remains a credible third option if the strategic case ever shifts.

Alternative 3: LiteLLM as the routing layer

Adopt LiteLLM as a managed multi-provider router behind the mitigation seam.

Strongest argument: LiteLLM is purpose-built for this — provider-agnostic SDK, built-in fallback logic, cost tracking, model-spec abstraction. Adopting it spares us writing routing logic ourselves and gives us provider portability for free.

Rejected because ADR-0005’s reasoning still holds at our scale. LiteLLM’s abstraction obscures the cost-discipline invariants (max_tokens enforcement, recorded cost per call) that today live in our wrapper, and our eval boundary needs one seam to test, not a router with its own routing logic on top. The in-house dispatcher costs about 50 lines of code and keeps every invariant under direct control. LiteLLM becomes reasonable to revisit if the provider list grows to four or more — at two providers, in-house is cheaper.

Alternative 4: Bedrock (Claude on AWS) as a redundancy path

Route Anthropic calls via AWS Bedrock as well as Anthropic direct, for vendor redundancy on the Anthropic model family itself.

Strongest argument: cleanest hedge against Anthropic-API-specific outages — same model family, different cloud’s SLA, no need to re-eval for the routed path.

Rejected as overreach for V1. The dependency we’re mitigating is on Anthropic-the-company (price changes, model deprecations, strategic shifts), not on the specific API endpoint. A Bedrock route doesn’t hedge against any of those — it only hedges against an API-endpoint outage, which Anthropic’s stated SLA already addresses adequately for our needs. Worth revisiting only if a customer SLA commitment ever requires sub-30-minute failover.

Consequences

What gets easier:

• Wrap-up generation (ADR 0019) lands on the right economic footing from day one, avoiding a “use Sonnet temporarily, migrate later” interim state.
• The Anthropic-dependency risk moves from unaddressed to actively mitigated. Counsel and Blair both have a concrete answer to “what if Anthropic doubles prices or has an outage.”
• Per-operation cost optimization becomes tractable. Each operation routes to the cheapest model that hits the quality bar, not to a default model regardless of job.
• Internal tooling — the engineering “minions” (eval-suite agent, support triage, ADR drafting) — gains a cheap home for high-volume work that doesn’t burn the advisory budget.

What gets harder or constrained:

• The eval suite grows. Every routed operation needs eval coverage on the provider it actually runs on; for provider-fallback paths, both paths need passing evals before the fallback is trusted in production.
• The TIERS dict grows in column count. The change is small (add provider alongside model_id) but every dict-test updates.
• Vertex AI quotas and rate limits become a second operational concern alongside Anthropic’s. The observability stack (ADR-0006) gains a second provider’s error category space.
• Counsel review at May 31 gains an item: confirm Vertex AI’s data-handling posture — training-on-customer-data opt-out, retention, regional pinning — meets our customer commitments. The default Vertex AI commercial posture satisfies this; getting it in writing matters.

Downstream commitments this creates:

• ModelHandle (ADR-0001) gains a provider field. Tests update.
• The TIERS dict (ADR-0002) gains provider alongside model_id. Single source of truth preserved.
• The CI guard rule (ADR-0005) extends: import google.cloud.aiplatform (or the chosen Vertex SDK) is allowed only in src/memberintel/llm/, alongside import anthropic.
• Cost recording schema gains provider as a dimension. Cost dashboards reflect per-provider spend.
• The eval suite gains provider-specific tests for each routed operation, plus a provider-fallback test exercising the fallback path end-to-end.
• A Vertex AI service account with KMS/CMEK enforcement, added to the infrastructure module. Wired via Workload Identity Federation (ADR-0009).
• Counsel review agenda gains a Vertex AI data-handling posture item.

Deferred deliberately:

• LiteLLM adoption. The in-house dispatcher is right at two providers; LiteLLM becomes reasonable at four or five.
• Multi-provider embedding routing. Embeddings are stable; switching providers is a re-embed-everything operation. Reserved as a switchable alternative, not a live multi-provider routing target.
• A third provider (OpenAI or Bedrock). Worth revisiting if a customer SLA, a strategic shift, or a major price change creates a real ask.

Conditions for revisiting

1. Vertex AI Gemini quality on routed operations falls below 85% eval pass rate sustained. The routing policy needs revision — either fall back to Anthropic for that operation or escalate prompt iteration before reverting.

2. Anthropic price changes or outage frequency exceed thresholds where one-provider fallback isn’t enough. Adds a third provider (Bedrock or OpenAI) to the rotation.

3. The provider list grows to four or more. LiteLLM or a similar managed router becomes economically right.

4. Counsel raises a Vertex AI data-handling concern that can’t be resolved via standard commercial terms. Reverts the Vertex commitment for the affected operation class; alternative is OpenAI or no second provider for that operation.

5. Cost/operation tracking shows the routing policy has stopped saving money. I.e., Flash usage has drifted up the price curve or Sonnet usage on advisory work hasn’t held. Triggers a per-operation routing review.

Related

• SPEC-v2 §8.4 (Model routing single source of truth)
• ADR-0001 (Entitlement service shape) — ModelHandle gains a provider field
• ADR-0002 (Model routing single source of truth) — TIERS dict extends
• ADR-0005 (Anthropic SDK direct, mitigation seam at llm.call) — this ADR fills the mitigation seam with a second provider; ADR-0005 stays Accepted
• ADR-0006 (Observability stack: Cloud Logging + OTel + BigQuery)
• ADR-0009 (CI/CD GitHub Actions with WIF) — Vertex AI auth via Workload Identity Federation
• ADR 0019 (Session continuity via structured wrap-up logs) — wrap-up generation routes to Vertex AI Gemini 2.5 Flash per this policy
• Counsel review agenda — Vertex AI data-handling posture
• PR #__ — provider field on ModelHandle and TIERS (added when implementation lands)
• PR #__ — Vertex AI provider dispatcher in src/memberintel/llm/ (added when implementation lands)
• PR #__ — eval coverage for routed operations on both providers (added when implementation lands)