ADR-0021: BigQuery ML as the home for predictive models

▶ Watch the 1:44 summary — ADR-0021 — BigQuery ML as the home for predictive models, explained

Status: Proposed
Date: 2026-05-17
Decider: Seth Shoultes (with Blair Williams approval per decision-rights matrix; counsel reviews model-as-derivative scope of cross-pollination training data)
Author: Seth Shoultes

Context

ADR-0006 commits MemberIntel to BigQuery as the warehouse for analytics and audit. ADR-0011 (in flight) defines the three-roles cross-pollination security boundary with a k-anonymity floor, establishing that aggregated cross-tenant analytical data lives in BigQuery behind the cross-pollination role — the one legitimate cross-tenant data flow in the system. Together these decisions create the data substrate but stop short of what the system does with it.

The product roadmap and the “minion” ideas being scoped for Phase 2/3 (Daily Briefing, Churn-Save, Outlier Detection, Weekly Insights) lean heavily on quantitative grounding: a churn risk score, an anomaly flag, an LTV estimate, a forecast range. Today the only mechanism MemberIntel has to produce these is an LLM, which is expensive (an LLM is a bad calculator), unreliable (LLMs confabulate numbers and shift on percentile thresholds), and economically wrong for high-volume per-tenant inference. ADR 0020 commits to multi-provider LLM routing but does not address predictive ML — it remains an open question where prediction-class work lives.

The decision the project hasn’t yet made is where predictive models live, how they’re trained, and how they’re served. This has implications for the analytics-warehouse architecture, the cross-pollination policy and its counsel review, the customer-facing minion infrastructure, and the unit economics of any product feature that depends on quantitative scoring. Phase 2 minion scoping depends on this resolving; the decision is easier to commit to now while the constraints are clear than to backfill once features are mid-build.

Three approaches are credible: BigQuery ML (BQML) native, Vertex AI custom training, or hand-coded prediction logic without an ML platform.

Decision

MemberIntel adopts BigQuery ML as the default home for tabular predictive models, with Vertex AI Forecast as the specialized home for time-series forecasting where BQML’s ARIMA_PLUS is insufficient. Models are registered, versioned, and tracked in Vertex AI Model Registry regardless of which training surface produced them.

Concrete model assignments:

• Churn risk scoring — BQML logistic regression or boosted trees on aggregated cross-tenant features. Per-member inference on a daily schedule. Output: probability plus feature attribution so the Churn-Save Minion can explain why a member is at risk.
• LTV prediction — BQML linear regression or GBDT on transaction-history features. Per-member inference.
• Anomaly detection — BQML’s ML.DETECT_ANOMALIES or matrix factorization on per-tenant time series. Surfaces refund spikes, signup drops, payment-volume anomalies for the Outlier Detection Minion.
• Cohort clustering — BQML K-means on member behavior features. Powers “members like this one” insights and the cross-pollination prior’s grouping logic.
• MRR and member-count forecasting — Vertex AI Forecast for the production path (better horizon and seasonality handling than BQML at the volumes we expect); BQML ARIMA_PLUS reserved as a fallback for simpler series.

Training data comes exclusively from the cross-pollination role’s view of aggregated tenant data, per ADR-0011 — every training example passes the k-anonymity floor before the model sees it. No raw tenant data flows into another tenant’s predictions; predictions are produced by the shared model plus that tenant’s own RLS-scoped feature vector at inference time.

Model serving: BQML models are queryable directly in BigQuery via ML.PREDICT. Predictions land in tenant-scoped tables (per ADR-0003 RLS posture) for application read. Latency-sensitive inference paths can later move to Vertex AI Endpoints, but V1 uses in-warehouse inference — same region as the warehouse, no separate serving fleet to operate.

Implementation is Phase 3 deliverable. The architectural commitment lands now so Phase 2 minion design can assume the prediction surface exists.

Alternatives considered

Alternative 1: Vertex AI Custom Training as the primary home

Train models in Vertex AI from the start, using BigQuery only as a data source. Models serve via Vertex AI Endpoints.

Strongest argument: Vertex is the more general platform. It supports any framework (TF, PyTorch, XGBoost), any model class (deep networks, recommenders, transformers), and provides a richer serving stack (autoscaling endpoints, monitoring, explanations). For a future with heavier ML investment, Vertex is the right home.

Rejected as the primary home for V1 because the gap to BQML is too large for our scale and our use cases. BQML covers logistic regression, linear regression, GBDT, K-means, matrix factorization, time-series, and anomaly detection — the entire envelope of what the minion roadmap needs. The data already lives in BigQuery. The pipeline cost of “export to GCS, train in Vertex, register, serve via endpoint, persist predictions back to BigQuery” is several engineering weeks of plumbing per model. BQML’s CREATE MODEL ... AS SELECT does the same work in one query. Vertex stays the right home for the specialized case (Forecast) and the future deep-learning case if one ever materializes; BQML carries the load that exists today.

Alternative 2: Build prediction logic ourselves (no ML platform)

Implement churn scoring, LTV, and anomaly detection as hand-coded analytical queries — logistic regression coefficients fit offline, served as SQL CASE WHEN clauses.

Strongest argument: maximum control, no platform dependency, low operational complexity, zero ML-platform cost. The math is well-understood and fits in a few hundred lines.

Rejected because the development cost is wrong. Hand-coding the training loop, cross-validation discipline, feature attribution, model versioning, and drift monitoring is multiple engineer-weeks per model — and we’d need to reinvent each as use cases multiply. BQML provides all of that as platform features. Engineering time on the V1 team is the binding constraint; spending it reinventing scikit-learn in SQL is not the right trade. Worth revisiting only if a specific BQML model-class limitation becomes binding.

Alternative 3: Use only the LLM (Gemini or Claude) for “predictive” work

Skip predictive ML entirely. Have the LLM read the relevant data and produce risk scores, anomaly flags, and forecasts directly.

Strongest argument: zero new infrastructure. The LLM already exists in the stack. For very small data volumes the LLM is plausibly accurate enough to fake it.

Rejected on three grounds. First, LLMs are unreliable on quantitative work — they confabulate numbers, drift on percentile thresholds, and produce different answers for the same input. The eval suite cannot establish confidence intervals for an LLM the way it can for a trained classifier. Second, the economics are wrong — an LLM call per member per day for churn risk is orders of magnitude more expensive than a BQML inference query at the volumes we expect (5K Pro × thousands of members each). Third, the cross-pollination prior (“businesses like yours typically see churn rise to X% before signup drop”) has no LLM substitute; the LLM has no aggregate to consult unless one is precomputed and handed to it — at which point a trained model is the better artifact.

Alternative 4: External ML platform (SageMaker or similar)

Use a non-GCP ML platform.

Strongest argument: AWS SageMaker is the best-known and most mature ML platform in the industry, with deep tooling around model lifecycle, A/B testing, and monitoring.

Rejected on cloud locality. MemberIntel is on GCP. Adding a second cloud for ML training and serving creates the same egress, IAM, audit, and counsel-review costs as adding a second cloud for LLM work — without a corresponding strategic benefit. SageMaker is reasonable to revisit only if a future ML use case has no plausible GCP equivalent, which is not the case for any model class on the roadmap.

Consequences

What gets easier:

• The Phase 2/3 minion roadmap becomes quantitatively grounded. Churn-Save speaks to a real risk number; Outlier Detection has statistical backing; Weekly Insights cites actual aggregates instead of LLM estimates.
• The cross-pollination data flow (ADR-0011) gains a concrete consumer. The k-anonymity floor isn’t theoretical — it’s the boundary between training data and the rest of the system.
• Cost economics for prediction-class work shift from “LLM call per inference” to “warehouse query per inference.” Orders of magnitude cheaper at the volumes we care about.
• Model lineage and reproducibility live in Vertex AI Model Registry. Counsel and customers both get a defensible answer to “where did this prediction come from.”
• The brain becomes citable on quantitative claims. The LLM is told “churn risk is 0.34 per the v3 churn model” instead of being asked to estimate.

What gets harder or constrained:

• BQML’s model-class envelope is finite. If a future use case needs a deep model (e.g., a transaction-sequence transformer analogous to Stripe’s payments foundation model), it falls outside BQML and lands in Vertex custom training. The boundary is fine; it just exists.
• BQML training cost is non-trivial at scale. Training jobs are billed in slot-time, and daily retraining on multi-TB feature tables can add up. Retraining cadence (weekly default, monthly drift review) is a cost-discipline decision that lands alongside the first model.
• The cross-pollination role’s training-data scope needs explicit counsel review beyond ADR-0011’s analytical-access framing. Specifically: is the trained model itself a derivative of the source data that carries the same protections? Default position: yes — treat the model as derivative, version it accordingly, and have counsel ratify the scope before training data flows.
• Inference latency from BigQuery is warehouse-query latency (hundreds of milliseconds). Acceptable for batch and scheduled inference; not acceptable for sub-100ms request paths. The architectural escape hatch (Vertex AI Endpoints) exists but is deferred from V1.
• The eval suite expands to cover models, not just prompts. ML models bring classification metrics (precision, recall, AUC), drift metrics, and fairness considerations. The release-gate posture extends to model evals as a peer of the prompt eval suite.

Downstream commitments this creates:

• A BQML training pipeline (likely Cloud Workflows or Cloud Composer) for each model, with versioned outputs registered in Vertex AI Model Registry.
• A predictions schema in BigQuery — tenant-scoped per ADR-0003, populated by scheduled ML.PREDICT jobs.
• A model-eval methodology distinct from the prompt eval suite. Lives alongside the AI eval suite and shares the release-gate status.
• A retraining cadence and drift-monitoring posture. Default proposal: weekly retraining, monthly drift review, alert on AUC drop > 5 points.
• A counsel review item: model-as-derivative treatment and training-data scope from the cross-pollination role.
• Cost monitoring on BQML slot-time usage, alongside the LLM cost dashboards (ADR-0006).

Deferred deliberately:

• Vertex AI Endpoints serving. V1 uses in-warehouse ML.PREDICT. Endpoints are the escape hatch when sub-100ms latency or non-tabular models force the move.
• Deep models / foundation-model-style training (the Stripe payments-foundation-model analog for membership businesses). Reserved for Phase 4+ if a tabular baseline is proven insufficient.
• Per-tenant fine-tuning or personalization layers on top of the shared model. Reserved for V1.5+.

Conditions for revisiting

1. A use case requires a model class BQML doesn’t support. Triggers a move to Vertex AI custom training for that specific model while BQML retains the tabular envelope. Most likely trigger: transaction-sequence modeling.

2. Inference latency on a prediction-driven feature exceeds product requirements. Triggers a move to Vertex AI Endpoints for that model. Threshold: sustained P95 latency above the user-facing budget for the feature.

3. BQML slot-time costs exceed 5% of total per-tenant infrastructure spend sustained. Triggers retraining-cadence reduction, feature-engineering work to shrink training tables, or a move to materialized incremental training.

4. Counsel raises a model-as-derivative concern that can’t be resolved with the proposed default treatment. Triggers either a tightened training-data scope, a different model-versioning posture, or a deferral of the predictive features to a later phase.

5. Model performance drifts below acceptance thresholds for two consecutive release windows. AUC, precision/recall, or whatever metric the model is gated on. Triggers feature-engineering review, retraining-frequency increase, or model-class change.

6. A product use case emerges that requires a deep-learning model trained on customer data. Most likely trigger: a MemberIntel domain foundation model (the Stripe payments-foundation-model analog). Triggers move to Vertex AI custom training and a substantial counsel re-review.

Related

• SPEC-v2 §6 (Cross-pollination)
• SPEC-v2 §10 (Compliance & Privacy)
• ADR-0003 (Per-tenant isolation: shared-schema with RLS) — prediction outputs live in tenant-scoped tables
• ADR-0006 (Observability stack: Cloud Logging + OTel + BigQuery) — the BigQuery warehouse this ADR builds on
• ADR-0009 (CI/CD GitHub Actions with WIF) — BQML/Vertex auth via Workload Identity Federation
• ADR-0011 (Cross-pollination security boundary: three-roles model) — training data flows through the cross-pollination role’s k-anonymized aggregate; counsel ratifies the model-as-derivative scope
• ADR 0020 (Multi-provider model routing — Vertex AI Gemini as second provider) — prediction work routes to BQML/Vertex rather than to any LLM provider
• Counsel review agenda — model-as-derivative scope added before May 31 engagement
• PR #__ — first BQML model and training pipeline (likely churn risk) (added when implementation lands)
• PR #__ — Vertex AI Model Registry integration (added when implementation lands)
• PR #__ — predictions table schema migration (added when implementation lands)
• PR #__ — model eval methodology and release-gate integration (added when implementation lands)