ADR-0035: Production eval monitoring — in-house sampled scoring, not Braintrust/LangSmith

Status: Proposed — ai-engineer review complete; ceo-blair sign-off pending (see Review log)
Date: 2026-07-14
Deciders: Seth (lead architect / ai-engineer review), Blair (ceo-blair approval — pending)

Review log

• 2026-07-14 — ai-engineer build-vs-buy review complete. Full write-up:
  docs/superpowers/reviews/2026-07-14-ai-engineer-braintrust-buildvsbuy.md.
  Recommends build in-house over Braintrust/LangSmith: the single llm/call.py
  chokepoint and the existing tests/evals/ code-assertion pattern make an
  in-house sampled job cheap to build (days, not weeks); third-party pricing
  is usage/trace-based, the same cost-surface risk the cost-control circuit
  breaker (caseproof/innovations#22) was built to guard against; and sending real customer site
  content + AI-generated summaries to a third-party SaaS is a data-egress
  question in the same category as the global-brain privacy reviews, which
  building in-house avoids entirely. Design detail (sampling rate, scoring
  mechanism per operation, storage, alerting, access control, retention) was
  then worked out collaboratively with Seth — full detail in
  docs/superpowers/specs/2026-07-14-production-eval-monitoring-design.md.
• 2026-07-14 — Blair, relayed by Seth: go ahead and build it. Framed
  explicitly as a staging-only trial — MemberIntel V1 has no live customers
  yet, launch is “the next couple months” out — build the in-house system,
  dogfood it internally on staging, evaluate whether it performs well. If
  it does, run with it; if not, revisit Braintrust. This is directional
  approval to proceed, not the same thing as Blair reading and approving
  this written ADR (see below) — recorded here as informal input shaping
  the rollout plan, not as formal ceo-blair sign-off.
• ceo-blair sign-off: still not formal. Two verbal go-aheads now (look
  into Braintrust; then build it/dogfood it) — real, useful direction, but
  per this repo’s own precedent (ADR-0032 distinguishes informal stand-in
  review from Blair’s actual sign-off), this ADR still needs Blair to read
  and approve the written artifact before it moves to Accepted. Flagging
  explicitly rather than treating a relayed conversation as sign-off.
• counsel review: timing revised. Originally scoped as fully
  build-blocking; given the trial is staging-only with the team’s own
  dogfooding traffic (no real customer content is sampled during this
  phase), building and testing on staging does not need to wait on
  counsel. Counsel review remains required — tracked as a sub-issue under
  caseproof/innovations#340 — before this system samples any real
  production customer traffic, i.e. before it ships live alongside V1’s
  actual launch. Not before the staging trial starts.

Context

Two quality signals exist today: pre-merge synthetic evals (tests/evals/,
real API calls, code-level assertions against hand-written fixtures, CI-only)
and production thumbs-up/down feedback (sparse, human-triggered). Nothing
continuously and automatically scores a sample of real production
completions and alerts on quality drift, hallucination, or genericness
regression between deploys — the v1-spec’s #1-ranked risk (the AI advisor
feeling like generic AI) goes unmonitored in the one place it matters most:
live customer traffic.

Blair asked Seth to evaluate Braintrust (an LLM eval/observability SaaS,
same category as LangSmith) for this. This ADR documents the resulting
build-vs-buy decision and the resulting system’s shape. Full design:
docs/superpowers/specs/2026-07-14-production-eval-monitoring-design.md.

Not the same decision as PR #329 / caseproof/innovations#230. That
proposal recommends Braintrust for a different, narrower problem — an
offline eval harness for experiment-vs-experiment diffing (e.g. proving
Sonnet 5 routing is as-good-or-better for caseproof/innovations#229), run against synthetic
sites and a de-identified staging clone, explicitly scoping out live
production eval as a non-goal. The data-egress and cost-scaling concerns
driving this ADR’s “build” decision barely apply there (no live customer
traffic, no real content unless a de-identified clone is counsel-cleared).
The two are complementary, not competing — caseproof/innovations#230/PR #329’s own
Braintrust recommendation stands on its own merits.

Not the same system as ADR-0030’s /brain/performance — that measures
which authored brain entries get cited/helpful-voted (content curation).
This measures whether a given generated response is grounded and
non-generic (response quality), sampled continuously from live traffic.

Decision

Build in-house. Do not adopt Braintrust or LangSmith.

Hook into src/memberintel/llm/call.py’s existing post-response seam (the
same point _record_success_metric() already runs) with a per-operation
probabilistic sampling decision, scoped in v1 to Operation.CHAT and
Operation.SITE_ANALYSIS only. Sampled calls are scored synchronously,
inline, with plain code-level assertions — no judge-model call, no LLM
involved in scoring:

• Shared: an anti-genericness denylist check, the same word-boundary
  substring-matching technique as tests/evals/test_site_analysis_differentiation.py.
• CHAT: citation validity — every [N] marker must reference a source
  actually retrieved for that turn, the same check test_citation_grounding.py
  runs in CI, applied to real conversations.
• SITE_ANALYSIS: a term-overlap groundedness proxy against the real
  fetched page text, since there’s no invented “canary fact” available on
  real customer sites.

call() gains one new optional parameter, eval_context, so the
groundedness checks have something to validate against; only the two
in-scope callers populate it, every other caller is unaffected.

Storage is a two-table split mirroring cost_breaker/llm_metrics_daily:
eval_sample (raw sampled content, 30-day retention, cleanup job) and
eval_sample_daily (aggregate pass/fail counts, indefinite retention).
Alerting mirrors the cost-breaker pattern exactly: structured log event →
Terraform logs-based metric → alert policy, same on-call channel. Admin
visibility reuses the existing observability admin access — no new
permission. New module: src/memberintel/eval_monitoring/.

Judge-model scoring is explicitly deferred, not rejected — named as a
future addition if code-assertions prove insufficient at scale.

Consequences

Positive:

• Closes a real, previously-unmonitored gap in production response quality
  for the two highest-differentiation-risk operations.
• No new vendor dependency, no new third-party billing surface that scales
  with traffic.
• No customer content leaves the company.
• Reuses proven infrastructure end to end (call.py chokepoint,
  tests/evals/ assertion patterns, cost_breaker/llm_metrics_daily
  storage split, the cost-breaker’s alerting pipeline) — small, well-bounded
  build.

Negative / costs:

• call()’s signature grows by one optional parameter — a small but real
  surface-area increase on the single most sensitive chokepoint in the
  codebase (architecture invariant #1).
• The SITE_ANALYSIS term-overlap check is a proxy, not a hard guarantee;
  it needs tuning against real samples post-launch, not fully specified
  up front.
• Storing sampled real customer content for 30 days, even internally, is a
  new data-retention practice requiring its own review (see counsel item
  below) and its own cleanup-job maintenance burden.
• Judge-model scoring, if added later, must move scoring off the
  synchronous inline path to a genuine background task — the current
  design’s “negligible added latency” claim depends on scoring staying
  cheap (string matching plus one Cloud SQL write, the same kind of
  write _record_success_metric already performs on every call today; no
  LLM calls and no third-party network requests).

Mitigations:

• eval_context is optional and additive; every existing caller that
  doesn’t pass it is completely unaffected, including all operations
  outside this ADR’s scope.
• 30-day retention + a scheduled cleanup job bounds the raw-content
  exposure window; the indefinite eval_sample_daily rollup carries no
  raw content, so long-term trend visibility doesn’t require long-term
  raw-content retention.
• A lightweight counsel first-pass review is required before this system
  samples any real production customer traffic — tracked as a sub-issue
  under caseproof/innovations#340 — but does not block building and
  testing on staging with internal/dogfooding traffic, since no real
  customer content is at stake during that phase.

Rollout plan and evaluation criteria (per Blair, 2026-07-14)

Staging-only trial: build the system, deploy it to staging, dogfood it
with the team’s own internal usage (no live customers yet — V1 hasn’t
launched). Evaluate whether it actually surfaces real quality issues
without excessive noise (false-positive alert rate, whether the
term-overlap proxy for SITE_ANALYSIS needs retuning, whether the
citation-validity check for CHAT catches anything real). If the trial
goes well, ship it as-is for V1 launch (after the counsel review closes,
per above). If it doesn’t — too noisy, insufficient signal, or the
build/maintenance cost turns out higher than expected — Braintrust
remains the explicit fallback, revisiting the build-vs-buy call with
real staging-trial evidence in hand rather than the pre-implementation
reasoning this ADR is based on.

Alternatives considered

• Braintrust / LangSmith (buy) — not adopted for v1, but an explicit,
  named fallback (see Rollout plan above) if the in-house staging trial
  underperforms. Usage/trace-based pricing is a cost-surface risk that
  scales with traffic (the same category of risk the cost-control circuit
  breaker exists to guard against), and it would mean sending real
  customer site content and AI-generated summaries to a third party — a
  data-egress question this repo already treats carefully for a different
  subsystem (global-brain de-identification). Full reasoning:
  docs/superpowers/reviews/2026-07-14-ai-engineer-braintrust-buildvsbuy.md.
• Judge-model scoring in v1 — rejected for now. Would add a second LLM
  call (cost/entitlement implications, its own Operation) for a problem
  the existing code-assertion pattern already handles reasonably for the
  two in-scope operations. Deferred, not rejected outright — revisit if
  code-assertions prove insufficient at scale.
• Separate background job/queue instead of the inline call.py hook —
  considered and set aside for v1. Fully decouples scoring from the request
  lifecycle, which would matter more once scoring gets heavier (e.g. a
  judge-model call), but adds infrastructure (a queue, a worker) this v1
  doesn’t need given scoring is cheap, synchronous string matching.
• Single-table storage (no separate daily rollup) — considered and set
  aside. Would lose trend history past the 30-day raw-retention window and
  re-aggregate on every admin panel load instead of reading a cheap
  pre-computed rollup, for no real simplicity win given the rollup pattern
  already exists in this codebase (llm_metrics_daily).