ADR-0034: Site-content-scan pipeline (structured backbone + rendered-front-end analysis)

Status: Accepted (2026-07-13) — ai-engineer APPROVE-WITH-CHANGES (folded in) + ceo-blair APPROVE. Staging build cleared to start (Layer 1, then Layer 2 immediately behind). Prod go-live remains gated on: counsel first-pass (new third-party-content surface), the consent surface shipping with Layer 2’s prod release, differentiation-eval coverage for the new prompt, and caseproof/innovations#323’s skip-rate number clearing product-lead’s ~30% threshold.
Date: 2026-07-13
Deciders: Seth Shoultes (Lead Architect), Blair Williams (CEO)
Author: Seth Shoultes

Context

The v1-spec (§6.2/§8.4) committed to an analyze_site capability that was
never built. What shipped instead (ADR-0028) is a silent, deterministic,
billing-derived classifier — industry/archetype/business_model inferred from
synced MemberPress data — used only to boost retrieval ranking. It has no
descriptive knowledge of what a customer’s site actually sells, publishes, or
looks like. The advisor can correctly rank a coaching playbook above a SaaS
playbook for a coaching customer, but it cannot answer “what does my homepage
say” or “does my pricing page mention my flagship offer” — the exact
differentiation-from-generic-AI gap the spec ranks as its #1 risk.

caseproof/innovations#314 tracked this divergence between the spec’s ambition
and what shipped. A three-persona review chain (ai-engineer → product-lead
→ ceo-blair) converged on building it, with the added requirement (from
Blair, who originated the analyze_site idea in the spec this project is
based on) that the scan cover the rendered front-end, not just stored WP
database content — a theme, page builder, or gated preview can make what a
visitor actually sees diverge materially from what wp_posts holds.

This ADR does not reopen ADR-0028. Declared industry/archetype remains the
retrieval-boost signal. This is a new, complementary signal that feeds the
chat context directly — the AI describing a customer’s actual business, not
re-ranking which playbook to surface.

Decision

Build a two-layer pipeline, staging-first, gated per the conditions below.

Layer 1 — structured backbone (deterministic, no LLM). Extend
wordpress/plugins/memberintel-connect with one new authenticated REST route
(alongside the existing mi/v1/me, /exchange-code, /info routes in
class-rest-controller.php, following the pattern already established for
data endpoints in class-data-controller.php — get_memberships,
get_transactions, etc.) exposing published posts/pages: title, categories,
publish date, and canonical URL. No content body, no LLM involvement. This
rides the existing trusted sync pipeline (src/memberintel/api/sites/sync.py
sync_site()), the same authenticated, API-key-gated channel every other
synced entity already uses. Persisted as a new SitePage rollup keyed on
(site_id, url), mirroring the existing per-entity sync tables
(_upsert_members/_upsert_memberships in sync.py) — one row per
published post/page, replaced wholesale each sync rather than diffed
incrementally. Output: an authoritative, deterministic URL list — the input
Layer 2 needs to know what to fetch without guessing at a sitemap or
crawling.

Layer 2 — presentation analysis (the actual ask). Server-side fetch the
real rendered front-end — homepage plus a bounded set of key pages chosen from
Layer 1’s URL list (not a crawl, not stored WP content) — reusing the
SSRF-guarded fetch already built and tested for onboarding site-connect:
src/memberintel/api/sites/url_guard.py (assert_safe_wp_url,
normalize_wp_base_url) + the httpx.AsyncClient pattern already used in
connect.py. Reuse note: assert_safe_wp_url validates the initial URL;
Layer 2 must re-validate on every redirect hop rather than flipping
follow_redirects=True — a redirect can point off-host after the initial
check passes.

HTML-reduction stage (required, not optional). A rendered WordPress page
is routinely 100–500KB of markup — an order of magnitude over the
SITE_ANALYSIS operation’s 8K-input cap (~32KB of text), and input_max in
tiers.py is advisory only (call.py clamps max_tokens — output — not
input size; the caller is responsible for staying inside the input budget).
Layer 2 must therefore run an explicit reduction pipeline before the LLM
call: strip <script>/<style>/nav-and-footer boilerplate, extract the
main-content region, then hard-truncate to fit the per-page share of the 8K
budget (e.g. homepage + 3 key pages ≈ 2K tokens/page after reduction). This
stage — its extraction heuristic and the exact page-count/budget split — is
Layer 2 build-time detail, not re-litigated here, but the pipeline’s
existence and the token-budget math above are load-bearing for this ADR’s
cost claim (condition 1) and must not be skipped.

Divergence signal — scoped to what Layer 1 actually carries. Layer 1 has
no content body (title/categories/date/URL only), so it cannot support a
body-level “what’s stored vs. what’s rendered” diff on its own. What it does
support for free: a synced URL that 404s, redirects, or renders behind a
gate when Layer 2 fetches it; a <title> mismatch between the stored post
title and the rendered page’s title tag. That structural/URL/title-level
divergence is the emergent, no-extra-scope signal this design produces.
Content-level positioning analysis (“does the homepage still say what the
customer thinks it says”) is Layer 2’s LLM analysis alone, not a diff.

The fetched HTML is analyzed through the already-reserved SITE_ANALYSIS
operation (src/memberintel/llm/operations.py, src/memberintel/entitlement/ tiers.py — Sonnet, 8K input / 4K output cap, 4 runs/mo Free, 30/mo Pro,
currently zero consumers). This is the only new anthropic-touching code
path; it goes through src/memberintel/llm/call.py like every other LLM call
(architecture invariant #1) — no new import site.

ceo-blair’s approval conditions (binding on this ADR)

1. Budget: capped at the already-reserved SITE_ANALYSIS slot — no new
   budget line, no new tier entry.
2. Kill-switch required — wire to the existing cost-control circuit
   breaker (src/memberintel/llm/cost_breaker.py, the call.py chokepoint,
   caseproof/innovations#22). Layer 2 is an LLM-call path like any other;
   check_cost_breaker must gate it identically.
3. Counsel first-pass required before PROD (not before staging/build) —
   this introduces a new data-usage surface (extracted third-party site
   copy). It becomes a harder gate the moment the LLM performs inference
   about positioning/strategy rather than surfacing extracted text verbatim —
   the same trigger ADR-0028’s classification component hit with counsel.
4. Consent surface ships with Layer 2’s prod release, not as a
   fast-follow — a “here’s what we understand about your business” view in
   Settings/Data Sources, part of the counsel packet (tracked as
   caseproof/innovations#322’s sub-task, this repo’s Task #34 in the current
   session’s tracking).
5. Free tier keeps baseline grounding — do not gate the existence of
   site knowledge behind Pro. The tier split (4 vs. 30 monthly runs) gates
   depth/freshness/on-demand refresh, not access.
6. Sequencing: Layer 1 lands first (cheap, de-risks the fetch) but is not
   a shippable milestone alone — Layer 2 follows immediately behind in the
   same effort, not a separately-scheduled follow-up.
7. Layer 2’s prod go-live is gated on the skip-rate telemetry number
   (caseproof/innovations#323 — now shipped, PR #410). Staging build proceeds
   regardless of that number; only the prod launch-blocking decision depends
   on it (product-lead’s stated threshold: >~30% of activations skipping the
   business_profile declare step makes this launch-blocking rather than
   merely desirable).

Consequences

Positive:

• Closes the v1-spec’s #1-ranked differentiation risk — the advisor gains
  real descriptive knowledge of a customer’s business instead of only a
  billing-derived classification.
• Reuses three pieces of already-built, already-tested infrastructure
  (SSRF guard, entitlement slot, cost breaker) — no new subsystem, no new
  budget line, minimal net-new surface area.
• The structural/URL/title-level divergence signal (see Decision) is a
  genuinely new capability — no other part of the system compares a synced
  post’s stored URL/title against what actually renders — and it falls out
  of the two-layer design without separate scope.

Negative / costs:

• New third-party-content data-usage surface — requires counsel sign-off
  before prod (condition 3), adding a review-cycle dependency this feature
  didn’t have before.
• Layer 2 is the second consumer of SITE_ANALYSIS capacity ever exercised
  in production (the slot has existed with zero consumers) — its real-world
  cost/latency/failure-mode profile is unproven until staging traffic
  exercises it.
• Layer 2 is a new prompt surface; the differentiation eval suite (guarding
  against the v1-spec’s #1 risk, which this ADR exists to address) must gain
  coverage for it before prod, per the eval-coverage decision gate.
• A consent-surface UI (condition 4) is now a hard dependency of Layer 2’s
  prod release, not an optional nice-to-have — adds scope that must land in
  the same push as the LLM-analysis code, not after.

Mitigations:

• Staging build and staging-only traffic can proceed without the counsel
  gate or the skip-rate threshold — only prod go-live is blocked, so the
  engineering work is not stalled waiting on either.
• The kill-switch (condition 2) and existing per-tier monthly caps bound
  worst-case cost exposure identically to every other SITE_ANALYSIS-style
  operation already in production.
• Redirect-hop re-validation (Layer 2 reuse note above) closes the one SSRF
  gap that reusing assert_safe_wp_url naively would reopen. Layer 2
  inherits connect.py’s existing check-then-fetch DNS-rebinding posture
  unchanged — this ADR does not worsen it, and closing it is out of scope
  here.

Alternatives considered

• Skip Layer 1, fetch rendered pages via a public sitemap or crawl.
  Rejected — no authoritative list of what’s actually published vs. draft/
  private, no trusted-channel guarantee (a public crawl isn’t behind the
  connect plugin’s API-key auth), and no natural place to bound “which pages
  matter” without re-deriving what Layer 1 already knows deterministically.
• Skip Layer 2, only ship the structured backbone (titles/categories/
  dates). Rejected by the persona review chain — this doesn’t touch the
  actual ask (Blair: “there might be a different view on the front end than
  in the back end”); titles and categories alone don’t tell the AI what a
  homepage or pricing page actually says.
• Reopen ADR-0028 and fold this into the existing profile classifier.
  Rejected — ADR-0028’s classifier is deterministic, billing-derived, and
  retrieval-scoped by design; conflating it with an LLM-based, content-scoped,
  chat-context signal would blur two decisions that should stay independently
  reviewable (and ADR-0028’s counsel gate is already narrowly scoped to
  billing-derived classification, not third-party site copy).