ADR-0022: Internal infrastructure authentication model

▶ Watch the 1:20 summary — ADR-0022 — Internal infrastructure authentication model, explained

Status: Proposed
Date: 2026-05-18
Decider: Seth Shoultes (with Blair Williams approval per decision-rights matrix; counsel reviews IAM posture before production launch)
Author: Seth Shoultes

Context

MemberIntel V1 began with a single Cloud Run service (memberintel-api-staging) handling all routes — public auth/chat/sites endpoints plus a single internal endpoint (/internal/wrap-up-sweep) called by Cloud Scheduler. The service ran as the GCP default compute service account. Secret material (8 Secret Manager secrets) was injected as plaintext env vars via Terraform’s value = var.* pattern. Cloud Scheduler authenticated to the internal endpoint via a shared bearer token in an X-Internal-Auth header, stored in a long-lived Secret Manager secret.

This worked for the first few PRs but accumulated several security debts that emerged during Copilot review cycles:

1. Plaintext secrets in Cloud Run revision config (visible to anyone with gcloud run revisions describe)
2. Plaintext secrets in Terraform state (sensitive but readable by anyone with state access)
3. Shared bearer in scheduler job definition (visible via gcloud scheduler jobs describe)
4. Default compute SA as Cloud Run identity (concentration risk — any service compromise grants broad project access)
5. No formal pattern for future internal endpoints (next scheduled job would copy the same shared-bearer mistake)

The decision the project hadn’t made was how internal infrastructure components authenticate to each other, and what the IAM topology should look like. PR #28 introduced the Cloud Scheduler tick-and-backfill pattern (with shared-bearer auth). PR #36 then refactored that pattern into the model this ADR codifies. Future internal endpoints — the wrap-up sweep advisory lock work in #30, the intra-session compaction job proposed in #20, and any future batch-style workload — need a documented pattern to follow.

Three architectural options were considered: (a) shared-bearer with rotation, (b) Cloud Run-native authenticated invocation with platform IAM, (c) OIDC tokens validated app-side. The decision below adopts (c) for reasons documented in the alternatives section.

Decision

MemberIntel adopts a three-service-account topology with Google OIDC tokens for service-to-service authentication and Secret Manager runtime references (not env-var injection) for secret material delivery.

Service account topology

Service account	Identity for	Roles (least-privilege)
memberintel-deploy (existing)	GitHub Actions WIF + manual operator tasks	artifactregistry.writer, cloudsql.admin, iam.serviceAccountUser, run.admin, secretmanager.secretAccessor, storage.admin
mi-runtime-${env} (new in PR #36)	Cloud Run service identity	per-secret secretmanager.secretAccessor (only the secrets the runtime reads — explicit allowlist in local.runtime_secrets), project-level cloudsql.client
mi-scheduler-${env} (new in PR #36)	Cloud Scheduler jobs	run.invoker on the specific Cloud Run service, iam.serviceAccountTokenCreator on itself (for OIDC minting)

Service account IDs are capped at 30 characters per GCP limits. The mi- prefix keeps mi-scheduler-production within bounds (23 chars).

Internal endpoint authentication

Cloud Run stays allUsers invoker — splitting the service to use platform-level IAM per route is not supported by Cloud Run (service-level only) and would force a public/internal service split that doesn’t pay for itself at V1 scale.

Scheduler-driven internal endpoints live under /internal/* (no API version prefix) in the same Cloud Run service. Each internal endpoint validates a Google-signed OIDC token in the Authorization: Bearer header. The validator checks: signature against Google’s certs, expiry, audience matches the Cloud Run URL, issuer is Google, email claim matches a configured expected service account email.

Historical note: an earlier /api/v1/internal/* route family briefly existed at src/memberintel/api/sites/router.py (a single legacy cron handler whose CRON_SECRET check had been commented out). Audit in #38 found it was unscheduled (no Terraform scheduler job) and unused (no in-repo callers) — but because Cloud Run’s invoker is allUsers and the route carried no auth dependency, it was nonetheless publicly reachable for the window it existed. PR #159 deleted it. The /internal/* (no version prefix) family below is now the only internal-routes surface, and the audit-trail + SA-binding claims in this ADR apply unconditionally.

The validator pattern (in src/memberintel/api/wrap_up/router.py):

def _require_oidc_token(authorization: str | None) -> None:
    if not authorization:
        raise HTTPException(status_code=401, detail="missing bearer token")
    if not authorization.lower().startswith("bearer "):
        raise HTTPException(
            status_code=401,
            detail="malformed authorization header (expected 'Bearer <token>')",
        )
    token = authorization.split(" ", 1)[1]

    audience = os.environ.get("INTERNAL_OIDC_AUDIENCE")
    expected_sa = os.environ.get("INTERNAL_OIDC_SCHEDULER_SA")
    if not audience or not expected_sa:
        raise HTTPException(status_code=500, detail="internal OIDC config missing")

    try:
        claims = id_token.verify_oauth2_token(token, _GOOGLE_REQUEST, audience=audience)
    except ValueError as exc:
        # Bad token (bad signature, wrong audience, expired). Log full reason
        # server-side; return a generic message to the public endpoint.
        logger.warning("OIDC token rejected: %s", exc)
        raise HTTPException(status_code=401, detail="invalid token") from exc
    except Exception as exc:
        # Transient upstream — Google identity service unreachable, cert-fetch
        # failed, network blip. NOT an auth failure. Scheduler retries on 503.
        logger.warning(
            "OIDC validation upstream error: %s: %s",
            type(exc).__name__, exc, exc_info=True,
        )
        raise HTTPException(
            status_code=503,
            detail="OIDC validation temporarily unavailable",
        ) from exc

    if claims.get("email") != expected_sa:
        raise HTTPException(status_code=401, detail="token issued by unexpected service account")
    if claims.get("iss") not in ("https://accounts.google.com", "accounts.google.com"):
        raise HTTPException(status_code=401, detail="invalid issuer")

The audience env var and the expected SA env var are populated from Terraform; both must be set or the endpoint returns 500 (deployment error, not auth failure).

Cloud Scheduler invocation

Scheduler jobs use the oidc_token block, not header injection:

http_target {
  uri = "${var.cloud_run_api_uri}/internal/wrap-up-sweep"
  http_method = "POST"
  body = base64encode(jsonencode({ mode = "tick" }))
  oidc_token {
    service_account_email = google_service_account.scheduler.email
    audience              = var.cloud_run_api_uri
  }
}

The audience comes from var.cloud_run_api_uri (single source of truth — same variable feeds the app’s INTERNAL_OIDC_AUDIENCE env). Drift between scheduler-issued audience and app-validated audience is impossible by construction.

Secret material delivery

Cloud Run env vars for sensitive material use value_source.secret_key_ref pointing at Secret Manager. Rotation = gcloud secrets versions add + Cloud Run revision roll. No Terraform apply required. ADR 0011 (Secrets Management — Secret Manager + KMS) prescribes a memberintel/{environment}/{SECRET_NAME} path-prefixed convention. This PR keeps the existing flat naming (memberintel-{purpose}-${var.environment}) to avoid a rotation event during pre-launch hardening; migrating to the path-prefixed convention is tracked as follow-up work under #37 and should land before production cutover.

env {
  name = "JWT_SECRET"
  value_source {
    secret_key_ref {
      secret  = google_secret_manager_secret.jwt_secret.secret_id
      version = "latest"
    }
  }
}

Per-secret IAM binding (not project-wide). The runtime SA gets secretmanager.secretAccessor only on the secrets it actually reads — list maintained in local.runtime_secrets.

Operator escape hatch

The tools/wrap_up_sweep.py CLI uses gcloud auth print-identity-token --impersonate-service-account=<scheduler-SA-email> --include-email to mint an OIDC token AS the scheduler SA. This impersonation requires the operator’s gcloud principal to hold roles/iam.serviceAccountTokenCreator on the scheduler SA. This grant is not provisioned by Terraform — it must be added out-of-band by an authorized administrator (typically scoped to the on-call group). The grant is kept narrow on purpose: broadly granting TokenCreator on the scheduler SA would let any deploy-SA holder mint tokens that bypass the per-SA audit trail this auth model establishes.

Pattern scope

This pattern applies to ALL future internal endpoints:

• Wrap-up sweep advisory lock work (#30) — same auth, same SAs
• Intra-session compaction batch job (#20) — same pattern when built
• Cross-pollination job (per ADR 0014) — when built, uses the scheduler SA via OIDC
• Any future scheduled / batch / cron-style workload

Specifically NOT for:

• Public user-facing endpoints (auth/chat/sites) — those continue to use JWT validation per ADR 0007
• WP plugin → API (/api/v1/auth/redeem-connect-code and friends) — uses PKCE per ADR 0017

Consequences

Positive:

• No plaintext secrets in Cloud Run revision config for the 7 migrated env vars (JWT_SECRET, ANTHROPIC_API_KEY, VOYAGE_API_KEY, RESEND_API_KEY, GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET, MP_ENCRYPT_KEY). DATABASE_URL remains a plaintext composite of host/user/db-name/password — refactoring to separate DB_PASSWORD + DB_HOST env vars is tracked under the existing ‘Conditions for revisiting’ clause and is out of scope for this PR.
• No shared bearer in scheduler job definitions
• Per-SA audit trail requires app-side logging of the validated email claim from _require_oidc_token. Because Cloud Run remains allUsers invoker, the platform request log does NOT populate principalEmail with the scheduler SA — the platform sees anonymous public requests. The per-SA trail is realized by the structured logging requirement tracked under #34; without it, only the FastAPI access log captures /internal/* calls (status code, path, timestamp — no caller identity). Operators investigating a failed scheduler tick must correlate Cloud Scheduler job logs (which DO record the scheduler SA mint event) with the app’s 200/401/500 responses on /internal/wrap-up-sweep.
• Future internal endpoints have a documented pattern — no per-feature security design needed
• Secret rotation decouples from Terraform applies
• SOC 2 / pen-test posture is materially better than the V1-kickoff state
• Cloud Run revision rollback is automatic when secret resolution fails — keeps prior revision serving

Negative / costs:

• Cloud Run cold-start gains a Secret Manager resolution step. Cloud Run resolves value_source.secret_key_ref references in parallel during container start, so the overhead is bounded by max(per-secret-fetch-latency) plus a small fixed overhead, not the sum. The actual figure is not yet measured on this app — we expect single-digit-hundreds-of-ms based on the GCP product docs but will establish the baseline empirically as part of #34
• App startup gains a dependency on Secret Manager API availability (mitigated by Cloud Run rollback)
• OIDC validation depends on Google’s identity service for cert distribution (cached after first call)
• More Terraform resources to maintain (3 SAs, per-secret IAM bindings, OIDC blocks)
• Operator CLI now requires iam.serviceAccountTokenCreator on the scheduler SA — explicit IAM step

Mitigations:

• Cold-start latency revisited if p99 crosses 5s (observability gap tracked in #34)
• Secret Manager outage = Cloud Run can’t scale new instances, but prior revision keeps serving
• Google identity service outage = scheduler ticks fail (visible in Cloud Scheduler execution history). The manual gcloud auth print-identity-token --impersonate-service-account=... escape hatch ALSO depends on the same Google identity service for token minting, so it does NOT provide independent recovery during such an outage. There is no fallback path during a Google-side outage other than waiting for restoration — this is an accepted vendor dependency. If durable independence becomes a requirement, the only workable answer is a non-OIDC out-of-band trigger (e.g., a self-hosted webhook with a separately-rotated bearer), which would re-introduce the shared-secret class of risks this ADR’s Alternative 1 rejected.
• IAM step for operators is documented in the runbook attached to the scheduler+OIDC hardening spec

Observability of auth-path failures

App-side rejections from _require_oidc_token (wrong audience, wrong SA email, expired/invalid token, malformed Authorization header) currently return 401 from FastAPI without emitting structured warnings — only the FastAPI access log captures the rejection. The pre-launch observability baseline (#34) tracks the requirement to add structured logging on each rejection path: timestamp, claim email (if extractable), claim aud (if extractable), rejection reason. Until that lands, operators investigating a failed scheduler tick should check (1) the Cloud Scheduler job’s execution history in gcloud scheduler jobs describe or the Cloud Console — that records whether the OIDC mint succeeded — and (2) the FastAPI access log on the Cloud Run service for the matching /internal/wrap-up-sweep request and its status code. The validated caller identity is not in either log until #34’s structured logging requirement lands.

Eval coverage

Negative-path coverage for _require_oidc_token lives in tests/unit/test_wrap_up_router.py: missing Authorization header, malformed header, invalid token (ValueError from google-auth), wrong SA email claim, wrong issuer claim, wrong-audience rejection, and an explicit assertion that the audience kwarg is passed to verify_oauth2_token. Expired-token coverage relies on google-auth’s internal exp claim check (mocked at the library boundary); a follow-up to add an explicit expired-token unit test should land before production cutover and is tracked under #34.

• test_internal_sweep_returns_503_on_cert_fetch_failure — non-ValueError exception from google-auth (simulated as ConnectionError) returns 503 instead of 401, so the scheduler retries on transient upstream blips.

Bootstrap and IAM propagation

Two foot-guns the next engineer building a new internal endpoint should know about:

1. First-apply ordering. Cloud Run’s value_source.secret_key_ref resolves version = "latest" at revision-create time. If Terraform creates the Cloud Run revision before the corresponding google_secret_manager_secret_version resource finishes, the revision fails to start. The Cloud Run service resource includes depends_on for both the per-secret IAM bindings AND the secret-version resources to enforce ordering.
2. IAM propagation lag. GCP IAM bindings can take up to ~7 minutes to propagate. A scheduler job whose SA was bound moments earlier can 403 on its first tick. Scheduler’s built-in retry policy absorbs this in practice; operators should not interpret a first-tick 403 in the minutes after a fresh deploy as a misconfiguration unless it persists past the propagation window.

Alternatives considered

Alternative 1: Shared-bearer with rotation (status quo from PR #28)

Keep the X-Internal-Auth header pattern but add a rotation schedule.

Rejected because the shared secret remains in scheduler job definitions, Terraform state, and Cloud Run env vars. Rotation reduces blast-radius window but doesn’t address visibility. OIDC’s per-call ephemeral tokens are structurally better.

Alternative 2: Cloud Run authenticated invocation, per-endpoint IAM

Drop allUsers invoker. Add allUsers IAM binding only for public paths. Internal endpoints rely on platform IAM (Cloud Run rejects unauthenticated requests before the app sees them).

Rejected because Cloud Run IAM is service-level only, not per-path. The only way to bind IAM per endpoint is to split the service in two (public + internal). That doubles the operational surface, doubles the Terraform footprint, and creates a deployment-coordination problem for shared code. App-side validation is the smaller change with equivalent security outcome.

Alternative 3: Hybrid bearer + OIDC (defense in depth)

Require BOTH a valid OIDC token AND a matching shared bearer. Reject if either fails.

Rejected because the bearer remains a long-lived shared secret to rotate. “Defense in depth” past two correctly-validated checks at the same trust boundary (HTTPS to Cloud Run) doesn’t add security — both checks share the same threat model. One correct check is better than two redundant ones plus the operational tax of rotating bearer.

Alternative 4: Single service account (operational simplicity)

Use memberintel-deploy for everything — GitHub Actions, Cloud Run runtime identity, Cloud Scheduler OIDC minting.

Rejected because the deploy SA already has broad project access (Artifact Registry write, Cloud SQL admin, GCS state, Run admin). A compromise of that SA — through a leaked WIF token, a malicious dependency, etc. — would grant the attacker the ability to mint OIDC tokens, read all secrets, redeploy any service, AND modify the Terraform state. The three-SA split bounds the blast radius of each compromise path. Concentration risk is what we’re explicitly trying to avoid.

Conditions for revisiting

1. Cloud Run cold-start p99 exceeds 5s once measured baseline is established (per #34) → revisit parallel secret-resolution behavior or cache secrets in a startup script. The 5s threshold is a placeholder until the empirical baseline lands; recalibrate to “baseline + N×stddev” once measured.
2. OIDC validation latency drift on /internal/* → pre-warm Google’s cert cache at app startup, or maintain an explicit cert cache
3. A new internal endpoint needs a different audience than the Cloud Run URL → cross-project OIDC or custom domain switch. Revisit the audience model
4. Auditor flags DATABASE_URL plaintext composite → refactor URL construction app-side using separate DB_PASSWORD (Secret Manager) + DB_HOST (plain) env vars
5. allUsers invoker becomes a compliance blocker → split the Cloud Run service into public and internal services, move internal to authenticated-only Cloud Run with platform IAM

Related

• ADR 0001 (Entitlement service shape — memberintel-deploy SA stays unchanged)
• ADR 0003 (Per-tenant isolation: shared-schema with RLS — unaffected, runtime SA still scoped per-request)
• ADR 0007 (Auth via JWT + Google OAuth + AI Foundation PKCE — public endpoint auth; complementary to this ADR’s internal pattern)
• ADR 0009 (CI/CD GitHub Actions WIF — deploy SA unchanged; this ADR adds runtime + scheduler SAs alongside)
• ADR 0010 (Observability: Cloud Logging + OTEL + BigQuery — per-SA audit logs flow to existing sink)
• ADR 0011 (Secrets management — Secret Manager + KMS; foundational for the value_source.secret_key_ref delivery pattern. The path-prefixed naming convention from ADR 0011 is not yet adopted in local.runtime_secrets — migration tracked under #37 per the Secret material delivery subsection above.)
• ADR 0013 (GCP project structure — single project for staging; production deploys use the same patterns)
• ADR 0014 (Cross-pollination security boundary — the future cross-pollination job uses the scheduler SA via OIDC per this pattern)
• ADR 0017 (MemberIntel Connect WordPress plugin as data source — PKCE for plugin auth; complementary to this ADR’s scheduler-OIDC pattern)
• ADR 0019 (Session continuity wrap-up logs — original consumer of the /internal/wrap-up-sweep endpoint)
• ADR 0020 (Multi-provider model routing — future cross-provider work uses runtime SA’s Secret Manager access pattern)
• Spec docs/superpowers/specs/2026-05-18-secrets-and-oidc-hardening-design.md — implementation detail for this ADR
• PR #28 (Cloud Scheduler initial wiring, used shared bearer — superseded by this ADR’s pattern)
• PR #36 (Implements this ADR)
• Issue #30 — wrap-up sweep advisory lock (next internal endpoint following this pattern)
• Issue #34 — pre-launch observability baseline (makes the latency thresholds enforceable)