ADR-0011: Secrets Management — Secret Manager + KMS + Path-Prefixed Naming

Status: Accepted
Date: 2026-05-14
Deciders: Seth (Lead Architect)

Context

The MemberIntel V1 deployment on GCP Cloud Run requires a secrets management strategy that is:

1. Auditable — privacy counsel (per ADR-0007 and the Blair/Seth working session) will review data-handling practices before GA
2. Environment-aware — staging and production must not share or leak secrets
3. Single-source-of-truth — the current Terraform configuration passes secrets as sensitive variables through state, which means they appear in the GCS backend bucket in plaintext unless remote state encryption is configured
4. Developer-friendly — local development must work without GCP credentials, using the canonical ~/.config/dev-secrets/ path convention established in the project’s CLAUDE.md

The current Terraform (infra/main.tf) creates Secret Manager resources but injects their values as plain environment variables on Cloud Run via var.* references. This means:

• Secret values transit through Terraform state (stored in a GCS bucket)
• Secret rotation requires a terraform apply with the new value
• The naming convention memberintel-{name}-{environment} is flat and error-prone when more environments or services are added

GCP KMS keyring memberintel-keys already exists with four keys: app-data-key, secrets-key, backups-key, audit-key. The secrets-key is unused; this ADR puts it to work.

Code that reads secrets uses environment variables — os.environ.get("ANTHROPIC_API_KEY") in src/memberintel/llm/call.py, os.environ.get("VOYAGE_API_KEY") in src/memberintel/api/retrieval/embed.py, and the FastAPI settings module. This pattern must be preserved; the injection mechanism changes, not the application interface.

Decision

GCP Secret Manager is the single source of truth for all production secrets, with KMS-encrypted payloads and path-prefixed naming. Applications read secrets via environment variables injected by Cloud Run’s native Secret Manager integration — never from Terraform variables, never from container images, never from .env files in production.

1. Path-prefixed naming convention

All secrets in GCP Secret Manager follow the pattern:

memberintel/{environment}/{SECRET_NAME}

Examples:

• memberintel/production/DATABASE_URL
• memberintel/staging/ANTHROPIC_API_KEY
• memberintel/production/MCP_CLIENT_SECRET

This replaces the current flat memberintel-{name}-{environment} convention. Path prefixes give us:

• Hierarchical IAM policies (grant a service account access to memberintel/production/* or memberintel/staging/*)
• Clear environment boundaries in the GCP console
• Natural extensibility for future services (e.g., memberintel/production/worker/REDIS_URL)

2. KMS encryption of secret payloads

The existing KMS keyring memberintel-keys (key secrets-key) encrypts secret payloads at rest in Secret Manager. Secret Manager supports CMEK (Customer-Managed Encryption Keys); each secret is configured with:

replication {
  automatic {
    customer_managed_encryption {
      kms_key_name = google_kms_crypto_key.secrets_key.id
    }
  }
}

This ensures GCP personnel cannot read secret values without access to the KMS key, satisfying privacy-counsel requirements for data-at-rest protection.

3. Cloud Run native Secret Manager injection

Cloud Run services mount secrets as environment variables via the secret block, replacing the current value = var.* pattern:

containers {
  env {
    name = "ANTHROPIC_API_KEY"
    value_source {
      secret_key_ref {
        secret  = google_secret_manager_secret.anthropic_api_key.secret_id
        version = "1"
      }
    }
  }
}

This eliminates secrets from Terraform state entirely — the Terraform resource creates the secret container and its version in one apply, but the Cloud Run service pulls the value at runtime from Secret Manager directly. The secret_key_ref block is the only connection; no sensitive variable holds the value.

4. Local development: .env files from canonical dev-secrets

Local development uses the pattern established in the project’s global CLAUDE.md:

# Source personal-everywhere keys, then project-isolated overrides:
set -a
source ~/.config/dev-secrets/secrets.env
source ~/.config/dev-secrets/memberintel/secrets.env
set +a

The project .env file (gitignored) is symlinked or copied from ~/.config/dev-secrets/memberintel/secrets.env. The .env.example file in the repo documents the required variable names with empty values.

Code reads exclusively from os.environ — never from a local vault, never from a .env parser that falls back to hardcoded defaults.

5. No secrets in code, containers, or Terraform state

Surface	Prohibited	Enforcement
Source code	Hardcoded keys, default key values, .env committed to git	git grep -i "api_key|secret|password" src/ in CI; .gitignore blocks .env and .env.local
Container images	.env files baked into Docker layers	Dockerfile has no COPY .env or ARG for secrets; multi-stage build discards build-time env
Terraform state	Secret values in terraform.tfstate	All secrets use value_source + secret_key_ref on Cloud Run; no sensitive variables hold actual values
Logs	Secret values in structured logs or error traces	Application logger redacts values matching known secret patterns; Cloud Logging exclusion filter on DATABASE_URL, *API_KEY, *SECRET

6. Secret inventory

The following secrets are managed under this convention:

Secret Name	Path	Purpose	Rotation
DATABASE_URL	memberintel/{env}/DATABASE_URL	Postgres connection string (Cloud SQL)	On DB password rotation
JWT_SECRET	memberintel/{env}/JWT_SECRET	HS256 signing key for access/refresh tokens	90 days, or on compromise
ANTHROPIC_API_KEY	memberintel/{env}/ANTHROPIC_API_KEY	LLM calls (src/memberintel/llm/call.py)	On key rotation in Anthropic console
VOYAGE_API_KEY	memberintel/{env}/VOYAGE_API_KEY	Embedding calls (src/memberintel/api/retrieval/embed.py)	On key rotation in Voyage console
GOOGLE_OAUTH_CLIENT_ID	memberintel/{env}/GOOGLE_OAUTH_CLIENT_ID	Google Sign-In (non-secret, but versioned for consistency)	On Google Cloud credential rotation
GOOGLE_OAUTH_CLIENT_SECRET	memberintel/{env}/GOOGLE_OAUTH_CLIENT_SECRET	Google OAuth callback	On Google Cloud credential rotation
ADMIN_API_KEY	memberintel/{env}/ADMIN_API_KEY	Internal admin endpoint authentication	90 days
CRON_SECRET	memberintel/{env}/CRON_SECRET	Cron trigger authentication (reseed, cleanup)	90 days
RESEND_API_KEY	memberintel/{env}/RESEND_API_KEY	Transactional email via Resend	On key rotation in Resend console
MP_ENCRYPT_KEY	memberintel/{env}/MP_ENCRYPT_KEY	Fernet key for MemberPress API key encryption	On compromise; never rotate without re-encrypting stored values
MCP_CLIENT_ID	memberintel/{env}/MCP_CLIENT_ID	Hive Mind MCP OAuth2 client ID	On credential rotation
MCP_CLIENT_SECRET	memberintel/{env}/MCP_CLIENT_SECRET	Hive Mind MCP OAuth2 client secret	On credential rotation
MCP_REFRESH_TOKEN	memberintel/{env}/MCP_REFRESH_TOKEN	Hive Mind MCP OAuth2 refresh token	On token revocation/renewal

7. Rotation procedure

1. Create a new secret version in GCP Secret Manager: gcloud secrets versions add memberintel/staging/ANTHROPIC_API_KEY --data-file=-
2. Update the Cloud Run service to reference the new version (or use version: "latest" for automatic pickup)
3. Verify the new value is live by hitting the /health endpoint and checking the application log for the expected initialization
4. Destroy old versions after the retention window (7 days for audit, then gcloud secrets versions destroy)
5. For DATABASE_URL rotation: change the password in Cloud SQL, update the secret, restart the Cloud Run service. The app_user password is managed by Terraform but its value comes from Secret Manager, not a Terraform variable.

For local development, rotation means updating ~/.config/dev-secrets/memberintel/secrets.env and restarting the dev server.

Consequences

Positive:

• Secrets are entirely absent from Terraform state — no plaintext secrets in the GCS backend bucket, no risk from state file leaks
• KMS CMEK provides an additional encryption layer under Team Caseproof’s control, satisfying privacy-counsel audit requirements
• Path-prefixed naming enables per-environment IAM policies (staging team can’t read production secrets)
• Cloud Run’s secret_key_ref integration means the application code reads plain os.environ.get() — no Secret Manager SDK dependency in application code
• Local development remains simple: source the canonical dev-secrets file, run the server
• Secret rotation is a Secret Manager operation, not a Terraform operation — gcloud CLI is sufficient

Negative / costs:

• Path-prefixed names are longer (memberintel/production/DATABASE_URL vs DATABASE_URL) — this only affects GCP console and CLI ergonomics, not the application
• KMS CMEK adds a dependency: if the secrets-key is destroyed or its IAM policy is misconfigured, the Cloud Run service cannot start. The keyring must have proper IAM bindings and deletion protection.
• Cloud Run secret_key_ref with a pinned version (e.g., version: "1") requires a redeploy when the secret is rotated; using version: "latest" avoids this but trades off immediate-auditability of which version is live
• Adding new secrets requires both a Terraform apply (to create the Secret Manager resource) and a gcloud secrets versions add (to populate the value) — a two-step process
• Local .env files can drift from production values — no automated sync mechanism

Mitigations:

• Pin Cloud Run secret_key_ref to version: "latest" for non-critical secrets (API keys) and to explicit versions for critical secrets (DATABASE_URL, JWT_SECRET) where auditability of “which version is live” matters
• Terraform prevent_destroy lifecycle on all google_secret_manager_secret resources prevents accidental KMS key deletion
• A CI check (scripts/check-secrets-sync.sh, to be created in Slice 1) compares .env.example variable names against the Secret Manager inventory and flags missing secrets
• KMS key secrets-key gets destroy_scheduled_duration = "86400s" (24-hour soft-delete window) and an IAM binding that restricts cloudkms.cryptoKeyDestroy to the memberintel-admin role only

Alternatives considered

• Current approach: Terraform sensitive variables passed as Cloud Run env vars — rejected: secrets appear in Terraform state (stored in GCS), rotation requires terraform apply, no CMEK encryption at rest, flat naming convention doesn’t scale
• HashiCorp Vault on GCP — rejected: operational complexity (Vault cluster, seal/unseal, HA); Secret Manager + KMS provides equivalent CMEK encryption without running a separate service
• AWS Secrets Manager — rejected: ADR-0004 chose GCP as the cloud provider; cross-cloud secrets management adds latency and a second vendor
• Doppler / Infisical (SaaS secrets manager) — rejected: adds a third-party dependency for something GCP Secret Manager does natively; the privacy counsel will prefer GCP-native encryption
• .env files in production (mounted via Cloud Run volumes) — rejected: no audit trail, no rotation without redeploy, no CMEK, no per-secret IAM policies
• Sealed Secrets (Bitnami) for Kubernetes — rejected: Cloud Run is not Kubernetes; no Kubernetes cluster in the architecture