ADR-0037: Staging-suffixed resources are production; add a sandbox tier

Status: Accepted
Date: 2026-07-21
Deciders: Seth

Context

No environment split was ever formally decided for MemberIntel V1’s GCP infrastructure. The -staging-suffixed Terraform resources (memberintel-api-staging, memberintel-db-staging, etc.) have been serving api.membersintel.com’s real, live traffic since before this was named or documented — several runbooks and CI config comments still asserted “there is no production environment,” which was already inaccurate by the time it was written down.

This left the team without a safe pre-prod tier: any infra or backend change had to be tested directly against what was, in fact, production. caseproof/innovations#369 scoped fixing this — not by renaming or migrating the live stack (real data-migration risk, zero user-facing benefit), but by formally naming today’s reality and adding a genuinely new, disposable tier for testing.

Decision

1. Formally document that the -staging-suffixed resources ARE production. No rename, no migration, no DNS change, no downtime. docs/runbooks/subdomain-map.md and docs/runbooks/01-deploy.md were updated to state this plainly instead of the stale “no production environment” framing.
2. Add a third environment value, sandbox, as a fully separate Cloud SQL + Cloud Run + VPC stack under *-sandbox.membersintel.com subdomains — a place to break things safely without touching what’s live. Sandbox reuses production’s third-party API keys and OAuth client (deferred credential isolation to a future need, not this plan’s scope) and is explicitly not drift-checked nightly (.github/workflows/terraform-drift.yml’s matrix stays [production, staging]) — it’s meant to be disposable and frequently broken by design, so nightly drift-checking it has less value than for the tiers that matter, and would mostly generate noise.
3. Isolate the project-global GCP singletons (Artifact Registry, BigQuery datasets/sinks, the tf-plan drift SA, the billing budget — resources with no environment suffix, shared across the one GCP project) into an optional project_globals module, owned exclusively by the existing -staging-named state. A new environment’s Terraform state sets manage_project_globals = false so it never tries to recreate — or accidentally destroy — those shared singletons.

Consequences

Positive:

• A real, isolated pre-prod tier exists today, reachable without any risk to live traffic.
• No migration risk taken on — production’s actual serving infrastructure is untouched by this decision.
• The project_globals split establishes the reusable pattern (manage_project_globals) that any future environment (including a genuine production tier) can adopt without re-deriving it.

Negative / costs:

• GCP console naming stays confusing (-staging = production) indefinitely, unless a future migration project is explicitly chartered.
• Sandbox has no drift detection — an out-of-band console change there can silently diverge from infra/ with no alert. Acceptable given sandbox’s disposable-by-design role; revisit only if it starts being treated as a long-lived shared resource rather than a personal test environment.
• Sandbox shares production’s third-party API keys and OAuth client rather than having its own — a real blast-radius tradeoff (a sandbox-side key leak or quota exhaustion affects production’s own usage) accepted to keep this plan’s scope to infra isolation, not credential provisioning across every vendor.

Mitigations:

• Sandbox’s Terraform state is fully separate (its own GCS backend prefix) from both staging and any future production state, so a sandbox terraform destroy can never reach the resources this decision protects.
• manage_project_globals = false is enforced per-environment at the tfvars level, making an accidental attempt to re-manage the shared singletons from a new environment’s state visible in terraform plan before it could ever apply.

Alternatives considered

• Rename -staging resources to -production. Rejected: real Cloud SQL data-migration risk and a coordinated domain-mapping cutover, for zero user-facing benefit — the resources already correctly serve production traffic under whatever name they carry.
• A genuinely separate, new production stack with a live cutover from the old -staging stack. Considered the more architecturally “correct” long-term fix, and has since been chartered as its own follow-on plan (docs/superpowers/plans/2026-07-16-three-tier-environment-architecture.md, caseproof/innovations#369’s Tasks 1-6 already merged as of this ADR’s date; the live cutover itself is still pending a dedicated session). Explicitly out of scope for the decision recorded here — this ADR covers only the sandbox-tier addition, not the eventual real production/staging split.
• A drift-check leg for sandbox, mirroring production/staging’s nightly terraform-drift.yml matrix. Rejected for now per the Decision section above; the plan’s own Task 7 recommendation was to skip and document rather than add machinery of unproven value for a disposable environment.