PSA/ee/docs/plans/2025-09-19-custom-portal-domains-plan.md

Title: Customer Portal Custom Domains – Design & Implementation Plan Date: 2025-09-19

Overview / Rationale

• Goal: Enable enterprise tenants to surface portal traffic on a vanity hostname while preserving seamless operation for CE tenants who remain on the default fleet domains.
• Approach: Tenants create a DNS CNAME that targets a canonical host we control (<tenant7>.portal.algapsa.com). We manage lifecycle state, DNS verification, and Istio configuration via Temporal so the portal can respond on both the canonical host and any approved vanity CNAMEs.
• Success Criteria: Exactly one active domain per tenant, accurate status visibility in the Client Portal settings UI, automated Istio/cert-manager reconciliation, and resilient observability hooks (OpenTelemetry traces, PostHog metrics) around the workflow.
• Non-goals: Supporting multiple domains per tenant, delegating DNS into customer zones, or replacing cert-manager.

Key Decisions & Clarifications

• Canonical target: Store <first 7 chars of tenant_id>.portal.algapsa.com in the database so DNS guidance remains stable even if tenant metadata shifts.
• Certificates: Re-use the existing wildcard *.portal.algapsa.com certificate for the canonical ingress path. Vanity domains CNAME to the canonical host; cert-manager issues per-domain certificates through ACME using HTTP-01, leveraging our ingress path. No new Route53 zones required.
• Reconciliation: Temporal activity generates desired manifests, writes them into the nm-kube-config/alga-psa repo, applies via kubectl/Helm, and commits back to Git for auditability (no standalone operator).
• Istio config source: nm-kube-config/alga-psa is the single source of truth; the activity rewrites YAML based on DB state, ensuring every change is versioned prior to applying in-cluster.
• Observability: Use our existing OpenTelemetry setup for workflow/activity spans and PostHog for counters and timings.
• Security: Gate all settings actions with RBAC (tenant admins only) and audit via existing logging helpers.

Current State Snapshot

• UI: CE build now shows the canonical hosted domain with an Enterprise badge; EE build ships a fully wired domain form (status, refresh/disable controls, DNS instructions) backed by server actions.
• CE vs EE: Webpack alias continues to map @ee/* to CE stubs, with an enterprise override providing the rich settings panel dynamically.
• Persistence & actions: portal_domains table, model helpers, and CE/EE server actions are implemented and committed.
• Temporal scaffolding: Workflow client, DNS verification activity, and reconciliation handle manifest generation; next iteration moves reconciliation to GitOps by rewriting nm-kube-config manifests, applying them, and committing the diff.
• Istio & cert-manager: algapsa-gateway, apps-gateway, and apps-gateway-auto continue to terminate traffic with cert-manager issued secrets. Per-tenant Gateway + VirtualService resources are generated automatically; HTTP-01 challenge routing can be enabled via PORTAL_DOMAIN_CHALLENGE_* env configuration.

Target Tenant Experience

1. Admin (EE tenant) opens Settings → Client Portal.
2. Page displays current domain state (none, pending, active, failed) plus canonical target instructions.
3. Admin submits a vanity host (single domain). The backend persists the request, kicks off Temporal, and shows pending_dns.
4. Temporal verifies the CNAME points at the canonical target; success transitions to certificate provisioning and reconciliation.
5. Once cert-manager reports Ready and Istio resources are synced, status flips to active. Failures surface actionable status_message strings.
6. Admin can trigger a refresh or remove the domain; removal re-runs reconciliation to prune Kubernetes resources.
7. CE tenants see only the default portal domain card with no editable controls.

Architecture & Components

Database Schema: portal_domains

• Migration via Knex (follow docs/AI_coding_standards.md). Lives in CE repo so schema exists everywhere; EE code governs usage.
• Columns:
  • id UUID PK
  • tenant_id FK → tenant.tenant_id
  • domain CITEXT unique (vanity hostname requested by tenant)
  • canonical_host CITEXT (stored <tenant7>.portal.algapsa.com), unique per tenant
  • status ENUM: pending_dns, verifying_dns, dns_failed, pending_certificate, certificate_issuing, certificate_failed, deploying, active, disabled
  • status_message TEXT (human-friendly guidance)
  • last_checked_at timestamptz
  • verification_method ENUM default cname
  • verification_details JSONB (e.g. { "expected_cname": "abc1234.portal.algapsa.com" })
  • certificate_secret_name TEXT (portal-domain-{tenant_id})
  • last_synced_resource_version TEXT (for VirtualService/Gateway tracking)
  • Timestamps: created_at, updated_at
  • Unique constraint (tenant_id)

Server Layer (RBAC-aware)

• Actions exposed under server/src/lib/actions/tenant-actions/portalDomainActions.ts:
  • getPortalDomainStatus (read current row + derived hints). Always safe for CE but returns read-only stub when edition ≠ EE.
  • requestPortalDomainRegistration(domain) (validate, ensure Admin RBAC, persist row, enqueue workflow).
  • refreshPortalDomainStatus() (poll DB + optionally trigger Temporal query to tighten freshness).
  • disablePortalDomain() (mark disabled, signal workflow, enqueue reconciliation).
• REST endpoints (optional) under /api/settings/client-portal/domain to support CLI/automation. Wrap each handler with RBAC guard + audit trail entry.
• Edition gating: CE build exports stubs returning the default hosted domain.

Temporal Workflow & Activities

• Workflow: PortalDomainRegistrationWorkflow located at ee/temporal-workflows/src/workflows/portal-domains/registration.workflow.ts.
• Activities (all in ee/temporal-workflows/src/activities/portal-domains):
  1. recordStatus(domainId, status, message?) – updates DB row via shared repository helper.
  2. verifyCname(domain, canonicalHost) – performs repeated DNS lookups until two consecutive matches (5 min backoff). On timeout → dns_failed with guidance.
  3. renderAndApplyKubernetesState() – fetches all managed rows, renders Istio Gateway/VirtualService and cert-manager Certificate manifests, writes optional JSON snapshots to the GitOps worktree (PORTAL_DOMAIN_MANIFEST_DIR), applies updates via the Kubernetes API client, records resource versions, and prunes anything no longer present.
  4. waitForCertificateReady(namespace, certificateName) – polls cert-manager status via K8s API, emits PostHog metrics (issuance duration) and OpenTelemetry spans, and confirms HTTP-01 challenges are reachable (failing fast with actionable messaging when challenge pods lack ingress).
  5. waitForIstioSync(domainId) – checks VirtualService/Gateway resourceVersion matches recorded value; updates DB.
  6. finalizeActivation(domainId) – optional HTTP probe to https://domain to confirm 200 status, then mark active.
  7. Shared handleFailure(domainId, error, stage) – surfaces sanitized details to status_message and emits PostHog failure metric.
• Signals:
  • removeDomain – invoked when tenant disables domain to short-circuit and move to tear-down path.
• Observability: Wrap workflow + activities with OpenTelemetry instrumentation (workflow.logger + trace) and PostHog event names (portal_domain.dns_verified, portal_domain.cert_ready, etc.).

Kubernetes & TLS Strategy

• Canonical host: Each tenant routes through <tenant7>.portal.algapsa.com. This stays backed by the wildcard certificate we already manage (*.portal.algapsa.com).
• Vanity host: When the tenant’s vanity domain CNAMEs to the canonical host, cert-manager issues an individual certificate using the HTTP-01 solver. We must ensure Istio routes /.well-known/acme-challenge/* to cert-manager’s challenge service; if that path is unavailable we will have the Temporal workflow render the required challenge response assets until issuance completes. Secrets named portal-domain-{tenant} live in namespace msp.
• Resources:
• Certificate per vanity domain (namespace msp, issuerRef from PORTAL_DOMAIN_CERT_* env, default letsencrypt-dns) with deterministic secret name portal-domain-<tenant7>.
• Gateway per domain (namespace istio-system by default) exposes HTTP→HTTPS redirect and SNI-bound TLS server using the generated secret.
• VirtualService per domain (namespace msp) routes vanity traffic to sebastian.msp.svc.cluster.local:3000; optional /.well-known/acme-challenge/* route forwards to PORTAL_DOMAIN_CHALLENGE_HOST when enabled.
• Reconciliation flushes the desired resource set on every run and prunes labelled resources that no longer map to active domains.

GitOps Workflow (nm-kube-config)

• Production manifests live under ~/nm-kube-config/alga-psa/portal-domains/<tenantSlug>.yaml. Each file contains the rendered Certificate, Gateway, and VirtualService separated by --- so kubectl/Helm can apply them directly.
• Staging (hv-dev2) mirrors the layout at ~/nm-kube-config/argo-workflow/alga-psa-dev/portal-domains/<tenantSlug>.yaml to keep dev/test traffic isolated. The workflow picks the target root based on environment.
• PORTAL_DOMAIN_MANIFEST_DIR points at the appropriate root folder; on every reconciliation the activity rewrites the per-tenant YAML from database state (sorted keys, deterministic ordering) so Git diffs stay readable.
• After files are updated the activity runs kubectl apply -f <tenantSlug>.yaml (or batched apply) against the cluster, stages the changes with git add portal-domains, commits with a message like chore(portal-domains): sync <tenantSlug>, and pushes to the shared repo.
• A helper CLI (pnpm nm-kube-sync) will encapsulate diff detection, safe commit messages, optional PR creation, and fall back to printing kubectl commands for manual review when auto-apply is disabled.
• Operational playbook: review the generated Git diff, merge/push (or approve the automation’s push), verify Argo/Flux sync health, then trigger the Temporal refresh action so DB status aligns with the cluster.

UI Updates (CE vs EE)

• CE (server/src/components/settings/general/ClientPortalSettings.tsx): Replace “Coming Soon” with a read-only card highlighting default portal address (<tenant7>.portal.algapsa.com) and note that custom domains require Enterprise.
• EE (ee/server/src/components/settings/general/ClientPortalSettings.tsx):
  • Status banner showing domain, canonical_host, status, last_checked_at, and status_message.
  • Form with input id="client-portal-domain-input", submit button id="client-portal-domain-submit", and optional id="client-portal-domain-refresh" button to poll immediately.
  • DNS instructions card (canonical host, sample CNAME record) and hints for propagation delays.
  • Error panel listing actionable steps (derived from status_message).
  • Remove button (id="client-portal-domain-remove") once status is non-pending.
  • Poll status with SWR (15s) while in non-terminal state; respect AI coding standards for naming, instrumentation IDs, and toast usage.

Security & Permissions

• RBAC: Reuse existing tenant admin role checks (requireTenantPermission('settings.manage_portal') or similar). Deny actions for non-admins with clear error.
• Audit logging: Insert entries via existing helper for every create, refresh, remove action.
• Validation: Enforce ASCII/LDH host rules, reject apex domains without CNAME capability, normalise to lowercase.

Observability

• Temporal: Use OpenTelemetry tracing already wired in ee/temporal-workflows to emit spans for each activity. Tag spans with tenant_id, domain, stage (avoid PII beyond hostnames).
• Metrics: Emit PostHog events (counts, durations, failure reasons). Add dashboards to monitor time-to-activate, failure rate, and number of active domains.
• Logging: Standardise structured logs with correlation IDs (workflow run ID).

Progress Update – 2025-09-19

• ✅ portal_domains migration, enums, and model helpers landed with canonical-host storage and normalization utilities.
• ✅ CE/EE server actions implemented with RBAC enforcement, Temporal enqueue hooks, and PostHog capture; CE build returns read-only status.
• ✅ CE UI now surfaces the canonical host; EE UI delivers status badges, form submission, refresh/disable actions, and onboarding guidance.
• ✅ Workflow client + Temporal workflow now render and apply Istio Gateway/VirtualService + Certificate resources via the Kubernetes API, prune stale objects, and capture manifest snapshots when configured.
• ✅ Support documentation added (ee/docs/guides/portal-domain-runbook.md) describing operational flows and observability.
• ✅ Optional GitOps export writes JSON manifests for each tenant into PORTAL_DOMAIN_MANIFEST_DIR, ready for nm-kube-config commits.
• 🟡 Automated tests remain sparse (initial manifest-render unit test added); workflow + UI coverage still outstanding.

Implementation Plan

1. Schema & Models – ✅ Completed (2025-09-19)
   • Migration and model helpers merged; canonical host stored per tenant.
2. Server Actions & API – ✅ Completed (2025-09-19)
   • RBAC-guarded EE actions and CE stubs implemented; PostHog instrumentation in place. REST surface still optional (not started).
3. Temporal Workflow & Activities – 🟡 In progress
   • DNS verification + reconciliation activities now apply/prune Kubernetes resources and write manifest snapshots; remaining work covers certificate readiness polling, richer status messaging, and workflow tests.
4. GitOps & Reconciliation Tooling – 🟡 In progress
   • Per-tenant YAML written to nm-kube-config/{alga-psa|argo-workflow/alga-psa-dev}/portal-domains; still need the CLI to diff/commit/apply and docs for the automation flow.
5. Kubernetes Reconciliation Templates & HTTP-01 Pathing – 🟡 In progress
   • Gateway/VirtualService templates implemented with optional HTTP-01 routing via PORTAL_DOMAIN_CHALLENGE_*; still need to standardise the challenge-serving workload and productionize readiness probes.
6. UI (CE + EE) – ✅ Completed (2025-09-19)
   • CE shows default host; EE form with status badges, refresh/disable flows, and guidance shipped. Cypress/Playwright coverage outstanding.
7. Testing & Verification – 🔄 Not started
   • Unit/integration tests, workflow harness, and staging checklist to be added.
8. Rollout – 🔄 Not started
   • Migration deployment, worker release, and manual onboarding plan remain after backend completion.

Terminal Status UX Matrix

Status	UI Treatment	Available Actions	Next-Step Guidance
active	Success banner with vanity + canonical host, "Active" badge, last verified time	Refresh, Remove	Inform user domain is live; suggest confirming CNAME remains pointed correctly.
disabled	Neutral banner noting custom domain disabled and default host in use	Submit new domain, Refresh	Explain portal serves canonical host; advise submitting a new domain when ready.
dns_failed	Error banner showing last resolved target + suggested TTL wait	Refresh, Remove, Retry Registration	Prompt user to correct DNS record to the canonical host, then retry once propagated.
certificate_failed	Error banner with cert-manager status message and timestamp	Refresh, Remove, Retry Registration	Direct user to ensure HTTP-01 path is reachable; recommend contacting support if blocked.

Resolved Questions

• Canonical target storage: store in schema (canonical_host).
• DNS / Certificate scope: rely on wildcard *.portal.algapsa.com for canonical ingress; vanity domains CNAME into it and are issued certs without extending Route53 zones.
• ACME challenge method: enforce HTTP-01 challenges by exposing /.well-known/acme-challenge/* through Istio or temporary challenge-serving workloads managed by the workflow.
• Reconciliation location: handled entirely inside the Temporal activity suite; no separate Kubernetes operator required.
• Resource teardown: reconciliation activity renders the full desired list and prunes anything missing, ensuring deleted domains remove all K8s resources automatically.

Remaining Work & Follow-ups

• Add cert-manager readiness polling + HTTP probes before marking domains active; emit PostHog timings + richer status messages for failure cases.
• Finalise HTTP-01 challenge serving (shared solver service or on-demand pod) and bake the required PORTAL_DOMAIN_CHALLENGE_* defaults into staging/production.
• Build the GitOps helper CLI (pnpm nm-kube-sync) to diff manifests, open PRs, and optionally apply changes; update runbook once available.
• Expand automated coverage: workflow unit/integration tests, CE/EE action tests, and mocked EE UI e2e flows plus a staging validation checklist.
• Validate the new base VirtualService redirect management in staging once rolled out; regression coverage lives in ee/temporal-workflows/src/activities/__tests__/portal-domain-activities.git.test.ts to guard the /client-portal/dashboard default route.
• Provide operational tooling (Temporal signal CLI/script) for forced reconciliation and document the procedure in the runbook.
• Plan rollout sequencing: migration deployment order, Temporal worker release, customer enablement messaging, and nm-kube-config PR cadence.