PSA/ee/docs/temporal-workflows/deployment.md

Temporal Worker Deployment Guide

This guide provides instructions for deploying the temporal worker to production using the Argo workflows and Helm charts created for the alga-psa project.

Prerequisites

Before deploying the temporal worker, ensure the following prerequisites are met:

1. Vault Setup

   • INTERNAL_API_SHARED_SECRET is stored in Vault at secret/data/alga-psa/temporal-worker
   • ALGA_AUTH_KEY is stored in Vault at secret/data/alga-psa/shared
   • Vault policy temporal-worker is created with read access to these paths
   • Service account is configured with appropriate Vault role

2. Harbor Registry

   • Harbor credentials are configured in the cluster
   • Access to push/pull from harbor.nineminds.com/nineminds/temporal-worker

3. Database Access

   • PostgreSQL cluster is accessible from the msp namespace
   • Database credentials are stored in the appropriate secrets

4. Temporal Server

   • Temporal server is running and accessible at temporal-frontend.temporal.svc.cluster.local:7233

Building the Temporal Worker

Manual Build

To manually trigger a build of the temporal worker:

# Submit the build workflow
kubectl create -n argo -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: temporal-worker-build-
spec:
  workflowTemplateRef:
    name: temporal-worker-build
  arguments:
    parameters:
    - name: repo-url
      value: "https://github.com/Nine-Minds/alga-psa.git"
    - name: commit-sha
      value: "main"  # or specific commit SHA
    - name: set-latest
      value: "true"  # Set to true to tag as latest
EOF

Automated Build with Main Pipeline

The temporal worker is automatically built when using the composite workflow if changes are detected:

# Submit the composite workflow
kubectl create -n argo -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: alga-psa-full-deploy-
spec:
  workflowTemplateRef:
    name: alga-psa-build-migrate-deploy-with-temporal
  arguments:
    parameters:
    - name: repo-url
      value: "https://github.com/Nine-Minds/alga-psa.git"
    - name: commit-sha
      value: "main"
    - name: environment
      value: "hosted"
    - name: helm-values-file
      value: "hosted.values.yaml"
    - name: namespace
      value: "msp"
    - name: build-temporal-worker
      value: "auto"  # auto-detect changes, or "true" to always build
EOF

Deploying the Temporal Worker

Manual Deployment

To manually deploy the temporal worker with a specific image tag:

# Submit the deployment workflow
kubectl create -n argo -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: temporal-worker-deploy-
spec:
  workflowTemplateRef:
    name: temporal-worker-deploy
  arguments:
    parameters:
    - name: image-tag
      value: "abc12345"  # Use the short SHA from build
    - name: environment
      value: "production"
    - name: namespace
      value: "msp"
    - name: helm-values-file
      value: "hosted.values.yaml"
    - name: rollback-on-failure
      value: "true"
EOF

Verifying Deployment

After deployment, verify the temporal worker is running:

# Check deployment status
kubectl get deployment alga-psa-temporal-worker -n msp

# Check pods
kubectl get pods -n msp -l app.kubernetes.io/component=temporal-worker

# Check logs
kubectl logs -n msp -l app.kubernetes.io/component=temporal-worker --tail=100

# Check health endpoint
kubectl exec -n msp deployment/alga-psa-temporal-worker -- wget -q -O- http://localhost:8080/health

Configuration

Environment Variables

The temporal worker uses the following environment variables (injected via Vault or secrets):

• TEMPORAL_ADDRESS: Temporal server address
• TEMPORAL_NAMESPACE: Temporal namespace (default: "default")
• TEMPORAL_TASK_QUEUE: Task queue name (default: "tenant-workflows")
• DB_HOST, DB_PORT, DB_NAME_SERVER: Database connection
• INTERNAL_API_SHARED_SECRET: API authentication secret
• ALGA_AUTH_KEY: Encryption key for passwords
• RESEND_API_KEY: Email service API key
• APPLICATION_URL: Base URL for email links
• NMSTORE_BASE_URL: NM Store integration URL

Scaling Configuration

The temporal worker is configured with:

• Initial replicas: 3 (production)
• HPA: Scales from 3 to 20 replicas based on CPU (70%) and memory (75%)
• PDB: Minimum 2 available pods during disruptions

To manually scale:

kubectl scale deployment alga-psa-temporal-worker -n msp --replicas=5

Troubleshooting

Common Issues

1. Worker fails to start

   • Check Vault secret injection: kubectl describe pod <pod-name> -n msp
   • Verify secrets are mounted: kubectl exec -n msp <pod-name> -- ls /vault/secrets/

2. Database connection errors

   • Verify database is accessible: kubectl exec -n msp <pod-name> -- nc -zv <db-host> 5432
   • Check database credentials in secrets

3. Temporal connection errors

   • Verify Temporal server is running: kubectl get pods -n temporal
   • Check network connectivity to Temporal

4. Health check failures

   • Check worker logs for startup errors
   • Verify port 8080 is accessible within the pod

Rollback Procedure

If deployment fails, the workflow automatically rolls back. To manually rollback:

# List Helm releases
helm history alga-psa -n msp

# Rollback to previous version
helm rollback alga-psa <revision-number> -n msp

Monitoring

Logs

Temporal worker logs are collected by the cluster logging solution. Key log patterns to monitor:

• "level":"error" - Error conditions
• "workflow":"started" - Workflow executions
• "activity":"completed" - Activity completions
• "health":"check" - Health check status

Metrics

The temporal worker exposes metrics that can be scraped by Prometheus:

• Worker task queue depth
• Workflow execution duration
• Activity execution duration
• Error rates

Alerts

Recommended alerts:

1. Worker Pod Down: Less than minimum replicas running
2. High Error Rate: More than 5% of workflows failing
3. Task Queue Backup: More than 100 pending tasks
4. Memory Pressure: Worker using >80% of memory limit

Security Considerations

1. Secret Rotation

   • INTERNAL_API_SHARED_SECRET should be rotated quarterly
   • Update in Vault and restart workers

2. Network Policies

   • Worker should only connect to:
     • PostgreSQL database
     • Temporal server
     • Redis (if caching enabled)
     • External APIs (Resend, NM Store)

3. RBAC

   • Service account has minimal permissions
   • No cluster-wide access required

Maintenance

Updating the Worker

To update the temporal worker code:

1. Make changes in ee/temporal-workflows/
2. Commit and push to repository
3. Run the build workflow with new commit SHA
4. Deploy using the deployment workflow

Database Migrations

Database migrations are handled by the main alga-psa migration workflow. The temporal worker uses the same database schema.

Performance Tuning

Adjust these values based on workload:

temporalWorker:
  temporal:
    maxConcurrentActivityExecutions: 20  # Increase for more parallelism
    maxConcurrentWorkflowTaskExecutions: 20
  resources:
    requests:
      cpu: 1000m  # Increase for CPU-intensive workflows
      memory: 2Gi  # Increase for memory-intensive workflows