PSA/ee/docs/plans/2025-01-26-temporal-workflows-database-migration.md

Temporal Workflows Database Migration Plan

Intro / Rationale

This plan outlines the migration of temporal workflows from using their own database connection logic (/ee/temporal-workflows/src/db/connection.ts) to using the shared database libraries (/shared/db/admin.ts). The primary goals are:

• Simplify database connection management by removing duplicate connection logic
• Standardize environment variables by eliminating ALGA_DB_* variables in favor of standard DB_* variables
• Improve maintainability by using the shared Knex-based connection pool
• Reduce configuration complexity in Kubernetes deployments

Success Criteria

• All temporal workflow database operations use the shared admin connection
• No ALGA_DB_* environment variables remain in the codebase
• All raw SQL queries are converted to Knex queries
• Temporal workflows continue to function correctly with the new connection method

Phased Implementation Checklist

Phase 1: Pre-Migration Verification

• Verify that shared/db/admin.ts is available in temporal workflow container
  • Check Dockerfile includes shared directory - Fixed to build from project root
  • Verify TypeScript compilation includes shared modules - Updated tsconfig.json
• Update Docker build process to include shared directory - Updated deploy.sh
• Test shared database connection in a development environment

Phase 2: Code Migration - Database Operations

• Update temporal-workflows package.json to ensure shared dependencies are included - Already has knex
• Migrate src/db/tenant-operations.ts:
  • Import getAdminConnection from shared/db/admin.ts
  • Convert createTenantInDB to use Knex queries
  • Convert setupTenantDataInDB to use Knex queries
  • Convert rollbackTenantInDB to use Knex queries
  • Remove executeTransaction wrapper, use Knex transactions
• Migrate src/db/user-operations.ts:
  • Import getAdminConnection from shared/db/admin.ts
  • Convert createAdminUserInDB to use Knex queries
  • Convert rollbackUserInDB to use Knex queries
  • Update password hashing to use proper bcrypt implementation - TODO later
• Delete src/db/connection.ts file entirely
• Run unit tests to verify query conversions

Phase 3: Environment Variable Updates

• Update temporal-workflows Kubernetes deployment (k8s/deployment.yaml):
  • Remove all ALGA_DB_* environment variables (lines 40-59)
  • Add standard DB_* environment variables:
    • DB_HOST
    • DB_PORT
    • DB_NAME_SERVER
    • DB_USER_SERVER
    • DB_PASSWORD_SERVER (from secret)
    • DB_USER_ADMIN
    • DB_PASSWORD_ADMIN (from secret)
  • Update secret references to use correct keys
• Search entire codebase for any remaining ALGA_DB_* references
  • None found except in this migration plan
  • No other configuration files need updating

Background Details / Investigation / Implementation Advice

Current Architecture

The temporal workflows currently use a custom PostgreSQL connection pool (pg library) with these characteristics:

• Direct PostgreSQL connections using pg.Pool
• Custom transaction wrapper functions
• Support for both regular and admin database connections
• Environment variables prefixed with ALGA_DB_*

Target Architecture

The shared database connection uses:

• Knex.js as the query builder and connection manager
• Built-in connection pooling with configurable limits
• Standard DB_* environment variables
• Integrated secret management via getSecret()

Code Conversion Examples

Converting Raw SQL to Knex

Current (Raw SQL):

const result = await client.query(
  'SELECT user_id FROM users WHERE email = $1 AND tenant = $2',
  [input.email, input.tenantId]
);

Target (Knex):

const result = await knex('users')
  .select('user_id')
  .where({ email: input.email, tenant: input.tenantId });

Converting Transactions

Current (pg transaction):

await executeTransaction(adminDb, async (client) => {
  await client.query('BEGIN');
  // operations
  await client.query('COMMIT');
});

Target (Knex transaction):

await knex.transaction(async (trx) => {
  // operations using trx instead of client
});

Environment Variable Mapping

• ALGA_DB_HOST → DB_HOST
• ALGA_DB_PORT → DB_PORT
• ALGA_DB_NAME → DB_NAME_SERVER
• ALGA_DB_USER → DB_USER_SERVER
• ALGA_DB_PASSWORD → DB_PASSWORD_SERVER
• ALGA_DB_ADMIN_USER → DB_USER_ADMIN
• ALGA_DB_ADMIN_PASSWORD → DB_PASSWORD_ADMIN

Potential Pitfalls

1. Import paths: Ensure correct relative imports from temporal-workflows to shared directory
2. TypeScript compilation: Verify shared modules are included in the build
3. Connection pooling: Monitor pool size as shared connection has different defaults
4. Error handling: Knex errors have different structure than pg errors
5. Query result format: Knex returns arrays directly, not { rows: [...] }

Testing Strategy

1. Unit tests with mocked database connections
2. Integration tests against test database
3. End-to-end workflow tests in Temporal

Implementer's Scratch Pad

Implementation Notes

Space for tracking observations during implementation

---

Issues Encountered

Document any problems and their resolutions

---

Deviations from Plan

Record any changes from the original plan and why

---

Performance Metrics

Before and after metrics

• Connection pool size:
• Average query time:
• Workflow execution time:
• Memory usage:

---

Questions for Review

Items needing clarification or decisions

---

Test Results

Summary of test outcomes

• Development environment:
• Staging environment:
• Load test results:

---

Deployment Log

Track deployment progress

• Development: Date/Time/Version
• Staging: Date/Time/Version
• Production: Date/Time/Version

---

Post-Deployment Observations

24-hour monitoring results

---