# Job Scheduler System ## Overview The job scheduler system provides a robust and scalable solution for managing background tasks and scheduled jobs. It supports two execution backends: - **PG Boss** (Community Edition) - PostgreSQL-based job queue - **Temporal** (Enterprise Edition) - Workflow orchestration with enhanced durability Both backends write to the same database tables (`jobs` and `job_details`), providing a unified monitoring experience regardless of the underlying execution engine. ## Key Features - Immediate job execution - Scheduled jobs with specific run times - Recurring jobs with cron-like syntax - Job monitoring and metrics - Automatic retries with backoff - Job history tracking - **Edition-based backend selection** (CE: PG Boss, EE: Temporal) - **Unified job monitoring dashboard** ## Architecture ### Community Edition (PG Boss) ``` ┌─────────────────────────────────────────────────────────────────┐ │ Next.js Application │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ JobRunnerFactory ││ │ │ Creates PgBossJobRunner ││ │ └──────────────────────────────┬──────────────────────────────┘│ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ PgBossJobRunner ││ │ │ - Wraps JobScheduler (pg-boss) ││ │ │ - Writes to jobs/job_details tables ││ │ └─────────────────────────────────────────────────────────────┘│ └─────────────────────────────────────────────────────────────────┘ ``` ### Enterprise Edition (Temporal) ``` ┌─────────────────────────────────────────────────────────────────┐ │ Next.js Application │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ JobRunnerFactory ││ │ │ Creates TemporalJobRunner ││ │ └──────────────────────────────┬──────────────────────────────┘│ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ TemporalJobRunner ││ │ │ - Schedules Temporal workflows ││ │ │ - Writes to jobs/job_details tables ││ │ └──────────────────────────────┬──────────────────────────────┘│ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ Temporal Worker (separate process) ││ │ │ - Executes genericJobWorkflow ││ │ │ - Updates job status in database ││ │ └─────────────────────────────────────────────────────────────┘│ └─────────────────────────────────────────────────────────────────┘ ``` ## Configuration ### Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `JOB_RUNNER_TYPE` | `pgboss` | Backend: `pgboss` or `temporal` | | `JOB_RUNNER_FALLBACK_TO_PGBOSS` | `true` | Fall back to PG Boss if Temporal unavailable | #### Temporal-specific (EE only) | Variable | Default | Description | |----------|---------|-------------| | `TEMPORAL_ADDRESS` | `temporal-frontend...` | Temporal server address | | `TEMPORAL_NAMESPACE` | `default` | Temporal namespace | | `TEMPORAL_JOB_TASK_QUEUE` | `alga-jobs` | Task queue for jobs | ## Usage ### Using the Job Runner Factory (Recommended) ```typescript import { JobRunnerFactory } from 'server/src/lib/jobs'; // Get the singleton job runner instance const runner = await JobRunnerFactory.getInstance(); // Schedule an immediate job await runner.scheduleJob('generate-invoice', { tenantId: 'tenant-123', billingCycleId: 'cycle-456', }); // Schedule a job for later await runner.scheduleJobAt('send-reminder', { tenantId: 'tenant-123', userId: 'user-789', }, new Date('2024-12-01T09:00:00Z')); // Schedule a recurring job await runner.scheduleRecurringJob('daily-cleanup', { tenantId: 'tenant-123', }, '0 0 * * *'); // Daily at midnight ``` ### Legacy API (PG Boss Direct) The legacy `JobScheduler` API is still available for backward compatibility: ```typescript import { JobScheduler } from 'server/src/lib/jobs'; const scheduler = await JobScheduler.getInstance(); // Immediate job await scheduler.scheduleImmediateJob('process-order', { orderId: 123 }); // Scheduled job const runAt = new Date(Date.now() + 3600 * 1000); await scheduler.scheduleScheduledJob('send-reminder', runAt, { userId: 456 }); // Recurring job await scheduler.scheduleRecurringJob('daily-report', '0 0 * * *', {}); ``` ## Registered Job Handlers | Job Name | Description | Timeout | |----------|-------------|---------| | `generate-invoice` | Generate invoices for billing cycles | 5 min | | `invoice_zip` | Create ZIP archives of invoices | 10 min | | `invoice_email` | Send invoice emails | default | | `expired-credits` | Mark expired credits | default | | `expiring-credits-notification` | Send expiration notifications | default | | `credit-reconciliation` | Reconcile credit balances | default | | `asset_import` | Process asset import batches | 10 min | | `reconcile-bucket-usage` | Reconcile usage records | default | | `cleanup-temporary-workflow-forms` | Clean up temporary forms | default | | `renew-microsoft-calendar-webhooks` | Renew MS calendar subscriptions | default | | `verify-google-calendar-pubsub` | Verify Google calendar setup | default | | `cleanup-ai-session-keys` | Clean up AI sessions (EE only) | default | | `createClientContractLineCycles` | Create billing cycles | default | | `createNextTimePeriods` | Create time periods | default | ## Adding a New Job Handler ### 1. Create the Handler ```typescript // server/src/lib/jobs/handlers/myNewHandler.ts import { BaseJobData } from '../interfaces'; export interface MyJobData extends BaseJobData { tenantId: string; customField: string; } export async function myNewHandler(data: MyJobData): Promise { // Your job logic here console.log('Processing job for tenant:', data.tenantId); } ``` ### 2. Register the Handler Add to `server/src/lib/jobs/registerAllHandlers.ts`: ```typescript import { myNewHandler, MyJobData } from './handlers/myNewHandler'; // Inside registerAllJobHandlers(): JobHandlerRegistry.register( { name: 'my-new-job', handler: async (_jobId, data) => { await myNewHandler(data); }, retry: { maxAttempts: 3 }, timeoutMs: 300000, // 5 minutes }, registerOpts ); ``` ### 3. Schedule the Job ```typescript const runner = await JobRunnerFactory.getInstance(); await runner.scheduleJob('my-new-job', { tenantId: 'tenant-123', customField: 'value', }); ``` ## Monitoring and Metrics ### Dashboard The job monitoring dashboard is available at `/msp/jobs` and displays: - Real-time job metrics (active, queued, completed, failed) - Job history with filtering - Job details and error inspection - Runner type indicator (PG Boss vs Temporal) ### Programmatic Access ```typescript const runner = await JobRunnerFactory.getInstance(); // Get job status const status = await runner.getJobStatus('job-id', 'tenant-id'); // { status: 'completed', progress: 100, metadata: {...} } // Check health const healthy = await runner.isHealthy(); ``` ### Database Queries ```sql -- Get all jobs for a tenant SELECT * FROM jobs WHERE tenant = 'tenant-id' ORDER BY created_at DESC; -- Get job details/steps SELECT * FROM job_details WHERE job_id = 'job-id' ORDER BY processed_at; -- Find jobs by runner type SELECT * FROM jobs WHERE runner_type = 'temporal'; -- Find job by Temporal workflow ID SELECT * FROM jobs WHERE external_id = 'workflow-id'; ``` ## Error Handling The system implements: - Automatic retries (3 attempts by default) - Exponential backoff between retries - Error logging and monitoring - Manual job cancellation - Graceful degradation (Temporal → PG Boss fallback) ### Retry Configuration ```typescript JobHandlerRegistry.register({ name: 'my-job', handler: async (jobId, data) => { /* ... */ }, retry: { maxAttempts: 5, backoffCoefficient: 2.0, initialIntervalMs: 1000, maxIntervalMs: 60000, }, timeoutMs: 600000, }); ``` ## Database Schema ### jobs Table ```sql CREATE TABLE jobs ( tenant UUID NOT NULL, job_id UUID DEFAULT gen_random_uuid() NOT NULL, type VARCHAR NOT NULL, metadata JSONB, status job_status NOT NULL, -- pending, processing, completed, failed, active, queued created_at TIMESTAMP DEFAULT NOW(), updated_at TIMESTAMP, user_id UUID NOT NULL, runner_type VARCHAR DEFAULT 'pgboss' NOT NULL, -- 'pgboss' or 'temporal' external_id VARCHAR, -- PG Boss job ID or Temporal workflow ID external_run_id VARCHAR, -- Temporal run ID PRIMARY KEY (tenant, job_id) ); ``` ### job_details Table ```sql CREATE TABLE job_details ( tenant UUID NOT NULL, detail_id UUID DEFAULT gen_random_uuid() NOT NULL, job_id UUID NOT NULL, step_name VARCHAR NOT NULL, status job_status NOT NULL, result JSONB, processed_at TIMESTAMP, retry_count INTEGER DEFAULT 0, metadata JSONB, PRIMARY KEY (tenant, detail_id), FOREIGN KEY (tenant, job_id) REFERENCES jobs(tenant, job_id) ); ``` ## Temporal-Specific Features (EE) When using Temporal as the backend, additional features are available: ### Workflow Signals ```typescript import { Client } from '@temporalio/client'; const client = new Client({ /* ... */ }); const handle = client.workflow.getHandle(workflowId); // Cancel a job await handle.signal('cancelJob', { reason: 'User requested cancellation', cancelledBy: 'admin@example.com', }); // Update progress await handle.signal('updateProgress', { progress: 50, message: 'Halfway done', }); ``` ### Workflow Queries ```typescript // Get current job state const state = await handle.query('getJobState'); // { step: 'executing', progress: 50, startedAt: '...' } ``` ## Troubleshooting ### Job Handler Not Found If you see "No handler registered for job: X": 1. Verify the handler is registered in `registerAllHandlers.ts` 2. Check that `registerAllJobHandlers()` was called at startup 3. For Temporal: ensure `initializeJobHandlersForWorker()` was called ### Jobs Not Processing 1. Check PG Boss is connected: verify database connection 2. Check Temporal worker is running: `kubectl get pods -l app=temporal-worker` 3. Check task queues match between scheduler and worker ### Temporal Connection Issues 1. Verify `TEMPORAL_ADDRESS` is correct 2. Check Temporal server is running 3. If `JOB_RUNNER_FALLBACK_TO_PGBOSS=true`, jobs will use PG Boss ## See Also - [Enterprise Temporal Documentation](../../ee/docs/temporal-workflows/job-runner-abstraction.md) - [Temporal Worker Deployment](../../ee/docs/temporal-workflows/deployment.md) - [Configuration Guide](../getting-started/configuration_guide.md)