PSA/docs/workflow/workflow-troubleshooting.md

Workflow Troubleshooting Guide

This document provides solutions to common issues encountered when working with the workflow system.

Table of Contents

1. Action Not Found Errors
2. Workflow Deserialization Errors
3. Database Update Issues
4. Test Environment Issues

Action Not Found Errors

Problem: actions.resolve_email_provider_defaults is not a function

Symptoms:

• Workflow fails with error: actions.resolve_email_provider_defaults is not a function
• Error occurs during email processing workflow execution

Root Cause: The workflow code in the database still references an old action name that has been deprecated or renamed.

Solution:

1. Update the workflow source code to use the new action name
2. Update the database using the appropriate update script
3. Verify the database contains the updated code

Example Fix:

// OLD (deprecated)
const ticketDefaults = await actions.resolve_email_provider_defaults({
  providerId: providerId,
  tenant: tenant
});

// NEW (correct)
const ticketDefaults = await actions.resolve_inbound_ticket_defaults({
  tenant: tenant
});

Verification Steps:

-- Check what action names are in the database
SELECT 
  sv.version_id,
  sv.is_current,
  LENGTH(sv.code) as code_length,
  CASE 
    WHEN sv.code LIKE '%resolve_email_provider_defaults%' THEN 'OLD_ACTION'
    WHEN sv.code LIKE '%resolve_inbound_ticket_defaults%' THEN 'NEW_ACTION'
    ELSE 'UNKNOWN'
  END as action_status
FROM system_workflow_registration_versions sv
WHERE sv.registration_id = '550e8400-e29b-41d4-a716-446655440001'
ORDER BY sv.created_at DESC;

Workflow Deserialization Errors

Problem: Unexpected token 'async'

Symptoms:

• Error: SyntaxError: Unexpected token 'async'
• Occurs when workflow runtime tries to deserialize workflow function
• Error in deserializeWorkflowFunction

Root Cause: The workflow function stored in the database contains incorrect syntax that conflicts with how the runtime wraps the function.

Solution: The stored function should NOT have async in the wrapper as the runtime adds it automatically.

// CORRECT - runtime adds async wrapper
const dbWorkflowCode = `function execute(context) {
  // workflow body
}`;

Problem: Failed to extract function body from compiled code

Symptoms:

• Error: Failed to extract function body from compiled code
• Occurs during workflow deserialization
• Error in deserializeWorkflowFunction

Root Cause: The workflow runtime expects stored functions to match the pattern async function <name>(context) { ... } but the stored function doesn't match this pattern.

Solution: Ensure the stored function matches the expected pattern:

// CORRECT - matches runtime expectations
const dbWorkflowCode = `async function execute(context) {
  ${functionBody}
}`;

Common Issues:

1. Missing async keyword: Function stored as function execute(context) instead of async function execute(context)
2. TypeScript imports: Import statements in the workflow code
3. Type annotations: TypeScript type annotations like context: WorkflowContext
4. Async function declarations within workflow: Helper functions declared as async function name() {} inside the main workflow

Problem: Unexpected token 'async' with helper functions

Symptoms:

• Error: SyntaxError: Unexpected token 'async'
• Occurs when workflow contains helper functions declared as async function
• Runtime fails to parse the wrapped function

Root Cause: Helper functions declared as async function name() {} within the workflow cause parsing errors when the runtime wraps the extracted function body.

Solution: Convert helper function declarations to function expressions:

// WRONG - causes parsing error
async function checkEmailThreading(emailData, actions) {
  // ...
}

// CORRECT - use function expressions
const checkEmailThreading = async (emailData, actions) => {
  // ...
};

// Also correct - function expression syntax
const checkEmailThreading = async function(emailData, actions) {
  // ...
};

Why this happens: The runtime wraps the extracted function body like this:

return (async function(context) {
  // Your extracted function body goes here
  async function helperFunction() {} // <-- This causes the error!
})(context);

Function declarations inside immediately invoked function expressions cause JavaScript parsing errors.

Issue 2: Remove imports and TypeScript syntax

// WRONG - TypeScript syntax
import type { WorkflowContext } from '../core/types.js';

export async function systemEmailProcessingWorkflow(context: WorkflowContext): Promise<void> {
  // workflow logic
}

// CORRECT - Plain JavaScript
export async function systemEmailProcessingWorkflow(context) {
  // workflow logic
}

Issue 3: Update tsconfig to exclude workflow files

{
  "exclude": ["node_modules", "dist", "workflow/workflows/system-email-processing-workflow.ts"]
}

Problem: Helper functions not accessible in workflow scope

Symptoms:

• Workflow deserialization succeeds initially
• Runtime errors when helper functions are called
• Functions appear undefined during execution
• Error: checkEmailThreading is not defined or similar

Root Cause: The workflow runtime extracts only the function body and wraps it in a new execution context. Helper functions defined outside the main workflow function are not included in the extracted body, making them inaccessible during execution.

Solution: All helper functions must be defined inside the main workflow function using function expressions:

export async function systemEmailProcessingWorkflow(context) {
  const { actions, data, logger, setState, events } = context;
  
  // CORRECT - Helper functions defined inside main function as const expressions
  const checkEmailThreading = async (emailData, actions) => {
    // helper function logic
  };
  
  const handleEmailReply = async (emailData, existingTicket, actions) => {
    // helper function logic
  };
  
  const findExactEmailMatch = async (emailAddress, actions) => {
    // helper function logic
  };
  
  // Main workflow logic starts here
  setState('PROCESSING_INBOUND_EMAIL');
  // ... rest of workflow
}

// WRONG - Helper functions outside main function won't be included
async function checkEmailThreading(emailData, actions) {
  // This function won't be available during execution!
}

export async function systemEmailProcessingWorkflow(context) {
  // Main workflow logic - but helper functions are missing!
}

Why this happens:

1. The runtime extracts only the content between the first { and last } of the main function
2. Functions defined outside the main function are not included in the extraction
3. The extracted body is wrapped in a new execution context where external functions don't exist

Key Requirements:

• Use const functionName = async () => {} syntax inside the main function
• Never use async function functionName() {} declarations inside the main function (causes parsing errors)
• Never define helper functions outside the main workflow function

Problem: Function body extraction issues

Symptoms:

• Error: Could not find systemEmailProcessingWorkflow function in source file
• Seed fails during workflow loading

Root Cause: The regex or string parsing logic in the seed file doesn't match the actual function signature.

Solution: Use simple string indexing instead of complex regex:

// Find the function declaration and extract everything after the opening brace
const functionStart = workflowContent.indexOf('export async function systemEmailProcessingWorkflow(context) {');
if (functionStart === -1) {
  throw new Error('Could not find systemEmailProcessingWorkflow function in source file');
}

// Find the opening brace and extract everything after it, minus the final closing brace
const openBraceIndex = workflowContent.indexOf('{', functionStart);
const functionContent = workflowContent.substring(openBraceIndex + 1);

// Remove the final closing brace
const functionBody = functionContent.substring(0, functionContent.lastIndexOf('}')).trim();

Database Update Issues

Problem: Workflow updates not taking effect

Symptoms:

• Updated workflow code in source file
• Ran update script successfully
• But runtime still executes old workflow code

Root Causes:

1. Multiple databases: Update script connected to different database than runtime
2. Test seeds overwriting: Test environment seeds overwrite manual updates
3. Caching issues: Workflow runtime caching old definitions

Solutions:

Issue 1: Verify correct database connection

# Check which databases are running
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep postgres

# Use correct environment variables
DB_HOST=localhost DB_PORT=5433 DB_NAME_SERVER=server_test DB_USER_ADMIN=postgres DB_PASSWORD_ADMIN=postpass123 node scripts/update-workflow-in-db.cjs

Issue 2: Update seed files to read from source Ensure seed files read from the actual TypeScript source instead of hardcoded workflow code:

// WRONG - hardcoded workflow
const dbWorkflowCode = `async function execute(context) {
  // hardcoded workflow logic here
}`;

// CORRECT - read from source file
function loadWorkflowCodeFromSource() {
  const workflowPath = path.join(__dirname, '../../../shared/workflow/workflows/system-email-processing-workflow.ts');
  const workflowContent = fs.readFileSync(workflowPath, 'utf8');
  // extract and convert workflow code
}

Issue 3: Check workflow version IDs Verify the update script is updating the correct version:

-- Check all workflow versions
SELECT 
  sv.version_id,
  sv.is_current,
  sv.created_at,
  sv.updated_at,
  LENGTH(sv.code) as code_length
FROM system_workflow_registration_versions sv
WHERE sv.registration_id = '550e8400-e29b-41d4-a716-446655440001'
ORDER BY sv.created_at DESC;

Test Environment Issues

Problem: Tests overwrite workflow updates

Symptoms:

• Manual workflow updates work
• Tests reset workflow to old version
• Tests fail with old action errors

Root Cause: Test environment runs seeds that overwrite workflow definitions with outdated code.

Solution: Update the seed file to load from the current source file instead of hardcoded workflow:

// File: /server/seeds/dev/004_email_processing_workflow_from_source.cjs

function loadWorkflowCodeFromSource() {
  const workflowPath = path.join(__dirname, '../../../shared/workflow/workflows/system-email-processing-workflow.ts');
  
  if (!fs.existsSync(workflowPath)) {
    throw new Error(`Workflow file not found: ${workflowPath}`);
  }
  
  console.log(`Reading workflow from source file: ${workflowPath}`);
  const workflowContent = fs.readFileSync(workflowPath, 'utf8');
  
  // Extract function body and create database-compatible version
  // ... (extraction logic)
  
  return dbWorkflowCode;
}

Problem: Multiple workflow registrations

Symptoms:

• Multiple entries for the same workflow
• Unclear which version is being executed

Diagnosis:

-- Check all email-related workflows
SELECT 
  sr.registration_id,
  sr.name,
  sr.status,
  sv.version_id,
  sv.is_current,
  LENGTH(sv.code) as code_length
FROM system_workflow_registrations sr
JOIN system_workflow_registration_versions sv ON sr.registration_id = sv.registration_id
WHERE sr.name LIKE '%email%' OR sv.code LIKE '%email%'
ORDER BY sr.name, sv.created_at DESC;

Solution: Ensure only one registration exists with the correct static UUID and that only one version is marked as current.

Debugging Tips

1. Check Workflow Runtime Logs

Look for these log patterns:

• [TENANT-DEBUG] WorkflowWorker about to start workflow
• Error deserializing workflow function
• Failed to load system workflow definition

2. Verify Database State

-- Check current workflow registration
SELECT * FROM system_workflow_registrations 
WHERE registration_id = '550e8400-e29b-41d4-a716-446655440001';

-- Check current version
SELECT * FROM system_workflow_registration_versions 
WHERE registration_id = '550e8400-e29b-41d4-a716-446655440001' 
AND is_current = true;

3. Test Workflow Updates

Use the improved update script with detailed logging:

# The script provides comprehensive feedback
node scripts/update-workflow-in-db.cjs

4. Validate Workflow Syntax

Before updating the database, ensure the workflow function:

• Has no TypeScript imports
• Uses plain JavaScript syntax
• Function signature matches: export async function systemEmailProcessingWorkflow(context)
• Contains expected action calls

Prevention Best Practices

1. Keep workflows simple: Avoid TypeScript syntax in workflow files
2. Use seed files that read from source: Don't hardcode workflow logic in seeds
3. Exclude workflow files from TypeScript compilation: Add to tsconfig exclude
4. Test workflow updates: Verify database state after updates
5. Use consistent database connections: Ensure update scripts target the correct database
6. Version control workflow changes: Track both source and database updates

Quick Reference

Common Commands

# Update workflow in database
DB_HOST=localhost DB_PORT=5433 DB_NAME_SERVER=server_test DB_USER_ADMIN=postgres DB_PASSWORD_ADMIN=postpass123 node scripts/update-workflow-in-db.cjs

# Check database workflows
docker exec sebastian_postgres_test psql -U postgres -d server_test -c "SELECT name, status FROM system_workflow_registrations;"

# Check running containers
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep postgres

Workflow File Requirements

• Plain JavaScript syntax only
• No TypeScript imports
• No type annotations
• Function signature: export async function systemEmailProcessingWorkflow(context)
• Store as function execute(context) in database (no async keyword)