PSA/ee/docs/plans/2026-04-08-workflow-v2-temporal-hard-cutover-remediation/SCRATCHPAD.md

SCRATCHPAD — Workflow V2 Temporal Hard-Cutover Remediation

Purpose

Focused follow-up plan for the remaining blockers after the broader Temporal-native runtime plan.

Parent plan:

• ee/docs/plans/2026-04-08-workflow-v2-temporal-native-runtime/

User-approved scope decisions

• Scope choice: B
  • include the 3 remaining review items plus adjacent operator/runtime cleanup needed to make hard cutover shippable
• Legacy operator behavior for Temporal runs: A — hard fail
  • do not translate old DB-runtime-style actions for Temporal-backed runs
• Event correlation source: C — support both
  • explicit correlation if present
  • configured derivation if explicit correlation is absent
• Expression parity posture: A — exact parity
  • Temporal must use the same expression contract as the canonical workflow runtime
• Cutover posture: hard cutover
  • shared / legacy DB-runtime authority can be removed for Workflow Runtime V2

Review findings this plan addresses

1. Expression engine parity gap

   • ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts
   • custom JSONata evaluator diverges from canonical expression contract in shared/workflow/runtime/expressionEngine.ts
   • known gap: missing nowIso() support
   • likely additional gap: skipped guardrails/allowed-function enforcement

2. Remaining DB/legacy authority over Temporal runs

   • ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts
   • resumeWorkflowRunAction() still mutates waits/runs and calls WorkflowRuntimeV2.executeRun(...)
   • retryWorkflowRunAction() still calls WorkflowRuntimeV2.executeRun(...)
   • submitWorkflowEventAction() still resolves waits through DB authority patterns
   • cancelWorkflowRunAction() still projects CANCELED even if Temporal cancel fails

3. Event ingress correlation bug

   • services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.ts
   • currently uses event.event_id as correlation key for audit, candidate lookup, and Temporal signal payload
   • real event.wait steps typically correlate on business keys like ticket/project/account identifiers

Constraints / implementation guardrails

• Prefer explicit failure over silent fallback.
• Do not preserve hybrid authority between Temporal and DB runtime for V2.
• DB tables remain projection/index/audit surfaces only.
• Candidate lookup may use DB indexes, but Temporal remains the only resume authority.
• Unsupported Temporal actions should fail loudly and clearly for operators/support.

Likely affected files

• ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts
• shared/workflow/runtime/expressionEngine.ts
• ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts
• ee/packages/workflows/src/lib/workflowRuntimeV2Temporal.ts
• services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.ts
• packages/event-bus/src/schemas/workflowEventSchema.ts
• shared/workflow/persistence/workflowRunWaitModelV2.ts

Expected test areas

• Temporal workflow/runtime unit tests
• server/API unit tests for run-control actions
• workflow worker event-ingress tests
• possibly one DB-backed integration path for correlation routing/projection sanity

Notes

• Existing broader runtime plan already covers the end-state architecture.
• This focused plan is meant to track the remaining hard-cutover blockers as a shippable remediation slice.

Progress — 2026-04-09 (Run-control hard cutover)

Completed scope in this pass

• Implemented hard-fail guardrails for legacy run-control actions on Temporal runs in:
  • ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts
• Added explicit unsupported-action boundary for Temporal runs:
  • resumeWorkflowRunAction now fails with 409 + code WORKFLOW_TEMPORAL_ACTION_UNSUPPORTED
  • retryWorkflowRunAction now fails with 409 + code WORKFLOW_TEMPORAL_ACTION_UNSUPPORTED
  • requeueWorkflowRunEventWaitAction now fails with 409 + code WORKFLOW_TEMPORAL_ACTION_UNSUPPORTED
• Fixed Temporal cancel authority semantics:
  • cancelWorkflowRunAction now calls Temporal cancel first and does not project DB CANCELED state if Temporal cancel fails.
  • Failure path now returns explicit 409 + code WORKFLOW_TEMPORAL_CANCEL_FAILED with actionable hint.

Key decisions and rationale

• Decision: treat legacy admin controls (resume, retry, requeue_event_wait) as unsupported for engine='temporal'.
  • Rationale: enforces single execution authority (Temporal) and prevents split-brain DB projection mutations.
• Decision: cancel remains supported for Temporal runs, but only when the Temporal cancel request succeeds.
  • Rationale: prevents false projection of cancellation when workflow execution is still active.

Tests added/updated

• Updated server/src/test/integration/workflowRuntimeV2.control.integration.test.ts:
  • New: Temporal resume hard-fails with explicit unsupported-action details and no wait/run mutation.
  • New: Temporal retry hard-fails with explicit unsupported-action details and keeps failed run projection unchanged.
  • New: Temporal cancel failure leaves run/waits in WAITING state and returns explicit error details.
• Added test mocking for Temporal cancel client call via:
  • vi.mock('@alga-psa/workflows/lib/workflowRuntimeV2Temporal', ...)

Commands / runbook used

• Focused integration run (full file):
  • cd server && npm run -s test -- src/test/integration/workflowRuntimeV2.control.integration.test.ts
  • Result: suite has a pre-existing unrelated failure (STARTED invocation with stale lease is treated as TransientError) not introduced by these changes.
• Focused validation for new Temporal tests only:
  • cd server && npm run -s test -- src/test/integration/workflowRuntimeV2.control.integration.test.ts -t "Temporal runs reject legacy|Temporal cancel failure"
  • Result: passed.

Current gaps / next work

• Event-ingress correlation/authority items (F018+) remain open.
• Expression parity items referenced here were addressed in the next section (Expression parity cutover).

Progress — 2026-04-09 (Expression parity cutover)

Completed scope in this pass

• Replaced Temporal workflow-local expression evaluation with canonical workflow expression contract in:
  • ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts
• Removed local Temporal-only JSONata helpers from workflow execution path:
  • removed local evaluator/normalizer and now uses canonical compileExpression from @alga-psa/workflows/runtime/expressionEngine
• Removed Temporal-only nowIso() control-flow ban.
  • Temporal workflow now accepts nowIso() with canonical function set/validation behavior.

Key decisions and rationale

• Decision: import expression contract from @alga-psa/workflows/runtime/expressionEngine directly instead of @alga-psa/workflows/runtime barrel.
  • Rationale: avoids pulling unrelated transitive runtime package entry dependencies into temporal workflow unit test bundle (@alga-psa/storage resolution issue).
• Decision: cache compiled expressions per source string in-workflow (Map<string, CompiledExpression>).
  • Rationale: preserves behavior and avoids repeated compile cost while remaining deterministic for replayed source strings.

Tests added/updated

• Updated ee/temporal-workflows/src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts:
  • nowIso parity: control.if accepts nowIso() via canonical engine.
  • normalization parity: == source normalization path works in Temporal execution.
  • disallowed-function parity: rejects unsupported functions with canonical validation.
  • output guardrail parity: expression result over max size fails with canonical guardrail error.

Commands / runbook used

• Temporal workflow unit suite (targeted):
  • cd ee/temporal-workflows && npx vitest run src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts
  • Result: passed (30 tests).

Current gaps / next work

• Determinism replay proof item (F009 / T003) is still open:
  • need a dedicated replay-focused test that demonstrates no expression-contract drift across replay.
• Event ingress correlation and API/stream alignment (F018+) remain open.

Progress — 2026-04-09 (Event-ingress authority guard for Temporal)

Completed scope in this pass

• Added Temporal guard in API event ingestion path (submitWorkflowEventAction) to prevent legacy DB-authoritative resume for Temporal runs:
  • file: ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts
  • behavior: if a matched candidate wait belongs to engine='temporal', ingestion fails explicitly (409) and records an audit error message; no wait/run projection mutation is performed for that Temporal run.

Key decisions and rationale

• Decision: hard-fail API resume attempts for Temporal candidate runs in the legacy DB event-resume code path.
  • Rationale: prevents WorkflowRuntimeV2.executeRun(...) from being an authority path for Temporal runs in operator/API control surfaces.

Tests added/updated

• Updated server/src/test/integration/workflowRuntimeV2.control.integration.test.ts:
  • New: Submit workflow event rejects legacy DB-authoritative resume for Temporal runs
  • Asserts:
    • explicit 409 failure
    • wait remains WAITING
    • run remains WAITING
    • runtime event audit row includes Temporal-unsupported error metadata

Commands / runbook used

• Focused server integration run for Temporal guard tests:
  • cd server && npm run -s test -- src/test/integration/workflowRuntimeV2.control.integration.test.ts -t "Temporal runs reject legacy|Temporal cancel failure|Submit workflow event rejects legacy DB-authoritative resume for Temporal runs"
  • Result: passed.

Progress — 2026-04-09 (Stream correlation contract)

Completed scope in this pass

• Extended workflow event stream schema/contracts to carry explicit workflow correlation:
  • packages/event-schemas/src/schemas/workflowEventSchema.ts
  • packages/event-bus/src/schemas/workflowEventSchema.ts
  • packages/event-schemas/src/schemas/eventBusSchema.ts (WorkflowPublishHooks.correlationKey, convertToWorkflowEvent mapping)
  • packages/event-bus/src/eventBus.ts (publishes workflow_correlation_key in stream payload)
• Updated stream ingestion worker correlation logic:
  • services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.ts
  • correlation resolution order:
    1. explicit (event.workflow_correlation_key, payload.workflowCorrelationKey, payload.correlationKey)
    2. configured derivation paths from WORKFLOW_RUNTIME_V2_EVENT_CORRELATION_PATHS_JSON (event-specific and * wildcard)
  • no fallback to event_id for wait routing
  • resolved key persisted in workflow_runtime_events.correlation_key
  • wait candidate lookup uses resolved key (tenant + event_name + resolved correlation)
  • Temporal signal payload carries resolved correlation key
  • unresolved correlation writes audit error metadata and skips wait routing/signaling

Key decisions and rationale

• Decision: use env-configured derivation paths as the immediate configurable source (WORKFLOW_RUNTIME_V2_EVENT_CORRELATION_PATHS_JSON).
  • Rationale: provides explicit, event-type-specific derivation without introducing schema migrations in this remediation slice.
• Decision: fail open for event-triggered workflow launches but fail closed for wait routing when correlation is missing.
  • Rationale: preserves event-triggered starts while preventing false-positive wait matches.

Tests added/updated

• Updated services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts:
  • explicit correlation routes candidate waits/signals by resolved key (not event_id)
  • configured derivation from payload path routes correctly
  • unresolved correlation writes audit error and skips wait-routing/signaling

Commands / runbook used

• Stream worker unit tests:
  • cd services/workflow-worker && npx vitest run src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts
  • Result: passed (4 tests).

Current gaps / next work

• API ingress still needs full correlation-derivation parity and Temporal-signal-only resume path alignment (F024, F025, F026, F028, T010).
• Determinism replay test (F009 / T003) and cutover regression/projection tests (T011, T012) remain open.

Progress — 2026-04-09 (Support posture documentation)

Completed scope in this pass

• Added hard-cutover support posture document:
  • ee/docs/plans/2026-04-08-workflow-v2-temporal-hard-cutover-remediation/HARD_CUTOVER_SUPPORT_POSTURE.md
• Document includes:
  • Temporal authority model
  • operator/API run-control support matrix (cancel supported, resume/retry/requeue_event_wait unsupported)
  • event-ingress correlation posture
  • support/debugging guidance for projection fields

Progress — 2026-04-09 (API/stream correlation + Temporal signal-only ingress)

Completed scope in this pass

• Aligned API event ingress with stream correlation contract using a shared resolver:
  • Added ee/packages/workflows/src/lib/workflowEventCorrelation.ts
  • Updated stream worker (services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.ts) to use shared resolver
  • Updated API action schema to accept explicit workflow correlation while supporting derivation fallback:
    • ee/packages/workflows/src/actions/workflow-runtime-v2-schemas.ts
• Reworked submitWorkflowEventAction to keep Temporal authority in Temporal:
  • file: ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts
  • Temporal candidate waits are no longer DB-resolved/advanced.
  • Temporal event waits are routed via signalWorkflowRuntimeV2Event(...).
  • Temporal human waits are routed via new signalWorkflowRuntimeV2HumanTask(...).
  • DB candidate lookup remains index/routing aid; payload-filter matching is no longer used as final authority for Temporal waits.
  • Missing correlation writes clear runtime-event audit metadata and skips wait routing.
• Added Temporal human-task signal helper:
  • ee/packages/workflows/src/lib/workflowRuntimeV2Temporal.ts

Determinism / replay test scope

• Added replay/checkpoint determinism coverage for canonical expressions in Temporal workflow unit suite:
  • ee/temporal-workflows/src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts
  • New test validates continue-as-new checkpoint replay path with canonical control.if expression semantics.

Tests added/updated

• server/src/test/integration/workflowRuntimeV2.control.integration.test.ts
  • Temporal API event ingress signals event waits without DB-authoritative mutation.
  • Derived correlation (WORKFLOW_RUNTIME_V2_EVENT_CORRELATION_PATHS_JSON) routes Temporal waits and avoids executeRun.
  • Temporal payload-filter matching stays inside Temporal wait contract (API still signals candidate wait).
  • Temporal human waits route through Temporal human-task signal.
  • Resume/retry unsupported Temporal actions assert no WorkflowRuntimeV2.executeRun(...) call.
• services/workflow-worker/src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts
  • remains passing with shared correlation helper integration.

Commands / runbook used

• cd services/workflow-worker && npx vitest run src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts
• cd ee/temporal-workflows && npx vitest run src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts
• cd server && mkdir -p coverage/.tmp && npm run -s test -- --coverage.enabled=false src/test/integration/workflowRuntimeV2.control.integration.test.ts -t "Submit workflow event|Temporal runs reject legacy admin resume|Temporal runs reject legacy retry"

Plan checklist updates

• Marked implemented: F009, F024, F025, F026, F028, F036
• Marked implemented: T003, T010, T011, T012

Remaining open items after this pass

• Features: F029, F030, F031, F032
• Tests: no remaining unchecked tests in this plan file.

Progress — 2026-04-09 (Legacy worker/runtime gating for Temporal runs)

Completed scope in this pass

• Gated legacy scheduler/runtime execution paths from affecting Temporal runs:
  • shared/workflow/workers/WorkflowRuntimeV2Worker.ts
    • due retry/time/event-time waits are now explicitly skipped when run.engine === 'temporal'
  • shared/workflow/runtime/runtime/workflowRuntimeV2.ts
    • acquireRunnableRun(...) excludes Temporal runs from runnable acquisition
    • executeRun(...) now hard-fails for Temporal runs with explicit unsupported-runtime error
  • services/workflow-worker/src/v2/WorkflowRuntimeV2Worker.ts
    • mirrored Temporal wait skip guardrails for service worker code path

Projection/supportability impact

• Temporal projections are no longer mutated by legacy worker timeout/retry polling paths.
• Runtime event projections continue to persist resolved correlation and matched run metadata for Temporal signal routing.
• Unsupported Temporal run-control actions continue returning actionable operator metadata (code/action/engine/runId).

Tests added/updated

• Updated server/src/test/integration/workflowRuntimeV2.control.integration.test.ts:
  • Added worker timeout polling does not execute Temporal runs through legacy runtime authority
  • Validates temporal wait remains WAITING, run remains WAITING, and WorkflowRuntimeV2.executeRun(...) is not invoked.

Commands / runbook used

• cd server && mkdir -p coverage/.tmp && npm run -s test -- --coverage.enabled=false src/test/integration/workflowRuntimeV2.control.integration.test.ts -t "Submit workflow event|Temporal runs reject legacy admin resume|Temporal runs reject legacy retry|timeout polling"
• cd services/workflow-worker && npx vitest run src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts src/index.startup.test.ts

Plan checklist updates

• Marked implemented: F029, F030, F031, F032

Remaining open items after this pass

• Features: none
• Tests: none