PSA/ee/docs/plans/2026-06-09-temporal-only-workflow-engine-design.md

# Temporal-only workflow engine

Workflow runtime V2 currently supports two execution engines: Temporal (the
default) and a legacy DB-polling engine. The split is selected per-process by
environment flags, which lets the API server and the workflow worker disagree
about who executes a run. When they disagree — or when the producer and worker
resolve different Temporal task queues — a click on **Run** or **Replay**
creates a `workflow_runs` row that nothing ever executes. The run sits in
`RUNNING` with zero steps, and to the operator the button "did nothing."

This change removes the DB engine entirely. Temporal becomes the only engine,
on one task queue, with no per-process configuration to get wrong.

## Failure modes this eliminates

1. **Engine flag mismatch.** The server defaults new runs to
   `engine='temporal'` unless `WORKFLOW_RUNTIME_V2_ENABLE_TEMPORAL_POLLING` is
   false (`ee/packages/workflows/src/lib/workflowRunLauncher.ts`), while the
   worker only starts the DB poller when
   `WORKFLOW_RUNTIME_V2_ENABLE_DB_POLLING=true`
   (`services/workflow-worker/src/index.ts`). Set the first flag to false on
   the server without setting the second on the worker and every new run is
   stranded.
2. **Task-queue split brain.** The worker honors a
   `WORKFLOW_RUNTIME_V2_TEMPORAL_TASK_QUEUE` override
   (`services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.ts`), but
   the producer hardcodes the contract constant
   (`ee/packages/workflows/src/lib/workflowRuntimeV2Temporal.ts`). Any
   environment that sets the variable starts runs on a queue nobody polls.
3. **Legacy run controls that 409 on Temporal runs.** Retry, Resume, and
   Requeue Event only work for DB-engine runs and hard-fail for Temporal runs
   via `assertLegacyRunControlSupported`
   (`ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts`), yet
   the Run Studio shows Retry on any FAILED run.
4. **Replay submits redacted payloads.** The Replay dialog pre-fills its
   payload editor from `run.input_json` as returned by `getWorkflowRunAction`,
   which applies `applyRunStudioRedactions`. Submitting unedited sends the
   redaction placeholders as an explicit payload override.
5. **Silent stuck runs.** A run accepted by Temporal but never picked up (a
   down worker, a queue backlog) shows as `RUNNING` with no steps and no
   warning.

## Design

### Launch path and task queue

`launchPublishedWorkflowRun` always writes `engine='temporal'` and always
starts the Temporal workflow; `isTemporalPollingEnabled()` is deleted.
`StartRunParams` loses its `engine` parameter — `startRun` writes
`'temporal'` unconditionally.

The task queue becomes the single constant in
`workflowRuntimeV2TemporalContract.ts`, used verbatim by both producer and
worker. The env override in `WorkflowRuntimeV2TemporalWorker` and the
`WORKFLOW_RUNTIME_V2_TEMPORAL_TASK_QUEUE` compose plumbing are removed.
Environment isolation remains the job of `TEMPORAL_ADDRESS` and
`TEMPORAL_NAMESPACE`.

### Worker service

`services/workflow-worker/src/index.ts` starts the Temporal worker and the
event-stream worker unconditionally. Both `WORKFLOW_RUNTIME_V2_ENABLE_*`
flags disappear. Both DB-poller classes are deleted:

- `services/workflow-worker/src/v2/WorkflowRuntimeV2Worker.ts`
- `shared/workflow/workers/WorkflowRuntimeV2Worker.ts`

The `workflow_data_store` expiry sweep currently embedded in the shared
poller is not engine work; it moves to a small standalone interval module in
the worker service.

### Server actions and event routing

Deleted, including API routes and UI entry points:

- `retryWorkflowRunAction` (`POST /api/workflow-runs/[runId]/retry`)
- `resumeWorkflowRunAction` (`POST /api/workflow-runs/[runId]/resume`)
- `requeueWorkflowRunEventWaitAction` (`POST /api/workflow-runs/[runId]/requeue`)
- `assertLegacyRunControlSupported` / `throwUnsupportedTemporalRunControlAction`

Simplified to the Temporal path only (engine branches removed):

- `cancelWorkflowRunAction`
- `resumeWorkflowRunFromQuotaPauseAction` (always signals quota resume)
- `submitWorkflowEventAction` and the event-stream worker's
  `engine !== 'temporal'` skip
- `server/src/lib/jobs/handlers/workflowQuotaResumeScanHandler.ts`

Replay (`replayWorkflowRunAction`) remains the operator recovery path for
failed runs: a fresh Temporal run with the original payload.

### DB interpreter deletion

The live Temporal path executes runs through the interpreter in
`ee/temporal-workflows/src/workflows/` plus per-step activities; it never
calls `WorkflowRuntimeV2.executeRun`. The only `executeRun` callers are the
legacy actions, the DB pollers, the quota handler's else-branch, and the
exported-but-never-invoked `executeWorkflowRuntimeV2Run` activity. With those
gone, the following are deleted from
`shared/workflow/runtime/runtime/workflowRuntimeV2.ts`:

- `executeRun`, `acquireRunnableRun`, `resumeRunFromEvent`,
  `resumeRunFromTimeout`
- the private step/action executor loop, `loadEnvelope`, `persistSnapshot`
- `executeWorkflowRuntimeV2Run` in
  `ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts`

`startRun` stays — the launcher and the child-run activity
(`startWorkflowRuntimeV2ChildRun`) both use it.

Kept as read-only history: the `engine` column and its `'db'` value, the
`workflow_run_snapshots` table and `WorkflowRunSnapshotModelV2` (Run Studio
still renders snapshots on pre-cutover runs; bulk-delete and tenant-deletion
cleanup still reference the table). Nothing writes new snapshots.

### Run Studio UI

`WorkflowRunDetailsPanel` drops the Retry, Resume, and Requeue Event buttons
and handlers; Replay, Cancel, and Export remain. Replay changes in two ways:

1. The payload is submitted only when the operator edits the pre-filled JSON.
   Unedited replays send no payload, and the server falls back to the
   original, unredacted `input_json`.
2. On success the UI navigates to the new run instead of staying on the old
   one.

### Stuck-run visibility

The run details panel shows a warning banner when a run is `RUNNING` with
zero steps and `started_at` is more than ~60 seconds old: the run is queued
and no worker has picked it up. This uses data the panel already fetches.

### Data migration

A migration finalizes stranded non-Temporal runs: rows with `engine` null or
`'db'` still in `RUNNING`/`WAITING` get `status='CANCELED'`,
`completed_at=now()`, and an `error_json` note that the DB execution engine
was removed; their open `workflow_run_waits` are resolved. Terminal
historical rows are untouched.

### Deployment

Temporal services merge from `docker-compose.temporal.ee.yaml` into
`docker-compose.ee.yaml` and the overlay file is deleted — an EE stack
without Temporal is not a valid deployment. The playwright workflow-deps
compose moves from DB polling to the Temporal stack. All engine env flags
disappear from compose files.

### Testing

- Delete `workflowEngineReferenceWorkflows.db.test.ts`; its coverage lives in
  the Temporal interpreter suite
  (`workflow-runtime-v2-run-workflow.test.ts`).
- Rewrite or delete the e2e/integration tests that drive
  `WorkflowRuntimeV2Worker`/`executeRun` directly
  (`server/src/test/e2e/workflowRuntimeV2.e2e.test.ts`,
  `server/src/test/integration/workflowRuntimeV2.control.integration.test.ts`,
  `server/src/test/integration/workflowRuntimeV2.publish.integration.test.ts`)
  — roughly 60–80 cases. Temporal-specific cases in those files survive.
- Remove flag-toggling cases from the launcher and worker-startup unit tests.
- Add: launcher always-temporal assertions, migration coverage, replay
  payload-dirty logic, stuck-run banner rendering.