PSA/ee/docs/plans/2026-06-09-temporal-only-workflow-engine/SCRATCHPAD.md

# Scratchpad: temporal-only workflow engine

## Origin

Customer video: Run button "did nothing"; internal testing couldn't reproduce.
Investigation concluded the UI path always toasts/navigates/disables — the
silent failure modes are downstream (run created, nothing executes it).
Design: `../2026-06-09-temporal-only-workflow-engine-design.md`.

## Key discoveries (verified against code, 2026-06-09)

- Producer hardcodes the task queue (`workflowRuntimeV2Temporal.ts:42-46`);
  worker honors `WORKFLOW_RUNTIME_V2_TEMPORAL_TASK_QUEUE`
  (`WorkflowRuntimeV2TemporalWorker.ts:61`). Override ⇒ split brain.
- Server defaults engine to temporal unless
  `WORKFLOW_RUNTIME_V2_ENABLE_TEMPORAL_POLLING` is falsy
  (`workflowRunLauncher.ts:20-22`); worker DB poller is opt-in via
  `WORKFLOW_RUNTIME_V2_ENABLE_DB_POLLING` (`index.ts:62-92`). Mismatch ⇒
  stranded `engine='db'` runs.
- `executeWorkflowRuntimeV2Run` (activities:36) is exported but never invoked
  by any workflow or worker registration — the live Temporal path is the
  interpreter + per-step activities. Hence the whole DB interpreter
  (`executeRun` & co.) is orphaned once legacy actions/workers go.
- Temporal activities never write `workflow_run_snapshots` (grep: zero
  snapshot references in activities). Snapshots are DB-interpreter artifacts;
  keep tables/models for historical reads only.
- TWO DB poller copies: `services/workflow-worker/src/v2/` (stripped) and
  `shared/workflow/workers/` (canonical; also sweeps `workflow_data_store`
  expiry — must be relocated, it is not engine work).
- Legacy controls hard-409 on temporal runs via
  `assertLegacyRunControlSupported` (actions:1259), but the panel shows
  Retry for ANY FAILED run (`canRetry`, panel:906) — guaranteed 409 trap.
- Replay pre-fills payload from `getWorkflowRunAction`'s **redacted**
  `input_json` (actions:2326-2332; panel:517-525) and always submits it
  (`hasExplicitReplayPayload`, actions:3253) ⇒ replays run with `[REDACTED]`
  placeholders when redaction configured.
- Replay success only toasts and refreshes the OLD run (panel:1002-1004) —
  no navigation to the new run. Big contributor to "did nothing" perception.
- `docker-compose.ee.yaml:240` sets TEMPORAL_POLLING=false — base EE compose
  relies on the DB engine unless `docker-compose.temporal.ee.yaml` overlay is
  applied. Appliance flux profile sets neither flag (defaults = temporal).
- Toasts are fine: `ThemedToaster` mounted in root layout at zIndex 999999,
  above dialog z-70. UI can't silently swallow errors.

## Decisions (Robert, 2026-06-09)

1. Scope: full engine removal (not config-only, not interpreter rewrite).
2. Retry/Resume/Requeue deleted; Replay is the operator recovery story.
3. Migration cancels stranded non-temporal RUNNING/WAITING runs.
4. Bundle the redacted-replay-payload fix and the stuck-run banner.
5. Merge Temporal services into base `docker-compose.ee.yaml`; delete overlay.

## Open questions / watch-outs for implementation

- `launchPublishedWorkflowRun`'s `execute?: boolean` param: check remaining
  callers (schedules?) before assuming always-start; `execute:false` + db
  engine used to mean "poller will pick it up" — that semantic dies with the
  poller.
- `workflowRunStartLimiter` and concurrency checks are duplicated between
  `startWorkflowRunAction` and `launchPublishedWorkflowRun` — possible
  follow-up simplification, out of scope here.
- i18n: removing panel buttons orphans `runDetails.actions.retry/resume/
  requeueEvent` + dialog keys across locale files — sweep them.
- e2e rewrite needs a Temporal test target: check what
  `WorkflowRuntimeV2TemporalWorker.integration.test.ts` uses (likely
  TestWorkflowEnvironment) and reuse the harness.
- External infra repos may still set the deleted env flags — harmless after
  removal (code ignores them), but sweep separately.
- Stranded-run migration: also check `workflow_run_waits` rows whose run is
  being canceled — resolve with a status that the run studio renders sanely.

## Implementation notes (2026-06-09)

- `launchPublishedWorkflowRun`'s `execute` flag was only ever passed as `true`
  (4 call sites) — removed along with the engine ternary. `executionKey` is
  genuinely used (schedules, webhooks, event launch) and stays.
- `WorkflowRuntimeV2` is now only the run-row projection writer (`startRun`);
  the whole DB interpreter (~1,300 lines) was deleted after confirming the
  Temporal interpreter never called it (`executeWorkflowRuntimeV2Run` was
  exported but never registered/invoked).
- `services/workflow-worker/src/v2/WorkflowRuntimeV2Worker.ts` was already
  unreferenced (index.ts imported the shared copy) — both deleted.
- Bulk Resume in `WorkflowRunList` used the legacy resume action — removed
  with the per-run buttons; bulk Cancel stays.
- Replay payload-dirty detection compares the textarea string against the
  pre-filled pristine string (ref) — exact-match is sufficient because
  untouched textareas don't reformat.
- `server/src/test/unit/workflowRunLauncher.unit.test.ts` fails to LOAD even
  unmodified (pre-existing: `@alga-psa/db/workDate` unresolvable through
  `importOriginal` of the runtime in this worktree). Updated for the new API
  anyway; failure is environmental, also see quota test's stale `tenant_id`
  field which suggests these server unit tests aren't in the active CI gate.
- Playwright workflow suites: host-run server needs `TEMPORAL_ADDRESS`
  pointing at the new `temporal-playwright` service (host port 17233 via
  `PLAYWRIGHT_TEMPORAL_PORT`) for replay/run-start flows to function.
- ee/server full typecheck needs `NODE_OPTIONS=--max-old-space-size=12288`.

## Remaining follow-ups (tests.json items still false)

- Migration coverage (T046–T049): the stranded-run migration has no automated
  test; needs the DB-backed integration harness.
- Stuck-run banner (T042–T045) and removed-button absence (T033–T035): no
  component tests written; behavior is hand-verifiable in the run studio.
- Replay UI dirty-detection (T036/T039): server-side contract is covered;
  the client-side "send nothing when unedited" path is not unit-tested.
- Full-stack runs (T030/T050/T052): need a live compose stack with Temporal;
  `docker compose -f docker-compose.ee.yaml up` then start a manual run.
- DB-backed integration suites (control/publish/e2e) were verified by
  typecheck + vitest collection only in this worktree (no DB available);
  CI run pending.
- EventStreamWorker vitest suite (7 cases) fails in this worktree on real
  Redis/mock-resolution grounds — confirmed identical failure on pre-change
  code; not a regression.

## Commands

- Find engine references: `grep -rn "engine.*'db'\|'db'.*engine" --include="*.ts" shared ee services server | grep -v node_modules`
- Flag sweep: `grep -rn "WORKFLOW_RUNTIME_V2_ENABLE" --include="*" . | grep -v node_modules | grep -v docs/plans`
- Stranded runs (prod triage): `select run_id, status, engine, started_at from workflow_runs where (engine is null or engine='db') and status in ('RUNNING','WAITING');`