PSA/ee/docs/plans/2026-04-09-workflow-v2-runtime-worker-ownership-split/SCRATCHPAD.md

# SCRATCHPAD — Workflow V2 Runtime Worker Ownership Split

## Purpose

Track the architecture change that moves authored Workflow Runtime V2 Temporal execution into `workflow-worker` and repairs the runtime/bootstrap package boundary.

Related plans:
- `ee/docs/plans/2026-04-08-workflow-v2-temporal-native-runtime/`
- `ee/docs/plans/2026-04-08-workflow-v2-temporal-hard-cutover-remediation/`

## User-approved decisions

- Authored Workflow Runtime V2 execution should move from `temporal-worker` to `workflow-worker`.
- `temporal-worker` should continue owning non-authored/domain Temporal workflows only.
- Ownership should be expressed in the same `workflow-worker` process/container, not a second sidecar process.
- This is a real architecture change for the codebase going forward, not just a local workaround.
- Preferred cleanup direction is the proper split:
  - worker-safe runtime core
  - separate bootstrap/app-wiring surface
- No more brainstorming needed; move directly to ALGA plan.

## Why this plan exists

Observed local Temporal behavior showed:
- authored runs launch into Temporal successfully
- Temporal UI reports `No Workers Running` for queue `workflow-runtime-v2`
- `workflow-worker` is up
- `temporal-worker` is not healthy

That exposed two separate but related issues:
1. operational ownership for authored runtime is in the wrong worker
2. `@alga-psa/workflows/runtime` is not a clean worker-safe boundary

## High-signal findings

### Current worker split
- `services/workflow-worker/src/index.ts`
  - currently boots authored workflow support duties
  - initializes workflow runtime bootstrap
  - starts event-stream worker
  - optionally starts legacy DB polling worker
  - does **not** currently poll Temporal authored queue

- `ee/temporal-workflows/src/worker.ts`
  - currently includes `workflow-runtime-v2` in its default queue list
  - also owns non-authored queues like tenant/domain/job queues

### Current runtime boundary smell
- `ee/packages/workflows/src/runtime/index.ts`
  - currently mixes runtime exports with bootstrap side effects
  - imports AI action registration
  - imports AI inference wiring from `packages/ee/src/services/workflowInferenceService`
  - re-exports a large shared runtime barrel

### Current layering leaks seen during worker boot investigation
- unresolved `@shared/*` imports in authored runtime paths
- repo-relative imports from built workflow dist back into source layout
- mixed app/runtime/bootstrap concerns pulled into worker startup

## Files likely involved

### Worker ownership / startup
- `services/workflow-worker/src/index.ts`
- `services/workflow-worker/Dockerfile`
- `docker-compose.ee.yaml`
- `docker-compose.temporal.ee.yaml`
- `ee/temporal-workflows/src/worker.ts`

### Authored Temporal runtime implementation
- `ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts`
- `ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts`
- `ee/temporal-workflows/src/workflows/index.ts`

### Runtime/package boundary split
- `ee/packages/workflows/src/runtime/index.ts`
- likely new worker-safe runtime core entrypoint under `ee/packages/workflows/src/runtime/`
- likely new bootstrap/app-wiring entrypoint under `ee/packages/workflows/src/runtime/` or nearby
- `ee/packages/workflows/package.json`
- `ee/packages/workflows/tsup.config.ts`

### Alias/import cleanup hotspots already observed
- `shared/workflow/runtime/nodes/registerDefaultNodes.ts`
- `shared/workflow/actions/emailWorkflowActions.ts`
- any worker-reachable path still importing `@shared/*`

## Constraints / guardrails

- Do not reintroduce legacy DB runtime authority for authored Workflow Runtime V2.
- Prefer explicit worker-safe boundaries over container-only symlink or build hacks.
- Keep authored queue name stable unless separately approved.
- Minimize blast radius to non-authored/domain Temporal workflows.
- Keep `workflow-worker` as a single process/container for authored support + authored queue polling.

## Validation targets

### Ownership validation
- `workflow-worker` logs show Temporal polling for `workflow-runtime-v2`
- `temporal-worker` logs/config no longer show authored queue ownership
- Temporal UI shows active workers for `workflow-runtime-v2`

### Behavior validation
- manual run launches Temporal workflow and progresses with `workflow-worker`
- authored `time.wait` / `event.wait` progress without `temporal-worker`
- non-authored/domain Temporal queues still run under `temporal-worker`

### Boundary validation
- worker-safe runtime imports do not require `@shared/*`
- worker-safe runtime imports do not require repo-relative source hops from dist
- worker-safe runtime imports do not drag UI/app-only modules into worker boot

## Useful commands

### Worker/container inspection
- `docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' | rg 'temporal-worker|temporal-dev|workflow-worker'`
- `docker logs --tail 120 alga-psa-local-test-workflow-worker-1`
- `docker logs --tail 120 alga-psa-local-test-temporal-worker-1`

### Queue ownership checks
- open Temporal UI at `http://localhost:8088`
- inspect queue `workflow-runtime-v2`
- confirm workers tab shows ownership from `workflow-worker` expectations

### Import graph checks
- `rg -n "@shared/|packages/ee/src|\.\./\.\./\.\./\.\./\.\./shared" ee/packages/workflows/src ee/temporal-workflows/src shared/workflow -g '!**/dist/**'`
- `node -e "import('./ee/packages/workflows/dist/runtime/index.mjs')"`

## Open follow-ups

- Decide whether authored workflow definitions/activities should remain physically under `ee/temporal-workflows` for now or move later.
- Decide whether to add a hard startup assertion against dual queue ownership.
- Decide whether `workflow-worker` should host only authored runtime queue polling or eventually more Temporal responsibilities.

## Implementation log (2026-04-09)

### Completed feature set in this checkpoint
- Ownership + queue split foundations: `F001`, `F002`, `F003`, `F004`, `F006`, `F007`, `F008`, `F009`, `F010`
- Runtime boundary split: `F011`, `F012`, `F013`, `F014`, `F015`, `F016`, `F017`
- Worker-safe import layering and startup dependency cleanup: `F018`, `F019`, `F020`, `F021`
- Contract/environment continuity: `F022`, `F023`, `F024`, `F025`, `F026`

### Key decisions and rationale
- `workflow-worker` now owns authored Temporal polling by adding a dedicated in-process Temporal poller (`WorkflowRuntimeV2TemporalWorker`) that starts alongside existing event-ingress workers.
  - Rationale: satisfy single-process ownership (`FR-3`) and make authored run debugging start from `workflow-worker`.
- `temporal-worker` now hard-fails if configured with `workflow-runtime-v2`.
  - Rationale: enforce no-dual-ownership risk mitigation (`F008`) instead of relying on convention.
- `@alga-psa/workflows/runtime` was split into explicit surfaces:
  - `runtime/core` for worker-safe initialization (no AI/bootstrap wiring)
  - `runtime/bootstrap` for app/server richer registration
  - `runtime/index` now re-exports bootstrap for backward compatibility.
  - Rationale: preserve existing server behavior while giving workers a safe core import path.
- `workflow-worker` switched imports from `@alga-psa/workflows/runtime` to `@alga-psa/workflows/runtime/core`.
  - Rationale: keep app/bootstrap-only side effects out of worker startup.

### Files changed (high signal)
- Worker ownership/polling:
  - `services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.ts` (new)
  - `services/workflow-worker/src/index.ts`
  - `services/workflow-worker/src/index.startup.test.ts`
  - `services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.test.ts` (new)
- Temporal worker scope:
  - `ee/temporal-workflows/src/workerConfig.ts` (new)
  - `ee/temporal-workflows/src/worker.ts`
  - `ee/temporal-workflows/src/__tests__/worker-queue-ownership.test.ts` (new)
- Runtime split:
  - `ee/packages/workflows/src/runtime/core.ts` (new)
  - `ee/packages/workflows/src/runtime/bootstrap.ts` (new)
  - `ee/packages/workflows/src/runtime/index.ts`
  - `ee/packages/workflows/package.json` exports update
  - `ee/packages/workflows/src/runtime/__tests__/runtimeEntryBoundaries.test.ts` (new)
- Compose/build wiring:
  - `docker-compose.ee.yaml`
  - `docker-compose.temporal.ee.yaml`
  - `services/workflow-worker/Dockerfile`
  - `services/workflow-worker/package.json`

### Commands and checks run
- `cd services/workflow-worker && npx vitest run src/index.startup.test.ts src/v2/WorkflowRuntimeV2EventStreamWorker.test.ts src/v2/WorkflowRuntimeV2TemporalWorker.test.ts`
- `cd services/workflow-worker && npm run build`
- `cd ee/temporal-workflows && npx vitest run src/__tests__/worker-queue-ownership.test.ts`
- `cd ee/temporal-workflows && npm run build`
- `cd ee/packages/workflows && npx vitest run src/runtime/__tests__/runtimeEntryBoundaries.test.ts`
- `cd ee/packages/workflows && npm run build`

### Tests checklist updates completed in this checkpoint
- `T001` implemented
- `T003` implemented
- `T004` implemented
- `T005` implemented
- `T006` implemented
- `T008` implemented
- `T010` implemented

### Gotchas discovered
- `ee/temporal-workflows/src/worker.ts` could not be directly imported in a lightweight config unit test due broader worker module dependency graph resolution; fixed by extracting queue config to `workerConfig.ts` for isolated testing.
- `services/workflow-worker/package.json` test script pointed at a missing `vitest.config.ts`; updated to plain `vitest`/`vitest --watch`.

### Remaining items after this checkpoint
- Features not yet implemented/verified: `F005`, `F027`, `F029`
- Tests not yet implemented/verified: `T002`, `T007`, `T009`, `T011`
- Added ownership/support documentation: `OWNERSHIP.md` (`F030`).
- Updated `ee/packages/workflows/src/lib/workflowRunLauncher.ts` to import runtime initialization from `@alga-psa/workflows/runtime/core` so worker startup paths do not transitively pull `runtime/bootstrap`.
- Strengthened `services/workflow-worker/scripts/validate-runtime-imports.mjs` with explicit checks for unresolved `@shared/*` aliases and bootstrap-only runtime dependency leakage (`registerAiActions`, `workflowInferenceService`, `runtime/bootstrap`).
- Added contract regression test for Temporal launch/signal queue authority:
  - `ee/packages/workflows/src/lib/__tests__/workflowRuntimeV2Temporal.contract.test.ts` (`T010`)
- Attempted local compose smoke for `F029`/`T009` with:
  - `docker compose -f docker-compose.ee.yaml -f docker-compose.temporal.ee.yaml up -d --build workflow-worker temporal-worker`
  - Blocked by missing compose env/secret context in this shell (`service "setup" refers to undefined secret postgres_password`).

## Implementation log (2026-04-09, follow-up checkpoint)

### Completed in this checkpoint
- Feature `F005` implemented via real Temporal integration coverage in `workflow-worker`.
- Feature `F027` implemented by removing authored-runtime modules from `temporal-worker` startup entrypoints.
- Test `T002` implemented with an automated authored-run execution smoke test (Temporal test environment + `WorkflowRuntimeV2TemporalWorker`).
- Test `T007` implemented with import-graph regression tests that validate dist-graph safety against unresolved `@shared/*` aliases and repo-layout-relative source hops.

### Decisions and rationale
- Added non-authored Temporal worker entrypoint barrels (`non-authored-index.ts`) and pointed `ee/temporal-workflows/src/worker.ts` to those barrels.
  - Rationale: `temporal-worker` should not carry authored-runtime startup/module baggage once authored queue ownership moved.
- Added an integration test that starts `WorkflowRuntimeV2TemporalWorker` against `TestWorkflowEnvironment` and executes `workflowRuntimeV2RunWorkflow` on queue `workflow-runtime-v2`.
  - Rationale: directly proves authored runtime tasks are picked up/progressed by `workflow-worker` without requiring `temporal-worker`.
- Made `validate-runtime-imports.mjs` accept override env `WORKFLOW_WORKER_VALIDATE_DIST_ROOT`.
  - Rationale: allows deterministic regression tests against fixture dist trees while preserving production behavior.

### Files changed in this checkpoint
- `ee/temporal-workflows/src/workflows/non-authored-index.ts` (new)
- `ee/temporal-workflows/src/activities/non-authored-index.ts` (new)
- `ee/temporal-workflows/src/worker.ts`
- `ee/temporal-workflows/src/__tests__/worker-queue-ownership.test.ts`
- `services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.integration.test.ts` (new)
- `services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.integration.workflows.mjs` (new)
- `services/workflow-worker/src/v2/WorkflowRuntimeV2TemporalWorker.integration.activities.mjs` (new)
- `services/workflow-worker/scripts/validate-runtime-imports.mjs`
- `services/workflow-worker/scripts/validate-runtime-imports.test.ts` (new)

### Commands and checks run
- `cd services/workflow-worker && npx vitest run src/v2/WorkflowRuntimeV2TemporalWorker.test.ts src/v2/WorkflowRuntimeV2TemporalWorker.integration.test.ts src/index.startup.test.ts scripts/validate-runtime-imports.test.ts`
- `cd ee/temporal-workflows && npx vitest run src/__tests__/worker-queue-ownership.test.ts`
- `cd services/workflow-worker && npm run build`
- `cd ee/temporal-workflows && npm run build`

### Compose smoke attempts and blockers (`F029`, `T009`, `T011`)
- Brought up compose with full base layering:
  - `docker compose -f docker-compose.base.yaml -f docker-compose.ee.yaml -f docker-compose.temporal.ee.yaml up -d --build workflow-worker temporal-worker temporal-ui temporal-dev`
- Resolved missing external volume blocker by creating:
  - `docker volume create workflow-wait-steps-productization_ngrok_data`
- Resolved Temporal host port collision by running compose with:
  - `EXPOSE_TEMPORAL_PORT=17233 EXPOSE_TEMPORAL_UI_PORT=18088 TEMPORAL_ADDRESS=temporal-dev:7233 ...`
- Remaining runtime blockers observed in logs:
  - `workflow-worker` fails at startup with Temporal native bridge load error:
    - `Error relocating ... @temporalio/core-bridge ... __register_atfork: symbol not found`
  - `temporal-worker` fails startup validation due missing required config/secrets in this shell context:
    - missing `ALGA_AUTH_KEY`, `NEXTAUTH_SECRET`, `APPLICATION_URL`
- Because workers are not both healthy in this environment, could not complete:
  - `F029` (UI active authored queue worker confirmation)
  - `T009` (compose/dev environment authored runtime ownership smoke)
  - `T011` (DB-backed integration sanity across workflow tables)

### Remaining items after this checkpoint
- Features not yet implemented/verified: `F029`
- Tests not yet implemented/verified: `T009`, `T011`

## Implementation log (2026-04-09, runtime packaging follow-up)

### What was changed
- Added a dedicated compose smoke harness script:
  - `scripts/workflow-runtime-v2-compose-smoke.mjs`
  - Added root script entry:
    - `package.json` → `test:workflow-runtime-v2-compose-smoke`
- Extended `workflow-worker` image build inputs and runtime dependencies:
  - `services/workflow-worker/Dockerfile`
  - Builds now include additional workspaces required by authored Temporal runtime paths (`@alga-psa/core`, `@alga-psa/types`, `@alga-psa/db`, `@alga-psa/formatting`, `@alga-psa/validation`, `@alga-psa/storage`, plus existing workflow/temporal/shared chain).
- Added temporal compose profile defaults for worker startup env:
  - `docker-compose.temporal.ee.yaml`
  - Includes local defaults for app/auth keys used during worker startup.
- Hardened `@alga-psa/core` runtime exports for worker containers:
  - `packages/core/package.json` now points runtime `import` exports to built JS under `dist/` instead of TS sources.
  - `packages/core/tsup.config.ts` now enables `addJsExtensions: true` so dist ESM imports are Node-resolvable.

### New findings
- The previous blocker (`ERR_UNKNOWN_FILE_EXTENSION` for `/app/packages/core/src/lib/logger.ts`) was due to `@alga-psa/core` exports resolving to TS source in standalone worker runtime.
- After redirecting core exports to dist, worker startup moved to the next failure:
  - `Cannot find module '/app/packages/core/dist/lib/secrets/EnvSecretProvider' imported from /app/packages/core/dist/lib/secrets/index.js`
  - Root cause: extensionless relative imports in core dist ESM output.
  - Fixed by enabling `addJsExtensions` in core tsup config.
- With those fixes in place, authored queue smoke is still blocked by compose-environment instability and repeated project collisions/port contention during iterative retries (not a single deterministic app-code failure yet for final `F029/T009/T011` sign-off).

### Current blocker state
- `F029`, `T009`, `T011` remain unflipped.
- Latest known high-signal blocker for clean verification is environment orchestration stability during long compose build/start loops (port collisions and overlapping compose projects), not a closed acceptance pass yet.

### Additional unblock attempt (same day)
- Updated `services/workflow-worker/Dockerfile` base image from Alpine to Debian slim to remove Temporal native bridge libc mismatch seen earlier (`__register_atfork`).
- Updated `docker-compose.temporal.ee.yaml` to provide local defaults for:
  - `APPLICATION_URL`
  - `NEXTAUTH_URL`
  - `NEXTAUTH_SECRET`
  - `ALGA_AUTH_KEY`
- Rebuilt `workflow-worker` image and re-ran compose smoke with explicit local env overrides:
  - `EXPOSE_TEMPORAL_PORT=17233 EXPOSE_TEMPORAL_UI_PORT=18088 TEMPORAL_ADDRESS=temporal-dev:7233 ALGA_AUTH_KEY=local-alga-auth-key NEXTAUTH_SECRET=local-nextauth-secret APPLICATION_URL=http://localhost:3000 docker compose -f docker-compose.base.yaml -f docker-compose.ee.yaml -f docker-compose.temporal.ee.yaml up -d workflow-worker temporal-worker temporal-ui temporal-dev`

### New observed blockers after unblock attempt
- `workflow-worker` still fails startup before stable queue polling due missing runtime modules in container image:
  - missing `@ee/lib` import from `registerEnterpriseStorageProviders`
  - missing `@alga-psa/types/dist/index.js` import from `@alga-psa/workflows/dist/runtime/index.mjs`
- `temporal-worker` startup validation reaches DB checks but fails with:
  - `password authentication failed for user "app_user"`
- Because of these unresolved startup/runtime issues, `F029` / `T009` / `T011` remain unverified.

## Implementation log (2026-04-09, runtime packaging follow-up 2)

### Additional runtime fixes applied
- `packages/types/tsup.config.ts`
  - Enabled `addJsExtensions: true` so `dist/index.js` and internal imports emit explicit `.js` specifiers for Node ESM.
- `packages/validation/tsup.config.ts`
  - Enabled `addJsExtensions: true` preemptively for the same worker-runtime ESM compatibility reason.

### New high-signal failure observed after core fix
- Worker startup advanced further but then failed on:
  - `Cannot find module '/app/packages/types/dist/lib/attributes' imported from /app/packages/types/dist/index.js`
- This confirmed the same extensionless-import class of failure now affected `@alga-psa/types`; fix above addresses that class.

### Verification state after this follow-up
- Full `F029/T009/T011` end-to-end acceptance is still not closed in this session.
- Latest blocker remains compose-heavy verification reliability (long build/start cycles and repeated project/port churn), with worker now moving through successive package-resolution failures as runtime packaging is hardened.

## Implementation log (2026-04-10, compose smoke closure)

### Completed in this checkpoint
- Feature `F029` implemented/verified.
- Tests `T009` and `T011` implemented/verified.

### Decisions and rationale
- Updated the compose smoke harness to allocate host ports dynamically (`DB`, `Redis`, `pgbouncer`, `Temporal`, `Temporal UI`, `server`, `hocuspocus`) so parallel local stacks no longer block the smoke with host-port collisions.
- Started `workflow-worker` with `--no-deps` and explicitly started only required services (`setup`, `redis`, `temporal-dev`, `temporal-ui`) to avoid unrelated `server`/`hocuspocus` startup coupling during authored-runtime validation.
- Hardened the smoke's DB fixture inserts to tolerate schema variants by filtering payload keys to actual table columns at runtime.
  - Rationale: this keeps `T011` meaningful even when migration shape differs slightly across branches.
- Fixed authored-runtime worker imports in Temporal activities/interpreter paths to use `@alga-psa/workflows/runtime/core` instead of the mixed bootstrap barrel.
  - Rationale: prevents bootstrap-only runtime dependencies from being pulled into worker startup and avoids ESM runtime failures.
- Fixed `@alga-psa/types` ESM barrel exports by using explicit `index` subpaths for directory exports (`constants`, `interfaces`).
  - Rationale: avoids unresolved `dist/constants.js` / `dist/interfaces.js` imports in worker containers.
- Updated smoke fixture workflow definition shape to omit `trigger` when absent (instead of `trigger: null`) so runtime schema parsing succeeds.

### Files changed in this checkpoint
- `scripts/workflow-runtime-v2-compose-smoke.mjs`
- `packages/types/src/index.ts`
- `ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts`
- `ee/temporal-workflows/src/workflows/workflow-runtime-v2-interpreter.ts`
- `ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts`
- `ee/docs/plans/2026-04-09-workflow-v2-runtime-worker-ownership-split/features.json`
- `ee/docs/plans/2026-04-09-workflow-v2-runtime-worker-ownership-split/tests.json`

### Commands and checks run
- `cd ee/temporal-workflows && npm run build`
- `cd packages/types && npm run build`
- `npm run -s test:workflow-runtime-v2-compose-smoke`
- Re-ran compose smoke after cleanup and confirmed pass again:
  - `npm run -s test:workflow-runtime-v2-compose-smoke`

### High-signal verification outcomes
- Compose smoke now passes end-to-end with exit code `0`.
- `workflow-worker` starts Temporal polling on `workflow-runtime-v2` in the compose profile and authored run execution proceeds.
- Smoke scenario confirms DB projection sanity for authored run + resumed event wait (`workflow_runs`, `workflow_run_waits`, `workflow_run_steps`).

### Gotchas captured
- Local schema in this branch does not include some previously assumed workflow columns (`published_version`, `definition_hash`), requiring schema-aware fixture inserts for stable integration smoke behavior.
- Runtime definition parser requires absent trigger to be omitted, not `null`.