PSA/ee/docs/plans/2026-04-28-workflow-step-quota-accounting/SCRATCHPAD.md

Scratchpad — Workflow Step Quota Accounting

• Plan slug: 2026-04-28-workflow-step-quota-accounting
• Created: 2026-04-28

What This Is

Rolling notes for the workflow step quota accounting plan. Keep decisions, discoveries, commands, links, and gotchas here as implementation proceeds.

Decisions

• (2026-04-28) Count every workflow step attempt at step start. Retries, failed attempts, forEach item attempts, and wait/human-task entry attempts all count. Quota-blocked steps do not count because they never start.
• (2026-04-28) Quota exhaustion pauses workflow runs instead of failing them. Runs remain at the current node_path with workflow_runs.status = 'WAITING' and a workflow_run_waits.wait_type = 'quota' record.
• (2026-04-28) Use Stripe subscription periods as the primary payment period source: stripe_subscriptions.current_period_start through current_period_end.
• (2026-04-28) Do not use contract recurring_service_periods, cadence ownership, invoice windows, or contract-line periods for this feature. Workflow quota is tenant platform licensing usage, not client contract billing usage.
• (2026-04-28) If no valid active Stripe period exists, fall back to the current UTC calendar month with tier defaults.
• (2026-04-28) Tier defaults are solo = 150, pro = 750, and premium = 10000 workflow step attempts per period.
• (2026-04-28) Use hybrid limit resolution: Stripe price metadata first, Stripe product metadata second, tier default last.
• (2026-04-28) Support workflow_step_limit=unlimited; unlimited tenants still record usage but are not quota-paused.
• (2026-04-28) Use a dedicated atomic counter table for enforcement and keep workflow_run_steps as the detailed audit/reconciliation ledger.
• (2026-04-28) Use column name tenant in new schema rather than tenant_id, per project schema convention and user instruction.
• (2026-04-28) Resume quota-paused workflows through both a recurring job and a manual resume action. Manual resume must not bypass quota.

Discoveries / Constraints

• (2026-04-28) DB workflow runtime creates step rows in shared/workflow/runtime/runtime/workflowRuntimeV2.ts inside executeRun() after resolveStepAtPath() and before executeStep().
• (2026-04-28) Temporal runtime creates projected step rows in ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts via projectWorkflowRuntimeV2StepStart().
• (2026-04-28) Existing workflow runtime tables include workflow_runs, workflow_run_steps, workflow_run_waits, workflow_action_invocations, workflow_run_snapshots, and workflow_runtime_events from migration server/migrations/20251221090000_create_workflow_runtime_v2_tables.cjs.
• (2026-04-28) Tenant tiers are defined in packages/types/src/constants/tenantTiers.ts as solo, pro, and premium.
• (2026-04-28) packages/types/src/constants/tierFeatures.ts already notes that WORKFLOW_DESIGNER is available to all tiers and that a usage cap was planned separately.
• (2026-04-28) Stripe subscription table is created in EE migration ee/server/migrations/20251014120000_create_stripe_integration_tables.cjs and includes current_period_start, current_period_end, stripe_price_id, status, and metadata.
• (2026-04-28) Existing scheduled job infrastructure includes server/src/lib/jobs/initializeScheduledJobs.ts, registerAllHandlers.ts, and jobHandlerRegistry.ts.

Commands / Runbooks

• (2026-04-28) Create plan folder: mkdir -p ee/docs/plans/2026-04-28-workflow-step-quota-accounting.
• (2026-04-28) Validate plan JSON manually or with the alga-plan validation helper after edits.

Links / References

• shared/workflow/runtime/runtime/workflowRuntimeV2.ts
• ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts
• server/migrations/20251221090000_create_workflow_runtime_v2_tables.cjs
• ee/server/migrations/20251014120000_create_stripe_integration_tables.cjs
• packages/types/src/constants/tenantTiers.ts
• packages/types/src/constants/tierFeatures.ts
• server/src/lib/jobs/initializeScheduledJobs.ts
• server/src/lib/jobs/registerAllHandlers.ts
• server/src/lib/jobs/jobHandlerRegistry.ts

Open Questions

• Should zero-limit metadata ever be valid, or should it remain invalid and fall back to tier default? Current PRD treats zero as invalid.
• What exact permission should gate manual quota resume if no dedicated workflow-run operation permission exists?
• Should rollout include an explicit enforcement feature flag, or is the plan to enforce immediately once shipped?

Implementation Notes (2026-04-28)

• Added shared quota service at shared/workflow/runtime/services/workflowStepQuotaService.ts.
• Resolver behavior implemented:
  • Preferred Stripe subscription period from stripe_subscriptions with status priority trialing > active > past_due > unpaid and valid current_period_start/end.
  • Fallback period to UTC month boundaries when Stripe tables/subscription period are unavailable.
  • Tier defaults from tenants.plan: solo=150, pro=750, premium=10000.
  • Metadata precedence implemented: stripe_prices.metadata.workflow_step_limit -> stripe_products.metadata.workflow_step_limit -> tier default.
  • workflow_step_limit=unlimited maps to effective_limit = null.
  • Invalid metadata safely ignored (falls through to next source).
• Atomic reservation implemented in same service:
  • Upserts workflow_step_usage_periods by (tenant, period_start, period_end).
  • Takes row lock (FOR UPDATE) before reservation decision.
  • Finite limits reject at/above limit without increment.
  • Unlimited limits always increment used_count.
• DB runtime enforcement integrated in shared/workflow/runtime/runtime/workflowRuntimeV2.ts:
  • Reservation now occurs before STARTED step row creation.
  • On exhaustion: run is set WAITING, node_path preserved, and quota wait (wait_type='quota') is created/reused.
• Temporal projection enforcement integrated in ee/temporal-workflows/src/activities/workflow-runtime-v2-activities.ts and workflow handling in ee/temporal-workflows/src/workflows/workflow-runtime-v2-run-workflow.ts:
  • Reservation happens before STARTED step projection.
  • On exhaustion: quota wait created/reused, run marked WAITING, activity returns quotaPaused result, workflow exits without treating it as failure.
• Added integration tests in server/src/test/integration/workflowStepQuotaService.integration.test.ts covering:
  • Stripe period resolution.
  • Fallback calendar month + tier default.
  • Metadata precedence and unlimited.
  • Finite reservation rejection-at-limit.
  • Unlimited reservation increments.
• Command run:
  • cd server && npm test -- src/test/integration/workflowStepQuotaService.integration.test.ts (pass).
• Added T005 coverage in server/src/test/integration/workflowStepQuotaService.integration.test.ts to verify uniqueness and upsert behavior on (tenant, period_start, period_end).
• Added structured observability logs in shared/workflow/runtime/services/workflowStepQuotaService.ts:
  • logger.debug on successful quota reservation with tenant/period/limit context.
  • logger.warn on quota exhaustion at reservation with tenant/period/used/limit context.
  • logger.warn when invalid workflow_step_limit metadata is detected on Stripe price/product and fallback resolution is used.
  • logger.info when resolver uses fallback UTC calendar periods due to missing Stripe period or missing Stripe tables.
• Validation command run:
  • cd server && npm test -- src/test/integration/workflowStepQuotaService.integration.test.ts (pass).
• Added scheduled quota resume scan job handler: server/src/lib/jobs/handlers/workflowQuotaResumeScanHandler.ts.
  • Scans workflow_run_waits with wait_type='quota' and status='WAITING'.
  • Uses FOR UPDATE SKIP LOCKED + configurable batchSize (default 100) for concurrency-safe scan batches.
  • Applies per-tenant capacity gating from workflowStepQuotaService.resolveQuotaSummary():
    • finite tenants resume up to effectiveLimit - usedCount
    • unlimited tenants resume all selected waits
  • Resolves selected waits, sets runs to RUNNING, writes run log entries, and re-enters DB runtime via WorkflowRuntimeV2.executeRun() without pre-consuming quota.
• Registered/scheduled job plumbing:
  • server/src/lib/jobs/registerAllHandlers.ts registration name workflow-quota-resume-scan.
  • server/src/lib/jobs/index.ts legacy registration + scheduleWorkflowQuotaResumeScanJob().
  • server/src/lib/jobs/initializeScheduledJobs.ts schedules recurring scan cron */5 * * * *.
• Added manual quota resume action:
  • ee/packages/workflows/src/actions/workflow-runtime-v2-actions.ts exports resumeWorkflowRunFromQuotaPauseAction.
  • Verifies permission and tenant ownership, requires WAITING + quota wait, checks current quota summary, returns structured exhausted-quota response when still blocked, otherwise resolves quota wait and executes runtime (no bypass of step-start reservation).
• Validation attempt:
  • cd server && npm test -- src/test/integration/workflowRuntimeV2.control.integration.test.ts (fails in current env due missing module resolution for @alga-psa/authorization/kernel, unrelated to quota changes).
• Added test coverage for concurrent finite quota reservations:
  • server/src/test/integration/workflowStepQuotaService.integration.test.ts
  • New case fires 12 concurrent reservations against finite limit 3 and asserts exactly 3 allows, 9 denies, and persisted used_count=3.
• Validation command run:
  • cd server && npm test -- src/test/integration/workflowStepQuotaService.integration.test.ts (pass).
• Added minimal run-level quota pause surfacing in workflow run detail responses:
  • listWorkflowRunStepsAction and exportWorkflowRunDetailAction now return quotaPause derived from active quota wait payload.
• Added reconciliation helper to quota service:
  • workflowStepQuotaService.reconcileUsagePeriod(tenant, periodStart, periodEnd) compares counter usage to step ledger count and returns drift.
• Added support/engineering documentation:
  • ee/docs/plans/2026-04-28-workflow-step-quota-accounting/OPERATIONS.md covering quota source/limits, pause/resume behavior, and reconciliation runbook.
• Added diagnostic/observability integration coverage in server/src/test/integration/workflowStepQuotaService.integration.test.ts:
  • Reconciliation drift test for reconcileUsagePeriod() (counterUsedCount, ledgerStepCount, drift).
  • Structured log assertions for invalid metadata fallback, reservation success, quota exhaustion, and fallback-calendar usage.
• Validation command run:
  • cd server && npm test -- src/test/integration/workflowStepQuotaService.integration.test.ts (pass after fixture fix for workflow_definitions.tenant_id).
• Added DB runtime quota integration coverage in server/src/test/integration/workflowRuntimeV2.control.integration.test.ts:
  • Reserve-before-step-start assertion (no workflow_run_steps rows exist before reservation callback returns).
  • Quota exhaustion on second step pauses run with wait_type='quota', keeps node_path, and avoids blocked-step row creation.
  • Retry and forEach attempt accounting assertions against workflow_step_usage_periods.used_count.
  • Event wait accounting assertion that first entry consumes quota and resume does not double count.
• Added Temporal coverage:
  • ee/temporal-workflows/src/activities/__tests__/workflow-runtime-v2-activities.test.ts adds projectWorkflowRuntimeV2StepStart tests for reserve-before-start and quota pause projection.
  • ee/temporal-workflows/src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts adds quota paused short-circuit behavior test (stepId:null, quotaPaused:true).
• Validation attempts:
  • cd server && npm test -- src/test/integration/workflowRuntimeV2.control.integration.test.ts fails in current env with known module resolution issue: @alga-psa/authorization/kernel missing from runtime action import graph.
  • cd ee/temporal-workflows && TEMPORAL_TEST_SKIP_ENV_BOOTSTRAP=1 npm test -- src/activities/__tests__/workflow-runtime-v2-activities.test.ts src/workflows/__tests__/workflow-runtime-v2-run-workflow.test.ts:
    • activities tests pass.
    • workflow tests fail in current env due package resolution of @alga-psa/workflows/lib/workflowRuntimeV2TemporalContract when run in isolation.
• Added manual quota resume integration coverage in server/src/test/integration/workflowRuntimeV2.control.integration.test.ts:
  • Resume succeeds when quota is available and runtime re-entry still calls quota reservation.
  • Resume returns quota_exhausted response with usedCount, effectiveLimit, periodStart, and periodEnd when quota remains exhausted.
• Added unit job coverage in server/src/test/unit/jobs/workflowQuotaResumeScanHandler.unit.test.ts:
  • Finite exhausted tenants are skipped while eligible tenants are resolved/resumed.
  • Repeated scans do not resolve/execute the same wait twice once status has moved to RESOLVED.
• Validation command run:
  • cd server && npm test -- src/test/unit/jobs/workflowQuotaResumeScanHandler.unit.test.ts (pass).
• Added Workflow Control Panel quota usage surfacing:
  • New read-only server action getWorkflowStepQuotaSummaryAction() returns current tenant period, limit, used count, remaining count, tier, and sources from workflowStepQuotaService.resolveQuotaSummary().
  • ee/server/src/components/workflow-designer/WorkflowDesigner.tsx fetches that action in control-panel mode and renders a compact "Workflow actions" summary with consumed/remaining values, finite-limit progress, and reset date.
  • Added a source-level compatibility re-export at packages/core/src/rateLimit/index.ts because Next's dev import map resolves @alga-psa/core/rateLimit to packages/core/src/rateLimit, while the implementation lives under packages/core/src/lib/rateLimit.
• Validation commands run:
  • cd ee/server && npm run typecheck (pass).
  • cd ee/server && NODE_ENV=test npm run test -- src/components/workflow-designer/__tests__/WorkflowDesigner.smoke.test.tsx (pass; running the same command without NODE_ENV=test loads React production test-utils in this workspace and fails before test execution).