PSA/ee/docs/plans/2026-03-04-invoice-workflow-expression-unification/SCRATCHPAD.md

SCRATCHPAD — Invoice Workflow Expression Binding Unification

• Plan slug: invoice-workflow-expression-unification
• Created: 2026-03-04

What This Is

Working notes for unifying invoice designer bindings and Workflow v2 expression authoring on a shared expression foundation while preserving each runtime/storage contract.

Decisions

• (2026-03-04) Use a shared abstraction centered on typed context paths + expression modes (path-only, template, expression) instead of forcing one persisted expression format across products.
• (2026-03-04) Shared primitives module location: shared/workflow/expression-authoring/* so both invoice (packages/billing) and workflow designer (ee/server) can import through @shared/workflow/....
• (2026-03-04) Shared context contract includes explicit SharedExpressionContextRoot and SharedExpressionPathOption interfaces plus deterministic root serialization to stabilize cross-editor behavior/tests.
• (2026-03-04) Shared path discovery contract uses deterministic lexical traversal with explicit array item marker segments ([]) and additional-property wildcard (*) to produce stable picker options.
• (2026-03-04) Shared insertion helper is split into pure value insertion (insertTextIntoValue) plus environment-specific adapters (insertTextIntoDomControl, insertTextIntoMonacoEditor) with explicit no-op reasons.
• (2026-03-04) Shared validation contract (SharedExpressionValidationResult) normalizes severity ordering and preserves path/root/range attribution for downstream UI rendering.
• (2026-03-04) Invoice adapter roots stay domain-native (invoice, customer, tenant, item) for this migration to preserve existing author mental model and template bindings.
• (2026-03-04) Workflow adapter includes dynamic loop roots (itemVar, indexVar) so forEach-scoped expressions can use the same shared path options as global roots.
• (2026-03-04) Invoice palette fields now derive from buildInvoiceExpressionPathOptions(...); static catalog import path is no longer used in rendering/search.
• (2026-03-04) Invoice FIELDS panel grouping now maps directly to shared root keys (invoice/customer/tenant/item) and only renders leaf path options from the shared model.
• (2026-03-04) DesignerShell now uses insertTextIntoDomControl and target-path mode inference: inspector fields ending in bindingKey receive raw path insertion (path-only).
• (2026-03-04) Invoice text fallback insertion path also uses shared insertion primitives (insertTextIntoValue) and enforces template token formatting for text/label nodes.
• (2026-03-04) Workspace AST export now parses moustache text edits into dynamic expressions: full-token moustache becomes path, mixed literal+moustache becomes template with deterministic arg keys.
• (2026-03-04) Existing roundtrip suites (workspaceAst.roundtrip.*) remain green after adapter/insertion changes, confirming deterministic export/import behavior.
• (2026-03-04) Shared path validator (validateSourcePaths) now powers invoice insertion feedback; unknown paths emit informational diagnostics without breaking insertion flow.
• (2026-03-04) Workflow step config validation now derives shared path options from DataContext via buildWorkflowExpressionPathOptions(...) and uses validateSourcePaths(...) in place of legacy ${...} scanners.
• (2026-03-04) Workflow step validation UI now consumes shared diagnostic objects directly (severity, code, message, path) and renders grouped Error/Warning/Info cards from the shared contract.
• (2026-03-04) Runtime function allowlist is now centralized in shared/workflow/runtime/expressionFunctions.ts and consumed by both runtime validation and Monaco completion generation.
• (2026-03-04) Workflow function completion/snippet suggestions are now constrained to runtime-allowlisted helpers ($nowIso, $coalesce, $len, $toString, $append).
• (2026-03-04) Workflow diagnostics now validate $function(...) tokens against runtime-allowlisted helpers only; non-allowlisted JSONata built-ins are flagged as unknown in-editor.
• (2026-03-04) Workflow persisted expression shape remains { $expr: string }; migration changes were limited to authoring/validation paths and retained ensureExpr + runtime schema guards.
• (2026-03-04) Added focused runtime guardrail tests (expressionEngine.test.ts) to lock function allowlist rejection, timeout enforcement, JSON-serializable result checks, and max output size checks.
• (2026-03-04) Shared schema/context-aware path checks (validateSourcePaths) are now used in both designers where author-time validation exists:
  • invoice insertion feedback (DesignerShell)
  • workflow step config validation (WorkflowDesigner)
• (2026-03-04) Workflow Monaco insertion paths now route through shared insertTextIntoMonacoEditor for both programmatic insert and drag/drop insert, with existing $0 placeholder stripping preserved.
• (2026-03-04) Invoice input/textarea insertion remains shared-helper based (insertTextIntoDomControl + insertTextIntoValue) for selection-aware replacement and repeated insert behavior.
• (2026-03-04) Invoice preview interpolation semantics remain intact post-migration (resolved known tokens + visibly marked unresolved tokens), validated by preview-mode suite.
• (2026-03-04) Workflow runtime expression suites (expressionEngine, mappingResolver, mappingValidator) remain green after migration; contracts for { $expr } evaluation/validation are preserved.
• (2026-03-04) Deprecated invoice static variable catalog path is removed; field discovery now only flows through shared adapter-driven options.
• (2026-03-04) Deprecated workflow ${...} scanner helpers are removed from step validation flow; only shared-path validation remains.
• (2026-03-04) Test implementation standards are codified in PRD.md and operationalized as explicit meta-test backlog items (T041, T043, T044) for checklist enforcement.
• (2026-03-04) Added shared expression-authoring foundational unit suites:
  • shared/workflow/expression-authoring/__tests__/coreContracts.test.ts
  • shared/workflow/expression-authoring/__tests__/insertionDom.test.ts These cover T001-T012 contract behaviors.
• (2026-03-04) T002 verified by coreContracts.test.ts roundtrip serialization/deserialization assertions for shared context roots.
• (2026-03-04) T003 verified by deterministic path ordering assertions in coreContracts.test.ts (buildPathOptionsFromContextRoots nested schema flattening).
• (2026-03-04) T004 verified by array marker segment assertions in coreContracts.test.ts (payload.items[] and payload.items[].id).
• (2026-03-04) T005 verified in insertionDom.test.ts with start/middle/end insertion coverage for insertTextIntoValue.
• (2026-03-04) T006 verified in insertionDom.test.ts by replacing an active range ({{invoice.total}} -> {{customer.name}}).
• (2026-03-04) T007 verified in insertionDom.test.ts with explicit readonly/disabled/unfocused no-op assertions for insertTextIntoDomControl.
• (2026-03-04) T008 verified via coreContracts.test.ts validation-result normalization assertions (severity fallback + deterministic diagnostic ordering).
• (2026-03-04) T009 verified via coreContracts.test.ts invoice adapter assertions (roots + canonical paths for invoice/customer/tenant/item).
• (2026-03-04) T010 verified via coreContracts.test.ts stable label assertions for known invoice adapter roots/paths.
• (2026-03-04) T011 verified via coreContracts.test.ts workflow adapter root emission assertions (payload/vars/meta/error + forEach vars).
• (2026-03-04) T012 verified via coreContracts.test.ts recursive vars-path assertions (vars.previous.account.id).
• (2026-03-04) T013 validated with ComponentPalette.fields.integration.test.tsx by exercising FIELDS tab insertion path callback using adapter-derived field options.
• (2026-03-04) T014 validated with ComponentPalette.fields.integration.test.tsx search assertions (query filters adapter-derived labels/paths).
• (2026-03-04) Preserve persisted contracts:
  • Invoice keeps AST value expressions (literal|binding|path|template).
  • Workflow keeps { $expr: string }.
• (2026-03-04) Migrate incrementally via adapters; no big-bang rewrite.
• (2026-03-04) Treat runtime allowlist as source of truth for workflow function capability; editor metadata must conform to it.
• (2026-03-04) Keep plan scope focused on logic/functionality and correctness; exclude metrics/observability/production-readiness additions.
• (2026-03-04) Replace doc-only feature/test entries with explicit test implementation standards and enforceable meta-tests (framework, selector strategy, anti-tautology, feature-test traceability).

Discoveries / Constraints

• (2026-03-04) This branch already had uncommitted invoice-designer WIP focused on static templateVariableCatalog and local insertion logic; foundational shared abstractions were still missing.
• (2026-03-04) Invoice currently uses multiple authoring paths:
  • metadata.bindingKey (path string)
  • moustache text in metadata.text
  • AST conversion in workspaceAst.ts.
• (2026-03-04) Invoice preview now resolves moustache interpolation in canvas preview path but this is distinct from export compilation.
• (2026-03-04) Workflow currently has duplicated expression validation paths:
  • Monaco diagnostics provider
  • Legacy ${...} extraction/validation in WorkflowDesigner.tsx.
• (2026-03-04) WorkflowDesigner.tsx no longer defines extractExpressionPaths / validateExpressionPath; step validation now emits issues from shared validation diagnostics.
• (2026-03-04) Editor metadata previously exposed broad JSONata helpers; completion hints were not runtime-aligned until runtime allowlist filtering was introduced.
• (2026-03-04) Workflow editor function metadata includes many JSONata functions, while runtime currently allowlists only a small helper set in shared/workflow/runtime/expressionEngine.ts.
• (2026-03-04) Removing drift between workflow editor and runtime is required to avoid misleading UX.

Commands / Runbooks

• Implement F001 shared mode contract:
  • shared/workflow/expression-authoring/modes.ts (EXPRESSION_MODES, ExpressionMode, isExpressionMode)
• Implement F002 context and path option contracts:
  • shared/workflow/expression-authoring/context.ts
• Implement F003 path discovery helper:
  • shared/workflow/expression-authoring/pathDiscovery.ts
• Implement F004 insertion helper API:
  • shared/workflow/expression-authoring/insertion.ts
• Implement F005 validation result contract:
  • shared/workflow/expression-authoring/validation.ts
• Implement F006 invoice adapter:
  • shared/workflow/expression-authoring/adapters/invoiceContextAdapter.ts
• Implement F007 workflow adapter:
  • shared/workflow/expression-authoring/adapters/workflowContextAdapter.ts
• Validate invoice field palette integration after adapter wiring:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/palette/ComponentPalette.fields.integration.test.tsx
• Validate invoice shell + fields insertion wiring:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/palette/ComponentPalette.fields.integration.test.tsx ../packages/billing/src/components/invoice-designer/DesignerShell.selectedContext.integration.test.tsx
• Validate moustache-to-AST behavior:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/ast/workspaceAst.test.ts
• Validate roundtrip determinism:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/ast/workspaceAst.roundtrip.templates.test.ts ../packages/billing/src/components/invoice-designer/ast/workspaceAst.roundtrip.nodes.test.ts ../packages/billing/src/components/invoice-designer/ast/workspaceAst.roundtrip.styles.test.ts
• Validate invoice insertion + validation integration:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/ast/workspaceAst.test.ts ../packages/billing/src/components/invoice-designer/palette/ComponentPalette.fields.integration.test.tsx ../packages/billing/src/components/invoice-designer/DesignerShell.selectedContext.integration.test.tsx
• Validate workflow editor test baseline after validation migration:
  • cd ee/server && npx vitest run src/components/workflow-designer/expression-editor/__tests__/completionProvider.test.ts src/components/workflow-designer/expression-editor/__tests__/diagnosticsProvider.test.ts
• Validate workflow designer type safety after shared validation UI refactor:
  • cd ee/server && npm run -s typecheck
• Validate allowlisted function completions and diagnostics baseline:
  • cd ee/server && npx vitest run src/components/workflow-designer/expression-editor/__tests__/completionProvider.test.ts src/components/workflow-designer/expression-editor/__tests__/diagnosticsProvider.test.ts
• Validate diagnostics allowlist behavior and type safety:
  • cd ee/server && npx vitest run src/components/workflow-designer/expression-editor/__tests__/diagnosticsProvider.test.ts
  • cd ee/server && npm run -s typecheck
• Audit persisted expression contract usage:
  • rg -n '\\$expr' ee/server/src/components/workflow-designer/WorkflowDesigner.tsx shared/workflow/runtime -g'*.ts*'
• Validate runtime expression guardrails:
  • npx vitest run --config shared/vitest.config.ts shared/workflow/runtime/__tests__/expressionEngine.test.ts shared/workflow/runtime/__tests__/mappingResolver.test.ts
• Audit shared path-validator usage in both designers:
  • rg -n "validateSourcePaths\\(" packages/billing/src/components/invoice-designer ee/server/src/components/workflow-designer -g'*.ts*'
• Validate workflow expression editor insertion refactor:
  • cd ee/server && npx vitest run src/components/workflow-designer/expression-editor/__tests__/completionProvider.test.ts src/components/workflow-designer/expression-editor/__tests__/diagnosticsProvider.test.ts && npm run -s typecheck
• Validate invoice preview interpolation behavior:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/canvas/DesignCanvas.previewMode.test.tsx
• Validate workflow runtime expression behavior:
  • npx vitest run --config shared/vitest.config.ts shared/workflow/runtime/__tests__/expressionEngine.test.ts shared/workflow/runtime/__tests__/mappingResolver.test.ts shared/workflow/runtime/__tests__/mappingValidator.test.ts
• Audit for deprecated invoice static catalog references:
  • rg -n "templateVariableCatalog|TEMPLATE_VARIABLE_OPTIONS" packages/billing/src/components/invoice-designer -g'*.ts*'
• Audit for deprecated workflow scanner helpers:
  • rg -n "extractExpressionPaths|validateExpressionPath|\\$\\{\\[^\\}\\]\\+\\}" ee/server/src/components/workflow-designer -g'*.ts*'
• Run shared expression-authoring foundational tests:
  • npx vitest run --config shared/vitest.config.ts shared/workflow/expression-authoring/__tests__/coreContracts.test.ts shared/workflow/expression-authoring/__tests__/insertionDom.test.ts
• Compare invoice/workflow binding/expression surfaces:
  • rg -n "binding|template|\{\{|\$expr|validateExpressionSource" packages/billing/src/components/invoice-designer ee/server/src/components/workflow-designer shared/workflow/runtime -g"*.ts*"
• Review workflow designer expression sections:
  • sed -n '820,980p' ee/server/src/components/workflow-designer/WorkflowDesigner.tsx
  • sed -n '6060,6665p' ee/server/src/components/workflow-designer/WorkflowDesigner.tsx
• Review workflow runtime constraints:
  • sed -n '1,220p' shared/workflow/runtime/expressionEngine.ts
• Review invoice AST compile/render paths:
  • sed -n '104,340p' packages/billing/src/components/invoice-designer/ast/workspaceAst.ts
  • sed -n '210,290p' packages/billing/src/lib/invoice-template-ast/react-renderer.tsx

Links / References

• Workflow designer:
  • ee/server/src/components/workflow-designer/WorkflowDesigner.tsx
  • ee/server/src/components/workflow-designer/expression-editor/
• Workflow runtime:
  • shared/workflow/runtime/expressionEngine.ts
  • shared/workflow/runtime/types.ts
• Invoice designer:
  • packages/billing/src/components/invoice-designer/DesignerShell.tsx
  • packages/billing/src/components/invoice-designer/palette/ComponentPalette.tsx
  • packages/billing/src/components/invoice-designer/ast/workspaceAst.ts
• Invoice AST renderer:
  • packages/billing/src/lib/invoice-template-ast/react-renderer.tsx

Open Questions

• Should invoice roots stay domain-native (invoice/customer/tenant/item) or converge on workflow-style roots for long-term consistency?
• Should workflow function metadata be hard-pruned immediately to runtime allowlist, or staged with compatibility warnings first?
• Should shared validation include a true parser abstraction now, or remain an interface with domain-specific parser implementations?
• Is invoice drag/drop path insertion in scope for this plan, or deferred after click-insert parity?
• (2026-03-04) T015 validated by DesignerShell.insertion.integration.test.tsx: bindingKey-target insertions write raw invoice.total path tokens without moustache wrapping.
• (2026-03-04) T016 validated by DesignerShell.insertion.integration.test.tsx: non-binding text targets insert moustache-wrapped tokens in template mode (Before {{invoice.total}}).
• (2026-03-04) T017 validated by DesignerShell.insertion.integration.test.tsx: when no insert target is focused, insertion falls back to appending template token into selected text-node metadata.text.
• (2026-03-04) T018 validated by workspaceAst.test.ts (compiles edited moustache text into a dynamic path expression): {{invoice.total}} compiles to AST content: { type: 'path', path: 'total' }.
• (2026-03-04) T019 validated by workspaceAst.test.ts (compiles mixed literal + moustache text into a template expression): mixed text exports as AST template with args including invoiceNumber and dueDate paths.
• (2026-03-04) T020 validated by workspaceAst.test.ts (preserves expression and format semantics when importing existing AST templates): import/export with unchanged imported content preserves original expression kinds/bindings (non-destructive text preview path).
• (2026-03-04) T021 validated by workspaceAst.test.ts (roundtrips exported AST deterministically (export -> import -> export)): repeated export/import cycles are byte-stable for AST shape.
• (2026-03-04) Added DesignerShell.insertion.integration.test.tsx coverage for T022: inserting invoice.missingField surfaces shared-validator feedback (Unknown path ...) and subsequent replacement with invoice.total succeeds without UI crash.
• (2026-03-04) Introduced workflow-designer/expressionValidation.ts and test coverage proving WorkflowDesigner.tsx no longer references legacy extractExpressionPaths/validateExpressionPath; step config validation uses shared validator wiring.
• (2026-03-04) T024 covered in expressionValidation.test.ts: partitionStepExpressionValidations consumes shared diagnostics contract (severity/message/code) and splits panel-facing error/warning/info groups deterministically.
• (2026-03-04) T025 covered in expressionValidation.test.ts: validateStepExpressions(...) now flags unknown.root and vars.missing.id via shared validator diagnostics while accepting known vars.previous.id.
• (2026-03-04) T026 validated by expression-editor/__tests__/completionProvider.test.ts: function completion items are constrained to runtime allowlisted helpers only.
• (2026-03-04) T027 validated by expression-editor/__tests__/diagnosticsProvider.test.ts: disallowed/non-allowlisted functions are diagnosed and runtime-allowlisted helpers pass.
• (2026-03-04) Added shared/workflow/runtime/__tests__/types.exprPersistence.test.ts for T028: step schemas persist expressions as { $expr: string } objects (and reject bare string Expr inputs).
• (2026-03-04) T029 validated by shared/workflow/runtime/__tests__/expressionEngine.test.ts: runtime validation rejects disallowed functions (e.g. $sum(...)).
• (2026-03-04) T030 validated by expressionEngine.test.ts: evaluation timeout and max output-size guardrails remain enforced post-migration.
• (2026-03-04) T031 added to coreContracts.test.ts: shared validateSourcePaths(...) returns info/unknown-path diagnostic for unresolved schema-aware path references (payload.user.missing).
• (2026-03-04) T032 covered by shared/workflow/expression-authoring/__tests__/insertionDom.test.ts: Monaco-style insertion now verifies shared helper replacement semantics and cursor/selection placement at the inserted text end.
• (2026-03-04) T033 validated via expression-editor/__tests__/insertionText.test.ts: trailing snippet placeholder ($0) is stripped before insertion, while non-trailing text remains unchanged.
• (2026-03-04) T034 added to DesignerShell.insertion.integration.test.tsx: repeated template insertions with manual typing in between remain cursor-accurate and produce expected final text.
• (2026-03-04) T035 validated by DesignCanvas.previewMode.test.tsx: preview rendering resolves known moustache tokens and leaves unresolved tokens visibly marked.
• (2026-03-04) T036 validated by DesignCanvas.previewMode.test.tsx regression run (17 passing tests), covering field/table/totals preview scaffolds.
• (2026-03-04) T037 validated by server/src/test/unit/workflowRuntimeV2.unit.test.ts (33 passing tests): workflow runtime v2 expression compile/evaluate/resolve unit suite remains green.
• (2026-03-04) T038 validated by server/src/test/integration/workflowRuntimeV2.control.integration.test.ts + workflowRuntimeV2.eventTrigger.integration.test.ts (86 passing tests total). Required fixes:
  • improved @alga-psa/auth test mock in server/src/test/setup.ts to provide tenant-aware withAuth/withOptionalAuth context (using mocked db tenant where available),
  • added @alga-psa/core alias coverage in server/vitest.config.ts to resolve document dependency imports in server Vitest runs,
  • updated integration test tenant mocks to use mockImplementation(() => tenantId) for mutable tenant scenarios,
  • updated event-trigger tests to use trigger sourcePayloadSchemaRef and schema-aware invalid-payload expectations aligned with current runtime contract.
• (2026-03-04) T039 validated by new repo-guard test templateVariableCatalog.removal.test.ts: static catalog file constants/templateVariableCatalog.ts is absent and no source files under invoice-designer reference legacy constants/templateVariableCatalog / TEMPLATE_VARIABLE_OPTIONS.
• (2026-03-04) T040 validated by ee/server/src/components/workflow-designer/__tests__/expressionValidation.test.ts (3 passing tests), including explicit source assertions that WorkflowDesigner.tsx no longer references legacy extractExpressionPaths or validateExpressionPath helpers.
• (2026-03-04) T041 validated by new meta-test testImplementationStandards.meta.test.ts: scoped React UI tests are enforced to import Vitest + Testing Library and, unless explicitly marked render-semantics-only, include role/automation-based selector signals.
• (2026-03-04) T042 validated in one local session by running invoice + workflow smoke insertion tests:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/DesignerShell.insertion.integration.test.tsx (5 passing tests),
  • cd ee/server && npx vitest run src/components/workflow-designer/expression-editor/__tests__/ExpressionEditor.smoke.test.tsx (1 passing test). This proves both designers render and accept at least one inserted binding path without runtime errors.
• (2026-03-04) T043 validated by new meta-guard packages/billing/src/components/invoice-designer/nonTautologyAssertions.meta.test.ts:
  • requires semantic assertion signals across scoped tests for AST behavior (workspaceAst.test.ts path/template checks),
  • model/feedback behavior (DesignerShell.insertion.integration.test.tsx store mutation, diagnostics, cursor assertions),
  • shared workflow diagnostics behavior (ee/server/.../expressionValidation.test.ts path attribution + severity partitioning). Verification run:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/nonTautologyAssertions.meta.test.ts ../packages/billing/src/components/invoice-designer/DesignerShell.insertion.integration.test.tsx ../packages/billing/src/components/invoice-designer/ast/workspaceAst.test.ts
  • cd ee/server && npx vitest run src/components/workflow-designer/__tests__/expressionValidation.test.ts
• (2026-03-04) T044 validated by new meta-test packages/billing/src/components/invoice-designer/featureTraceability.meta.test.ts:
  • loads plan features.json + tests.json,
  • asserts every Fxxx in features has at least one reference in tests featureIds,
  • asserts no tests reference unknown feature IDs. Verification run:
  • cd server && npx vitest run ../packages/billing/src/components/invoice-designer/featureTraceability.meta.test.ts