PSA/ee/docs/plans/2025-12-27-workflow-trigger-payload-mapping/PRD.md

# PRD — Stable Workflow Payload Across Trigger Schema Changes

- Slug: `workflow-trigger-payload-mapping`
- Date: `2025-12-27`
- Status: Draft

## Summary
Workflows have a `payloadSchemaRef` that should remain stable over time, even if the trigger event’s schema evolves. This plan introduces an explicit “source payload schema” concept for event triggers and an optional trigger-level mapping (event payload → workflow payload). When the source schema ref matches the workflow payload schema ref, “no mapping” is a valid state and execution uses the payload as-is.

## Problem
Today, event-triggered workflows implicitly assume the inbound event payload already matches the workflow’s `payloadSchemaRef`. When the trigger event’s schema changes (or differs between tenants/system), workflows can silently stop running or become confusing to debug. Users also lack a clear UX for: (a) seeing the trigger’s source schema, (b) deciding whether they need a mapping, and (c) understanding whether a workflow is runnable for a given event payload schema.

## Goals
- Preserve the invariant: **workflow payload is stable** (`payloadSchemaRef` is the canonical contract for steps, mapping UI, expression autocomplete, etc.).
- Allow trigger event schemas to change independently of workflows, without breaking stable workflows when a mapping exists.
- Make “no mapping” a valid state when `sourcePayloadSchemaRef === payloadSchemaRef`.
- Block execution (not saving) when required mapping is missing or invalid; surface this via validation status/details.
- Record enough schemaRef/mapping provenance on events/runs for debugging and audit.

## Non-goals
- Automatic migration of existing workflows when event schemas change.
- Schema compatibility inference beyond `schemaRef` equality (i.e., “structurally compatible” schemas without mapping).
- Backward-compatibility layers for legacy event shapes (explicitly not required).
- A generalized “schema transform language” beyond existing mapping primitives (expressions / field refs / literals).

## Users and Primary Flows
### Users
- MSP admin / automation builder
- Internal support / engineers debugging automation

### Primary flows
1. **Build workflow**: choose trigger event → see source schemaRef → choose/confirm workflow payloadSchemaRef → if mismatch, define trigger mapping; if match, leave mapping empty.
2. **Publish workflow**: validator computes runnable status. If mismatch + missing mapping: publish fails; draft saves with invalid status.
3. **Ingest event**: event arrives with eventName + payload (and optionally a source schemaRef). System resolves `sourcePayloadSchemaRef`, then for each matching workflow version:
   - If refs match and no mapping: payload passes through unchanged.
   - If mapping exists: map → validate → start run (even when refs match).
   - If refs differ and mapping missing/invalid: do not start run.
4. **Run workflow (Run dialog)**: choose event type to seed “source schema”, enter source payload, run uses trigger mapping to produce workflow payload.
5. **Debug**: event list/run logs show source schemaRef used and whether mapping was applied.

## UX / UI Notes
- In the workflow designer trigger section:
  - Display the trigger’s **source schema ref** (from event catalog; optionally override-able).
  - Display the workflow’s **payload schema ref** (existing picker).
  - If refs match: show “Identity (no mapping required)” and hide mapping editor by default.
  - If refs differ: show required mapping editor (event → workflow payload) with schema-aware field pickers.
- Trigger mapping expressions evaluate with an `event` root (the full incoming event). The source payload is available at `event.payload` and is typed from the trigger’s source schema ref. The workflow payload root variable (`payload`) continues to refer to the workflow payload inside normal workflow steps.
- Validation messaging should explicitly mention:
  - Missing/unknown source schema ref
  - Mismatch requiring mapping
  - Mapping gaps (missing required fields) and expression errors

## Requirements

### Functional Requirements
- Event ingestion resolves a `sourcePayloadSchemaRef` with precedence:
  1) event submission payloadSchemaRef, 2) event catalog payload_schema_ref, 3) unknown (validation error).
- If event submission provides a schemaRef that differs from the event catalog, **accept the event** and record a **warning** (and telemetry) indicating the conflict; submission schemaRef still wins precedence.
- Workflows can optionally define a trigger-level mapping from source payload → workflow payload.
- “No mapping” is valid when schema refs match; execution uses source payload as workflow payload.
- When schema refs match, trigger mapping remains optional; if provided, it is applied and validated like any other mapping.
- When refs differ, mapping is required for execution and is validated deeply (including nested required fields).
- Missing secrets referenced by trigger mapping are validation **errors**.
- Validation details are persisted and shown in the designer.
- Run dialog can build a source-payload form from the trigger’s source schema ref, then run with mapping applied.
- Trigger mapping expressions must use `event` as the root variable; the incoming payload is accessed via `event.payload` (no `payload` alias in the trigger mapping context to avoid confusion with workflow payload).

### Non-functional Requirements
- Mapping evaluation and schema validation must be deterministic and auditable (store schemaRef + mapping provenance on run/events).
- Avoid runtime surprises: if mapping is required and invalid, the workflow does not start.

## Data / API / Integrations
- Add `payload_schema_ref` to workflow runtime event records (v2) so events retain source schema provenance.
- Extend workflow trigger shape to include optional `sourcePayloadSchemaRef` override and optional `payloadMapping`.
- Expose `payload_schema_ref` in workflow event list APIs so UI can show it.

## Security / Permissions
- Creating/editing trigger mapping requires workflow `manage` permission.
- Viewing schema refs and mapping details requires workflow `read` (consistent with existing behavior).

## Observability
- Audit log for publish/save already exists; extend run/event metadata to include:
  - sourcePayloadSchemaRef used
  - whether mapping was applied
  - catalog-vs-submission schemaRef conflict warning (when present)
  - validation failures preventing start (for debugging)

## Rollout / Migration
- DB migration to add `payload_schema_ref` to workflow runtime events table (v2).
- Backfill is not required; existing events can have null source schemaRef.
- Existing system event catalog entries should include `payload_schema_ref` going forward (inbound email already does).

## Open Questions
- Trigger mapping root variable name: **`event`** (resolved). Steps continue to use `payload` for the workflow payload; trigger mapping uses `event.payload` for the source payload.
- If event submission provides a schemaRef that conflicts with event catalog: **accept and warn** (resolved). Submission wins precedence; record warning/telemetry for observability.
- If schemaRefs match, do we allow an optional mapping anyway? **Yes** (resolved). Mapping remains optional but is validated/applied when provided.

## Acceptance Criteria (Definition of Done)
- When trigger source schemaRef matches workflow payload schemaRef, a workflow with no trigger mapping is valid and runnable.
- When source schemaRef differs, publish fails without trigger mapping; draft can save but is invalid for execution.
- Event-triggered runs apply trigger mapping (when present) and validate the resulting payload against workflow payloadSchemaRef.
- Workflow event list/run views show the source schemaRef used and whether mapping was applied.
- Designer clearly communicates whether mapping is required and guides users to fix invalid states.