PSA/ee/docs/plans/2025-12-28-workflow-payload-contract-inference/PRD.md

PRD — Draft-Time Payload Inference + Publish-Time Payload Contract

• Slug: workflow-payload-contract-inference
• Date: 2025-12-28
• Status: Draft
• Depends on:
  • ee/docs/plans/2025-12-21-workflow-overhaul.md
  • ee/docs/plans/2025-12-27-workflow-trigger-payload-mapping/PRD.md

Summary

Business owners should be able to start building an automation by selecting an event trigger and adding steps, without having to fully specify the workflow payload schema up front. This plan makes the workflow payload schema a publish-time contract while enabling a rich, typed design-time experience via an effective schema computed from the trigger + step outputs.

Published workflows still have a stable payloadSchemaRef contract that remains stable across trigger schema changes.

Problem

Today, the workflow designer expects a workflow payloadSchemaRef to be specified early (and users can be asked to type schema ref strings). In practice, business owners typically do not know the final “shape” of the workflow payload before they build the steps. This creates friction and leads to incorrect / arbitrary schema choices that harm validation and mapping UX.

Goals

• Keep the invariant: Published workflow payload is stable via payloadSchemaRef.
• Allow creating and editing draft workflows without explicitly selecting a workflow payload schema ref.
• Maintain a strong design-time experience (field picking, autocomplete, mapping, validation) using an effective payload schema that is inferred from:
  • trigger event payload schema (event.payload)
  • step outputs (vars.*) derived from registries
• Preserve and extend the trigger payload mapping policy:
  • If trigger source schemaRef matches workflow payload schemaRef, “no mapping” is valid.
  • Conflicts accept + warn; missing secrets are errors; missing schemas for system events are errors.
• Generate a publish-time payload contract schema automatically (or allow pinning for advanced users).

Non-goals

• Perfect whole-program type inference across arbitrary expressions (we’ll prefer “known where possible; unknown where not”).
• Auto-migrating existing workflows to new schemas (explicit mapping is still the mechanism).
• Supporting “structural compatibility” between schema refs without explicit mapping.

Users and Primary Flows

Users

• Business owner / automation builder
• MSP admin
• Support/engineering (debugging and governance)

Primary flows

1. Create workflow from event
   • Choose event trigger (must have known schema for system events).
   • Start adding steps immediately.
   • Designer shows available fields from event.payload and from vars.* as steps are added.
2. Publish workflow
   • System locks the payload contract schema ref (pinned or inferred from the trigger event).
   • Validator enforces execution requirements (mapping, secrets, schema presence).
3. Run workflow (test run)
   • Run dialog uses trigger schema to build event payload input.
   • Trigger mapping (if present) produces workflow payload; run executes using the published payload contract.
4. Schema evolution
   • Trigger schema changes do not change published payload contract.
   • Workflows remain runnable when required mappings exist and validate.

UX / UI Notes

Workflow Designer — Trigger

• Trigger event is selected from the event catalog (not typed).
• Under the trigger selector, show:
  • event schema ref (required; status badge)
  • “View schema” link/modal
• If event schema is missing/unknown:
  • show error banner and disable publish/run (system events: cannot select; tenant events: selectable but cannot publish/run).

Workflow Designer — Workflow Payload Contract

• Replace the current “Payload schema” requirement with a contract section:
  • Default: “Contract is inferred from the selected trigger event.”
  • Advanced: “Pin payload schema ref” (optional; power users)
• Schema preview / modal shows:
  • “Effective schema (design time)” when inferred
  • “Pinned contract schema” when pinned
  • “Published contract schema” when viewing published versions

Field Picker / Mapping / Expressions

• The “available data” palette is driven by an effective data context:
  • event.payload typed from trigger source schema
  • vars.<saveAs> typed from registry output schemas
  • (Optionally) payload typed from effective payload schema (inferred or pinned)
• If something is unknown, label it as “Unknown type” and degrade type-compat validation to warnings for that segment.

Requirements

Design-Time Behavior (Drafts)

• Draft workflows can be created/saved with no explicit payloadSchemaRef.
• The designer computes an effective payload schema used for:
  • mapping UI field options
  • expression autocomplete contexts
  • type compatibility checks (where type info exists)
• Effective payload schema sources:
  1. pinned payload schema ref (advanced)
  2. inferred payload schema derived from the trigger event’s payload schema ref
• If the trigger is an event, trigger source schema inference must come from:
  • event submission schemaRef (if provided) OR
  • event catalog schemaRef (required for system events; required for tenant events under current policy)
• If the workflow has no event trigger (manual / future trigger types), inference cannot determine a payload contract.
  • Draft save is allowed, but publish/run must require a pinned payload schema ref.

Publish-Time Behavior (Contract)

• When publishing, a workflow must end up with a stable payloadSchemaRef:
  • if pinned: use pinned ref
  • if not pinned: infer from the selected trigger event’s schemaRef at publish time
• Publishing must validate:
  • trigger schema presence policy (system events cannot publish without known schema)
  • mapping requirements (trigger→payload mapping required when schema refs differ)
  • missing secrets referenced by mappings/expressions are errors
  • deep nested required fields for mappings
• After publish, the workflow payload contract is stable and used for runs.
  • The published version stores a payload schema JSON snapshot (payload_schema_json) for audit/debugging; runtime execution does not depend on the schema registry.

Type Compatibility

• Validator should treat type mismatches as:
  • errors when both source and target types are known and incompatible
  • warnings when types are unknown (e.g., unknown step output schema)

Data Model / API / Integrations

• Draft-level payload schema fields:
  • Add an explicit payloadSchemaMode (or equivalent) to distinguish:
    • inferred (default)
    • pinned (advanced)
• Published versions must store:
  • payload_schema_ref (contract)
  • provenance: pinned vs inferred
  • deterministic payload schema snapshot (payload_schema_json) and hash (for drift detection / auditing)

Admin / Operator Notes

• There is no dynamic schema generation/registration in the schema registry for workflow contracts.
• Published workflow versions store the payload contract schema snapshot on workflow_definition_versions.payload_schema_json.
• Runtime execution should not depend on the schema registry being available.

Security / Permissions

• Only users with workflow:manage can pin payload schema refs and edit mappings/steps.
• Only users with workflow:publish can publish contracts.
• Tenant context is inferred from the authenticated session; tenant must not be exposed in simulation/test event builders.

Observability

• Emit telemetry/audit for:
  • publish payload schema ref + mode/provenance (and version)
  • publish blocked due to missing trigger schema/mapping/secrets
  • trigger schema conflicts (submission vs catalog)
  • inferred vs pinned mode

Rollout / Migration

• DB migrations:
  • allow draft workflows to have null/empty payload schema ref where needed
  • add any new metadata fields required for payloadSchemaMode and contract provenance
• Existing published workflows remain unchanged; existing drafts may remain with explicit payloadSchemaRef.

Open Questions

Resolved:

1. Pinning payload schema ref is available to all users with workflow:manage (no extra governance restriction).
2. Start with payload contract schemas that only include the payload contract (no event/vars embedding).
3. Show inferred/effective schema as JSON preview (no requirement for a virtual schemaRef).
4. Custom events policy is intentionally left flexible for now (we may remove custom events); avoid extra work tied to custom-event draft states.

Acceptance Criteria (Definition of Done)

• Users can create a workflow from an event and add steps without selecting a workflow payload schema ref.
• Designer provides a usable field picker/autocomplete experience based on trigger schema + step outputs.
• Publishing a workflow without a pinned schema infers and stores a stable payload contract schema ref from the trigger event.
• Published workflows remain stable across trigger schema changes; execution uses trigger mapping when needed.
• Missing trigger schema (system events), missing required mapping, and missing secrets block publish/run with clear validation errors.