PSA/ee/docs/plans/2026-01-24-inbound-email-in-app-processing/SCRATCHPAD.md

Scratchpad — 2026-01-24 Inbound Email In-App Processing

Context / Decision

• We are choosing Option B for now: bypass workflows entirely and call existing email “domain” functions directly in the webhook request flow.
• Goal remains to make workflow-worker v2-only and to remove inbound-email dependency on legacy workflow runtimes.

Current wiring notes (file pointers)

• Gmail webhook route: server/src/app/api/email/webhooks/google/route.ts → packages/integrations/src/webhooks/email/google.ts
• Microsoft webhook route: server/src/app/api/email/webhooks/microsoft/route.ts → packages/integrations/src/webhooks/email/microsoft.ts
• V2 email workflow definition exists (not used in Option B): shared/workflow/runtime/workflows/email-processing-workflow.v2.json
• Legacy email workflow exists (not used in Option B): shared/workflow/workflows/system-email-processing-workflow.ts
• Domain helpers we will call:
  • shared/workflow/actions/emailWorkflowActions.ts
  • HTML conversion utility: @alga-psa/shared/lib/utils/contentConversion (see shared/workflow/init/registerWorkflowActions.ts for how it’s used)

Commands I used during investigation

• rg -n "INBOUND_EMAIL_RECEIVED|webhooks/email|email-processing-workflow.v2.json" -S server packages shared services
• sed -n '1,220p' shared/workflow/actions/emailWorkflowActions.ts

Open questions to resolve before implementation

• What do we do for unmatched senders (no contact match)?
• What is the canonical idempotency key per provider (Gmail historyId vs messageId)?
• Do we want comment bodies stored as BlockNote JSON (preferred), or raw text/html?

Progress log

• 2026-01-24: Implemented F001 by defining the normalized processInboundEmailInApp contract/types in shared/services/email/processInboundEmailInApp.ts (input + union result shape).
• 2026-01-24: Implemented F002 by adding processInboundEmailInApp() in shared/services/email/processInboundEmailInApp.ts with in-app routing (existing ticket reply vs new-ticket creation).
• 2026-01-24: Implemented F003 by invoking parseEmailReplyBody() and falling back to raw body content on parser failures (logs + continue).
• 2026-01-24: Implemented F004 by resolving existing tickets via reply token first (if present), then falling back to thread headers via findTicketByEmailThread.
• 2026-01-24: Implemented F005 by creating reply comments with BlockNote JSON content (HTML→blocks when available, text→blocks otherwise) and storing message threading metadata on the comment.
• 2026-01-24: Implemented F006 by processing reply attachments per item via processEmailAttachment with error-continue semantics.
• 2026-01-24: Implemented F010 by requiring inbound ticket defaults per provider via resolveInboundTicketDefaults(tenantId, providerId) and skipping processing when missing.
• 2026-01-24: Implemented F011 by matching sender email to an existing contact via findContactByEmail and using its client_id/contact_id on ticket creation.
• 2026-01-24: Implemented F012 decision: if sender email does not match an existing contact, create the ticket under provider defaults client_id with contact_id unset, and mark the initial comment metadata with unmatchedSender: true for manual triage.
• 2026-01-24: Implemented F013 by creating tickets via createTicketFromEmail and persisting tickets.email_metadata (messageId/threadId/inReplyTo/references/providerId) for future threading queries.
• 2026-01-24: Implemented F014 by creating the initial ticket comment via createCommentFromEmail with BlockNote JSON content derived from the inbound email body.
• 2026-01-24: Implemented F015 by processing new-ticket attachments per item via processEmailAttachment with error-continue semantics.
• 2026-01-24: Implemented F020 by de-duping reply comment creation via a DB lookup keyed on {tenant, ticket_id, metadata.email.messageId} before inserting.
• 2026-01-24: Implemented F021 by de-duping new-ticket creation via a DB lookup on tickets.email_metadata.messageId+providerId before creating a ticket.
• 2026-01-24: Implemented F022 by re-enabling Gmail Pub/Sub historyId de-dupe (skip when gmail_processed_history already contains the notification historyId). Microsoft continues using email_processed_messages PK de-dupe.
• 2026-01-24: Implemented F030 by wiring Gmail webhook message handling to call processInboundEmailInApp when the in-app flag is enabled (otherwise preserves legacy INBOUND_EMAIL_RECEIVED publish).
• 2026-01-24: Implemented F031 by wiring Microsoft webhook notifications to call processInboundEmailInApp when enabled and to persist ticket linkage back to email_processed_messages.
• 2026-01-24: Implemented F040 by adding env-driven, tenant/provider-scoped gating in shared/services/email/inboundEmailInAppFeatureFlag.ts (INBOUND_EMAIL_IN_APP_PROCESSING_ENABLED, INBOUND_EMAIL_IN_APP_TENANT_IDS, INBOUND_EMAIL_IN_APP_PROVIDER_IDS).
• 2026-01-24: Implemented F041 by supporting provider-level rollout via INBOUND_EMAIL_IN_APP_PROVIDER_IDS allowlist.
• 2026-01-24: Implemented F050 by defaulting workflow-worker deployments to WORKFLOW_WORKER_MODE=v2 in compose configs (docker-compose.ce.yaml, docker-compose.ee.yaml, and prebuilt variants).
• 2026-01-24: Implemented F051 by disabling legacy system email workflow updates by default in services/workflow-worker/src/init/updateWorkflows.ts (requires LEGACY_SYSTEM_EMAIL_WORKFLOW_ENABLED=true to opt back in).
• 2026-01-24: Implemented T001 with an integration test covering Gmail webhook → in-app processing path and asserting exactly one ticket and one initial comment (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T002 with an integration test covering Microsoft webhook → in-app processing path and asserting exactly one ticket and one initial comment (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T003 by adding an integration test for reply-token threading to ensure the in-app processor creates exactly one comment on the matched ticket (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T004 by adding an integration test for header-based threading (In-Reply-To/References) to ensure the in-app processor creates exactly one comment on the matched ticket (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T010 with a unit test ensuring parseEmailReplyBody produces non-empty sanitized text for plain-text emails (server/src/test/unit/email/inboundEmailBodyParsing.test.ts).
• 2026-01-24: Implemented T011 with a unit test ensuring parseEmailReplyBody produces sanitized HTML when HTML input is present (server/src/test/unit/email/inboundEmailBodyParsing.test.ts).
• 2026-01-24: Implemented T020 with an integration test verifying sender→contact matching overrides default client/contact on ticket creation (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T021 with an integration test verifying unmatched senders create a ticket under provider defaults without a contact and without throwing (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T022 with an integration test verifying missing inbound ticket defaults causes processing to skip without creating tickets/comments (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T030 with an integration test verifying attachment processing errors do not prevent ticket + initial comment creation (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T031 with an integration test verifying attachment processing errors do not prevent reply comment creation (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T040 with an integration test verifying reply idempotency (replaying the same email id does not create duplicate comments) (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T041 with an integration test verifying new-email idempotency (replaying the same email id does not create duplicate tickets/comments) (server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts).
• 2026-01-24: Implemented T042 with an integration test ensuring the env-based feature flag switches Gmail webhook behavior between legacy publish and in-app processing (server/src/test/integration/inboundEmailInApp.featureFlag.integration.test.ts).
• 2026-01-24: Implemented T050 with a smoke integration test asserting compose defaults WORKFLOW_WORKER_MODE=v2 and that processInboundEmailInApp successfully creates a ticket/comment without any worker running (server/src/test/integration/workflowWorkerV2.inboundEmailSmoke.integration.test.ts).
• 2026-01-24: Test env note: the three DB-backed integration tests in this plan now auto-skip when the configured Postgres host/port is unreachable (to avoid hard failures in dev setups without a local test DB).