284313f908
Some checks are pending
Bidi Control Character Guard / bidi-control-guard (push) Waiting to run
Circular Dependency Check / Check for new circular dependencies (push) Waiting to run
Citus Migration Smoke / Combined migrations on single-node Citus (push) Waiting to run
E2E Fresh Install Tests / fresh-install-e2e (push) Waiting to run
ext-v2 guardrails / Run ext-v2 guard and ESLint (push) Waiting to run
Integration Tests / Check for relevant changes (push) Waiting to run
Integration Tests / ${{ (github.event_name == 'schedule' || github.event.inputs.suite == 'full') && 'Full integration suite' || 'Tier-1 integration subset' }} (push) Blocked by required conditions
Mobile checks / Mobile lint + typecheck (push) Waiting to run
Mobile checks / Mobile unit tests (push) Waiting to run
Mobile checks / Mobile dependency audit (report) (push) Waiting to run
Mobile checks / Mobile reproducibility checks (push) Waiting to run
Secrets guard (env backups) / Ensure no tracked env backup files (push) Waiting to run
Temporal Readiness / fast-readiness (push) Waiting to run
Temporal Readiness / docker-parity (push) Waiting to run
TypeScript Type Check / Nx affected typecheck (push) Waiting to run
Unit Tests / Skipped-test budget (push) Waiting to run
Unit Tests / Nx affected unit tests (push) Waiting to run
Unit Tests / Server unit coverage (informational) (push) Waiting to run
Validate Tenant Management Schema / Check for relevant changes (push) Waiting to run
Validate Tenant Management Schema / Validate Tenant Management Schema (push) Blocked by required conditions
EE Workflows Build Guard / ee-workflows-build-guard (push) Waiting to run
Excluded: .git, node_modules, secrets/, compose.env, assemblyscript tgz Source: /opt/alga-psa on psa.joliet.tech
68 lines
9.1 KiB
Markdown
68 lines
9.1 KiB
Markdown
# Scratchpad — 2026-01-24 Inbound Email In-App Processing
|
||
|
||
## Context / Decision
|
||
|
||
- We are choosing **Option B** for now: bypass workflows entirely and call existing email “domain” functions directly in the webhook request flow.
|
||
- Goal remains to make workflow-worker **v2-only** and to remove inbound-email dependency on legacy workflow runtimes.
|
||
|
||
## Current wiring notes (file pointers)
|
||
|
||
- Gmail webhook route: `server/src/app/api/email/webhooks/google/route.ts` → `packages/integrations/src/webhooks/email/google.ts`
|
||
- Microsoft webhook route: `server/src/app/api/email/webhooks/microsoft/route.ts` → `packages/integrations/src/webhooks/email/microsoft.ts`
|
||
- V2 email workflow definition exists (not used in Option B): `shared/workflow/runtime/workflows/email-processing-workflow.v2.json`
|
||
- Legacy email workflow exists (not used in Option B): `shared/workflow/workflows/system-email-processing-workflow.ts`
|
||
- Domain helpers we will call:
|
||
- `shared/workflow/actions/emailWorkflowActions.ts`
|
||
- HTML conversion utility: `@alga-psa/shared/lib/utils/contentConversion` (see `shared/workflow/init/registerWorkflowActions.ts` for how it’s used)
|
||
|
||
## Commands I used during investigation
|
||
|
||
- `rg -n "INBOUND_EMAIL_RECEIVED|webhooks/email|email-processing-workflow.v2.json" -S server packages shared services`
|
||
- `sed -n '1,220p' shared/workflow/actions/emailWorkflowActions.ts`
|
||
|
||
## Open questions to resolve before implementation
|
||
|
||
- What do we do for **unmatched** senders (no contact match)?
|
||
- What is the canonical **idempotency key** per provider (Gmail historyId vs messageId)?
|
||
- Do we want comment bodies stored as BlockNote JSON (preferred), or raw text/html?
|
||
|
||
## Progress log
|
||
|
||
- 2026-01-24: Implemented F001 by defining the normalized `processInboundEmailInApp` contract/types in `shared/services/email/processInboundEmailInApp.ts` (input + union result shape).
|
||
- 2026-01-24: Implemented F002 by adding `processInboundEmailInApp()` in `shared/services/email/processInboundEmailInApp.ts` with in-app routing (existing ticket reply vs new-ticket creation).
|
||
- 2026-01-24: Implemented F003 by invoking `parseEmailReplyBody()` and falling back to raw body content on parser failures (logs + continue).
|
||
- 2026-01-24: Implemented F004 by resolving existing tickets via reply token first (if present), then falling back to thread headers via `findTicketByEmailThread`.
|
||
- 2026-01-24: Implemented F005 by creating reply comments with BlockNote JSON content (HTML→blocks when available, text→blocks otherwise) and storing message threading metadata on the comment.
|
||
- 2026-01-24: Implemented F006 by processing reply attachments per item via `processEmailAttachment` with error-continue semantics.
|
||
- 2026-01-24: Implemented F010 by requiring inbound ticket defaults per provider via `resolveInboundTicketDefaults(tenantId, providerId)` and skipping processing when missing.
|
||
- 2026-01-24: Implemented F011 by matching sender email to an existing contact via `findContactByEmail` and using its `client_id`/`contact_id` on ticket creation.
|
||
- 2026-01-24: Implemented F012 decision: if sender email does not match an existing contact, create the ticket under provider defaults `client_id` with `contact_id` unset, and mark the initial comment metadata with `unmatchedSender: true` for manual triage.
|
||
- 2026-01-24: Implemented F013 by creating tickets via `createTicketFromEmail` and persisting `tickets.email_metadata` (messageId/threadId/inReplyTo/references/providerId) for future threading queries.
|
||
- 2026-01-24: Implemented F014 by creating the initial ticket comment via `createCommentFromEmail` with BlockNote JSON content derived from the inbound email body.
|
||
- 2026-01-24: Implemented F015 by processing new-ticket attachments per item via `processEmailAttachment` with error-continue semantics.
|
||
- 2026-01-24: Implemented F020 by de-duping reply comment creation via a DB lookup keyed on `{tenant, ticket_id, metadata.email.messageId}` before inserting.
|
||
- 2026-01-24: Implemented F021 by de-duping new-ticket creation via a DB lookup on `tickets.email_metadata.messageId+providerId` before creating a ticket.
|
||
- 2026-01-24: Implemented F022 by re-enabling Gmail Pub/Sub historyId de-dupe (skip when `gmail_processed_history` already contains the notification historyId). Microsoft continues using `email_processed_messages` PK de-dupe.
|
||
- 2026-01-24: Implemented F030 by wiring Gmail webhook message handling to call `processInboundEmailInApp` when the in-app flag is enabled (otherwise preserves legacy `INBOUND_EMAIL_RECEIVED` publish).
|
||
- 2026-01-24: Implemented F031 by wiring Microsoft webhook notifications to call `processInboundEmailInApp` when enabled and to persist ticket linkage back to `email_processed_messages`.
|
||
- 2026-01-24: Implemented F040 by adding env-driven, tenant/provider-scoped gating in `shared/services/email/inboundEmailInAppFeatureFlag.ts` (`INBOUND_EMAIL_IN_APP_PROCESSING_ENABLED`, `INBOUND_EMAIL_IN_APP_TENANT_IDS`, `INBOUND_EMAIL_IN_APP_PROVIDER_IDS`).
|
||
- 2026-01-24: Implemented F041 by supporting provider-level rollout via `INBOUND_EMAIL_IN_APP_PROVIDER_IDS` allowlist.
|
||
- 2026-01-24: Implemented F050 by defaulting workflow-worker deployments to `WORKFLOW_WORKER_MODE=v2` in compose configs (`docker-compose.ce.yaml`, `docker-compose.ee.yaml`, and prebuilt variants).
|
||
- 2026-01-24: Implemented F051 by disabling legacy system email workflow updates by default in `services/workflow-worker/src/init/updateWorkflows.ts` (requires `LEGACY_SYSTEM_EMAIL_WORKFLOW_ENABLED=true` to opt back in).
|
||
- 2026-01-24: Implemented T001 with an integration test covering Gmail webhook → in-app processing path and asserting exactly one ticket and one initial comment (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T002 with an integration test covering Microsoft webhook → in-app processing path and asserting exactly one ticket and one initial comment (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T003 by adding an integration test for reply-token threading to ensure the in-app processor creates exactly one comment on the matched ticket (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T004 by adding an integration test for header-based threading (In-Reply-To/References) to ensure the in-app processor creates exactly one comment on the matched ticket (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T010 with a unit test ensuring `parseEmailReplyBody` produces non-empty sanitized text for plain-text emails (`server/src/test/unit/email/inboundEmailBodyParsing.test.ts`).
|
||
- 2026-01-24: Implemented T011 with a unit test ensuring `parseEmailReplyBody` produces sanitized HTML when HTML input is present (`server/src/test/unit/email/inboundEmailBodyParsing.test.ts`).
|
||
- 2026-01-24: Implemented T020 with an integration test verifying sender→contact matching overrides default client/contact on ticket creation (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T021 with an integration test verifying unmatched senders create a ticket under provider defaults without a contact and without throwing (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T022 with an integration test verifying missing inbound ticket defaults causes processing to skip without creating tickets/comments (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T030 with an integration test verifying attachment processing errors do not prevent ticket + initial comment creation (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T031 with an integration test verifying attachment processing errors do not prevent reply comment creation (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T040 with an integration test verifying reply idempotency (replaying the same email id does not create duplicate comments) (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T041 with an integration test verifying new-email idempotency (replaying the same email id does not create duplicate tickets/comments) (`server/src/test/integration/inboundEmailInApp.webhooks.integration.test.ts`).
|
||
- 2026-01-24: Implemented T042 with an integration test ensuring the env-based feature flag switches Gmail webhook behavior between legacy publish and in-app processing (`server/src/test/integration/inboundEmailInApp.featureFlag.integration.test.ts`).
|
||
- 2026-01-24: Implemented T050 with a smoke integration test asserting compose defaults `WORKFLOW_WORKER_MODE=v2` and that `processInboundEmailInApp` successfully creates a ticket/comment without any worker running (`server/src/test/integration/workflowWorkerV2.inboundEmailSmoke.integration.test.ts`).
|
||
- 2026-01-24: Test env note: the three DB-backed integration tests in this plan now auto-skip when the configured Postgres host/port is unreachable (to avoid hard failures in dev setups without a local test DB).
|