284313f908
Some checks are pending
Bidi Control Character Guard / bidi-control-guard (push) Waiting to run
Circular Dependency Check / Check for new circular dependencies (push) Waiting to run
Citus Migration Smoke / Combined migrations on single-node Citus (push) Waiting to run
E2E Fresh Install Tests / fresh-install-e2e (push) Waiting to run
ext-v2 guardrails / Run ext-v2 guard and ESLint (push) Waiting to run
Integration Tests / Check for relevant changes (push) Waiting to run
Integration Tests / ${{ (github.event_name == 'schedule' || github.event.inputs.suite == 'full') && 'Full integration suite' || 'Tier-1 integration subset' }} (push) Blocked by required conditions
Mobile checks / Mobile lint + typecheck (push) Waiting to run
Mobile checks / Mobile unit tests (push) Waiting to run
Mobile checks / Mobile dependency audit (report) (push) Waiting to run
Mobile checks / Mobile reproducibility checks (push) Waiting to run
Secrets guard (env backups) / Ensure no tracked env backup files (push) Waiting to run
Temporal Readiness / fast-readiness (push) Waiting to run
Temporal Readiness / docker-parity (push) Waiting to run
TypeScript Type Check / Nx affected typecheck (push) Waiting to run
Unit Tests / Skipped-test budget (push) Waiting to run
Unit Tests / Nx affected unit tests (push) Waiting to run
Unit Tests / Server unit coverage (informational) (push) Waiting to run
Validate Tenant Management Schema / Check for relevant changes (push) Waiting to run
Validate Tenant Management Schema / Validate Tenant Management Schema (push) Blocked by required conditions
EE Workflows Build Guard / ee-workflows-build-guard (push) Waiting to run
Excluded: .git, node_modules, secrets/, compose.env, assemblyscript tgz Source: /opt/alga-psa on psa.joliet.tech
118 lines
5.5 KiB
Markdown
118 lines
5.5 KiB
Markdown
# PRD — Unified Inbound Email Queue with Pointer Jobs
|
|
|
|
- Slug: `unified-inbound-email-pointer-queue`
|
|
- Date: `2026-03-01`
|
|
- Status: Draft
|
|
|
|
## Summary
|
|
|
|
Unify inbound email handling so Microsoft callbacks, Google callbacks, and the IMAP listener all enqueue pointer-based jobs into one Redis-backed queue. A single email processor consumes jobs, fetches full email content at consume time, and runs existing in-app processing.
|
|
|
|
## Problem
|
|
|
|
Inbound email ingestion currently has split execution paths, including direct callback processing and IMAP-specific in-memory async queue behavior. This causes inconsistent reliability, duplicate handling differences, and different failure semantics across providers.
|
|
|
|
## Goals
|
|
|
|
- Move all providers to one ingestion contract: ingress validates and durably enqueues pointer jobs.
|
|
- Process email in one consumer path using consume-time idempotency.
|
|
- Remove dependence on in-memory queue acceptance semantics.
|
|
- Keep queue payloads lightweight by storing pointers, not full email bytes.
|
|
- Preserve existing ticket/comment/document outcomes produced by in-app processing.
|
|
|
|
## Non-goals
|
|
|
|
- Rewriting ticket/comment/document business logic.
|
|
- Introducing a brand-new external broker beyond existing Redis.
|
|
- Guaranteeing immutable IMAP message content between ingest and consume.
|
|
- Adding new UI surfaces for queue management in this scope.
|
|
|
|
## Users and Primary Flows
|
|
|
|
1. Email provider webhook/listener ingress
|
|
- Microsoft webhook receives notification pointer and enqueues.
|
|
- Google webhook receives Pub/Sub pointer (`historyId`/address) and enqueues.
|
|
- IMAP listener observes new message pointer and enqueues.
|
|
|
|
2. Email processing worker
|
|
- Worker pops job, resolves provider context, fetches full email content from source.
|
|
- Worker performs consume-time idempotency check.
|
|
- Worker runs inbound in-app processing pipeline.
|
|
- Worker ACKs on success, retries on failure, and sends to DLQ on terminal failure.
|
|
|
|
3. Retry and duplicate handling
|
|
- If ingress cannot durably enqueue, it returns non-2xx so sender retries.
|
|
- If consumer fails before ACK, job resurfaces and retries.
|
|
- If duplicate message is consumed, idempotency no-ops duplicate processing.
|
|
|
|
## UX / UI Notes
|
|
|
|
- No net-new user-facing workflow is required.
|
|
- Existing ticket and document surfaces should continue to show the same outcomes as current in-app processing.
|
|
|
|
## Requirements
|
|
|
|
### Functional Requirements
|
|
|
|
- Add a unified inbound queue job schema supporting provider-specific pointers:
|
|
- Microsoft: provider/subscription + message id pointer.
|
|
- Google: provider + history pointer (and any derived message pointer metadata).
|
|
- IMAP: provider + mailbox/UID/message pointer.
|
|
- Update all three ingress paths to enqueue pointer jobs only.
|
|
- Ingress endpoints/listeners must only report success after durable queue enqueue success.
|
|
- Implement a queue consumer that:
|
|
- claims jobs,
|
|
- fetches full email payload from source,
|
|
- applies consume-time idempotency,
|
|
- executes existing `processInboundEmailInApp` pipeline,
|
|
- ACKs only on successful handling.
|
|
- Support retry with bounded attempts and DLQ transfer.
|
|
- Support job resurfacing when ACK is not recorded.
|
|
- Handle unavailable/moved/deleted source messages by recording a deterministic skipped outcome.
|
|
- Replace or retire IMAP in-memory async queue code path in favor of the unified queue.
|
|
|
|
### Non-functional Requirements
|
|
|
|
- Queue payloads remain pointer-only and do not include attachment bytes or MIME bodies.
|
|
- Queue and consumer behavior must be deterministic enough to support at-least-once delivery semantics.
|
|
- Callback/listener latency should remain bounded by enqueue-only work.
|
|
|
|
## Data / API / Integrations
|
|
|
|
- Redis is the durable temporary queue store.
|
|
- Existing provider APIs remain source of truth for message content retrieval at consume time.
|
|
- Consumer invokes existing in-app inbound processing contracts after payload enrichment.
|
|
- Add/extend persistence for consume-time idempotency records keyed by normalized external message identity.
|
|
|
|
## Security / Permissions
|
|
|
|
- Preserve existing webhook authentication/verification rules for Microsoft, Google, and IMAP.
|
|
- Ensure queued job payload contains only minimally necessary metadata.
|
|
- Maintain tenant isolation in queue job schema and consume-time fetch.
|
|
|
|
## Observability
|
|
|
|
- Structured logs at ingress enqueue, consume start, retry, ack, and DLQ events.
|
|
- Include provider, tenant, pointer identifiers, attempt count, and terminal status reason.
|
|
|
|
## Rollout / Migration
|
|
|
|
- Introduce feature flag/env control for unified queue path.
|
|
- Migrate Microsoft and Google callback paths to enqueue-only mode.
|
|
- Migrate IMAP listener path to enqueue-only mode and disable legacy in-memory queue mode.
|
|
- Validate end-to-end behavior in local GreenMail + provider test paths before wider rollout.
|
|
|
|
## Open Questions
|
|
|
|
- Whether Google pointer jobs should carry only `historyId` or also optional snapshot of discovered message ids when available.
|
|
- Whether to expose DLQ re-drive tooling in this scope or leave as operator script only.
|
|
|
|
## Acceptance Criteria (Definition of Done)
|
|
|
|
- Microsoft, Google, and IMAP ingress all enqueue pointer jobs and do not perform full business processing inline.
|
|
- Ingress success response is tied to durable enqueue success.
|
|
- Consumer fetches email at consume time and executes existing in-app processing for all three providers.
|
|
- Consumer idempotency prevents duplicate downstream processing for the same message identity.
|
|
- Failed jobs retry and eventually land in DLQ after max attempts.
|
|
- IMAP in-memory async queue path is no longer used for production processing.
|