PSA/ee/docs/plans/2026-03-01-unified-inbound-email-pointer-queue/PRD.md

PRD — Unified Inbound Email Queue with Pointer Jobs

• Slug: unified-inbound-email-pointer-queue
• Date: 2026-03-01
• Status: Draft

Summary

Unify inbound email handling so Microsoft callbacks, Google callbacks, and the IMAP listener all enqueue pointer-based jobs into one Redis-backed queue. A single email processor consumes jobs, fetches full email content at consume time, and runs existing in-app processing.

Problem

Inbound email ingestion currently has split execution paths, including direct callback processing and IMAP-specific in-memory async queue behavior. This causes inconsistent reliability, duplicate handling differences, and different failure semantics across providers.

Goals

• Move all providers to one ingestion contract: ingress validates and durably enqueues pointer jobs.
• Process email in one consumer path using consume-time idempotency.
• Remove dependence on in-memory queue acceptance semantics.
• Keep queue payloads lightweight by storing pointers, not full email bytes.
• Preserve existing ticket/comment/document outcomes produced by in-app processing.

Non-goals

• Rewriting ticket/comment/document business logic.
• Introducing a brand-new external broker beyond existing Redis.
• Guaranteeing immutable IMAP message content between ingest and consume.
• Adding new UI surfaces for queue management in this scope.

Users and Primary Flows

1. Email provider webhook/listener ingress

• Microsoft webhook receives notification pointer and enqueues.
• Google webhook receives Pub/Sub pointer (historyId/address) and enqueues.
• IMAP listener observes new message pointer and enqueues.

2. Email processing worker

• Worker pops job, resolves provider context, fetches full email content from source.
• Worker performs consume-time idempotency check.
• Worker runs inbound in-app processing pipeline.
• Worker ACKs on success, retries on failure, and sends to DLQ on terminal failure.

3. Retry and duplicate handling

• If ingress cannot durably enqueue, it returns non-2xx so sender retries.
• If consumer fails before ACK, job resurfaces and retries.
• If duplicate message is consumed, idempotency no-ops duplicate processing.

UX / UI Notes

• No net-new user-facing workflow is required.
• Existing ticket and document surfaces should continue to show the same outcomes as current in-app processing.

Requirements

Functional Requirements

• Add a unified inbound queue job schema supporting provider-specific pointers:
  • Microsoft: provider/subscription + message id pointer.
  • Google: provider + history pointer (and any derived message pointer metadata).
  • IMAP: provider + mailbox/UID/message pointer.
• Update all three ingress paths to enqueue pointer jobs only.
• Ingress endpoints/listeners must only report success after durable queue enqueue success.
• Implement a queue consumer that:
  • claims jobs,
  • fetches full email payload from source,
  • applies consume-time idempotency,
  • executes existing processInboundEmailInApp pipeline,
  • ACKs only on successful handling.
• Support retry with bounded attempts and DLQ transfer.
• Support job resurfacing when ACK is not recorded.
• Handle unavailable/moved/deleted source messages by recording a deterministic skipped outcome.
• Replace or retire IMAP in-memory async queue code path in favor of the unified queue.

Non-functional Requirements

• Queue payloads remain pointer-only and do not include attachment bytes or MIME bodies.
• Queue and consumer behavior must be deterministic enough to support at-least-once delivery semantics.
• Callback/listener latency should remain bounded by enqueue-only work.

Data / API / Integrations

• Redis is the durable temporary queue store.
• Existing provider APIs remain source of truth for message content retrieval at consume time.
• Consumer invokes existing in-app inbound processing contracts after payload enrichment.
• Add/extend persistence for consume-time idempotency records keyed by normalized external message identity.

Security / Permissions

• Preserve existing webhook authentication/verification rules for Microsoft, Google, and IMAP.
• Ensure queued job payload contains only minimally necessary metadata.
• Maintain tenant isolation in queue job schema and consume-time fetch.

Observability

• Structured logs at ingress enqueue, consume start, retry, ack, and DLQ events.
• Include provider, tenant, pointer identifiers, attempt count, and terminal status reason.

Rollout / Migration

• Introduce feature flag/env control for unified queue path.
• Migrate Microsoft and Google callback paths to enqueue-only mode.
• Migrate IMAP listener path to enqueue-only mode and disable legacy in-memory queue mode.
• Validate end-to-end behavior in local GreenMail + provider test paths before wider rollout.

Open Questions

• Whether Google pointer jobs should carry only historyId or also optional snapshot of discovered message ids when available.
• Whether to expose DLQ re-drive tooling in this scope or leave as operator script only.

Acceptance Criteria (Definition of Done)

• Microsoft, Google, and IMAP ingress all enqueue pointer jobs and do not perform full business processing inline.
• Ingress success response is tied to durable enqueue success.
• Consumer fetches email at consume time and executes existing in-app processing for all three providers.
• Consumer idempotency prevents duplicate downstream processing for the same message identity.
• Failed jobs retry and eventually land in DLQ after max attempts.
• IMAP in-memory async queue path is no longer used for production processing.