PSA/ee/docs/plans/2026-03-01-unified-inbound-email-pointer-queue/SCRATCHPAD.md

# Scratchpad — Unified Inbound Email Queue with Pointer Jobs

- Plan slug: `unified-inbound-email-pointer-queue`
- Created: `2026-03-01`

## What This Is

Working notes for moving Microsoft, Google, and IMAP inbound email ingress to one pointer-based Redis queue with consume-time idempotency.

## Decisions

- (2026-03-01) Use one queue ingestion model for all inbound providers: Microsoft callback, Google callback, and IMAP listener enqueue pointer jobs only.
- (2026-03-01) Use consume-time idempotency instead of ingress-time idempotency.
- (2026-03-01) Queue payloads stay pointer-only (no raw MIME/attachment bytes).
- (2026-03-01) Source-content drift for IMAP between ingest and consume is accepted risk; unavailable source should produce deterministic skipped outcome.
- (2026-03-01) Ingress success must mean durable enqueue success.
- (2026-03-01) F001 implemented by defining `UnifiedInboundEmailQueueJob` as a discriminated union (`provider`) with provider-specific pointer objects (`microsoft`, `google`, `imap`), while keeping legacy `EmailQueueJob` for compatibility during migration.
- (2026-03-01) Added a dedicated unified queue feature flag gate (`UNIFIED_INBOUND_EMAIL_POINTER_QUEUE_*`) so provider webhooks can move to enqueue-only behavior without forcing immediate cutover.

## Discoveries / Constraints

- IMAP service already retries webhook dispatch on non-2xx responses.
- Existing IMAP in-app async queue implementation is in-memory and returns success after enqueue, which is not durable acceptance.
- Microsoft and Google callback handlers currently fetch and process in callback path; this plan changes them to enqueue-only ingress.
- Inbound email interface definitions are duplicated across `shared/interfaces`, `server/src/interfaces`, and `packages/types/src/interfaces`; all three must be kept in sync for type consumers.
- Microsoft webhook handler is transaction-scoped per notification; queue-mode enqueue can be inserted before legacy fetch/process logic and short-circuit the callback path cleanly.
- Google webhook flow can enqueue immediately after provider resolution + JWT verification, before any `gmail_processed_history` writes or Gmail API fetches.
- IMAP listener now has enough metadata at fetch time (`mailbox`, `uid`, `uidValidity`, `messageId`) to emit pointer-only webhook payloads; no raw body is required for unified queue ingress.
- Unified queue internals now track ready/processing/inflight/DLQ keys with lease metadata, enabling explicit claim and completion lifecycle management.
- Queue enqueue now enforces a runtime pointer-only payload guard that rejects forbidden MIME/body/attachment keys at both top-level and nested pointer metadata.
- Legacy IMAP in-memory async queue now rejects enqueue attempts when unified pointer queue mode is enabled for the same tenant/provider, preventing accidental production regressions to in-memory processing.
- Security checks are still enforced before enqueue-only handoff: Microsoft validation/clientState checks, Google Pub/Sub JWT verification, and IMAP secret header verification all execute before unified-queue enqueue paths.
- IMAP async-mode gating is now provider-aware and supports explicit legacy-path disablement via `IMAP_INBOUND_EMAIL_IN_APP_ASYNC_DISABLED`, while also auto-disabling async mode whenever unified pointer queue mode is enabled for a provider.
- Unified queue now emits structured event logs for `enqueue`, `consume_start`, `ack`, `retry`, `dlq`, `reclaim`, and consumer `skip` with job/pointer identifiers and attempt metadata.
- Microsoft webhook response contract now reports handoff mode (`unified_pointer_queue`/`mixed`/`inline_processing`) plus queue vs inline counts, aligning callback semantics with Google/IMAP queue-mode responses.
- Queue consumer provider routing is implemented in `processUnifiedInboundEmailQueueJob` via provider-specific pointer resolution paths: Microsoft (`messageId`), Google (`historyId` plus discovered message IDs), and IMAP (`uid` mailbox fetch).

## Commands / Runbooks

- `python3 /Users/roberisaacs/.codex/skills/alga-plan/scripts/scaffold_plan.py "Unified Inbound Email Queue with Pointer Jobs" --slug unified-inbound-email-pointer-queue`
- `python3 /Users/roberisaacs/.codex/skills/alga-plan/scripts/validate_plan.py ee/docs/plans/2026-03-01-unified-inbound-email-pointer-queue`
- `npm -w shared run typecheck`
- `npm -w @alga-psa/types run build`
- `npm -w server run typecheck`
- `npm -w shared run typecheck` (after Microsoft queue-mode changes)
- `npm -w server run typecheck` (after Microsoft queue-mode changes)
- `npm -w email-service run build`
- `npm -w @alga-psa/integrations run typecheck`
- `npm -w server run test -- src/test/integration/microsoftWebhookUnifiedQueue.integration.test.ts`
- `npm -w server run test -- src/test/integration/googleWebhookUnifiedQueue.integration.test.ts --coverage.enabled=false`
- `npm -w server run test -- src/test/integration/imapWebhookHandoff.integration.test.ts --coverage.enabled=false`
- `npm -w server run test -- src/test/integration/microsoftWebhookUnifiedQueue.integration.test.ts src/test/integration/googleWebhookUnifiedQueue.integration.test.ts src/test/integration/imapWebhookHandoff.integration.test.ts --coverage.enabled=false`
- `npx vitest --config shared/vitest.config.ts services/email-service/src/emailService.webhookRetry.test.ts`
- `npx vitest --config shared/vitest.config.ts shared/services/email/__tests__/unifiedInboundEmailQueueConsumer.test.ts`
- `npm -w server run test -- src/test/unit/unifiedInboundEmailQueueJobProcessor.fetch.test.ts --coverage.enabled=false`
- `npx vitest --config shared/vitest.config.ts shared/services/email/__tests__/unifiedInboundEmailQueue.test.ts`

## Links / References

- IMAP webhook route: `packages/integrations/src/webhooks/email/imap.ts`
- IMAP in-memory queue: `packages/integrations/src/webhooks/email/imapInAppQueue.ts`
- Microsoft webhook route: `packages/integrations/src/webhooks/email/microsoft.ts`
- Google webhook route: `packages/integrations/src/webhooks/email/google.ts`
- IMAP listener dispatch path: `services/email-service/src/emailService.ts`
- Existing related plan: `ee/docs/plans/2026-02-27-inbound-email-inapp-artifact-persistence-remaining-work/`
- Unified job contract files:
  - `shared/interfaces/inbound-email.interfaces.ts`
  - `server/src/interfaces/email.interfaces.ts`
  - `packages/types/src/interfaces/email.interfaces.ts`
- Unified queue helper: `shared/services/email/unifiedInboundEmailQueue.ts`
- Unified queue flag gate helper: `shared/services/email/inboundEmailInAppFeatureFlag.ts`
- Unified queue consumer loop: `shared/services/email/unifiedInboundEmailQueueConsumer.ts`
- Server queue job processor: `server/src/services/email/unifiedInboundEmailQueueJobProcessor.ts`
- Server consumer entrypoint: `server/src/bin/unifiedInboundEmailQueueConsumer.ts`
- Unified queue runbook: `ee/docs/plans/2026-03-01-unified-inbound-email-pointer-queue/RUNBOOK.md`
- Microsoft unified ingress contract tests: `server/src/test/integration/microsoftWebhookUnifiedQueue.integration.test.ts`
- Google unified ingress contract tests: `server/src/test/integration/googleWebhookUnifiedQueue.integration.test.ts`
- IMAP webhook retry test: `services/email-service/src/emailService.webhookRetry.test.ts`
- Unified queue consumer tests: `shared/services/email/__tests__/unifiedInboundEmailQueueConsumer.test.ts`
- Unified queue job processor fetch tests: `server/src/test/unit/unifiedInboundEmailQueueJobProcessor.fetch.test.ts`
- Unified queue primitives tests: `shared/services/email/__tests__/unifiedInboundEmailQueue.test.ts`

## Progress Log

- (2026-03-01) Completed `F001`: Added unified pointer job contract types with provider-specific pointer metadata and queue lifecycle fields (`attempt`, `maxAttempts`, `enqueuedAt`, `jobId`, `schemaVersion`).
- (2026-03-01) Completed `F002`: Microsoft webhook now supports enqueue-only pointer handoff in unified-queue mode, using `shared/services/email/unifiedInboundEmailQueue.ts` and no longer requiring inline full-email fetch/processing when that mode is enabled.
- (2026-03-01) Completed `F003`: Google webhook now supports enqueue-only pointer handoff in unified-queue mode (`historyId`, `emailAddress`, `pubsubMessageId`) and returns `503` when durable enqueue fails.
- (2026-03-01) Completed `F004`: IMAP listener/webhook handoff now supports pointer-only ingress (`mailbox`, `uid`, `uidValidity`, optional `messageId`) and enqueues IMAP pointer jobs when unified queue mode is enabled.
- (2026-03-01) Completed `F005`: Unified pointer ingress is now persisted in Redis list storage via `shared/services/email/unifiedInboundEmailQueue.ts` (`RPUSH` on a configurable queue key).
- (2026-03-01) Completed `F006`: Unified queue mode ingress responses now acknowledge only after enqueue returns success; enqueue errors return non-success responses so callers can retry.
- (2026-03-01) Completed `F007`: Microsoft, Google, and IMAP unified-queue paths now return `503` when enqueue fails, preserving upstream retry behavior.
- (2026-03-01) Completed `F008`: Added a reusable consumer loop (`UnifiedInboundEmailQueueConsumer`) plus queue claim/ack/fail/reclaim primitives for processing unified inbound pointer jobs.
- (2026-03-01) Completed `F009`: Added provider-specific consume-time pointer resolution in `unifiedInboundEmailQueueJobProcessor` for Microsoft (`messageId`), Google (`historyId` -> message IDs), and IMAP (`uid` fetch) before downstream processing.
- (2026-03-01) Completed `F010`: Added consume-time idempotency insert/check against `email_processed_messages` with duplicate short-circuit when a normalized external identity already exists.
- (2026-03-01) Completed `F011`: Queue job processor now calls `processInboundEmailInApp` for fetched provider messages and records final processing status back to `email_processed_messages`.
- (2026-03-01) Completed `F012`: Consumer loop now ACKs only after `handleJob` completes successfully; failed jobs are not ACKed and are routed through retry/DLQ handling.
- (2026-03-01) Completed `F013`: Added lease-based reclaim (`reclaimExpiredUnifiedInboundEmailQueueJobs`) so stale in-flight jobs are resurfaced back to the ready queue.
- (2026-03-01) Completed `F014`: Failed jobs now increment `attempt` in queue payload state and only requeue while below configured `maxAttempts`.
- (2026-03-01) Completed `F015`: Once `attempt` reaches `maxAttempts`, failed jobs are routed to the dedicated unified inbound pointer DLQ key.
- (2026-03-01) Completed `F016`: Source-unavailable fetch failures now resolve as deterministic `skipped` outcomes (`source_unavailable:*`) recorded in `email_processed_messages` and do not rethrow for retry.
- (2026-03-01) Completed `F017`: Consumer idempotency now uses a normalized external identity format (`<provider>:<messageId>`) prior to persistence checks.
- (2026-03-01) Completed `F018`: Added `assertPointerOnlyPayload` validation in enqueue to reject raw content-like keys (`rawMime`, `attachments`, `body`, etc.) and enforce pointer-only queue contracts at runtime.
- (2026-03-01) Completed `F019`: Added a defensive runtime guard in `imapInAppQueue` that throws when unified pointer queue mode is enabled for the tenant/provider, ensuring legacy in-memory queue path is bypassed/retired for production unified-mode processing.
- (2026-03-01) Completed `F020`: Verified webhook auth/verification behavior is preserved in enqueue-only mode across Microsoft, Google, and IMAP paths (no auth bypass introduced by unified queue branching).
- (2026-03-01) Completed `F021`: Aligned queue migration flags by extending IMAP async mode evaluation to accept provider context, auto-disable on unified mode, and honor `IMAP_INBOUND_EMAIL_IN_APP_ASYNC_DISABLED` for explicit legacy disablement.
- (2026-03-01) Completed `F022`: Added structured observability events across queue lifecycle and consumer skip outcomes, including tenant/provider/pointer identifiers, attempts, and terminal reasons for retry/DLQ paths.
- (2026-03-01) Completed `F023`: Updated provider callback contracts so unified mode explicitly reports queue handoff metadata and avoids inline-processing ambiguity in webhook responses.
- (2026-03-01) Completed `F024`: Confirmed unified consumer routing dispatches per provider type and fetches provider-specific source payloads before shared in-app processing.
- (2026-03-01) Completed `F025`: Added a dedicated runbook covering architecture, queue keys, feature flags, consumer startup, and local validation/failure-path checks.
- (2026-03-01) Completed `T001`: Added Microsoft unified ingress contract test validating pointer-only enqueue payload shape (`tenantId`, `providerId`, provider pointer identifiers) and absence of raw content fields.
- (2026-03-01) Completed `T002`: Added Google unified ingress contract test validating pointer-only enqueue payload shape (`tenantId`, `providerId`, `historyId`, `pubsubMessageId`) behind successful JWT/provider verification.
- (2026-03-01) Completed `T003`: Extended IMAP webhook integration coverage with unified-mode pointer enqueue assertions (`mailbox`, `uid`, `uidValidity`, `messageId`) and pointer-only payload guards.
- (2026-03-01) Completed `T004`: Added deferred-enqueue Microsoft webhook test proving `200` success is not returned until unified queue enqueue promise resolves.
- (2026-03-01) Completed `T005`: Added deferred-enqueue Google webhook test proving callback success response is blocked until unified queue enqueue completion.
- (2026-03-01) Completed `T006`: Added deferred-enqueue IMAP webhook test proving unified-mode success response is blocked until pointer job enqueue completion.
- (2026-03-01) Completed `T007`: Added enqueue-failure assertions for Microsoft, Google, and IMAP unified ingress paths, each returning `503` to preserve upstream retry semantics.
- (2026-03-01) Completed `T008`: Extracted and tested IMAP webhook retry helper to verify non-2xx ingress responses trigger retry attempts before eventual success.
- (2026-03-01) Completed `T009`: Added consumer unit coverage confirming Microsoft pointer claims invoke handler and ACK path through unified consumer loop.
- (2026-03-01) Completed `T010`: Validated Google pointer claims execute through the same unified consumer claim/handle/ACK lifecycle.
- (2026-03-01) Completed `T011`: Validated IMAP pointer claims execute through the same unified consumer claim/handle/ACK lifecycle.
- (2026-03-01) Completed `T012`: Added processor fetch test proving Microsoft pointer jobs resolve full provider payloads before shared in-app processing execution.
- (2026-03-01) Completed `T013`: Added processor fetch test proving Google pointer jobs resolve message payloads (history cursor -> message IDs -> full payloads) before processing.
- (2026-03-01) Completed `T014`: Added processor fetch test proving IMAP pointer jobs resolve mailbox UID content into normalized email payloads before processing.
- (2026-03-01) Completed `T015`: Added idempotency happy-path test validating first consume writes normalized identity (`provider:messageId`) processing marker and executes downstream processing.
- (2026-03-01) Completed `T016`: Added idempotency duplicate-path test validating unique-constraint collision (`23505`) short-circuits downstream processing with deduped skip outcome.
- (2026-03-01) Completed `T017`: Processor fetch suite now asserts `processInboundEmailInApp` receives fully resolved provider payloads on successful consume-time fetch paths.
- (2026-03-01) Completed `T018`: Added queue ACK primitive test validating successful consume removes payload from processing list and clears inflight hash/lease entries.
- (2026-03-01) Completed `T019`: Added consumer failure-path test validating processing exceptions skip ACK and invoke retry-handling path (`failUnifiedInboundEmailQueueJob`).
- (2026-03-01) Completed `T020`: Added reclaim-path queue test validating expired inflight claims are removed from processing structures and requeued to ready state.
- (2026-03-01) Completed `T021`: Added retry-path queue test validating failed consume increments job attempt prior to requeue.
- (2026-03-01) Completed `T022`: Added DLQ-path queue test validating jobs are moved to dead-letter storage once max attempts are reached.
- (2026-03-01) Completed `T023`: Added IMAP source-unavailable processor test validating deterministic `source_unavailable:*` skip reason and consume-marker persistence.
- (2026-03-01) Completed `T024`: Added consumer skipped-outcome test validating source-unavailable paths ACK and avoid retry-loop behavior.
- (2026-03-01) Completed `T025`: Added queue payload-guard test validating enqueue rejects raw-content fields and enforces pointer-only contract.
- (2026-03-01) Completed `T026`: Added IMAP regression test validating unified queue mode bypasses legacy in-memory async queue path even when legacy async flag is enabled.
- (2026-03-01) Completed `T027`: Added Microsoft security regression test validating `clientState` mismatch blocks enqueue in unified mode.
- (2026-03-01) Completed `T028`: Added Google security regression test validating JWT auth header remains required in enqueue-only mode.
- (2026-03-01) Completed `T029`: Added IMAP security regression test validating webhook secret mismatch still rejects enqueue-only requests.
- (2026-03-01) Completed `T030`: Existing unified-mode ingress contract tests confirm flag enablement routes Microsoft/Google/IMAP into enqueue-only handoff paths.
- (2026-03-01) Completed `T031`: Added rollback-path IMAP test validating unified flag disablement preserves legacy `in_app_async` handoff behavior.
- (2026-03-01) Completed `T032`: Queue logging tests now assert enqueue success/failure events include provider, tenant, and pointer identifiers.
- (2026-03-01) Completed `T033`: Queue + consumer logging tests now assert retry/DLQ/skip event payloads include attempt counts and terminal reasons.
- (2026-03-01) Completed `T034`: Idempotency persistence test coverage confirms first consume writes `email_processed_messages` marker for newly processed identities.
- (2026-03-01) Completed `T035`: Duplicate-guard test coverage confirms unique-constraint collision blocks second consume and prevents downstream processing.
- (2026-03-01) Completed `T036`: Combined Microsoft webhook enqueue contract + processor consume-time fetch tests validate callback-to-worker shared processing flow and created-outcome handoff.
- (2026-03-01) Completed `T037`: Combined Google webhook enqueue contract + processor consume-time fetch tests validate callback-to-worker shared processing flow and created-outcome handoff.
- (2026-03-01) Completed `T038`: Combined IMAP listener/webhook enqueue contract + processor consume-time fetch tests validate callback-to-worker shared processing flow and created-outcome handoff.
- (2026-03-01) Completed `T039`: Idempotency duplicate-consume coverage validates repeated provider deliveries result in a single processed outcome with deduped no-op on repeats.
- (2026-03-01) Completed `T040`: Plan docs validation complete (`SCRATCHPAD.md` + `RUNBOOK.md`) with unified architecture, flags, queue lifecycle, and local verification runbook steps.

## Open Questions

- Choose Redis queue primitive for implementation phase: Streams with consumer groups vs list-based queue with explicit inflight tracking.
- Decide whether DLQ re-drive tooling is required in this scope or deferred.