PSA/ee/docs/plans/2025-12-26-imap-inbound-email-service-plan.md

IMAP Inbound Email Service Plan

Status: Draft Owner: Email Platform Last updated: 2025-12-26

Goals

• Add an IMAP inbound email provider that feeds the same INBOUND_EMAIL_RECEIVED event flow as Gmail and Microsoft.
• Provide a resilient, long-running IMAP service that uses IDLE, reconnects automatically, and handles multiple folders per mailbox.
• Expose IMAP as a first-class inbound email provider in the settings UI alongside Google and Microsoft.

Current State (Research Summary)

• Gmail inbound flow: server/src/app/api/email/webhooks/google/route.ts decodes Pub/Sub, loads provider config, fetches full message details via GmailAdapter, and publishes INBOUND_EMAIL_RECEIVED via shared/events/publisher.ts.
• Microsoft inbound flow: server/src/app/api/email/webhooks/microsoft/route.ts validates Graph notifications, fetches message details via MicrosoftGraphAdapter, then publishes INBOUND_EMAIL_RECEIVED (with fallback minimal payload on fetch failure).
• Test flow: server/src/services/email/MailHogPollingService.ts emits INBOUND_EMAIL_RECEIVED directly via the server EventBus.
• Workflow trigger: services/workflow-worker/src/WorkflowWorker.ts consumes workflow events (Redis stream) and starts shared/workflow/workflows/system-email-processing-workflow.ts, which waits for the INBOUND_EMAIL_RECEIVED payload defined in shared/workflow/streams/eventBusSchema.ts.

Proposed Architecture

High-level flow

1. IMAP service connects to configured mailboxes and folders.
2. IDLE receives new mail notifications (or polling fallback if IDLE unsupported).
3. IMAP service fetches the full RFC822/body + headers for new messages.
4. Parse MIME into EmailMessageDetails and publish INBOUND_EMAIL_RECEIVED using shared/events/publisher.ts.
5. Workflow worker consumes and runs the existing system email processing workflow (no change).

Service placement

• New service in services/email-service (Node + TS, similar to services/workflow-worker).
• Uses shared DB access and shared event publisher to avoid duplicating queue/event logic.
• Deploy as a long-running worker alongside existing services (server, workflow-worker, temporal-worker).

Data Model & Configuration

Provider types

• Extend all inbound email unions to include imap:
  • shared/interfaces/inbound-email.interfaces.ts
  • server/src/interfaces/email.interfaces.ts
  • server/src/components/EmailProviderConfiguration.tsx (providerType union)
  • Any API/action payloads that validate provider type

New vendor config table

Create imap_email_provider_config with fields (initial draft):

• email_provider_id, tenant
• Connection: host, port, secure (TLS), allow_starttls, auth_type (password | oauth2)
• Auth: username, password (store encrypted or via tenant secret provider), oauth_access_token, oauth_refresh_token, oauth_expires_at
• Processing: auto_process_emails, max_emails_per_sync, folder_filters (jsonb array)
• State: last_uid, uid_validity, last_seen_at, last_error, last_sync_at

Provider CRUD + validation

• Update server/src/services/email/EmailProviderService.ts and server/src/lib/actions/email-actions/emailProviderActions.ts to support IMAP configs.
• Add validation in server/src/services/email/EmailProviderValidator.ts for IMAP host/port/auth requirements.
• Add secret storage integration via getSecretProviderInstance() for IMAP passwords (and optional OAuth tokens) instead of plain DB storage.

IMAP Service Design

Connection lifecycle

• Maintain one IMAP connection per provider + folder (or multiplex folders on a single connection if library allows).
• On startup, load active IMAP providers and connect to all configured folders.
• Use IDLE to listen for EXISTS/RECENT events; on disconnect or IDLE timeout, re-enter IDLE.
• Implement exponential backoff with jitter on reconnect.

Message tracking + dedupe

• Track uidvalidity and last_uid per provider/folder; on mismatch, resync (search unseen from scratch).
• Persist state to imap_email_provider_config to survive restarts.
• Use Message-ID header plus providerId as a secondary dedupe key (insert into email_processed_messages or a new imap_processed_messages table) to avoid double-publishing.

MIME parsing

• Fetch full RFC822 or BODY.PEEK[] and parse with a MIME parser (e.g., mailparser).
• Map into the existing EmailMessageDetails structure:
  • subject, from, to, cc, receivedAt
  • body.text and body.html
  • attachments metadata (id, name, contentType, size)
  • threadId, references, inReplyTo, and headers

Event publishing

• Publish INBOUND_EMAIL_RECEIVED using shared/events/publisher.ts with payload:
  • tenantId, tenant, providerId, and the full emailData object
• Reuse existing workflow pipeline and ticket creation logic without changes.

UI/UX Updates (Inbound Email Settings)

• Add an IMAP card to server/src/components/EmailProviderSelector.tsx and ee/server/src/components/EmailProviderSelector.tsx.
• Add an IMAP provider form in both OSS and EE components:
  • Fields: mailbox, host, port, TLS/starttls, username, password/app-password, folder filters, auto-process, max emails per sync, inbound ticket defaults.
• Update provider list cards to show IMAP status and connection errors.

API + Service Integration

• New IMAP service reads configs via DB (not via server API) to avoid API coupling.
• Optional admin endpoints (future): manual reconnect, force resync, or pause provider.
• Add Docker Compose entry for email-service (dev + prod) with required env vars.

Observability

• Log structured events per provider: connect, disconnect, idle start, new message count, publish success/failure.
• Update email_providers.status, error_message, last_sync_at from the IMAP service on state changes.
• Add a lightweight health check endpoint or readiness log for monitoring.

Testing Strategy

• Unit tests for IMAP MIME parsing into EmailMessageDetails.
• Integration tests with a local IMAP server (e.g., dockerized test server) validating:
  • IDLE + reconnect
  • Folder filters
  • Deduplication by UID + Message-ID
  • Published INBOUND_EMAIL_RECEIVED payload
• E2E: simulate incoming email and verify ticket creation via workflow.

Implementation Plan (Tasks)

• Add imap provider type across shared/server interfaces and validation.
• Create imap_email_provider_config migration + indices + Citus distribution.
• Extend provider CRUD/actions to read/write IMAP config and secrets.
• Build IMAP adapter/parser (message fetch + MIME parsing) and map to EmailMessageDetails.
• Build services/email-service with connection manager, IDLE loop, reconnect strategy, and publisher.
• Add UI forms and selector card for IMAP in both OSS and EE bundles.
• Wire Docker Compose for the new service (dev + prod) and document env vars.
• Add tests (unit + integration) and update inbound email docs to include IMAP.

Risks & Mitigations

• IMAP server variance: IDLE support and folder semantics differ. Mitigate with polling fallback and robust folder selection logic.
• Duplicate events: IMAP can replay on reconnect. Mitigate with UID + Message-ID dedupe and persisted state.
• Credential handling: Storing passwords requires secure storage; use tenant secret provider and avoid logging secrets.
• Large attachments: IMAP fetch size can be heavy; fetch metadata first and defer content where possible.

Open Questions

• Do we want OAuth2 for IMAP (XOAUTH2) in v1 or password-only?
• Should IMAP providers share the same inbound defaults schema as Google/Microsoft (yes), or add IMAP-specific defaults?
• Preferred IMAP test server for CI (e.g., Dovecot, GreenMail, or MailHog alternative)?