PSA/ee/docs/plans/imap-inbound-email-scratchpad.md

IMAP Inbound Email Scratchpad

Last updated: 2026-01-11 Owner: Email Platform

Use this scratchpad to capture key findings, decisions, TODOs, and file references while implementing the IMAP inbound email service.

---

Scope Notes

• IMAP provider must feed existing INBOUND_EMAIL_RECEIVED workflow event pipeline (same as Gmail/Microsoft).
• IMAP v1 must include OAuth2 (XOAUTH2) support.
• IMAP service should be resilient: IDLE, reconnect, backoff, dedupe, and per-folder monitoring.
• auto_process_emails is not exposed in the UI (processing is gated by provider Active flag).

---

Current Event Flow (Research)

• Gmail webhook: server/src/app/api/email/webhooks/google/route.ts
  • Decodes Pub/Sub notification, loads provider config, fetches message details via GmailAdapter, publishes INBOUND_EMAIL_RECEIVED via shared/events/publisher.ts.
• Microsoft webhook: server/src/app/api/email/webhooks/microsoft/route.ts
  • Validates Graph notification, fetches message details via MicrosoftGraphAdapter, publishes INBOUND_EMAIL_RECEIVED (fallback minimal payload on fetch failure).
• Workflow worker: services/workflow-worker/src/WorkflowWorker.ts
  • Consumes workflow events (Redis stream) and starts shared/workflow/workflows/system-email-processing-workflow.ts.
• Event schema: shared/workflow/streams/eventBusSchema.ts
  • INBOUND_EMAIL_RECEIVED payload expects { providerId, emailData{...} }.

---

Important Files / Entry Points

• Provider types + interfaces
  • shared/interfaces/inbound-email.interfaces.ts
  • server/src/interfaces/email.interfaces.ts
• Email provider CRUD + orchestration
  • server/src/services/email/EmailProviderService.ts
  • server/src/lib/actions/email-actions/emailProviderActions.ts
• Gmail integration (baseline)
  • server/src/services/email/providers/GmailAdapter.ts
  • server/src/services/email/GmailWebhookService.ts
• Microsoft integration (baseline)
  • shared/services/email/providers/MicrosoftGraphAdapter.ts
• Event publishing
  • shared/events/publisher.ts
• Inbound email docs
  • docs/inbound-email/README.md
  • docs/inbound-email/architecture/workflow.md
  • docs/inbound-email/development/adapters.md

---

Data Model Notes

• Existing tables:
  • email_providers (provider_type: google|microsoft)
  • google_email_provider_config
  • microsoft_email_provider_config
• IMAP will need imap_email_provider_config (new migration + Citus distribution).
  • Implemented migration: server/migrations/20251226121000_create_imap_email_provider_config.cjs
  • Includes host/port/TLS/auth_type/username/folder_filters + OAuth fields + state tracking.
  • Added folder state jsonb for per-folder UID tracking: server/migrations/20251226124500_add_imap_folder_state.cjs.
  • Added runtime metadata columns (leases, capabilities, timeouts): server/migrations/20251226140000_add_imap_runtime_columns.cjs.

---

UI/UX Touchpoints

• Provider selection: server/src/components/EmailProviderSelector.tsx
• Provider configuration shell: server/src/components/EmailProviderConfiguration.tsx
• Provider forms: server/src/components/GmailProviderForm.tsx, server/src/components/MicrosoftProviderForm.tsx
• Provider bundle entry: packages/product-email-providers/{oss,ee}/entry.tsx
  • Added IMAP forms: server/src/components/ImapProviderForm.tsx and ee/server/src/components/ImapProviderForm.tsx

---

OAuth2 for IMAP (v1)

• Need IMAP OAuth2 flow (likely XOAUTH2, provider-specific endpoints).
• Determine if we must support generic OAuth2 endpoints per provider config (authorize/token URLs).
• Decide how to store IMAP OAuth client secrets + refresh tokens (tenant secrets provider preferred).
  • Secret keys used by actions: imap_password_<providerId>, imap_oauth_client_secret_<providerId>, imap_refresh_token_<providerId>
  • OAuth endpoints:
    • POST /api/email/oauth/imap/initiate
    • GET /api/email/oauth/imap/callback

---

Implementation Notes (Initial)

• Service location proposal: services/email-service (standalone worker like services/workflow-worker).
• IMAP service should publish INBOUND_EMAIL_RECEIVED via shared/events/publisher.ts.
• Read-only IMAP fetch should use BODY.PEEK[] to avoid marking as read.
  • IMAP service uses imapflow + mailparser to fetch/parse messages and publish events.
  • Per-folder listeners are created for each configured folder (fallback to INBOX).
  • Folder state (UIDVALIDITY/last UID) persisted in imap_email_provider_config.folder_state.
  • IMAP service acquires DB leases per provider (lease_owner, lease_expires_at) to avoid double-processing.
  • IMAP OAuth re-connect flow is available in the IMAP provider form + card action.
  • Local test IMAP server: GreenMail (greenmail/standalone:latest) added to dev compose.
    • SMTP: host port ${EXPOSE_IMAP_TEST_SMTP_PORT:-3025} → container 3025
    • IMAP: host port ${EXPOSE_IMAP_TEST_IMAP_PORT:-3143} → container 3143
    • IMAPS: host port ${EXPOSE_IMAP_TEST_IMAPS_PORT:-3993} → container 3993
    • For SMTP injection, use recipient local-part imap_user (GreenMail stores To: imap_user in headers).
  • IMAP runtime tuning is intentionally not exposed in the provider UI:
    • IMAP_CONNECTION_TIMEOUT_MS (default: 10000)
    • IMAP_MAX_EMAILS_PER_SYNC (default: 5)
    • IMAP_SOCKET_KEEPALIVE (default: true, set to false to disable)
  • To avoid thundering herd across instances/providers, IMAP service uses jittered timers:
    • IMAP_TIMER_JITTER_PCT (default: 0.1) applies to provider refresh, heartbeat, poll sleep, and IDLE NOOP interval.
    • IMAP_STARTUP_STAGGER_MS (default: 2000) staggers folder listener startup per provider/folder.
    • IMAP_RECONNECT_JITTER_PCT (default: 0.5) randomizes reconnect delay within [baseDelay*(1-jitterPct), baseDelay].
  • Initial connect and manual resync behavior:
    • When cursor state is empty, IMAP listener starts from the most recent IMAP_MAX_EMAILS_PER_SYNC window (based on mailbox uidNext) instead of replaying the whole mailbox from UID 1.
    • last_uid advances to the highest UID observed even when a message is skipped/deduped, to avoid repeatedly re-scanning the same window.
  • Kubernetes deployment assets:
    • Helm chart: ee/helm/email-service (includes liveness/readiness probes via /health).
    • Argo CI/CD (nm-kube-config): argo-workflow/alga-psa-dev/templates/build/email-service-ci-cd-sebastian.yaml.
    • Helm values (nm-kube-config): email-service/{hosted,sebastian,prod}.values.yaml.

---

Open Questions

• Which IMAP client library should we standardize on (imapflow vs node-imap)?
• OAuth2 endpoints: per-provider config vs pre-baked defaults?
• UID tracking per folder: store as map in JSONB or separate table?
• Do we want a lease/lock (Redis/DB) to prevent multi-instance double processing?

---

TODO Log

• Add IMAP provider type to shared/server interfaces and UI enums.
• Create imap_email_provider_config migration.
• Add IMAP provider form (OSS + EE).
• Implement IMAP OAuth2 flow (init + callback + token refresh).
• Stand up services/email-service with IDLE + reconnect + publish.

---

Decisions

• 2025-12-26: IMAP v1 must include OAuth2 support (XOAUTH2).