PSA/docs/plans/2026-06-12-rmm-alert-handling/PRD.md

# PRD — RMM Alert Handling

- Slug: `2026-06-12-rmm-alert-handling`
- Date: 2026-06-12
- Status: Approved
- Branch: `feature/rmm-alerts-sync`
- Design doc: `docs/plans/2026-06-12-rmm-alert-handling-design.md`

## Summary

Turn RMM alerts into tickets automatically. A provider-generic pipeline in
`shared/rmm/alerts/` ingests normalized alert events from the NinjaOne and
TacticalRMM webhooks, evaluates tenant-defined rules, creates or updates
tickets with dedup, and keeps alert and ticket lifecycles in sync in both
directions. Tenants manage rules from the RMM integration settings UI.

## Problem

Alert ingestion is scaffolded but broken on main: the webhook writes columns
that don't exist in `rmm_alerts`, the rules engine expects a JSONB schema the
migration never created, and nothing connects alerts to tickets except a manual
button. There is no dedup, no auto-close, no outbound reset, and no rules UI.
Competing PSAs (ConnectWise, Autotask, Halo) treat all of this as table stakes.

## Goals

- Webhook-delivered alerts create tickets automatically per tenant-defined rules.
- Repeat firings of the same condition on the same device land on the existing
  open ticket instead of creating ticket storms.
- Alert resets close untouched tickets and annotate touched ones.
- Closing an alert-linked ticket resets the alert in the RMM (per-rule opt-out).
- Rules are manageable from the integration settings UI by admins.
- Alert events are first-class workflow v2 triggers; matched rules can notify users.
- Maintenance windows suppress alert ticketing for a client, asset, or
  integration during planned work, without losing the alerts.
- Scheduled polling reconciles missed webhooks: missed triggers become tickets,
  missed resets close stale tickets, and post-window still-active alerts get
  processed.
- One pipeline serves NinjaOne and TacticalRMM; a third provider only needs a
  normalizer and an optional outbound adapter.

## Non-goals

- RMM device-count billing integration, scheduled device sync, or org auto-matching.
- Per-rule dedup configuration (dedup behavior is fixed).
- Migrating NinjaOne device sync onto `sharedAssetIngestionService` (separate effort).

## Users and Primary Flows

- **MSP admin** configures alert rules per RMM integration: match conditions,
  ticket routing, lifecycle flags, notifications.
- **Dispatcher/tech** works alert tickets like any other ticket: sees occurrence
  comments on flapping conditions, sees resolution comments when alerts reset,
  and closing a ticket clears the alert in the RMM.
- **Automation builder** uses `RMM_ALERT_TRIGGERED`/`RMM_ALERT_RESOLVED`
  workflow triggers and the `rmm.alerts.create_ticket` action for custom flows.

## UX / UI Notes

- New "Alert Rules" section in RMM integration settings (next to the org-mapping
  manager), rendered for NinjaOne and TacticalRMM.
- Priority-ordered rules list: active toggle, reorder controls, edit/delete.
- Rule editor dialog with a Match group (severities, activity types, alert
  classes, source types, organization picker fed from
  `rmm_organization_mappings`, keywords, message regex) and an Actions group
  (create-ticket toggle, board picker, priority override, assignee,
  title/description templates with placeholder hints, auto-resolve toggle,
  reset-on-close toggle, notify-users picker).
- Save-time validation errors (e.g., bad regex) shown inline in the dialog.
- "Maintenance Windows" subsection beside Alert Rules: a list plus an editor
  with client/asset scope pickers and a one-off or weekly recurring schedule
  (with timezone).
- Alert polling enable/disable and interval (5–60 minutes) in integration
  settings.
- Existing per-asset `AssetAlertsSection` remains the alert-viewing surface.

## Requirements

### Functional Requirements

#### FR-1 Schema

One additive migration. `rmm_alerts` gains `activity_type`, `acknowledged_at`,
`acknowledged_by`, `dedup_key` (indexed with tenant + integration), `occurrence_count`
(default 1), `last_occurrence_at`, `matched_rule_id`, `auto_ticket_created`,
and `suppressed_by_window_id` (status gains a `suppressed` value).
Raw payloads standardize on the existing `metadata` jsonb. `rmm_alert_rules`
gains `conditions` and `actions` jsonb and drops the eleven flat filter/action
columns. New `rmm_maintenance_windows` table (FR-10). The deployed
`20251124000001` migration is not rewritten; no backfill.

#### FR-2 Contracts and normalizers

`shared/rmm/alerts/contracts.ts` defines `NormalizedRmmAlertEvent`
(kind: triggered | reset | acknowledged) and the optional per-provider
`RmmAlertOutboundAdapter` (`resetAlert`). Shared Zod schemas define the rule
`conditions`/`actions` shapes (see design doc for exact fields). NinjaOne and
TacticalRMM webhook routes map their payloads to the contract; existing webhook
auth, tenant resolution, and tier gating are unchanged.

#### FR-3 Ingest pipeline

`processRmmAlertEvent()` runs synchronously in the webhook request. Triggered:
upsert `rmm_alerts` on `(tenant, integration_id, external_alert_id)`, compute
and store `dedup_key` (device + condition identity; NinjaOne: `statusCode`
falling back to `activityType`), evaluate active rules first-match by
`priority_order` (empty conditions = catch-all; a rule that fails to evaluate
is logged and skipped), store `matched_rule_id`, then act. Replayed webhooks
are no-ops (idempotent ingest).

#### FR-4 Ticketing and dedup

If the matched rule creates tickets: an alert whose `dedup_key` matches an
alert with a still-open linked ticket joins that ticket (link, increment
`occurrence_count`, internal "re-triggered — Nth occurrence" comment).
Otherwise create a ticket honoring `boardId`, `priorityOverride` (else
severity→priority mapping), `assignToUserId`, and the title/description
templates with `{{device}}`/`{{message}}`/`{{severity}}`/`{{organization}}`
placeholders. Created tickets get source + source_reference, an asset
association, an initial internal comment, and client resolution from the asset
or the org mapping.

#### FR-5 Lifecycle

Reset marks the alert resolved. With a linked ticket and `autoResolveTicket`:
always comment; close (via `autoResolveStatusId`, else the tenant's first
is_closed status) only if the ticket is untouched — no human comments, no time
entries, no manual status change; rule auto-assignment doesn't count.
Acknowledged events stamp `acknowledged_at`/status. A ticket-closed event-bus
subscriber resets still-active linked alerts in the RMM via the provider's
outbound adapter when the matched rule's `resetAlertOnTicketClose` is true
(default). Outbound failures log and stamp alert `metadata`; they never block
the close. Providers without an adapter are skipped.

#### FR-6 Rules CRUD

List/create/update/delete/reorder server actions in `packages/integrations`,
admin-gated, Zod-validated, regex validated at save time.

#### FR-7 Rules UI

The settings section and editor described in UX notes, shared across providers.

#### FR-8 Workflow v2 and notifications

`RMM_ALERT_TRIGGERED` and `RMM_ALERT_RESOLVED` registered in the workflow v2
catalog with provider-generic payloads and published by the pipeline (replacing
the orphaned legacy-bus publishes). New `rmm.alerts.create_ticket` workflow
action invokes the shared ticket creator by alert ID. New `rmm-alert`
notification category delivers in-app + email to a matched rule's
`notifyUserIds`, honoring per-user preferences.

#### FR-9 Hardening and cleanup

Implement CSRF validation in the NinjaOne OAuth callback. Move
`ninjaone/alerts/*` logic into the shared module and remove the superseded
`resetInNinjaOne` TODO. Deprecate `rmm_organization_mappings.auto_create_tickets`
(no read paths remain).

#### FR-10 Maintenance windows

New `rmm_maintenance_windows` table: optional `integration_id`/`client_id`/
`asset_id` scopes (null = all of that dimension), one-off `starts_at`/`ends_at`
or weekly `recurrence` jsonb (days, time range, timezone), `name`, `is_active`.
The pipeline checks windows before rule matching: an alert matching all
non-null scopes of an active window at its `occurredAt` is stored with
`status = 'suppressed'` and `suppressed_by_window_id` — no ticket, no
notifications, no workflow events. A reset for a suppressed alert resolves it
quietly. Window CRUD server actions are admin-gated and Zod-validated, with the
settings UI described in UX notes.

#### FR-11 Alert polling (reconciliation)

A per-integration Temporal scheduled workflow (Entra per-tenant schedule
pattern), default on for connected integrations, every 15 minutes (configurable
5–60), created on connect and removed on disconnect. Each cycle, through the
same pipeline: (1) upsert RMM-active alerts missing locally as `triggered`
events; (2) synthesize `reset` events for local active alerts no longer active
in the RMM; (3) process still-active suppressed alerts whose window ended
through the normal rules path. Webhooks stay primary; ingest idempotency makes
overlap harmless.

### Non-functional Requirements

- Webhook ingest path makes no external API calls; webhook latency stays
  bounded. RMM API calls happen only in the poller and the ticket-close
  subscriber, both off the request path.
- All queries tenant-scoped (CitusDB composite keys: `tenant` + entity id).
- Webhook response semantics preserved: 200 unmapped org, 200 success, 500
  unexpected error (RMM retries; ingest idempotency makes this safe).

## Data / API / Integrations

See FR-1 for schema and the design doc for exact JSONB shapes. External APIs:
`NinjaOneClient.resetAlert()` (exists); TacticalRMM alert resolution if its API
supports it (open question in SCRATCHPAD — adapter is optional by design).

## Security / Permissions

- Rule CRUD requires admin permission; all actions tenant-scoped.
- Webhook auth unchanged (HMAC signature / shared-secret header).
- OAuth callback CSRF validation (FR-9).

## Observability

Pipeline logs rule-evaluation skips and outbound reset failures. No new
metrics/monitoring infrastructure.

## Rollout / Migration

Single additive migration; deployed tables hold negligible data, so no
backfill. No feature flag: with zero rules configured, the pipeline stores
alerts without creating tickets, which matches today's effective behavior.

## Open Questions

Tracked in `SCRATCHPAD.md` (Tactical outbound capability; exact ticket-closed
event name).

## Acceptance Criteria (Definition of Done)

- A NinjaOne CONDITION TRIGGERED webhook for a mapped org creates an
  `rmm_alerts` row and, when a rule matches, a correctly-routed ticket.
- The same condition re-firing while that ticket is open adds an occurrence
  comment and creates no new ticket; after the ticket closes, a new firing
  creates a new ticket.
- CONDITION RESET resolves the alert, comments the ticket, and closes it only
  if untouched.
- Closing an alert-linked ticket resets the alert in NinjaOne unless the rule
  opted out.
- A TacticalRMM alert webhook flows through the same pipeline end to end.
- Admins manage rules entirely from the settings UI; invalid rules are rejected
  at save time.
- Alert workflows can trigger on `RMM_ALERT_TRIGGERED`/`RMM_ALERT_RESOLVED`
  and call `rmm.alerts.create_ticket`.
- Matched rules with `notifyUserIds` produce in-app and email notifications per
  user preference.
- An alert firing inside a matching maintenance window creates no ticket and no
  notifications; the same alert outside the window processes normally; a
  condition still firing after its window ends becomes a ticket via the poller.
- With webhooks disabled, a poll cycle turns RMM-active alerts into tickets per
  the rules and closes stale tickets whose alerts cleared in the RMM.
- All features in `features.json` implemented; the automated core in
  `tests.json` passes; the `SMOKE_TESTS.md` checklist has been executed against
  a live stack.