PSA/docs/plans/2026-06-12-rmm-alert-handling/SCRATCHPAD.md

Scratchpad — RMM Alert Handling

• Plan slug: 2026-06-12-rmm-alert-handling
• Created: 2026-06-12

What This Is

Working memory for the RMM alert handling effort. Approved design lives at docs/plans/2026-06-12-rmm-alert-handling-design.md (commit c717a4fd50).

Decisions

• (2026-06-11) Dedup is one open ticket per (device, condition). Repeats append a comment and bump occurrence_count. Always on, not per-rule configurable.

• (2026-06-11) Auto-close on alert reset: always comment; close only if no human touched the ticket (no human comments, no time entries, no manual status change). Rule-driven auto-assignment does not count as touched.

• (2026-06-11) Outbound reset on ticket close is a per-rule actions.resetAlertOnTicketClose flag, default true.

• (2026-06-11) Pipeline is provider-generic in shared/rmm/alerts/; NinjaOne and TacticalRMM both wired this branch.

• (2026-06-11) Processing runs synchronously in the webhook request (Approach A). All ingest work is local DB; the only external call (outbound reset) happens on the ticket-close bus subscriber.

• (2026-06-11) Schema direction: rules use JSONB conditions/actions (the alertProcessor model wins). One additive corrective migration; the deployed 20251124000001 migration is never rewritten. Deployed data is negligible — no backfill.

• (2026-06-11) Raw alert payload standardizes on the existing metadata jsonb column; code that wrote source_data changes to metadata.

• (2026-06-11) rmm_organization_mappings.auto_create_tickets is deprecated; rules with organizationIds conditions are the single source of truth.

• (2026-06-12) Maintenance windows and alert polling added to scope (originally non-goals) for competitor parity. Windows suppress before rule matching; suppressed alerts are stored but produce no ticket/notification/workflow event. The reconciliation poller owns window-end processing of still-active suppressed alerts.

• (2026-06-12) Polling is a per-integration Temporal schedule (Entra per-tenant pattern): default on, 15-minute default interval, 5–60 configurable, created on connect / removed on disconnect. Cycles upsert missed triggers and synthesize resets for stale active alerts, all through the same pipeline.

• (2026-06-12) Merged origin/main (443 commits) before implementing — the snapshot was stale: main added Huntress + Level.io providers, redesigned the RMM settings page to master-detail, and fixed Tactical/Level alert writers to be (mostly) schema-compliant. Local .env.localtest (wirein to the alga-psa-local-test stack, port 5472) preserved through the merge.

• (2026-06-12) There are now FOUR rmm_alerts writers: NinjaOne (still writes nonexistent source_data), Tactical (still writes source_data), Level.io (compliant), Huntress (compliant but raw severity/status values). Huntress has a complete incident→ticket path (incidentPlan/incidentProcessor/ticketCreator, pg-boss polling) configured via rmm_integrations.settings — leave it intact this branch; the shared pipeline wires NinjaOne + Tactical (+ Level.io if cheap). Folding Huntress in is a follow-up.

• (2026-06-12) Ticket facts from main: tickets.source exists; source_reference does NOT — store in attributes JSONB (Huntress convention). Ticket creator should take the caller's trx (Huntress ticketCreator pattern, not NinjaOne's self-managed one). Internal notes: comment_threads + comments with is_internal/is_system_generated (helper addTicketInternalNote). Closed statuses: statuses where item_type='ticket' and is_closed=true. TICKET_CLOSED is published via publishWorkflowEvent at packages/tickets/src/actions/ticketActions.ts:1062 (also optimizedTicketActions.ts:2508).

• (2026-06-12) RMM_ALERT_TRIGGERED/RESOLVED Zod schemas already exist (packages/event-schemas/src/schemas/eventBusSchema.ts:966-981) but are not in system_event_catalog (needs a registration migration, pattern: 20250130201000_register_email_system_events.cjs) and not published anywhere via publishWorkflowEvent.

• (2026-06-12) Shared import alias: @alga-psa/shared/* → shared/* (server + ee/server tsconfigs). Settings UI: providerSettingsComponents map in packages/integrations/src/components/settings/integrations/RmmIntegrationsSetup.tsx; per-provider detail panes are where Alert Rules / Maintenance Windows sections plug in.

• (2026-06-12) Testing strategy is 80/20: tests.json holds a 32-test automated core (logic permutations, idempotency, lifecycle, tenant isolation, one E2E per direction); UI, live RMM round-trips, Temporal schedule lifecycle, email, and migrations are manual flows in SMOKE_TESTS.md, each tied to a named business risk. The old 114-test list was consolidated, not expanded — table-driven tests absorb the per-permutation entries.

• (2026-06-12, Robert's call) Reversed the pg-boss deviation: both RMM alert reconciliation AND Huntress incident polling now run as per-integration Temporal schedules (Entra pattern). Key enabler discovered during investigation: ee/temporal-workflows imports the full ee/server tree via its @ee/@/ tsconfig aliases (the ninjaone-token-refresh activity precedent), so the activities are thin wrappers over the existing poll logic — no duplication. setupSchedules.ts reconciles schedules at worker boot; activities re-check is_active/polling-enabled per run so stale schedules no-op; NinjaOne connect/disconnect ensure/remove dynamically (ee/server/src/lib/integrations/rmm/alertPollingSchedule.ts). Caveats: interval changes saved via updateRmmAlertPollingSettings apply at next worker boot (the action lives in packages/ and can't reach the Temporal client); Tactical/Huntress schedule creation after a fresh connect also waits for worker boot (manual "poll now"/backfill covers the gap). CE deployments without Temporal workers get no polling — webhooks remain primary.

• (2026-06-12, Robert's catch) Final polling architecture: the per-integration polls ride the IJobRunner abstraction (packages/jobs interfaces; PgBossJobRunner CE / TemporalJobRunner EE, selected by JOB_RUNNER_TYPE else isEnterprise) — which I'd initially missed because the older IJobScheduler (pg-boss-only) looked like "the" scheduler abstraction. Same handler code both editions (server/src/lib/jobs/handlers/rmmAlertPollingHandlers.ts, registered in registerAllHandlers + initializeJobHandlersForWorker); cron intervals; reconcileRmmPollingSchedules() control loop (5-min tick on the legacy scheduler in initializeApp + boot pass + NinjaOne connect/disconnect hooks) converges jobs onto rmm_integrations state, so interval/toggle changes apply within minutes hands-off. TemporalJobRunner.scheduleRecurringJob fixed to update existing schedule specs. The bespoke temporal workflows/setupSchedules blocks and the ee alertPollingSchedule helper from the interim iteration are deleted; CE stubs added for the ninjaone fetcher + huntress poller dynamic @enterprise imports. CE Tactical polling restored.

Discoveries / Constraints

• (2026-06-11) Alert ingestion is broken on main today: webhookHandler.ts:671-674 inserts activity_type and source_data into rmm_alerts, but the only migration creating that table (server/migrations/20251124000001_create_rmm_integration_tables.cjs) has neither column.
• (2026-06-11) alertProcessor.ts reads JSONB conditions/actions from rmm_alert_rules; the migration created flat text[]/scalar columns instead. The processor is imported only by tests — never called from the webhook path.
• (2026-06-11) ticketCreator.ts works (manual button in AssetAlertsSection.tsx uses it) and is the basis for the shared creator.
• (2026-06-11) TacticalRMM's webhook (server/src/app/api/webhooks/tacticalrmm/route.ts) writes rmm_alerts directly, including the nonexistent columns — same schema bug.
• (2026-06-11) NinjaOne condition identity for dedup: statusCode, falling back to activityType. NinjaOne sends a fresh CONDITION TRIGGERED per firing, so a flapping check fires many times a day.
• (2026-06-11) NinjaOne webhook returns 200 for unmapped orgs (suppresses retries), 500 for unexpected errors (NinjaOne retries). Keep this; idempotent ingest makes at-least-once safe.
• (2026-06-11) NinjaOneClient.resetAlert() exists (POST /api/v2/alert/{uid}/reset); the only caller today is the ninjaone.alerts.reset workflow action.
• (2026-06-11) Known TODOs in adjacent code: CSRF validation in the NinjaOne OAuth callback; resetInNinjaOne in alertProcessor.resolveAlert() (superseded by the outbound adapter).
• (2026-06-11) Precedent for shared provider-agnostic RMM code: shared/rmm/contracts.ts + shared/rmm/sharedAssetIngestionService.ts (used by Tanium and Tactical device sync). The alert pipeline mirrors this layout.
• (2026-06-11) Legacy-bus events RMM_ALERT_TRIGGERED/RMM_ALERT_RESOLVED are published today but have no subscribers and are not in the workflow v2 catalog. Workflow v2 currently only sees INTEGRATION_WEBHOOK_RECEIVED.

Implementation status (2026-06-12, mid-flight)

Done and committed: corrective migration (20260612090000) + event-catalog (…0100) and notification (…0200) migrations; shared pipeline in shared/rmm/alerts (contracts, evaluator, windowMatcher, dedup, ticketCreator, untouched, processRmmAlertEvent, reconciliation, createTicketForAlertId, outboundRegistry); NinjaOne/Tactical/Level webhooks normalized into it; TICKET_CLOSED outbound-reset subscriber + NinjaOne adapter (CE stub in packages/ee); rmmAlertNotificationSubscriber (in-app + email); rules+windows CRUD actions (packages/integrations/src/actions/integrations/rmmAlertRuleActions.ts); rmm.alerts.create_ticket workflow action; pg-boss reconciliation dispatcher (EE init, NinjaOne fetcher) — deviation: pg-boss instead of design-doc Temporal (Huntress precedent, CE compatibility; F083 connect/disconnect becomes "dispatcher polls only active integrations"); OAuth CSRF fix (ninjaone_oauth_state tenant secret, one-time use); legacy ee ninjaone alertProcessor/ticketCreator deleted, manual button + workflow action use shared createTicketForAlertId.

Update (2026-06-12, later): FR-7 UI shipped (RmmAlertAutomationSettings in packages/integrations, rendered in Tactical/NinjaOne/Level panes; asset-scope picker for windows deferred — F076 false). Tactical reconciliation fetcher shipped (reuses the backfill-verified alerts endpoint; backfill now runs a reconciliation cycle). Test status: unit suites (rule matrix, window matcher, dedup, schemas) + adapted tactical webhook/backfill tests green (70/71 in tactical+rmmalerts — the 1 failure, tacticalDeviceSync.fullSync, is pre-existing on merged main, unrelated); DB-backed integration suite (rmmAlertPipeline, 10/10) proves the migration + pipeline spine. tests.json: 27/32 automated (subscriber, reconciliation, untouched branches, normalizer, events, severity fallback all landed); the remaining 5 (T026/T027 CRUD action tests, T029 workflow action, T030 OAuth CSRF route, T031 notification subscriber) need bespoke harnesses and are the only un-automated items; the pre-existing tacticalDeviceSync.fullSync unit failure was inherited from main (fails without this branch's changes). Integration tests need env: DB_HOST=localhost DB_PORT=5472 DB_USER_ADMIN=postgres DB_PASSWORD_ADMIN=$(cat secrets/postgres_password).

Previously remaining: FR-7 settings UI (Alert Rules + Maintenance Windows + polling settings section in provider panes — providerSettingsComponents map in packages/integrations RmmIntegrationsSetup.tsx); test core T001–T032 (old tactical webhook unit tests will need adapting to the pipeline; old ee alertProcessor tests reference deleted module — replace with shared-module tests); flip features.json/tests.json implemented flags; update design doc for the pg-boss deviation + id-space caveat.

Verification caveats for smoke testing: NinjaOne webhook external ids are activity ids while the alerts API returns uids — reconciliation only trusts poller-ingested ids for staleness (RECONCILIATION_INGEST_MARKER in metadata) and dedup absorbs cross-source duplicates; verify uid/id behavior against a live sandbox. Tactical/Level reconciliation fetchers intentionally deferred until their list-alerts API shapes are verified live (F082 open).

Commands / Runbooks

• Run server migrations: from server/, npx knex migrate:latest (see existing env scripts; use the worktree's compose stack via the alga-env-manager skill).
• Webhook entry for local testing: POST /api/webhooks/ninjaone?tenant=<tenantId> with X-Alga-Webhook-Secret header (secret in rmm_integrations.settings.webhookSecret).

Links / References

• Design doc: docs/plans/2026-06-12-rmm-alert-handling-design.md
• Migration with current (broken) schema: server/migrations/20251124000001_create_rmm_integration_tables.cjs
• NinjaOne webhook: ee/server/src/app/api/webhooks/ninjaone/route.ts, handler ee/server/src/lib/integrations/ninjaone/webhooks/webhookHandler.ts
• Rules engine to move: ee/server/src/lib/integrations/ninjaone/alerts/alertProcessor.ts
• Ticket creator to move: ee/server/src/lib/integrations/ninjaone/alerts/ticketCreator.ts
• Tactical webhook: server/src/app/api/webhooks/tacticalrmm/route.ts
• Shared device-ingest precedent: shared/rmm/sharedAssetIngestionService.ts
• Provider registry (capability flags): packages/integrations/src/lib/rmm/providerRegistry.ts
• Asset alert UI: ee/server/src/components/assets/AssetAlertsSection.tsx
• NinjaOne client (resetAlert): ee/server/src/lib/integrations/ninjaone/ninjaOneClient.ts

Open Questions

• Does TacticalRMM's API expose alert resolution for the outbound adapter? If not, ship the adapter for NinjaOne only and mark the capability off for Tactical (pipeline skips it cleanly).
• Exact event-bus event name for ticket closure (TICKET_UPDATED with closed status vs. a dedicated TICKET_CLOSED) — confirm against server/src/lib/eventBus/ when wiring the subscriber.
• Confirm Tactical's alerts API supports listing active alerts for reconciliation (NinjaOne's getAlerts() already exists in the client).
• FR-11 scheduler choice: the design says Temporal (Entra pattern), but Huntress's incident poller uses pg-boss (server-side, CE-compatible — Tactical is a CE provider and CE deployments may not run Temporal workers). Decide at FR-11; pg-boss looks like the better precedent. Flag the deviation to Robert either way.
• Huntress provider registry flags are stale (deviceSync/events false despite reality) — out of scope here, but worth a drive-by fix or follow-up.
• Reconciliation poller and resolveAlert semantics: a poller-synthesized reset should be distinguishable in the ticket comment ("alert no longer active in RMM" vs. "alert reset received").

Smoke run (2026-06-12) — all 8 runbook flows PASS after fixes

Executed /tmp/rmm-alert-smoke-tests.md end-to-end (algadev, local-test stack, EE + JOB_RUNNER_TYPE=pgboss). Bugs found and fixed on this branch:

• statuses.item_type drift (4 call sites): main's board-scoped statuses migration left item_type NULL; live data uses status_type + board_id. Fixed ticketCreator (now delegates to TicketModel.getDefaultStatusId), resolveCloseStatusId, untouched.ts, the rule-form closedStatuses query, and the Huntress ticket creator. Symptoms: webhook 500 on first trigger; auto-close never closed; untouched check ignored status moves.
• Dedup primary selection: sibling lookup ordered created_at desc, bumping the newest absorbed row instead of the ticket-owning primary. Now asc; occurrence counts and "occurrence N" comments track the primary.
• Severity→priority: exact-name match missed "P1 - Critical"-style names; added substring fallback pass.
• Polling settings first save was a no-op: jsonb_set can't create the alertPolling parent key; now merges jsonb_build_object into the parent.
• Reconciler tick never recurred: legacy IJobScheduler.scheduleRecurringJob is a one-shot delayed send (cron coerced to '24 hours', singletonHours 24, no re-fire). The tick is now a process-local 5-min setInterval in initializeApp (+ boot pass); reconcile is idempotent so multi-replica ticks are safe.
• PgBossJobRunner recurring-record lifecycle: every cron fire flipped the schedule's jobs row to completed (runs share jobServiceId), so the reconciler saw "no job" → recreated rows while enabled and leaked the pgboss schedule on disable. Fires now carry jobRecurring: true and the worker returns recurring rows to queued; cancelJob exempts recurring records from the completed/failed guard, unschedules, and nulls external_id (the live-schedule pointer the reconciler now also filters on). Ineligible-branch cancels via newest record of any status.
• Rule dialog stale patternError: dialog component stays mounted; a cancelled invalid-regex edit blocked the next Add rule. Reset on isOpen.
• Button ids: 14 missing required id props in RmmAlertAutomationSettings.
• Integration test seed: statuses now board-scoped; 20/20 green again.

Environment notes for future smoke runs: DB_NAME_SERVER is required by migrate:ee (else knex hits the default postgres DB — an accidental full migration run was left there; safe to drop that DB's public schema); Tactical row needs NEXT_PUBLIC_FORCE_FEATURE_FLAGS=tactical-rmm-integration:true; EE without Temporal needs JOB_RUNNER_TYPE=pgboss; seed tenant had no closed ticket status (marked "Enchanted Closure" is_closed=true); Test Connection (GET /api/beta/v1/client/) is what activates the integration. Tactical backfill button label is "Sync Alerts".

Validation: rmmalerts unit 40/40; tactical unit 36/37 (1 pre-existing main failure); rmmAlertPipeline integration 20/20; tsc clean on touched packages. Flow 6 observed live: interval change converged ~3 min, disable unscheduled, pre-fix leaked schedule self-healed on first tick.

Deferred: "run workflow" as an alert-rule action (2026-06-12)

Considered and deliberately held off. Coverage today: workflows can already trigger on RMM_ALERT_TRIGGERED/RESOLVED (system event catalog) and on ticket creation, and can call rmm.alerts.create_ticket; the rule action would mainly add rule-level filtering as the trigger plus rule context in the payload.

If revisited, the design direction we converged on: run-workflow as an independent action toggle (composes with create-ticket; "replace" = create ticket off + workflow on), launched fire-and-forget via launchPublishedWorkflowRun after the processing transaction commits. Two gaps must be closed for replace mode to be safe: (1) storm protection without a ticket — dedup should treat an active same-dedup-key alert as an occurrence even before any ticket exists, else flap storms launch one workflow per firing during the async gap; (2) lifecycle hooks — workflow- created tickets keep auto-resolve/reset-on-close only when created through rmm.alerts.create_ticket (which maintains rmm_alerts.ticket_id). The transactional-vs-eventual ticket guarantee is the documented trade-off.