PSA/ee/docs/plans/2026-03-26-ninjaone-proactive-token-refresh/PRD.md

PRD — NinjaOne Proactive Token Refresh

• Slug: ninjaone-proactive-token-refresh
• Date: 2026-03-26
• Status: Draft

Summary

Add per-integration proactive NinjaOne OAuth token refresh scheduling through Temporal so connected NinjaOne integrations refresh access and refresh tokens before expiry instead of waiting for a user-triggered sync or webhook processing path to hit the expired token.

Problem

NinjaOne credentials are currently refreshed lazily inside the API client when a request notices the token is near expiry or when a request receives a 401. This means:

• the first user-visible action after expiry pays the refresh cost;
• refresh-token failures surface during organization/device syncs instead of being handled as background maintenance;
• there is no dedicated lifecycle owner for NinjaOne token refreshes in Temporal;
• failures are hard to distinguish from sync failures until worker logs are inspected.

Recent production evidence showed a Temporal organization sync reaching the worker successfully, then failing while refreshing the NinjaOne token at https://ca.ninjarmm.com/oauth/token with 400 Bad Request and error: invalid_token. That proves the current path does attempt refresh, but only on demand and too late for a good operator or user experience.

Goals

• Refresh NinjaOne OAuth credentials proactively before expires_at using Temporal worker-owned execution.
• Schedule refreshes per integration, not via a global polling scanner.
• Persist newly rotated access tokens and refresh tokens after each successful refresh.
• Reschedule the next refresh automatically after each successful refresh.
• Keep current lazy refresh logic as a fallback path if a scheduled run is missed.
• Make refresh failure state explicit enough that operators and future code can distinguish reconnect-required credentials from ordinary sync failures.

Non-goals

• Replacing the existing lazy refresh logic in the NinjaOne client.
• Building a generic cross-provider token lifecycle framework in this scope.
• Adding a full user-facing token-health dashboard.
• Introducing a broad periodic scanner over all integrations.
• Auto-reconnecting or auto-reauthorizing NinjaOne after a terminal refresh-token failure.

Users and Primary Flows

1. Connected tenant with active NinjaOne integration

• OAuth callback stores credentials and marks the integration active.
• The system schedules one delayed Temporal refresh workflow for that integration before token expiry.

2. Background refresh lifecycle

• The delayed workflow wakes up before expiry.
• The worker loads current NinjaOne credentials, refreshes them through the NinjaOne OAuth token endpoint, persists the rotated tokens, and computes the next refresh time.
• The worker schedules the next one-off refresh workflow for the same integration.

3. Failure and reconnect flow

• If refresh fails with a retryable infrastructure error, the workflow retries according to Temporal activity/workflow policy.
• If refresh fails with a non-retryable token/provider error such as invalid_token, the integration is marked as requiring reconnect and no further future refresh is scheduled until a reconnect or manual recovery path resets the lifecycle.
• User-triggered syncs still use lazy refresh fallback, but should usually find a fresh token already present.

4. Disconnect / reconnect flow

• Disconnecting NinjaOne cancels or invalidates future scheduled refreshes for that integration.
• Reconnecting NinjaOne creates a new valid credential set and seeds a new proactive refresh schedule.

UX / UI Notes

• No new user-facing page is required in this scope.
• Existing sync flows should fail less often for expired tokens because refresh should already have happened in the background.
• When a refresh token is invalid and the integration needs reconnect, server actions should continue to return a clear reconnect-style error rather than a generic sync failure where practical, but a broader UI redesign is not part of this scope.

Requirements

Functional Requirements

• Introduce a dedicated NinjaOne token refresh workflow/activity in Temporal.
• Schedule one delayed refresh workflow per active NinjaOne integration using the credential expires_at value and a configurable safety buffer.
• Seed or reschedule that delayed workflow when:
  • OAuth callback stores fresh credentials,
  • a proactive refresh succeeds,
  • a lazy refresh succeeds in the client.
• Ensure only one future proactive refresh is considered active for a given integration at a time.
• Refresh logic must reload the latest stored credential set at execution time rather than trusting stale workflow input.
• On successful refresh, persist:
  • new access token,
  • new refresh token,
  • new expiry timestamp,
  • unchanged instance URL unless the provider response or current stored credentials require otherwise.
• On terminal provider/token failure, record reconnect-required state in integration-owned metadata and stop automatic rescheduling until the integration is reconnected or explicitly reset.
• Disconnecting NinjaOne must cancel, invalidate, or safely no-op any in-flight future refresh workflow for that integration.
• Reconnecting NinjaOne must replace stale lifecycle state and create a fresh future refresh schedule.
• Existing organization/device sync and webhook-triggered client calls must keep the current lazy refresh fallback path.
• Refresh scheduling and execution must emit structured logs with tenant, integration, workflow identity, schedule target time, attempt outcome, and provider error payload details where safe.

Non-functional Requirements

• Scheduling must be precise enough that refresh normally occurs before expiry with reasonable clock skew tolerance.
• The design must avoid a global high-frequency poller over all NinjaOne integrations.
• The implementation must be idempotent under duplicate workflow starts, repeated reconnects, or retries.
• Workflow ownership and cleanup semantics must survive worker restarts and deploys without orphaning endless refresh loops.

Data / API / Integrations

• Current NinjaOne credentials live in the tenant secret ninjaone_credentials and contain:
  • access_token
  • refresh_token
  • expires_at
  • instance_url
• Current rmm_integrations rows do not store OAuth expiry directly. This plan should store schedule/lifecycle metadata in provider settings or another integration-owned persistence field that is available without reading secrets for every UI/status read.
• The proactive refresh workflow should use the same NinjaOne OAuth refresh contract already used by the client:
  • POST {instanceUrl}/oauth/token
  • grant_type=refresh_token
  • refresh_token
  • client_id
  • client_secret
• The workflow should run on the existing app Temporal worker/task queue used for NinjaOne sync workflows unless a more specific queue is already required by runtime conventions.

Security / Permissions

• Do not duplicate raw tokens into rmm_integrations or other broadly-readable tables.
• Any status or lifecycle metadata persisted outside secrets must exclude access tokens and refresh tokens.
• Failure logs should capture provider error codes and safe response body fragments, but must not log secret values or full request bodies containing credentials.

Observability

• Log schedule creation/reschedule/cancel decisions with tenant and integration IDs.
• Log workflow execution start with tenant, integration, scheduled refresh target, and current token expiry.
• Log successful refresh completion with old/new expiry timestamps and next scheduled refresh time.
• Log terminal failure with provider status, provider error body, and whether the integration was marked reconnect-required.
• Reuse existing integration token lifecycle events where they fit, and add a NinjaOne-specific refresh-scheduled/refreshed signal only if needed for implementation clarity.

Rollout / Migration

• Implement the workflow and scheduling path without removing lazy refresh.
• Backfill existing active NinjaOne integrations by seeding a future refresh workflow from their currently stored secret expiry.
• Treat integrations missing credentials or missing expiry as unschedulable and surface that as reconnect-required or configuration error rather than silently looping forever.
• Deploy with conservative scheduling buffer and validate on one integration before broad production reliance.

Open Questions

• Whether schedule/lifecycle metadata should live in rmm_integrations.settings or in a dedicated table for token lifecycle state.
• Whether a terminal invalid_token refresh error should update sync_error, a new reconnect-required field in settings, or both.
• Whether disconnected integrations should actively cancel existing Temporal handles or rely on workflow/activity guards plus idempotent no-op behavior.

Acceptance Criteria (Definition of Done)

• A newly connected NinjaOne integration automatically gets a future proactive refresh workflow scheduled before token expiry.
• A successful proactive refresh rotates and persists credentials, then schedules the next future refresh without user action.
• Existing active integrations can be seeded into the proactive schedule lifecycle after rollout.
• Lazy refresh remains available and continues to work as a fallback for missed schedules.
• A terminal refresh-token failure is recorded as reconnect-required state and no longer appears as an opaque sync-only failure.
• Disconnect and reconnect flows do not leave duplicate or stale future refresh executions for the integration.