PSA/ee/docs/plans/2026-03-26-ninjaone-proactive-token-refresh/SCRATCHPAD.md

Scratchpad — NinjaOne Proactive Token Refresh

• Plan slug: ninjaone-proactive-token-refresh
• Created: 2026-03-26

What This Is

Rolling notes for implementing per-integration proactive NinjaOne OAuth token refresh scheduling through Temporal.

Decisions

• (2026-03-26) Use per-integration delayed Temporal refresh scheduling rather than a global scanner.
• (2026-03-26) Keep existing lazy refresh in NinjaOneClient as fallback; proactive refresh reduces user-facing failures but does not replace request-time safety checks.
• (2026-03-26) Treat terminal provider responses like invalid_token as reconnect-required lifecycle failures, not ordinary sync failures.
• (2026-03-26) Store proactive lifecycle state in rmm_integrations.settings.tokenLifecycle (non-secret metadata only) with a monotonic scheduleNonce and activeWorkflowId so stale scheduled workflows safely no-op.
• (2026-03-26) Use one-off delayed workflows (startDelay) on the existing TEMPORAL_JOB_TASK_QUEUE (alga-jobs default) and terminate prior active workflow handles during reschedule to keep at most one active future refresh.
• (2026-03-26) Consider NinjaOne refresh failures terminal when provider response indicates invalid refresh token (400 invalid_token / invalid_grant), mark reconnect-required, and stop rescheduling until reconnect.

Discoveries / Constraints

• (2026-03-26) Current NinjaOne credentials are stored only in tenant secret storage under ninjaone_credentials; rmm_integrations does not currently store OAuth expiry.
• (2026-03-26) Production logs from msp/temporal-worker-868df5f5fb-g2744 showed a new organization sync starting at 2026-03-26T19:00:19.652Z.
• (2026-03-26) Production logs from msp/temporal-worker-868df5f5fb-jbrx9 showed the sync failing during token refresh at 2026-03-26T19:00:20.071Z with 400 Bad Request, ERR_BAD_REQUEST, and data: { error: 'invalid_token' } from https://ca.ninjarmm.com/oauth/token.
• (2026-03-26) That proves NinjaOne refresh is already attempted inside the Temporal worker during sync execution, but only on demand.
• (2026-03-26) The current code already emits INTEGRATION_TOKEN_EXPIRING and INTEGRATION_TOKEN_REFRESH_FAILED, but no code currently schedules a future refresh off those signals.
• (2026-03-26) ee/temporal-workflows imports ee/server NinjaOne integration code at runtime; avoid @/ aliases in shared NinjaOne modules and prefer relative imports so temporal workspace compilation can resolve modules consistently.
• (2026-03-26) Local test/typecheck execution is currently environment-limited by missing workspace deps (e.g. pathe, knex, @temporalio/* resolution in this checkout), so verification is restricted to static inspection and targeted test file creation in this run.

Commands / Runbooks

• (2026-03-26) Confirm current NinjaOne refresh implementation:
  • rg -n "refreshAccessToken\\(|grant_type: 'refresh_token'|oauth/token" ee/server/src/lib/integrations/ninjaone/ninjaOneClient.ts
• (2026-03-26) Confirm NinjaOne sync workflow entrypoints:
  • rg -n "ninjaOneSyncWorkflow|syncOrganizations" ee/server/src/lib/integrations/ninjaone/sync ee/temporal-workflows/src/workflows ee/temporal-workflows/src/activities
• (2026-03-26) Inspect recent MSP Temporal worker logs:
  • kubectl -n msp logs temporal-worker-868df5f5fb-g2744 -c temporal-worker --since=10m --timestamps
  • kubectl -n msp logs temporal-worker-868df5f5fb-jbrx9 -c temporal-worker --since=10m --timestamps
• (2026-03-26) Validate plan artifacts:
  • python3 /Users/roberisaacs/.codex/skills/alga-plan/scripts/validate_plan.py ee/docs/plans/2026-03-26-ninjaone-proactive-token-refresh
• (2026-03-26) Validate new proactive scheduling implementation references:
  • rg -n "ninjaone-token-refresh|ninjaOneProactiveTokenRefreshWorkflow|proactiveNinjaOneTokenRefreshActivity|scheduleNinjaOneProactiveRefresh" ee/server/src ee/temporal-workflows/src
• (2026-03-26) Attempt targeted tests (blocked by missing local dependency graph):
  • cd ee/server && npm run test:unit -- src/__tests__/unit/ninjaoneProactiveRefresh.schedule.test.ts src/__tests__/unit/ninjaOneClient.baseUrl.test.ts
  • cd ee/temporal-workflows && npm run test -- src/__tests__/worker-registration.test.ts
• (2026-03-26) Attempt type checks (blocked by pre-existing missing modules in workspace bootstrap):
  • cd ee/server && npm run typecheck
  • cd ee/temporal-workflows && npm run type-check

Implementation Notes

• (2026-03-26) Added new server lifecycle module: ee/server/src/lib/integrations/ninjaone/proactiveRefresh.ts
  • Computes refresh target using configurable buffer/min-delay.
  • Starts delayed Temporal workflow per integration and records lifecycle metadata in settings.tokenLifecycle.
  • Reloads latest credentials on execution, refreshes token, persists rotated credentials, updates lifecycle metadata, and reschedules next refresh.
  • Marks terminal failures reconnect-required and prevents further scheduling until reconnect.
  • Marks unreadable/missing credentials unschedulable with explicit lifecycle failure metadata.
• (2026-03-26) Added dedicated Temporal pair:
  • Workflow: ee/temporal-workflows/src/workflows/ninjaone-token-refresh-workflow.ts
  • Activity: ee/temporal-workflows/src/activities/ninjaone-token-refresh-activities.ts
  • Exported from workflow/activity indexes and worker-registration coverage.
• (2026-03-26) Wired scheduling hooks:
  • OAuth callback now clears reconnect-required state and seeds proactive refresh schedule after successful connect/reconnect.
  • Lazy refresh success in NinjaOneClient.refreshAccessToken now reschedules proactive workflow.
  • Disconnect action now cancels/inactivates pending proactive refresh workflow lifecycle.
• (2026-03-26) Added unit test coverage:
  • ee/server/src/__tests__/unit/ninjaoneProactiveRefresh.schedule.test.ts verifies delayed scheduling on connect path semantics and previous-workflow termination during reschedule.
• (2026-03-26) Added proactive execution/backfill tests:
  • ee/server/src/__tests__/unit/ninjaoneProactiveRefresh.execution.test.ts
    • verifies runtime credential reload from secret storage;
    • verifies rotated credential persistence;
    • verifies successful proactive refresh reschedules next workflow and increments lifecycle nonce;
    • verifies terminal invalid_token marks reconnect-required and avoids further scheduling;
    • verifies missing credentials become unschedulable without reschedule loop;
    • verifies integration settings lifecycle metadata excludes raw token material.
  • ee/temporal-workflows/src/schedules/__tests__/setupSchedules.ninjaone-backfill.test.ts
    • verifies rollout backfill seeding runs for active integrations lacking lifecycle ownership and skips reconnect-required/already-owned lifecycle rows.
• (2026-03-26) Added lazy fallback schedule handoff test:
  • ee/server/src/__tests__/unit/ninjaOneClient.proactiveSchedule.test.ts
    • verifies successful lazy NinjaOne token refresh triggers proactive reschedule (source: lazy_refresh_success) for the same integration.
• (2026-03-26) Expanded proactive suite coverage:
  • ee/server/src/__tests__/unit/ninjaoneProactiveRefresh.execution.test.ts
    • includes inactive integration no-op assertions for disconnect protection semantics.
  • ee/server/src/__tests__/unit/ninjaoneProactiveRefresh.schedule.test.ts
    • includes reconnect lifecycle reset + fresh schedule seeding assertions.
  • ee/temporal-workflows/src/workflows/__tests__/ninjaone-token-refresh-workflow.test.ts
    • verifies structured workflow start/success logs contain tenant/integration/schedule context.
• (2026-03-26) Added rollout backfill implementation in Temporal startup schedule bootstrap (setupSchedules) that seeds proactive NinjaOne refresh for active integrations without active lifecycle ownership.
• (2026-03-26) Added proactive refresh failure event publication (INTEGRATION_TOKEN_REFRESH_FAILED) in proactive path so failures are visible outside worker logs.

Links / References

• NinjaOne client refresh implementation: ee/server/src/lib/integrations/ninjaone/ninjaOneClient.ts
• NinjaOne sync strategy: ee/server/src/lib/integrations/ninjaone/sync/syncStrategy.ts
• NinjaOne OAuth callback: ee/server/src/app/api/integrations/ninjaone/callback/route.ts
• Temporal schedules bootstrap: ee/temporal-workflows/src/schedules/setupSchedules.ts
• Temporal delayed scheduling patterns: ee/server/src/lib/jobs/runners/TemporalJobRunner.ts
• NinjaOne proactive refresh plan: ee/docs/plans/2026-03-26-ninjaone-proactive-token-refresh/PRD.md
• Official NinjaOne public API docs: https://www.ninjaone.com/docs/application-programming-interface-api/public-api-operations/
• Official NinjaOne OAuth configuration docs: https://www.ninjaone.com/docs/application-programming-interface-api/oauth-token-configuration/

Open Questions

• Should lifecycle metadata live in rmm_integrations.settings or a dedicated table if we need stronger scheduling introspection later?
• Should a reconnect-required token failure also write sync_error, or should it be tracked separately to avoid conflating token lifecycle state with sync state?