PSA/ee/docs/plans/2026-05-05-api-rate-limiting-and-ticket-webhooks/SCRATCHPAD.md

# Scratchpad — API Rate Limiting and Outbound Ticket Webhooks

- Plan slug: `api-rate-limiting-and-ticket-webhooks`
- Created: `2026-05-05`
- Source plans (kept for diff/history; this folder is canonical going forward):
  - `/Users/natalliabukhtsik/Desktop/projects/alga-psa/.ai/api-rate-limiting-plan.md`
  - `/Users/natalliabukhtsik/Desktop/projects/alga-psa/.ai/ticket-webhooks-plan.md`

## What This Is

Rolling notes for the combined effort. Append decisions and discoveries as
implementation progresses; update earlier entries when something changes.

## Decisions

- (2026-05-05) **Combined into one plan.** The two source plans share
  infrastructure (`TokenBucketRateLimiter` namespace work — features F001–F005
  — must land before either feature can use namespaced buckets). Splitting the
  features into one plan avoids re-stating the foundation.
- (2026-05-05) **Queue: Redis ZSET, not BullMQ or Temporal.** BullMQ is not a
  current dependency; adding it would introduce a third queue paradigm.
  Temporal is in use for `workflow-worker` but webhook delivery is "POST +
  retry," not multi-step. The `DelayedEmailQueue` ZSET pattern
  (`packages/email/src/DelayedEmailQueue.ts`) is the closest analog and
  reuses the existing Redis client. User confirmed this on 2026-05-05.
- (2026-05-05) **Signing secret stored via secret provider, not hashed.**
  HMAC requires the plaintext on every delivery — hashing breaks signing.
  Mirror the Stripe integration (`webhook_secret_vault_path` column,
  resolved through `getSecretProviderInstance()`). Fixed during plan review.
- (2026-05-05) **Reuse `TooManyRequestsError`, don't add a parallel
  `RateLimitError`.** It already exists at `apiMiddleware.ts:101-111` with
  the right shape. Plumb headers through `ApiError.headers` instead.
- (2026-05-05) **Subscribe to `TICKET_STATUS_CHANGED` directly.** It's a
  first-class internal event (`eventBusSchema.ts:170`) — don't synthesize it
  from `TICKET_UPDATED.changes.status_id`.
- (2026-05-05) **Three auth surfaces, one helper.**
  `enforceApiRateLimit(req, ctx)` is called from `ApiBaseController.authenticate`,
  `withApiKeyAuth` (both branches), and `withAuth`. NM Store path uses
  sentinel subjectId `'nm_store'` since it has no `apiKeyId`.
- (2026-05-05) **Defer to v2 by removing routes, not by leaving 501s.**
  Discovered 14+ TODO stubs in `ApiWebhookController`. The deferred ones
  (transformations, bulk ops, templates marketplace, etc.) get their route
  files deleted so OpenAPI doesn't advertise them.
- (2026-05-05) **Rate-limiter and webhooks share the
  `TokenBucketRateLimiter` namespace work.** The webhook per-webhook outbound
  cap (namespace `'webhook-out'`) depends on F001–F005 being merged first.
- (2026-05-06) **Place the v1 webhook admin UI under Security, next to API Keys.**
  The open question remained unresolved, and the existing `/msp/security-settings`
  surface already hosts the external API admin controls. Reusing that location
  avoids inventing a second admin-only settings entry point during the MVP.

## Discoveries / Constraints

- (2026-05-05) `TokenBucketRateLimiter` is at
  `packages/email/src/TokenBucketRateLimiter.ts`. Bucket key prefix is
  `alga-psa:ratelimit:bucket:` and TTL is 3600s. The `BucketConfigGetter`
  signature is `(tenantId) => BucketConfig` — must widen to
  `(tenantId, subjectId?) => BucketConfig` for per-key/per-webhook overrides.
- (2026-05-05) Existing email rate-limit defaults are `maxTokens=60,
  refillRate=1`. New API defaults are deliberately higher (`120, 1`) — API
  bursts are expected to be larger than email bursts.
- (2026-05-05) `WebhookService.checkRateLimit` (line 1056) queries
  `webhook_deliveries`, which doesn't exist yet — it would throw if called.
  Latent bug: nothing currently calls into the delivery path.
- (2026-05-05) `WebhookService.performWebhookDelivery` (line 950) is mocked
  — sleeps 100 ms and returns `{ success: true, status_code: 200 }`. No real
  HTTP request happens today.
- (2026-05-05) `webhookEventTypeSchema` lacks `ticket.comment.added`. F023
  must extend the enum or webhook creation requests for that event type
  fail validation.
- (2026-05-05) Existing distribution pattern for tenant-scoped tables:
  `notification_settings` is in `20250805000019_distribute_final_tables.cjs`.
  Migration extension is `.cjs`, not `.ts`. Citus distribution lives in
  `ee/server/migrations/citus/`, separate from the create migration in
  `server/migrations/`.
- (2026-05-05) PostgreSQL `UNIQUE (tenant, api_key_id)` would allow multiple
  `(tenant, NULL)` tenant-default rows. The migration needs a separate unique
  partial index on `tenant WHERE api_key_id IS NULL` to make the null fallback
  row actually unique.
- (2026-05-05) The current secret-provider API resolves tenant secrets by
  `(tenant, secretName)`, not by an arbitrary vault path. For webhook signing
  secrets, `signing_secret_vault_path` therefore acts as stored metadata; the
  DAL resolves the actual secret by taking the basename of the stored path and
  calling `getTenantSecret(tenant, basename(path))`.
- (2026-05-05) `undici` is already available in the server runtime, so the
  real webhook transport can use `undici.fetch` + `Agent` for the
  `verify_ssl=false` path without introducing a new dependency.
- (2026-05-05) Node's `net.BlockList` is sufficient for the required SSRF
  address classes. The helper now blocks RFC1918, loopback, link-local, and
  CGNAT IPv4 ranges plus `::1` and `fe80::/10`, and it short-circuits all of
  those checks when `WEBHOOK_SSRF_ALLOW_PRIVATE=true`.
- (2026-05-05) The repo still had an older generic webhook validator that
  expected `sha256=<hex>`. F030 replaces that with the PRD-specific outbound
  format `t=<unix>,v1=<hex>` and routes the leftover schema helper through the
  new shared implementation so future controller work doesn't split the
  signature recipe again.
- (2026-05-05) The ticket webhook surface now has a single canonical
  translation layer under `eventBus/subscribers/webhook/`; future subscriber
  fan-out code can map one internal event to one or more public webhook events
  without duplicating string switches.
- (2026-05-05) The placeholder retry math in `WebhookService` was still using
  generic exponential/linear config fields. F039 replaces that with the PRD's
  fixed retry cadence and exposes it as a shared helper for the future Redis
  queue worker.
- (2026-05-05) `initializeApp.ts` is a poor `tsx` smoke-import target in this
  repo because importing the full app graph pulls Next/UI assets like
  `react-day-picker/src/style.css`. For F031 validation, focused imports of the
  new rate-limit getter and the touched service file are the useful checks.
- (2026-05-05) `ApiBaseController.authenticate` is **not** the universal
  hook point — `withApiKeyAuth` and `withAuth` in `apiMiddleware.ts:144,201`
  are independent paths, and the NM Store branch in `withApiKeyAuth`
  produces a context with `apiKeyId === undefined`. Verified by reading
  service-types and test-auth routes.
- (2026-05-05) `/api/v1/test-auth` does not use the same `withApiKeyAuth`
  helper as `service-types`; it goes through the older
  `server/src/lib/api/middleware/apiAuthMiddleware.ts`. Rate-limit wiring has
  to cover that legacy wrapper too or the planned cross-surface test would
  split buckets by middleware implementation.
- (2026-05-05) Several `/api/v1` route families still bypassed the three
  shared auth surfaces even after F018: asset routes and contract-line routes
  were calling controllers that expect `req.context` but never authenticated,
  and a handful of direct route handlers (`tickets/priorities`,
  `tickets/statuses`, ticket comment reactions, storage routes, and several
  mobile moderation/push/account routes) were validating API keys inline
  without invoking the limiter.
- (2026-05-05) Internal event vocabulary is much larger than the v1 public
  surface. `TICKET_REOPENED`, `TICKET_ESCALATED`, `TICKET_PRIORITY_CHANGED`,
  `TICKET_UNASSIGNED`, `TICKET_QUEUE_CHANGED`, `TICKET_TAGS_CHANGED`,
  `TICKET_RESPONSE_STATE_CHANGED`, `TICKET_ADDITIONAL_AGENT_ASSIGNED` exist
  in `EVENT_TYPES` but are deferred to v2 (rolled into `ticket.updated`).
- (2026-05-05) `TICKET_COMMENT_ADDED` currently reaches subscribers through
  the legacy `TicketEventPayloadSchema` shape from `TicketService`: it
  includes `payload.comment.{content,author,isInternal}` but not a persisted
  comment timestamp. The webhook payload builder therefore uses
  `payload.occurredAt` / event timestamp for `comment.timestamp`.
- (2026-05-05) `TICKET_STATUS_CHANGED` payloads may arrive in either the new
  domain shape (`previousStatusId`) or an older `changes.status_id.from`
  style. The webhook payload builder now accepts both so subscriber output
  stays stable across publishers while the event vocabulary converges.
- (2026-05-05) `webhookSubscriber.ts` needs a queue boundary before the full
  poller lands. I added `WebhookDeliveryQueue.enqueue()` as the initial Redis
  storage contract now, and the later F037 work will extend that same class
  with claim/process/retry behavior instead of swapping subscriber behavior.
- (2026-05-05) Importing `server/src/lib/eventBus/subscribers/index.ts`
  through `tsx` drags a large app/UI graph and currently trips the same
  unrelated `react-day-picker/src/style.css` loader issue seen with broad
  `initializeApp` smoke imports. The narrower `webhookSubscriber.ts` module
  import remains the useful compile smoke for webhook subscriber changes.
- (2026-05-05) `ApiWebhookController.ts` imports can hit that same broad
  `.css` loader issue under `tsx`. For controller TODO replacements, the
  narrower DAL/helper module smokes plus `git diff --check` are the reliable
  local validation path unless we run the full server test suite.
- (2026-05-05) The webhook signature-verify route now supports both the plan's
  direct `secret_vault_path` input and a safer `webhook_id` lookup. Both
  paths resolve to the same tenant secret provider and use the shared
  `verifyWebhookSignature()` helper after normalizing the header format.
- (2026-05-05) The remaining read-side webhook controller stubs can stay thin:
  delivery details come straight from `webhook_deliveries`, health derives
  from the webhook stats columns already maintained by the delivery processor,
  subscriptions are just `webhook.event_types`, and available events come
  from `webhookEventTypeSchema.options`.
- (2026-05-05) Deferred webhook TODOs are now route-level cleanup, not
  controller cleanup. The implemented surface keeps nested delivery/health/
  subscriptions reads plus create/list/test/verify, and drops the transform,
  filter, validate, bulk, search, export, trigger, and system-health routes
  so they naturally 404 instead of advertising dead handlers.
- (2026-05-05) The nested webhook test route now diverges from the older
  generic `/api/v1/webhooks/test` helper: `/[id]/test` always uses the stored
  webhook URL + live signing secret, emits `event_type='webhook.test'`,
  records `is_test=true`, and intentionally skips outbound bucket
  consumption.
- (2026-05-05) Broad imports through `server/src/lib/jobs/index.ts` also hit
  the same unrelated `react-day-picker` CSS loader issue under `tsx`, so the
  cleanup-job service module is the reliable smoke target for scheduled-job
  additions in this environment.
- (2026-05-05) I could not find a dedicated operational metrics client/facade
  in this repo. For the v1 observability items, the fallback is structured log
  emission with stable metric names/labels rather than a Prometheus/StatsD
  sink.
- (2026-05-05) Webhook observability follows that same fallback pattern:
  queue depth is emitted from the Redis ZSET wrapper, delivery totals and
  durations from the delivery processor, and auto-disable counts from the
  state transition helper.
- (2026-05-05) Public docs for this plan now live at
  `docs/api/api-rate-limiting-and-ticket-webhooks.md` and are linked from
  `docs/api/api_overview.md`.
- (2026-05-05) `WebhookDeliveryQueue` now owns the retry loop contract:
  processors now return explicit `delivered` / `retry` / `abandoned`
  outcomes. The queue handles atomic `zRem` claims, caps active work at 50
  in-process jobs, and re-enqueues attempts 2..5 with
  `computeBackoff(attempt)`.
- (2026-05-05) Auto-disable must follow a continuous failure streak, not just
  "some failures in the last day." `maybeAutoDisable()` therefore keys off
  the first non-delivered attempt since `last_success_at` and disables only
  once that streak has remained all-failure for 24 hours.
- (2026-05-05) Added feature `F052` after discovering a plan/code mismatch:
  `webhookSchemas.ts` already exposed `event_filter.entity_ids`, but the
  `webhooks` table migration and `webhookModel` never persisted `event_filter`
  at all. The subscriber-side entity filter needs that durable field first.
- (2026-05-05) The v1 subscriber filter stops at `event_filter.entity_ids`.
  Generic `conditions`, `tags`, and `entity_types` remain schema-only for now
  per the PRD; the enqueue path simply treats an empty/missing `entity_ids`
  list as "match all."
- (2026-05-06) The new webhook settings tab uses tenant-authenticated server
  actions instead of the standalone `/api/v1/webhooks` controller surface.
  That keeps the admin UI on the same auth model as the rest of settings while
  still reusing the shared DAL, delivery transport, signing helper, and queue.
- (2026-05-06) `tsx` import-smoke of the new client component still trips the
  repo's unrelated `react-day-picker/src/style.css` loader issue. The focused
  validation path for F047 is therefore `git diff --check` plus a direct smoke
  import of the new server-action module, matching the earlier UI validation
  limitation already documented for this repo.
- (2026-05-06) The first API rate-limit integration harness now uses a minimal
  `ApiBaseController` subclass plus mocked auth/RBAC/data-service edges. That
  keeps `T007` focused on the shared authenticate/throttle/response path
  without having to pull the full tickets stack or a database-backed route into
  the fixture.
- (2026-05-06) `T016` exercises the per-key override path by spying on
  `apiRateLimitSettingsReadOps.getForKey` and wiring the bucket to the *real*
  `apiRateLimitConfigGetter`. Both the limit-header lookup and the bucket's
  internal lookup share the same in-process cache, so a single seeded row
  drives both consumption (`tryConsume`) and the `X-RateLimit-Limit` value
  emitted on every response — no additional fixture is required.
- (2026-05-06) `T017` covers the rate-limit server-action contract at the
  cache + DAL seam rather than through the `withAuth` wrapper. The session
  machinery used by `setApiRateLimitForKey` / `clearApiRateLimitForKey`
  (`getCurrentUser`, `getUserRoles`, `assertApiKeyExists`) is session-coupled
  and out of scope for vitest in this repo; the load-bearing assertion
  ("subsequent enforce call sees new limit immediately, not after 30s") lives
  in the `invalidateApiRateLimitConfig` step the actions perform after each
  upsert/clear, so the test simulates that exact write+invalidate sequence
  and verifies the bucket honours the new limit on the very next
  `tryConsume`.
- (2026-05-06) Reusable webhook delivery test fixture: in-memory mock Redis
  implementing `RedisClientLike` with full ZSET semantics (`zAdd`/`zRem`/
  `zRangeByScore`/`zCard`) plus an ephemeral `node:http` stub server keyed
  off `WEBHOOK_SSRF_ALLOW_PRIVATE=true`. The webhook model + autoDisable are
  mocked at the module boundary and the queue is given a 999_999 ms
  `checkIntervalMs` so the `setInterval` poller never races a manual
  `queue.process()` call. `(WebhookDeliveryQueue as any).instance = null` is
  required between tests to reset the singleton; the public API has no
  `resetInstance`.
- (2026-05-06) `T026` exercises tenant isolation at the subscriber/event-bus
  seam without standing up a real Redis-backed event bus: mocking
  `@/lib/eventBus` to a stub that records `subscribe()` callbacks lets the
  test invoke the captured handler directly with a forged
  `TICKET_ASSIGNED` event. `webhookModel.listForEventType` then proves the
  query is scoped to the publishing tenant and `WebhookDeliveryQueue.enqueue`
  is spied to verify only the matching-tenant webhook gets a job.
- (2026-05-06) `T027` skips `vi.useFakeTimers` in favour of
  fast-forwarding the mock-Redis ZSET scores between iterations
  (`fastForwardAll()`); the queue's claim/process cycle is what matters and
  it's already deterministic once `checkIntervalMs` is set to `999_999`. A
  small `waitFor` polling helper drains in-flight deliveries between
  attempts. This keeps the retry-cadence assertion ("score equals
  `now + computeBackoff(attempt)`") honest without the cross-test
  contamination fake timers tend to introduce.
- (2026-05-06) `T030` simulates two pods racing on the same job by
  initializing two `WebhookDeliveryQueue` instances against the same shared
  mock Redis (clear `(WebhookDeliveryQueue as any).instance = null` between
  the two `getInstance()` calls). A custom processor spy passed to
  `initialize()` lets the test assert "exactly one of the two workers ran
  the processor" without spinning up the full delivery stack.
- (2026-05-06) `T031` mocks `undici` at the package boundary so
  `assertSafeWebhookTarget` can be exercised end-to-end through
  `performWebhookDeliveryRequest`. The blocked path proves the SSRF guard
  fires before `fetch` is reached (spy unused) and returns
  `error_type='ssrf'`; the bypassed path proves the override path lets the
  fetch through. The `Agent` constructor is mocked alongside `fetch` so
  `verify_ssl=false` paths don't touch the real undici Agent.
- (2026-05-06) `T036` cannot use `vi.spyOn(ApiBaseController.prototype,
  ...)` because `ApiWebhookController` declares its OWN `private`
  `authenticate` and `checkPermission` that shadow the base class — the
  spy must be on `ApiWebhookController.prototype` directly. URLs in
  controller tests must use a real UUID for the `[id]` segment because
  `extractIdFromPath` validates against `^[0-9a-f]{8}-...$`.
- (2026-05-06) `T037` audits the migration source files instead of
  spinning up a Citus-aware test database, since the vitest harness here
  doesn't have Citus available. The audit verifies the table-creation +
  partial unique-index + distribute_table contracts that real migrations
  enforce; if/when a Citus test DB lands, this test should be replaced
  with a real `migrate:up` smoke + a `pg_dist_partition` query.

## Commands / Runbooks

- (2026-05-05) Run a single integration test:
  `cd server && npx vitest run src/test/integration/apiRateLimit.headers.test.ts`
- (2026-05-05) Run all webhook integration tests:
  `cd server && npx vitest run src/test/integration/webhook*`
- (2026-05-05) Run unit tests for the rate limiter package:
  `cd packages/email && npx vitest run src/__tests__/TokenBucketRateLimiter*`
- (2026-05-05) Apply migrations against a local dev database — see existing
  migrate flow in `server/package.json` (knex CLI driven by `migrations/`
  and `ee/server/migrations/citus/`).
- (2026-05-05) Toggle observation mode locally: `RATE_LIMIT_ENFORCE=false`
  in `server/.env`. Toggle SSRF bypass for staging:
  `WEBHOOK_SSRF_ALLOW_PRIVATE=true`.
- (2026-05-05) Tail Redis bucket state during integration tests:
  `redis-cli --scan --pattern 'alga-psa:ratelimit:bucket:*' | xargs -L1 redis-cli get`
- (2026-05-05) Run the namespace foundation unit suite without coverage noise:
  `cd server && npx vitest run --coverage.enabled=false src/test/unit/notifications/tokenBucketRateLimiter.test.ts ../packages/email/src/__tests__/TokenBucketRateLimiter.namespaces.test.ts ../packages/email/src/__tests__/TokenBucketRateLimiter.subjectId.test.ts ../packages/email/src/__tests__/TokenBucketRateLimiter.email-regression.test.ts`
- (2026-05-05) Run the API response-header unit test:
  `cd server && npx vitest run --coverage.enabled=false src/test/unit/api/apiMiddleware.responseHeaders.test.ts`
- (2026-05-05) Run the API rate-limit config getter unit tests:
  `cd server && npx vitest run --coverage.enabled=false src/lib/api/rateLimit/__tests__/configGetter.cache.test.ts src/lib/api/rateLimit/__tests__/configGetter.invalidate.test.ts src/lib/api/rateLimit/__tests__/configGetter.fallback.test.ts`
- (2026-05-05) Run the API rate-limit enforcement helper tests:
  `cd server && npx vitest run --coverage.enabled=false src/lib/api/rateLimit/__tests__/enforce.test.ts src/test/unit/api/apiMiddleware.responseHeaders.test.ts`
- (2026-05-05) Smoke-load the webhook payload builder:
  `cd server && npx tsx -e "import('./src/lib/eventBus/subscribers/webhook/webhookTicketPayload.ts').then(() => console.log('payload-ok'))"`
- (2026-05-05) Smoke-load the webhook subscriber + queue storage layer:
  `cd server && npx tsx -e "import('./src/lib/webhooks/processWebhookDeliveryJob.ts').then(() => console.log('processor-ok'))"`
  `cd server && npx tsx -e "import('./src/lib/webhooks/autoDisable.ts').then(() => console.log('auto-disable-ok'))"`
  `cd server && npx tsx -e "import('./src/lib/webhooks/WebhookDeliveryQueue.ts').then(() => console.log('queue-ok'))"`
  `cd server && npx tsx -e "import('./src/lib/eventBus/subscribers/webhookSubscriber.ts').then(() => console.log('subscriber-ok'))"`
- (2026-05-05) `cd server && npx tsc --noEmit --pretty false` currently OOMs
  in this repo, and even targeted `tsc` entrypoint checks surface existing
  package-resolution / JSX-config errors unrelated to this feature slice, so
  compile verification here is limited to focused runtime/unit checks plus
  manual review.
- (2026-05-05) Smoke-import the webhook DAL after edits:
  `cd server && npx tsx -e "import('./src/lib/webhooks/webhookModel.ts').then(() => console.log('ok'))"`
- (2026-05-05) Smoke-import the webhook delivery transport after edits:
  `cd server && npx tsx -e "import('./src/lib/webhooks/delivery.ts').then(() => console.log('delivery-ok'))"`
- (2026-05-06) Smoke-import the webhook admin server actions after edits:
  `npx tsx -e "import('./packages/auth/src/actions/webhookActions.ts').then(() => console.log('webhook-actions-ok'))"`
- (2026-05-05) Quick SSRF helper smoke:
  `cd server && npx tsx -e "import('./src/lib/webhooks/ssrf.ts').then(async ({ assertSafeWebhookTarget }) => { await assertSafeWebhookTarget('https://example.com'); console.log('public-ok'); try { await assertSafeWebhookTarget('http://127.0.0.1'); process.exit(1); } catch (error) { console.log((error && error.name) || 'error'); } })"`
- (2026-05-05) Quick signing helper smoke:
  `cd server && npx tsx -e "import('./src/lib/webhooks/sign.ts').then(({ signRequest, verifyWebhookSignature }) => { const header = signRequest('shh', '{\\\"a\\\":1}', 1700000000); console.log(header); console.log(verifyWebhookSignature(header, '{\\\"a\\\":1}', 'shh')); })"`
- (2026-05-05) Quick event-map smoke:
  `cd server && npx tsx -e "import('./src/lib/eventBus/subscribers/webhook/webhookEventMap.ts').then(({ publicEventsFor }) => { console.log(publicEventsFor('TICKET_ASSIGNED').join(',')); console.log(publicEventsFor('NOPE').length); })"`
- (2026-05-05) Quick backoff helper smoke:
  `cd server && npx tsx -e "import('./src/lib/webhooks/backoff.ts').then(({ computeBackoff }) => { console.log([1,2,3,4,5].map(computeBackoff).join(',')); })"`
- (2026-05-05) Quick webhook rate-limit getter smoke:
  `cd server && npx tsx -e "import('./src/lib/webhooks/rateLimitConfig.ts').then(({ DEFAULT_WEBHOOK_RATE_LIMIT_PER_MIN }) => console.log(DEFAULT_WEBHOOK_RATE_LIMIT_PER_MIN))"`

## Links / References

- Source plans:
  - `.ai/api-rate-limiting-plan.md`
  - `.ai/ticket-webhooks-plan.md`
- Key files:
  - `packages/email/src/TokenBucketRateLimiter.ts` — bucket implementation.
  - `packages/email/src/DelayedEmailQueue.ts` — pattern for
    `WebhookDeliveryQueue`.
  - `server/src/lib/initializeApp.ts:144-168` — singleton init site.
  - `server/src/lib/api/controllers/ApiBaseController.ts:44-87` — auth surface 1.
  - `server/src/lib/api/middleware/apiMiddleware.ts:101-111` —
    `TooManyRequestsError`; lines 144 & 201 — auth surfaces 2 & 3.
  - `server/src/lib/api/services/WebhookService.ts:950, 1056` — mock + broken
    rate limit.
  - `server/src/lib/api/controllers/ApiWebhookController.ts` — 14+ TODOs.
  - `packages/event-schemas/src/schemas/eventBusSchema.ts:157-184` — internal
    `EVENT_TYPES`.
  - `server/src/lib/api/schemas/webhookSchemas.ts:21-60` — public enum to
    extend.
  - `ee/server/migrations/20251014120000_create_stripe_integration_tables.cjs:28`
    — `webhook_secret_vault_path` precedent.
  - `server/src/lib/webhooks/webhookModel.ts` — tenant-scoped webhook DAL and
    signing-secret resolution helpers.
  - `server/src/lib/webhooks/delivery.ts` — shared outbound HTTP transport
    for webhook delivery with timeout/TLS/error classification.
  - `server/src/lib/webhooks/ssrf.ts` — outbound target validation for
    webhook delivery and test-send flows.
  - `server/src/lib/webhooks/sign.ts` — outbound request signing and
    signature verification helper for webhook deliveries.
  - `server/src/lib/eventBus/subscribers/webhook/webhookEventMap.ts` —
    canonical mapping from internal ticket events to public webhook events.
  - `server/src/lib/webhooks/backoff.ts` — shared retry schedule helper for
    the outbound webhook queue.
  - `server/src/lib/webhooks/rateLimitConfig.ts` — shared token-bucket config
    getter for the `webhook-out` namespace.

## Open Questions

- (2026-05-05) IA placement of the new admin UIs — Settings → Security or
  Settings → Integrations? Confirm with design before F022/F047 lands.
- (2026-05-05) Per-tenant cap on top of per-key buckets? Defer until
  Stage 1 observation data justifies it.
- (2026-05-05) Per-endpoint cost weights (search costs more than get)?
  Defer until observation data shows pressure differences.
- (2026-05-05) Expose `ticket.deleted` in v1? Decision: defer unless the
  noisy poller specifically asks during migration.
- (2026-05-05) Per-tenant webhook count cap — proposed 50; confirm before
  F047 lands.

## Progress Log

- (2026-05-05) **F001 complete.** `TokenBucketRateLimiter` now requires an
  explicit `namespace` on `tryConsume`, `getState`, `getBucketKey`, and
  `getBucketConfig`. Redis keys now include the namespace segment
  (`alga-psa:ratelimit:bucket:{namespace}:{tenant}[:{subject}]`) so future
  API/webhook buckets cannot collide with the existing email path.
- (2026-05-05) **F002 complete.** `BucketConfigGetter` now receives
  `(tenantId, subjectId?)`, which lets the limiter surface per-key and
  per-webhook configuration decisions without additional key parsing.
- (2026-05-05) **F003 complete.** `TokenBucketRateLimiter.initialize()`
  now accepts a namespace-to-getter map, and lookup/fail-open behavior stays
  centralized inside the shared limiter instead of spreading per-namespace
  branching to callers.
- (2026-05-05) **F004 complete.** `initializeApp()` now registers the
  existing email tenant-config getter under namespace `email` and a temporary
  hard-coded API getter under namespace `api`, so startup is already wired
  for the upcoming API limiter without altering email defaults.
- (2026-05-05) **F005 complete.** `TenantEmailService.checkRateLimits()`
  now consumes tokens from namespace `email`, preserving the pre-existing
  per-tenant/per-user email semantics after the limiter API change.
- (2026-05-05) **T001 complete.** Added
  `packages/email/src/__tests__/TokenBucketRateLimiter.namespaces.test.ts`
  to prove the same tenant/subject can exhaust `email` without consuming the
  `api` bucket.
- (2026-05-05) **T002 complete.** Added
  `packages/email/src/__tests__/TokenBucketRateLimiter.subjectId.test.ts`
  to verify namespace getters receive `subjectId` and that API-key buckets
  are keyed as `...:api:{tenant}:{subject}`.
- (2026-05-05) **T003 complete.** Added
  `packages/email/src/__tests__/TokenBucketRateLimiter.email-regression.test.ts`
  with fake time pinned to confirm the email namespace preserves the legacy
  60-token burst / 1-token-per-second refill behavior at calls 1, 30, 60,
  and 61.
- (2026-05-05) **F006 complete.** `ApiError` now supports optional response
  headers and `handleApiError()` forwards them into `NextResponse.json()`,
  which lets later rate-limit errors attach `Retry-After` and
  `X-RateLimit-*` metadata without a parallel error class.
- (2026-05-05) **F007 complete.** `createSuccessResponse()` and
  `createPaginatedResponse()` now accept optional `extraHeaders` as a final
  parameter, preserving existing controller call sites while opening a clean
  path for rate-limit headers on successful responses.
- (2026-05-05) **F008 complete.** Added
  `server/migrations/20260505123000_create_api_rate_limit_settings.cjs` with
  tenant-scoped rate-limit columns plus separate unique indexes for per-key
  rows and the `(tenant, NULL)` tenant default row.
- (2026-05-05) **F009 complete.** Added
  `ee/server/migrations/citus/20260505123100_distribute_api_rate_limit_settings.cjs`
  so the new settings table is distributed on `tenant` when Citus is present.
- (2026-05-05) **F010 complete.** Added
  `server/src/lib/api/rateLimit/apiRateLimitSettingsModel.ts` with exact-row
  reads/writes plus a fallback resolver that checks `(tenant, apiKeyId)`,
  then `(tenant, NULL)`, then the hard defaults `{ maxTokens: 120, refillRate: 1 }`.
- (2026-05-05) **F011 complete.** Added
  `server/src/lib/api/rateLimit/apiRateLimitConfigGetter.ts` with a 1000-entry,
  30-second TTL cache, exact-entry invalidation, tenant-prefix invalidation,
  and `initializeApp()` now uses it for the `api` namespace.
- (2026-05-05) **T004 complete.** Added
  `server/src/lib/api/rateLimit/__tests__/configGetter.cache.test.ts` to
  verify identical cached lookups hit the settings resolver once.
- (2026-05-05) **T005 complete.** Added
  `server/src/lib/api/rateLimit/__tests__/configGetter.invalidate.test.ts`
  to prove tenant-wide invalidation clears only that tenant and single-key
  invalidation clears only the targeted key.
- (2026-05-05) **T006 complete.** Added
  `server/src/lib/api/rateLimit/__tests__/configGetter.fallback.test.ts`
  to verify the resolver order is per-key override, then tenant default, then
  the hard-coded API defaults.
- (2026-05-05) **F012 complete.** Added
  `server/src/lib/api/rateLimit/enforce.ts` as the shared API limiter entry
  point. It resolves the `api` namespace bucket, skips configured bypass
  paths, computes rate-limit header values, and either throws
  `TooManyRequestsError` or returns a `RateLimitDecision`.
- (2026-05-05) **F013 complete.** `enforceApiRateLimit()` now treats
  `RATE_LIMIT_ENFORCE=false` as observation mode: it logs the throttle with
  tenant/api-key/retry metadata and returns a decision instead of throwing.
- (2026-05-05) **F014 complete.** The NM Store branch in
  `apiMiddleware.withApiKeyAuth()` now stamps `rateLimitSubjectId='nm_store'`
  before calling the limiter so all global-key traffic shares one tenant
  bucket instead of bypassing per-subject accounting.
- (2026-05-05) **F015 complete.** `shouldBypassRateLimit()` now centralizes
  the bypass prefixes for health endpoints, mobile auth, and runner-internal
  endpoints so future auth wrappers reuse one rate-limit allowlist.
- (2026-05-05) **F016 complete.** Rate-limit denials now throw the existing
  `TooManyRequestsError` with `details.retry_after_ms`, `details.remaining`,
  and the full header set attached on `error.headers`.
- (2026-05-05) **F017 complete.** `ApiBaseController.authenticate()` now
  enforces the API bucket immediately after building request context and stores
  the resulting decision on `apiRequest.context.rateLimit`.
- (2026-05-05) **F018 complete.** The middleware auth wrappers now call
  `enforceApiRateLimit()` as soon as context is available. I also wired the
  legacy `apiAuthMiddleware.ts` path so `/api/v1/test-auth` stays in the same
  bucket family as the newer wrappers.
- (2026-05-05) **F019 complete.** `createSuccessResponse()` and
  `createPaginatedResponse()` now emit `X-RateLimit-Limit` and
  `X-RateLimit-Remaining` automatically when the passed request carries
  `context.rateLimit`, and the generic `ApiBaseController` create/update
  paths now pass `apiRequest` through to the helper.
- (2026-05-05) **F020 complete.** Added reusable legacy auth helpers:
  `authenticateApiKeyRequest()` for inline API-key handlers,
  `withApiKeyRouteAuth()` for route files that need `req.context`, and
  `appendRateLimitHeaders()` for direct `NextResponse` routes. Wrapped the
  entire asset and contract-line `/api/v1` route families so they now
  authenticate through the shared legacy middleware and emit rate-limit
  headers. I also migrated the remaining direct `/api/v1` handlers that were
  doing inline API-key validation (ticket priorities/statuses/reactions,
  storage routes, and the non-mobile-auth mobile moderation/push/account
  routes) onto the shared helper so they consume the same `api` bucket.
- (2026-05-05) **F021 complete.** Added tenant-admin server actions in
  `packages/auth/src/actions/apiKeyRateLimitActions.ts`:
  `getApiRateLimitForKey`, `setApiRateLimitForKey`,
  `setTenantDefaultApiRateLimit`, and `clearApiRateLimitForKey`. They verify
  admin access, scope API key IDs to the current tenant, use the
  `api_rate_limit_settings` model for reads/writes, and invalidate the
  in-process API rate-limit config cache immediately after every write so UI
  updates do not wait on the 30s TTL.
- (2026-05-05) **F022 complete.** `AdminApiKeysSetup` now loads each key's
  effective API rate-limit settings plus live bucket state and renders a new
  "Rate Limit" column with inline override editing and reset. The column
  shows the effective burst / refill values, the config source
  (per-key override vs tenant default vs hard default), and the current
  remaining tokens from `TokenBucketRateLimiter.getState('api', tenant,
  apiKeyId)`.
- (2026-05-05) **F023 complete.** The public webhook event enum now includes
  `ticket.comment.added`, so webhook create/update validation no longer
  rejects the v1 ticket-comment subscription event.
- (2026-05-05) **T018 complete.** Added
  `server/src/lib/api/schemas/__tests__/webhookSchemas.test.ts` to lock in
  acceptance of the new `ticket.comment.added` enum member.
- (2026-05-05) **F024 complete.** Added
  `server/migrations/20260505140000_create_webhook_tables.cjs` with the base
  `webhooks` subscription table: tenant-scoped primary key, event list,
  signing-secret vault path, retry/rate-limit config, activation flag, rolling
  delivery stats, auto-disable timestamp, and creator/audit timestamps.
- (2026-05-05) **F025 complete.** Expanded the same webhook migration to add
  `webhook_deliveries` with tenant/webhook foreign key wiring, request +
  response capture columns, retry scheduling fields, `is_test`, and the three
  queue-oriented indexes required by the PRD (`webhook+attempted_at`,
  `event_id`, and partial pending/retrying `next_retry_at`).
- (2026-05-05) **F026 complete.** Added
  `ee/server/migrations/citus/20260505140100_distribute_webhook_tables.cjs`
  to distribute both `webhooks` and `webhook_deliveries` on `tenant`, with
  the same Citus-enabled / already-distributed guards used by the earlier
  rate-limit distribution migration.
- (2026-05-05) **F027 complete.** Added
  `server/src/lib/webhooks/webhookModel.ts` as the first non-mock webhook
  foundation: public reads omit `signing_secret_vault_path`, inserts persist
  signing secrets via `getSecretProviderInstance()`, delivery attempts write
  to `webhook_deliveries`, stats updates increment the rolling counters on
  `webhooks`, and `getSigningSecret()` resolves the stored path-style
  reference back to the tenant secret name.
- (2026-05-05) **F028 complete.** Added
  `server/src/lib/webhooks/delivery.ts` and rewired
  `WebhookService.performWebhookDelivery()` to use it. Deliveries now perform
  a real `undici.fetch` call with a 10s timeout, preserve response status and
  headers, truncate stored response bodies to 8 KB, classify DNS/connect/TLS/
  timeout failures, and disable certificate verification only when
  `verify_ssl=false`.
- (2026-05-05) **F029 complete.** Added
  `server/src/lib/webhooks/ssrf.ts` and enforced it in the shared delivery
  transport before any outbound fetch. Targets must now use `http(s)`,
  reject `localhost`/loopback/private/link-local/CGNAT destinations after DNS
  resolution, and only bypass those checks when
  `WEBHOOK_SSRF_ALLOW_PRIVATE=true`.
- (2026-05-05) **F030 complete.** Added
  `server/src/lib/webhooks/sign.ts` with the PRD's `X-Alga-Signature`
  contract: `t=<timestamp>,v1=<sha256 hex>` over `${timestamp}.${body}`.
  `webhookSchemas.validateWebhookSignature()` now delegates to the same helper
  instead of preserving the old `sha256=<hex>` comparison logic.
- (2026-05-05) **F032 complete.** Added
  `server/src/lib/eventBus/subscribers/webhook/webhookEventMap.ts` with the
  v1 ticket-event translation table and a `publicEventsFor()` helper that
  returns a fresh array for each lookup, making the mapping ready for the
  upcoming event-bus subscriber.
- (2026-05-05) **F039 complete.** Added
  `server/src/lib/webhooks/backoff.ts` with the PRD retry schedule
  (1m, 5m, 30m, 2h, 12h) and pointed the scaffolded
  `WebhookService.calculateNextRetryTime()` method at that helper so old
  placeholder retry math no longer diverges from the intended queue behavior.
- (2026-05-05) **F031 complete.** Added
  `server/src/lib/webhooks/rateLimitConfig.ts`, registered the new
  `'webhook-out'` namespace in `initializeApp()`, and replaced the stale
  delivery-count query in `WebhookService.checkRateLimit()` with
  `TokenBucketRateLimiter.tryConsume('webhook-out', tenant, webhookId)`.
  The delivery path now applies the shared per-webhook bucket instead of the
  mocked `webhook.rate_limit.enabled` branch.
- (2026-05-05) **F033 complete.** Added
  `server/src/lib/eventBus/subscribers/webhook/webhookTicketPayload.ts`,
  which builds the PRD's curated ticket snapshot for webhook fan-out,
  normalizes `ticket.updated` change diffs, includes
  `ticket.comment.added` comment metadata without attachments, resolves tags
  from `tag_mappings`, and caches the base `(tenant,ticket_id)` snapshot for
  60 seconds so a multi-subscriber fan-out does not repeat the same joins.
- (2026-05-05) **F034 complete.** `ticket.status_changed` payloads from
  `webhookTicketPayload.ts` now include `previous_status_id` plus a
  tenant-scoped lookup of `previous_status_name`, using either
  `payload.previousStatusId` or the older `payload.changes.status_id.from`
  compatible shape when deriving the prior status.
- (2026-05-05) **F035 complete.** Added
  `server/src/lib/eventBus/subscribers/webhookSubscriber.ts`, which
  subscribes to the six v1 ticket events, builds the curated webhook payload
  once per internal event, filters subscribers by `(tenant, public event
  type)`, and enqueues one delivery job per matching active webhook. I also
  introduced the initial `server/src/lib/webhooks/WebhookDeliveryQueue.ts`
  storage contract so the subscriber already targets the eventual Redis ZSET
  queue instead of a temporary inline-delivery path.
- (2026-05-05) **F036 complete.** Registered the webhook subscriber in
  `server/src/lib/eventBus/subscribers/index.ts` so the existing
  register-all / unregister-all lifecycle now includes webhook ticket events
  alongside the other subscriber families.
- (2026-05-05) **F037 complete.** Expanded
  `server/src/lib/webhooks/WebhookDeliveryQueue.ts` from storage-only enqueue
  support into the actual Redis ZSET poller: `initialize(getRedisClient,
  processFn)` now starts a 2s processing loop, claims ready jobs via
  `zRangeByScore` + `zRem`, limits active processor promises to 50, retries
  failed jobs up to five total attempts using the shared backoff helper, and
  drains in-flight work for up to 30 seconds on shutdown / `SIGTERM`.
- (2026-05-05) **F038 complete.** `initializeApp()` now boots the webhook
  delivery queue with `getRedisClient` plus a real
  `processWebhookDeliveryJob()` callback, and the existing SIGTERM/SIGINT
  cleanup path now shuts the queue down alongside the email retry queues.
- (2026-05-05) **F040 complete.** Added
  `server/src/lib/webhooks/autoDisable.ts` and wired it into
  `processWebhookDeliveryJob()`. Failed deliveries now advance the webhook's
  rolling stats, and once the first failure since the last success has aged
  past 24 hours the webhook is auto-disabled exactly once and the owning user
  receives a direct notification email via the system email service.
- (2026-05-05) **F052 complete.** Updated the base webhook migration plus
  `server/src/lib/webhooks/webhookModel.ts` so webhook rows now persist and
  return `event_filter` JSON. That closes the storage gap under
  `event_filter.entity_ids` before the subscriber starts enforcing it.
- (2026-05-05) **F041 complete.** `webhookSubscriber.ts` now enforces
  `event_filter.entity_ids` before enqueueing jobs: when a webhook row carries
  a non-empty allowlist, only matching ticket IDs are queued. Missing/empty
  allowlists still receive all matching event types.
- (2026-05-05) **F042 complete.** `ApiWebhookController.rotateSecret()` now
  performs a real secret rotation: it generates a 32-byte base64url secret,
  updates the webhook through `webhookModel.update(..., { signingSecret })`,
  and returns the plaintext once in the response instead of the old timestamp
  stub.
- (2026-05-05) **F043 complete.** `ApiWebhookController.verifySignature()`
  now resolves the signing secret from either `webhook_id` or
  `secret_vault_path`, normalizes split signature inputs into the
  `t=...,v1=...` header format when needed, and returns the real HMAC match
  result instead of the old always-true stub.
- (2026-05-05) **F044 complete.** Replaced four controller TODOs:
  `getDelivery()` now loads a concrete `webhook_deliveries` row via
  `webhookModel.getDeliveryById()`, `getHealth()` derives a stable health
  summary from the webhook stats columns, `getSubscriptions()` returns the
  stored `event_types` for the webhook, and `listEvents()` returns the public
  enum from `webhookEventTypeSchema`.
- (2026-05-05) **F045 complete.** Deleted the deferred webhook route handlers
  for transform/filter validation, system health, global/nested subscription
  creation, bulk/search/export, and manual event triggering. The nested
  `[id]/subscriptions` route now exposes only `GET`, and the removed paths
  will 404 instead of surfacing TODO-backed handlers.
- (2026-05-05) **F046 complete.** `ApiWebhookController.testById()` now sends
  a real signed `webhook.test` request to the configured webhook URL, records
  the attempt in `webhook_deliveries` with `is_test=true`, and returns the
  observed transport result. It reuses the live signing/header and SSRF-guard
  path but skips the outbound rate-limit bucket and does not mutate webhook
  delivery stats.
- (2026-05-05) **F048 complete.** Added
  `server/src/services/cleanupWebhookDeliveriesJob.ts` plus scheduler wiring
  in `server/src/lib/jobs/index.ts` and
  `server/src/lib/jobs/initializeScheduledJobs.ts`. The new system-wide job
  runs every 15 minutes and deletes `webhook_deliveries` rows older than
  30 days in batches of 10,000 until the backlog is gone.
- (2026-05-05) **F049 complete.** `enforceApiRateLimit()` now emits
  structured fallback metric logs for
  `api_rate_limit_consumed_total`, `api_rate_limit_remaining`, and
  `api_rate_limit_redis_unavailable_total`, using stable label fields
  (`tenant`, `api_key_id`, `outcome`) alongside the existing throttle WARN.
- (2026-05-05) **F050 complete.** Added
  `server/src/lib/webhooks/metrics.ts` and wired structured fallback metric
  logs into `WebhookDeliveryQueue`, `processWebhookDeliveryJob()`, and
  `maybeAutoDisable()`. That now emits
  `webhook_queue_depth`, `webhook_deliveries_total`,
  `webhook_delivery_duration_ms`, and
  `webhook_auto_disabled_total`.
- (2026-05-05) **F051 complete.** Added
  `docs/api/api-rate-limiting-and-ticket-webhooks.md` with the public
  rate-limit contract, webhook event examples, HMAC verification recipes,
  idempotency/ordering notes, and retry schedule; linked it from
  `docs/api/api_overview.md`.
- (2026-05-06) **F047 complete.** Added tenant-authenticated webhook admin
  actions in `packages/auth/src/actions/webhookActions.ts`, a new
  `AdminWebhooksSetup` settings component with create/edit, test-send,
  secret rotation, pause/resume, delete, delivery history, and manual retry
  enqueue, plus Security settings tab wiring in
  `server/src/components/settings/security/SecuritySettingsPage.tsx`. The DAL
  now also exposes tenant-scoped webhook listing and paginated delivery
  history helpers for the UI.
- (2026-05-06) **T007 complete.** Added
  `server/src/test/integration/apiRateLimit.headers.test.ts`, which drives the
  real `ApiBaseController.list()` auth path 121 times under one tenant/API key
  and asserts the 121st response is a 429 with `Retry-After`,
  `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, and the
  expected `RATE_LIMITED` error envelope details.
- (2026-05-06) **T008 complete.** Extended
  `server/src/test/integration/apiRateLimit.headers.test.ts` with the success
  case assertion: an allowed authenticated request now proves
  `X-RateLimit-Limit=120` and `X-RateLimit-Remaining=119` are attached on the
  200 response from the same controller path.
- (2026-05-06) **T009 complete.** Extended the same
  `apiRateLimit.headers.test.ts` harness to swap API key identities within one
  tenant and prove bucket isolation: with a 5-token config, key A throttles on
  request 6 while key B still gets a 200 and its own `remaining=4` header.
- (2026-05-06) **T010 complete.** The same harness now also forces the
  tenant-scoped API-key auth branch via `x-tenant-id` and proves the bucket key
  includes tenant: exhausting tenant A with a shared `api_key_id` no longer
  affects tenant B, which still succeeds with its own `remaining=4` header.
- (2026-05-06) **T011 complete.** Extended the rate-limit integration harness
  to drive an exhausted bucket and then swap only the request pathname to a
  bypassed route (`/api/v1/meta/health`). Those calls stay 200 and the next
  ticket-path request is still 429, proving bypasses do not consume tokens.
- (2026-05-06) **T012 complete.** Added observation-mode coverage to the same
  rate-limit integration file by mocking the shared logger: with
  `RATE_LIMIT_ENFORCE=false`, the 121st request now stays 200 with
  `remaining=0`, and the test asserts the structured throttle WARN still
  carries `tenant`, `api_key_id`, and `retry_after_ms`.
- (2026-05-06) **T013 complete.** Added a broken-Redis branch to the same
  harness: 200 authenticated requests now stay 200 with
  `X-RateLimit-Remaining=-1`, and the mocked logger proves the
  `api_rate_limit_redis_unavailable_total` metric payload is emitted on the
  fail-open path.
- (2026-05-06) **T014 complete.** The rate-limit harness now also imports the
  shared middleware wrappers and proves bucket sharing across all three auth
  surfaces: after five mixed requests through `ApiBaseController`,
  `withApiKeyAuth`, and `withAuth`, the next request on each surface returns
  429 from the same `(tenant, api_key_id)` bucket.
- (2026-05-06) **T015 complete.** Added NM Store coverage to the same
  integration file by mocking `getAppSecret('nm_store_api_key')`: the
  `withApiKeyAuth({ allowNmStore: true })` branch now throttles the shared
  sentinel bucket after five requests, while a normal API key in the same
  tenant still succeeds with its own `remaining=4` header.