PSA/ee/docs/plans/2026-05-05-api-rate-limiting-and-ticket-webhooks/PRD.md

PRD — API Rate Limiting and Outbound Ticket Webhooks

• Slug: api-rate-limiting-and-ticket-webhooks
• Date: 2026-05-05
• Status: Draft
• Source plans:
  • /Users/natalliabukhtsik/Desktop/projects/alga-psa/.ai/api-rate-limiting-plan.md
  • /Users/natalliabukhtsik/Desktop/projects/alga-psa/.ai/ticket-webhooks-plan.md

Summary

Two complementary protections against "noisy poller" pressure on the public REST API and the underlying Citus cluster:

1. API rate limiting (guardrail) — a per-(tenant, api_key_id) token-bucket rate limit on every authenticated /api/v1/* request, fail-open on Redis outage, configurable per tenant.
2. Outbound ticket webhooks (cure) — finish the partially-scaffolded webhook delivery pipeline so well-behaved customers can subscribe to ticket events instead of polling. Includes signed HTTP delivery, retries with backoff, a minimal admin UI, and a curated payload shape.

The two ship as one plan because they share infrastructure (the TokenBucketRateLimiter namespace work is required by both — the rate limiter uses namespace 'api', the webhook delivery worker uses namespace 'webhook-out' for per-webhook outbound caps).

Problem

Production telemetry on 2026-05-04 traced intermittent Citus "remaining connection slots are reserved for non-replication superuser connections" errors to a single external integration polling GET /api/v1/tickets/<id> for 6 specific ticket IDs once per minute from 52.53.71.0. When the 6-call burst lands at a minute boundary alongside other work, ~18% of the calls fail with HTTP 500.

Today there is no rate limit on the public REST API and no working outbound webhooks (the system is scaffolded but the delivery method is mocked and the data tables don't exist), so:

• Customers have no choice but to poll if they need near-real-time data.
• A single key can pressure the cluster regardless of other work.

Goals

• Stop a single tenant or API key from monopolizing Citus worker connections via runaway polling.
• Give well-behaved customers a way to subscribe to ticket lifecycle events (ticket.created, ticket.updated, ticket.assigned, ticket.status_changed, ticket.closed, ticket.comment.added) and receive signed HTTP POSTs instead of polling.
• Migrate the noisy customer to webhooks once shipped.
• Ship the rate-limit guardrail first (smaller, unblocks the immediate problem); ship webhook delivery next (larger, removes the cause).

Non-goals

• Not implementing custom payload templates / Handlebars rendering for webhooks in v1.
• Not implementing webhook subscriptions for non-ticket entities in v1 (projects / clients / contacts / invoices come later — pattern is the same).
• Not changing internal email/notification subscribers — webhooks are an additional output, not a replacement.
• Not changing UI / Server-Action traffic — rate limiting only protects the external API surface (x-api-key-authenticated endpoints).
• Not building a customer-facing "webhook explorer" UI in v1; admin can manage via API and a basic settings page.
• Not changing Citus connection-pool config (separate but complementary work).
• Not introducing new queue/runtime dependencies — reuse the existing Redis client and the DelayedEmailQueue ZSET pattern; do not add BullMQ.

Users and Primary Flows

Tenant administrator (target persona)

1. Set or override an API rate limit: Settings → API Keys → "Rate Limit" → set per-key max_tokens and refill_per_min, or set a tenant-wide default.
2. Create a webhook subscription: Settings → Webhooks → "New Webhook" → name, URL, event types (multi-select), custom headers, retry config → save → copy plaintext signing secret (shown once).
3. Verify a webhook: "Send Test" delivers a synthetic payload to the configured URL using the live signing secret; result shown inline.
4. Inspect deliveries: Webhook detail → paginated history with status, response body, retry button.
5. Rotate signing secret: one click → new plaintext returned once.

External integration (consumer of webhooks and rate limits)

1. Hit a 429: receives Retry-After and X-RateLimit-* headers; backs off and retries.
2. Receive webhook: HTTP POST with X-Alga-Signature: t=<ts>,v1=<hex>; verifies HMAC; idempotently processes by event_id.

Internal noisy poller (the 52.53.71.0 integration)

• Continues working under sane defaults (6 calls/min is well under the limit).
• Receives a heads-up + migration guide pointing at the new webhook flow.

UX / UI Notes

• API Keys settings: add a "Rate Limit" column to the existing AdminApiKeysSetup table, plus an inline edit form. Surface the current remaining tokens (read via getState).
• Webhooks settings: new section under Settings, modeled on AdminApiKeysSetup. Must include create form, list view (status, last delivery, success rate), delivery history, secret reveal/rotate, pause/resume.
• Neither feature requires a new top-level navigation entry — both live under Settings → Security or Settings → Integrations (TBD with design; tracked in Open Questions).

Requirements

Functional Requirements

API rate limiting

• Every authenticated /api/v1/* request consumes 1 token from the (tenant, api_key_id) bucket in namespace 'api'.
• 429 response on denial includes Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset.
• Successful responses include X-RateLimit-Limit and X-RateLimit-Remaining.
• Defaults: max_tokens = 120, refill_per_min = 60 (1 RPS sustained, 120 burst).
• Per-(tenant, api_key_id) overrides via api_rate_limit_settings. Falls back to (tenant, NULL), then to hard-coded defaults.
• Bypass list: health/version endpoints, internal-runner endpoints, mobile auth.
• NM Store global-key path uses sentinel subjectId 'nm_store' so its traffic shares one tenant-scoped bucket.
• Observation mode: RATE_LIMIT_ENFORCE=false logs denials instead of throwing. Default false until Stage 3 of rollout.
• Fail-open if Redis is unavailable.

Outbound webhooks

• Tenant admins create webhook subscriptions for ticket lifecycle events.
• Internal events publish via the existing event bus (publishEvent({ eventType: 'TICKET_*', payload })); a new subscriber fans out to active webhooks.
• Delivery is queued, not inline (Redis ZSET, mirrored on DelayedEmailQueue).
• Each delivery is HMAC-SHA256 signed: X-Alga-Signature: t=<unix>,v1=<hex> over t + "." + body.
• Per-webhook rate limit using the TokenBucketRateLimiter namespace 'webhook-out', dimension (tenant, webhookId). Default rate_limit_per_min = 100.
• Retry policy: 1m → 5m → 30m → 2h → 12h, then abandon. 5 attempts total. Configurable per webhook via retry_config.
• Auto-disable a webhook after 24h of all-failure deliveries; email the owning user.
• Curated payload shape (see source plan §3.3) — stable subset of ticket fields plus changes diff for ticket.updated events. Comments include text/author/timestamp/internal flag, never attachments.
• POST /api/v1/webhooks/[id]/test sends a synthetic webhook.test payload, recorded with is_test = true.
• SSRF guard rejects RFC1918, loopback, link-local, CGNAT, and non-http(s) schemes before delivery (real and test).
• Signing secret stored via the existing secret provider; column holds signing_secret_vault_path, never plaintext or hash.

Non-functional Requirements

• Rate limiter adds ≤2 ms p99 to authenticated API requests when Redis is healthy.
• Webhook delivery latency floor ~2 s (ZSET poll interval); acceptable.
• At-least-once delivery semantics; idempotency key is event_id.
• Tenant isolation is preserved end-to-end (subscriber filters by tenant, payload builder uses tenant-scoped getConnection).

Data / API / Integrations

New tables

• api_rate_limit_settings (tenant-distributed): (tenant, api_key_id NULL, max_tokens, refill_per_min) with UNIQUE (tenant, api_key_id).
• webhooks (tenant-distributed): subscription rows with signing_secret_vault_path, event_types text[], retry/rate-limit config, rolling stats columns, auto_disabled_at.
• webhook_deliveries (tenant-distributed): one row per attempt; is_test boolean, next_retry_at, response capture.

New / modified APIs

• TokenBucketRateLimiter: signature change — namespace as required first parameter on tryConsume / getState / getBucketKey / getBucketConfig. BucketConfigGetter widened to (tenantId, subjectId?) => BucketConfig. initialize() accepts Record<namespace, BucketConfigGetter>.
• ApiError interface: optional headers?: Record<string, string>; handleApiError merges them into the NextResponse.
• createSuccessResponse / createPaginatedResponse: optional extraHeaders parameter.
• New helper enforceApiRateLimit(req, context) called from ApiBaseController.authenticate, withApiKeyAuth, and withAuth.
• webhookEventTypeSchema extended with 'ticket.comment.added'.
• Webhook controller stubs implemented (or removed): getDeliveryDetails, getWebhookHealth, getWebhookSubscriptions (read-only), rotateWebhookSecret, verifyWebhookSignature, listAvailableEvents. Deferred stubs (bulk, templates, transformations, etc.) have their routes deleted, not left as 501s.

Reused infrastructure (no new dependency)

• getRedisClient for both buckets and the webhook ZSET.
• DelayedEmailQueue pattern for the webhook poller class.
• getSecretProviderInstance for signing-secret storage.
• Existing event bus (publishEvent / getEventBus().subscribe) and the TICKET_* schemas in packages/event-schemas.

Security / Permissions

• Webhook signing secrets never leave the secret provider once written; GET responses for webhook rows must omit the column entirely (covered by an integration assertion).
• SSRF protection enforced server-side in both real and test delivery, with an env-var escape hatch (WEBHOOK_SSRF_ALLOW_PRIVATE) for staging/local only.
• Rate-limit configuration writes scoped to tenant admins (RBAC: existing api_keys.update permission applies; webhook CRUD uses webhook.* — added if not present).
• signing_secret_vault_path column never exposed in the API response shape.

Observability

• Rate-limit metrics: api_rate_limit_consumed_total{tenant,api_key_id,outcome}, api_rate_limit_remaining{tenant,api_key_id}, api_rate_limit_redis_unavailable_total.
• Webhook metrics: webhook_deliveries_total{tenant,webhook_id,outcome}, webhook_delivery_duration_ms histogram, webhook_queue_depth gauge, webhook_auto_disabled_total{tenant,webhook_id}.
• Structured WARN log on every throttle and on every Redis fail-open.
• Grafana panel: top throttled (tenant, api_key_id) and top failing webhooks.

Rollout / Migration

1. Citus worker max_connections bump — out of band, ~1 hour, eliminates the immediate 500s. Not part of this plan.
2. Rate limiter MVP — Stages 1–3:
   • Stage 1 (observation): RATE_LIMIT_ENFORCE=false. Measure for one week.
   • Stage 2 (notify outliers): email tenants whose keys would have been throttled.
   • Stage 3 (enforce): flip RATE_LIMIT_ENFORCE=true.
   • Stage 4: remove the env-var bypass after ~2 weeks stable.
3. Webhook MVP — Stages 1–4:
   • Stage 1 (dark launch): ship behind a feature flag; internal testing against webhook.site.
   • Stage 2 (invite-only beta): enable for a handful of API-heavy tenants, including the noisy poller.
   • Stage 3 (GA): open to all tenants; publish docs.
   • Stage 4: tighten polled REST rate limits once webhook adoption is healthy.
4. The noisy poller (52.53.71.0) gets a personal note + migration guide.

Open Questions

1. Should we expose ticket.deleted? Internal TICKET_DELETED exists. Defer to v2 unless the noisy poller specifically asks.
2. Per-tenant webhook count limit? Cap at 50 per tenant default.
3. Should ticket.status_changed include previous_status_id/ previous_status_name? Yes — captured as a feature.
4. Webhook-side filtering by entity_ids? Implement in v1 — directly addresses the noisy poller's "tell me about these 6 tickets" pattern.
5. IA placement of the new settings UI sections — Settings → Security or Settings → Integrations? Confirm with design.
6. Per-tenant rate-limit cap on top of per-key buckets? Defer until data shows we need it.
7. Per-endpoint cost weights (/search costs more than /get)? Defer until observation data shows pressure differences.

Acceptance Criteria (Definition of Done)

Rate limiter

• A test API key making 121 requests in 60 seconds receives 429 on the 121st with the documented headers, and a different key in the same tenant is unaffected.
• The email path's existing rate-limit behavior is unchanged on a baseline notification_settings.rate_limit_per_minute value.
• RATE_LIMIT_ENFORCE=false lets denials through but emits the same metrics and headers as enforce mode.
• An api_rate_limit_settings row with (tenant, api_key_id) overrides the tenant default; clearForKey returns to the tenant default within the cache TTL.

Webhooks

• TICKET_ASSIGNED published in tenant A enqueues a delivery job for an active webhook in tenant A subscribed to ticket.assigned, and does not enqueue a job for any webhook in tenant B.
• The WebhookDeliveryQueue poller successfully delivers to a stubbed HTTP server, persists a row in webhook_deliveries, and updates webhook stats columns.
• A webhook URL pointing at 127.0.0.1 or 10.0.0.5 is rejected before delivery (production mode), and accepted with WEBHOOK_SSRF_ALLOW_PRIVATE=true.
• HMAC verification with the documented recipe matches the server signature byte-for-byte.
• Signing secret never appears in any GET webhook response body.
• 5 failed attempts mark the delivery abandoned; 24h of all-failure deliveries auto-disable the webhook.
• Webhooks settings UI: an admin can create, view deliveries, send a test, rotate the secret, and pause/resume a webhook.