PSA/ee/docs/plans/bucket-overlays-migration-plan.md

Bucket Overlays Migration Plan

This plan transitions “Bucket” from a standalone plan type into an overlay on top of Hourly and Usage contract lines. The outcome is a simpler UX, clearer billing semantics, and reduced disambiguation complexity while preserving bucket usage tracking and overage billing.

This is a clean cutover and redesign without any requirement to preserve backward compatibility or legacy behavior.

The plan is phased to minimize risk and to keep tests green throughout. Each phase lists: goals, tasks, code touchpoints, acceptance criteria, and rollout notes.

---

Phase 1 — Foundations

Goals

• Align interfaces and test utilities on the new overlay model.

Tasks

• Confirm current tables used by buckets (no schema change now):
  • plan_service_configuration (with configuration_type = 'Bucket')
  • plan_service_bucket_config (stores per-line bucket config: total_minutes/units, overage_rate, allow_rollover, billing_period, etc.)
  • bucket_usage (per period usage results, minutes/units used and overage)
• Establish coding conventions for overlay editor reuse and per-line validation.

Schema verification (2025-02-14)

• plan_service_bucket_config exists with (tenant, config_id) PK, FK back to plan_service_configuration, and bucket fields: total_minutes, billing_period, overage_rate, allow_rollover, timestamps.
• contract_line_service_configuration exists with (tenant, config_id) PK but currently lacks FKs to contract_lines and service_catalog, and has no supporting indexes on contract_line_id / service_id.
• contract_line_service_bucket_config mirrors the plan table structure (total_minutes, billing_period, overage_rate, allow_rollover, timestamps) but is missing the (tenant, config_id) FK back to contract_line_service_configuration.
• Recommended follow-up migrations for Phase 2+: add (tenant, contract_line_id) FK + index, (tenant, service_id) FK + index, and unique (tenant, contract_line_id, service_id) constraint on contract_line_service_configuration; add (tenant, config_id) FK on contract_line_service_bucket_config; audit bucket_usage to ensure it will link to contract-line overlays post-cutover.

Acceptance

• Documentation notes captured in this plan; team aligned on scope.

Rollout

• N/A

---

Phase 2 — Test Helpers & Unit Tests First

Goals

• Migrate shared test helpers to create “bucket overlays” on Hourly/Usage lines instead of a dedicated Bucket plan type.

Tasks

• Update helper in server/test-utils/billingTestHelpers.ts:
  • Replace createBucketPlanAssignment(...) with createBucketOverlayForPlan(...) that:
    • Ensures an Hourly or Usage plan exists (create if needed).
    • Inserts plan_services entries for target services (if missing).
    • Upserts plan_service_configuration with configuration_type = 'Bucket' for each service.
    • Inserts plan_service_bucket_config with: total_minutes (for hours) or units as “minutes” generic quantity for usage, overage_rate, allow_rollover, billing_period.
  • Remove the old function usage across the suite; do not maintain a shim.
• Update/adjust unit tests that explicitly assert plan_type = 'Bucket':
  • server/src/test/unit/contractLineDisambiguation.test.ts: remove bucket-plan special cases; add overlay preference tests (prefer lines with active bucket balance).
  • server/src/test/unit/timeEntryBillingPlanSelection.test.tsx: mock overlay presence (not plan type) and assert default selection preference.
  • server/src/test/unit/billingPlanSelection.test.ts: same as above.
• server/src/test/unit/billingEngine.test.ts: replace “calculateBucketPlanCharges” expectations with overlay consumption assertions (time/usage paths).
• server/src/test/unit/bucketUsageService.test.ts: confirm logic still works with overlays.
• Progress (2025-02-14):
  • Helper rewritten as createBucketOverlayForPlan and wired to both contract-line and legacy plan tables; usage helper now records contract_line_id.
  • Invoice infrastructure test (usageBucketAndFinalization) migrated to build overlays on top of fixed lines and seed usage via overlay helper.
  • Selector/disambiguation unit tests now mock has_bucket_overlay metadata instead of contract_line_type === 'Bucket'; pending updates to the production disambiguation logic and remaining billing engine/unit suites.
  • Billing engine unit coverage updated to treat overlays as the source for bucket charges (calculateBucketPlanCharges test now stubs overlay rows and bucket usage, and calculateBilling aggregates the overlay charge).
  • UI/service layers now consume overlay metadata: contract-line selectors prefer overlays, usage/time-entry actions update bucket usage when overlays exist (rather than relying on contract_line_type), and API imports point at contractLineDisambiguation.
• Follow-up work to finish ripping out bucket plans
  1. Contract wizard + Playwright flows
     • Update contractWizardActions so bucket configuration toggles add overlays to the existing fixed contract line (never create a contract_line_type = 'Bucket' row).
     • Remove bucket-plan assertions from the Playwright suite (ee/server/src/__tests__/integration/contract-wizard-happy-path.playwright.test.ts) and replace them with overlay verifications (contract_line_service_configuration + contract_line_service_bucket_config joins).
     • Adjust wizard helpers/seeders to account for overlay toggles and ensure monthly-fee behaviour still covered.
     • Drop bucket_contract_lines from test cleanup lists once the wizard no longer creates them.
  2. Reporting/analytics surfaces
     • Switch getRemainingBucketUnits, client-portal billing metrics, and account dashboards to join contract_line_service_configuration + bucket overlays (remove .where(plan_type = 'Bucket') or contract_line_type === 'Bucket' filters).
     • Update any UI components (e.g. BucketContractLineConfiguration) that guard on plan type so they inspect overlay metadata instead.
  3. Service APIs & helpers
     • Finish migrating remaining server actions and utilities that still query contract_line_type (search codebase for 'Bucket' checks) – specifically ContractLineServiceConfigurationService, report actions, constants, and interfaces in billing.interfaces.ts.
     • Update TypeScript enums/interfaces to drop 'Bucket' from contract_line_type once overlay consumers are in place.
  4. Database/tests cleanup
     • Remove dependencies on the bucket_contract_lines table from tests; plan a follow-up migration to archive/drop the table after the data migration script (Phase 8) runs.
     • Sweep remaining tests/integration helpers that reference legacy plan tables (e.g. bundle_billing_plans lookups) and ensure they validate overlays instead.

Acceptance

• Unit tests compile and run; bucket assertions are overlay-based.

Rollout

• N/A

---

Phase 3 — Contract Wizard: Per-Line Bucket Overlays

Goals

• Replace the separate “Bucket Services” step with per-line overlays inside Hourly and Usage steps.

Tasks

• Remove the dedicated wizard step (UI only change):
  • server/src/components/billing-dashboard/contracts/ContractWizard.tsx
    • Update STEPS to drop “Bucket Services” and adjust navigation/validation.
    • Update validateStep logic to stop validating the removed step.
• Introduce per-line bucket overlay in:
  • FixedFeeServicesStep.tsx and HourlyServicesStep.tsx: Add a toggle and fields per service for included hours (stored as minutes), overage rate (cents), and rollover support.
  • UsageBasedServicesStep.tsx: Provide the same overlay toggle with included units (stored as generic minutes), overage rate, and rollover toggle.
• Create a reusable BucketOverlayFields component that renders the shared overlay inputs and automation hooks; reuse it across the fixed, hourly, and usage steps.
• Update the review step to summarize overlays under each line item.

Acceptance

• Wizard lets users add bucket overlays per Hourly/Usage line with proper validation.
• No separate bucket step shown when the flag is ON.

Rollout

• N/A

---

Phase 4 — Wizard Submission & Persistence

Goals

• Persist per-line overlays instead of creating a Bucket plan.

Tasks

• Update ContractWizardData and the submission payload to capture a bucket_overlay object per fixed/hourly/usage service with fields:
  • { total_minutes?: number, overage_rate?: number, allow_rollover?: boolean, billing_period?: 'monthly' | 'weekly' }
• Modify server/src/lib/actions/contractWizardActions.ts:
  • Remove creation of a dedicated Bucket plan.
  • After creating each contract-line service configuration, upsert a matching contract_line_service_configuration + contract_line_service_bucket_config row when bucket_overlay is provided, using a shared helper to insert/update the overlay metadata.
• Monthly fee handling (decision):
  • Option A (preferred for now): bill the “bucket monthly fee” by adding/allocating a Fixed Fee line scoped to that service (documented on the plan). No schema change needed; engine already handles fixed fees.
  • Option B (future): add monthly_fee to plan_service_bucket_config (requires migration and engine update). Leave as a future enhancement.

Acceptance

• New contracts with overlays persist correctly and no Bucket plan is created.

Rollout

• N/A

---

Phase 5 — Billing Engine: Consume Overlays

Goals

• Calculate included minutes/units and overage from overlay configs during invoice generation; eliminate reliance on a Bucket plan type.

Tasks

• In server/src/lib/billing/billingEngine.ts:
  • Remove/retire calculateBucketPlanCharges (or convert into an internal helper that aggregates overlay consumption per line).
  • For time-based charges:
    • Fetch applicable overlay configs for the Hourly line’s plan/service during the billing period.
    • Compute available included minutes from plan_service_bucket_config minus previously used (bucket_usage) for that period.
    • Allocate billable time first against remaining bucket minutes; compute overage minutes and bill at overage_rate.
  • For usage-based charges:
    • Same pattern using “units” (represented as minutes in total_minutes for parity).
  • Write/update bucket_usage for the period with new consumption and overage.
  • Ensure tax logic remains consistent; overage lines should apply the service’s tax config.

Acceptance

• Engine produces correct totals for lines with bucket overlays (included + overage).
• Infrastructure invoice tests pass with overlays.

Rollout

• N/A

---

Phase 6 — Plan Disambiguation Update

Goals

• Simplify plan selection by preferring plans with active overlays (balance available), removing bucket plan special-cases.

Tasks

• In server/src/lib/utils/contractLineDisambiguation.ts:
  • determineDefaultBillingPlan(clientId, serviceId):
    • If a single eligible plan → select it.
    • If multiple: prefer the plan where the overlay exists and has remaining balance in the entry’s period (requires a helper to query overlay and usage balance).
    • Otherwise, return null and require explicit selection.
  • getEligibleBillingPlansForUI: include an overlay marker and, if feasible, a quick “has_balance” boolean for UX hints.
• Update unit tests to reflect new rules.

Acceptance

• Disambiguation chooses the overlay plan when sensible; tests cover tie cases.

Rollout

• N/A

---

Phase 7 — Time Entry Plan Selector Defaults

Goals

• Keep the selector optional, default intelligently to overlay-backed plan with balance.

Tasks

• In server/src/components/time-management/time-entry/time-sheet/TimeEntryEditForm.tsx:
  • When multiple plans are eligible, default to the one with an active overlay and remaining balance (fall back to previous heuristics if unknown).
  • Show the selector only when >1 eligible plans.
  • Tooltip copy to reflect overlay preference.

Acceptance

• UI defaults match disambiguation; selector appears only when needed.

Rollout

• N/A

---

Phase 8 — Integration & Invoice Tests

Goals

• Ensure end-to-end generation with overlays passes and equals legacy behavior for equivalent configurations.

Tasks

• Update integration/infrastructure tests to use createBucketOverlayForPlan:
  • usageBucketAndFinalization.test.ts
  • prepaymentInvoice.test.ts
  • Invoice subtotal/tax/consistency/edge-case tests that reference bucket tables.
• Keep assertions on bucket_usage updates and invoice line item correctness (included vs overage).

Acceptance

• Tests pass with overlays; no regressions in totals or tax.

Rollout

• N/A

---

Phase 9 — Cleanup

Goals

• Remove Bucket from plan_type usages in code after the feature is stable and migration is complete.

Tasks

• Remove bucket plan special-cases across code and tests so overlays are the sole bucket source:
  • server/src/lib/actions/contractWizardActions.ts: collapse the “bucket plan” branch so the wizard only ever mutates the fixed/usage line and appends overlay configuration rows; purge any fallbacks that still create a contract_line_type = 'Bucket'.
  • ee/server/src/__tests__/integration/contract-wizard-happy-path.playwright.test.ts: update assertions/helper flows to verify overlay persistence (contract_line_service_configuration + contract_line_service_bucket_config) and stop expecting a bucket contract line or plan type string.
  • Reporting and analytics callers (server/src/lib/actions/client-portal-actions/client-billing-metrics.ts, server/src/lib/actions/report-actions/getRemainingBucketUnits.ts, CSV/BI exports) need to replace contract_line_type === 'Bucket' filters with overlay joins and expose overlay metadata to the UI.
  • Billing/usage helpers (server/src/lib/utils/contractLineDisambiguation.ts, server/src/lib/services/contractLineServiceConfigurationService.ts, server/src/lib/billing/billingEngine.ts, server/src/lib/services/bucketUsageService.ts) should look up overlays via contract_line_service_configuration and never branch on plan type.
  • Test + seed utilities (server/test-utils/billingTestHelpers.ts, infrastructure invoice suites, nightly cleanup scripts) must drop bucket_contract_lines cleanup and rely solely on overlay tables.
• Update enum/documentation to drop Bucket as a plan type once callers above run overlay-only logic (e.g. server/src/interfaces/billing.interfaces.ts, server/src/constants/billing.ts, API surface types).
• Delete legacy UI step/components that are no longer referenced (wizard bucket step, bucket-plan filters, etc.).
• Add guardrails: introduce an automated check (rg "contract_line_type.*Bucket") in CI or lint scripts so new plan-type branches cannot be reintroduced.

Acceptance

• Repository compiles and tests green without any plan_type='Bucket' coupling.

Rollout

• N/A

Verification

• rg "contract_line_type.*'Bucket'" and rg "plan_type.*Bucket" return no results in TypeScript/TSX sources (excluding documentation).
• Playwright contract wizard regression passes while persisting overlays.
• Billing smoke tests confirm bucket invoices derive from overlay rows only.

---

Phase 10 — Documentation & Rollout

Goals

• Provide clear internal and customer-facing docs on overlays.

Tasks

• Internal docs:
  • Architecture overview (overlay model, configs, usage ledger).
  • Disambiguation rules, defaulting, and reporting by billing_plan_id.
• Customer-facing docs (if applicable):
  • “Bucket of Hours/Units” under Hourly/Usage lines; effective rate; overage; rollover.
• Release notes and upgrade guidance for existing tenants.

Acceptance

• Docs published and linked in relevant UI tooltips.

Rollout

• N/A

---

Appendix — Implementation Notes

• Table references (no schema changes required for MVP):
  • plan_service_configuration (ensure per plan/service configuration_type = 'Bucket' exists)
  • plan_service_bucket_config (uses total_minutes for hours or usage units; overage_rate, allow_rollover, billing_period)
  • bucket_usage (minutes/units used + overage per period)
• Monthly fee for buckets:
  • MVP: bill via Fixed Fee lines (per plan or per service allocation); engine already handles fixed charges.
  • Future: add monthly_fee to plan_service_bucket_config if we want it stored alongside the overlay.
• Plan disambiguation/UI:
  • getEligibleBillingPlansForUI may include overlay hints: { has_overlay: boolean, has_balance?: boolean } to aid defaults and UX copy.
• Reporting:
  • Keep billing_plan_id on time entries/usage for plan-level reporting; overlays don’t change that contract.

---

Success Criteria

• Users configure bucket hours/units directly on Hourly/Usage lines.
• Time entries/usage automatically consume included amounts; overage is billed correctly.
• Default plan selection prefers overlay-backed lines when appropriate.
• All tests pass; migrations maintain historical accuracy; no regressions in invoice totals or taxes.