PSA/ee/docs/plans/bucket-overlays-migration-plan.md

# Bucket Overlays Migration Plan

This plan transitions “Bucket” from a standalone plan type into an overlay on top of Hourly and Usage contract lines. The outcome is a simpler UX, clearer billing semantics, and reduced disambiguation complexity while preserving bucket usage tracking and overage billing.

This is a clean cutover and redesign without any requirement to preserve backward compatibility or legacy behavior.

The plan is phased to minimize risk and to keep tests green throughout. Each phase lists: goals, tasks, code touchpoints, acceptance criteria, and rollout notes.

---

## Phase 1 — Foundations

Goals
- Align interfaces and test utilities on the new overlay model.

Tasks
- Confirm current tables used by buckets (no schema change now):
  - `plan_service_configuration` (with `configuration_type = 'Bucket'`)
  - `plan_service_bucket_config` (stores per-line bucket config: total_minutes/units, overage_rate, allow_rollover, billing_period, etc.)
  - `bucket_usage` (per period usage results, minutes/units used and overage)
- Establish coding conventions for overlay editor reuse and per-line validation.

Schema verification (2025-02-14)
- `plan_service_bucket_config` exists with `(tenant, config_id)` PK, FK back to `plan_service_configuration`, and bucket fields: `total_minutes`, `billing_period`, `overage_rate`, `allow_rollover`, timestamps.
- `contract_line_service_configuration` exists with `(tenant, config_id)` PK but currently lacks FKs to `contract_lines` and `service_catalog`, and has no supporting indexes on `contract_line_id` / `service_id`.
- `contract_line_service_bucket_config` mirrors the plan table structure (`total_minutes`, `billing_period`, `overage_rate`, `allow_rollover`, timestamps) but is missing the `(tenant, config_id)` FK back to `contract_line_service_configuration`.
- Recommended follow-up migrations for Phase 2+: add `(tenant, contract_line_id)` FK + index, `(tenant, service_id)` FK + index, and unique `(tenant, contract_line_id, service_id)` constraint on `contract_line_service_configuration`; add `(tenant, config_id)` FK on `contract_line_service_bucket_config`; audit `bucket_usage` to ensure it will link to contract-line overlays post-cutover.

Acceptance
- Documentation notes captured in this plan; team aligned on scope.

Rollout
- N/A

---

## Phase 2 — Test Helpers & Unit Tests First

Goals
- Migrate shared test helpers to create “bucket overlays” on Hourly/Usage lines instead of a dedicated `Bucket` plan type.

Tasks
- Update helper in `server/test-utils/billingTestHelpers.ts`:
  - Replace `createBucketPlanAssignment(...)` with `createBucketOverlayForPlan(...)` that:
    - Ensures an Hourly or Usage plan exists (create if needed).
    - Inserts `plan_services` entries for target services (if missing).
    - Upserts `plan_service_configuration` with `configuration_type = 'Bucket'` for each service.
    - Inserts `plan_service_bucket_config` with: `total_minutes` (for hours) or units as “minutes” generic quantity for usage, `overage_rate`, `allow_rollover`, `billing_period`.
  - Remove the old function usage across the suite; do not maintain a shim.
- Update/adjust unit tests that explicitly assert `plan_type = 'Bucket'`:
  - `server/src/test/unit/contractLineDisambiguation.test.ts`: remove bucket-plan special cases; add overlay preference tests (prefer lines with active bucket balance).
  - `server/src/test/unit/timeEntryBillingPlanSelection.test.tsx`: mock overlay presence (not plan type) and assert default selection preference.
  - `server/src/test/unit/billingPlanSelection.test.ts`: same as above.
- `server/src/test/unit/billingEngine.test.ts`: replace “calculateBucketPlanCharges” expectations with overlay consumption assertions (time/usage paths).
- `server/src/test/unit/bucketUsageService.test.ts`: confirm logic still works with overlays.
- Progress (2025-02-14):
  - Helper rewritten as `createBucketOverlayForPlan` and wired to both contract-line and legacy plan tables; usage helper now records `contract_line_id`.
  - Invoice infrastructure test (`usageBucketAndFinalization`) migrated to build overlays on top of fixed lines and seed usage via overlay helper.
  - Selector/disambiguation unit tests now mock `has_bucket_overlay` metadata instead of `contract_line_type === 'Bucket'`; pending updates to the production disambiguation logic and remaining billing engine/unit suites.
  - Billing engine unit coverage updated to treat overlays as the source for bucket charges (`calculateBucketPlanCharges` test now stubs overlay rows and bucket usage, and `calculateBilling` aggregates the overlay charge).
  - UI/service layers now consume overlay metadata: contract-line selectors prefer overlays, usage/time-entry actions update bucket usage when overlays exist (rather than relying on `contract_line_type`), and API imports point at `contractLineDisambiguation`.
- Follow-up work to finish ripping out bucket plans
  1. **Contract wizard + Playwright flows**
     - Update `contractWizardActions` so bucket configuration toggles add overlays to the existing fixed contract line (never create a `contract_line_type = 'Bucket'` row).
     - Remove bucket-plan assertions from the Playwright suite (`ee/server/src/__tests__/integration/contract-wizard-happy-path.playwright.test.ts`) and replace them with overlay verifications (`contract_line_service_configuration` + `contract_line_service_bucket_config` joins).
     - Adjust wizard helpers/seeders to account for overlay toggles and ensure monthly-fee behaviour still covered.
     - Drop `bucket_contract_lines` from test cleanup lists once the wizard no longer creates them.
  2. **Reporting/analytics surfaces**
     - Switch `getRemainingBucketUnits`, client-portal billing metrics, and account dashboards to join `contract_line_service_configuration` + bucket overlays (remove `.where(plan_type = 'Bucket')` or `contract_line_type === 'Bucket'` filters).
     - Update any UI components (e.g. `BucketContractLineConfiguration`) that guard on plan type so they inspect overlay metadata instead.
  3. **Service APIs & helpers**
     - Finish migrating remaining server actions and utilities that still query `contract_line_type` (search codebase for `'Bucket'` checks) – specifically `ContractLineServiceConfigurationService`, report actions, constants, and interfaces in `billing.interfaces.ts`.
     - Update TypeScript enums/interfaces to drop `'Bucket'` from `contract_line_type` once overlay consumers are in place.
  4. **Database/tests cleanup**
     - Remove dependencies on the `bucket_contract_lines` table from tests; plan a follow-up migration to archive/drop the table after the data migration script (Phase 8) runs.
     - Sweep remaining tests/integration helpers that reference legacy plan tables (e.g. `bundle_billing_plans` lookups) and ensure they validate overlays instead.

Acceptance
- Unit tests compile and run; bucket assertions are overlay-based.

Rollout
- N/A

---

## Phase 3 — Contract Wizard: Per-Line Bucket Overlays

Goals
- Replace the separate “Bucket Services” step with per-line overlays inside Hourly and Usage steps.

Tasks
- Remove the dedicated wizard step (UI only change):
  - `server/src/components/billing-dashboard/contracts/ContractWizard.tsx`
    - Update `STEPS` to drop “Bucket Services” and adjust navigation/validation.
    - Update `validateStep` logic to stop validating the removed step.
- Introduce per-line bucket overlay in:
  - `FixedFeeServicesStep.tsx` and `HourlyServicesStep.tsx`: Add a toggle and fields per service for included hours (stored as minutes), overage rate (cents), and rollover support.
  - `UsageBasedServicesStep.tsx`: Provide the same overlay toggle with included units (stored as generic minutes), overage rate, and rollover toggle.
- Create a reusable `BucketOverlayFields` component that renders the shared overlay inputs and automation hooks; reuse it across the fixed, hourly, and usage steps.
- Update the review step to summarize overlays under each line item.

Acceptance
- Wizard lets users add bucket overlays per Hourly/Usage line with proper validation.
- No separate bucket step shown when the flag is ON.

Rollout
- N/A

---

## Phase 4 — Wizard Submission & Persistence

Goals
- Persist per-line overlays instead of creating a `Bucket` plan.

Tasks
- Update `ContractWizardData` and the submission payload to capture a `bucket_overlay` object per fixed/hourly/usage service with fields:
  - `{ total_minutes?: number, overage_rate?: number, allow_rollover?: boolean, billing_period?: 'monthly' | 'weekly' }`
- Modify `server/src/lib/actions/contractWizardActions.ts`:
  - Remove creation of a dedicated `Bucket` plan.
  - After creating each contract-line service configuration, upsert a matching `contract_line_service_configuration` + `contract_line_service_bucket_config` row when `bucket_overlay` is provided, using a shared helper to insert/update the overlay metadata.
- Monthly fee handling (decision):
  - Option A (preferred for now): bill the “bucket monthly fee” by adding/allocating a Fixed Fee line scoped to that service (documented on the plan). No schema change needed; engine already handles fixed fees.
  - Option B (future): add `monthly_fee` to `plan_service_bucket_config` (requires migration and engine update). Leave as a future enhancement.

Acceptance
- New contracts with overlays persist correctly and no `Bucket` plan is created.

Rollout
- N/A

---

## Phase 5 — Billing Engine: Consume Overlays

Goals
- Calculate included minutes/units and overage from overlay configs during invoice generation; eliminate reliance on a `Bucket` plan type.

Tasks
- In `server/src/lib/billing/billingEngine.ts`:
  - Remove/retire `calculateBucketPlanCharges` (or convert into an internal helper that aggregates overlay consumption per line).
  - For time-based charges:
    - Fetch applicable overlay configs for the Hourly line’s plan/service during the billing period.
    - Compute available included minutes from `plan_service_bucket_config` minus previously used (`bucket_usage`) for that period.
    - Allocate billable time first against remaining bucket minutes; compute overage minutes and bill at `overage_rate`.
  - For usage-based charges:
    - Same pattern using “units” (represented as minutes in `total_minutes` for parity).
  - Write/update `bucket_usage` for the period with new consumption and overage.
  - Ensure tax logic remains consistent; overage lines should apply the service’s tax config.

Acceptance
- Engine produces correct totals for lines with bucket overlays (included + overage).
- Infrastructure invoice tests pass with overlays.

Rollout
- N/A

---

## Phase 6 — Plan Disambiguation Update

Goals
- Simplify plan selection by preferring plans with active overlays (balance available), removing bucket plan special-cases.

Tasks
- In `server/src/lib/utils/contractLineDisambiguation.ts`:
  - `determineDefaultBillingPlan(clientId, serviceId)`:
    - If a single eligible plan → select it.
    - If multiple: prefer the plan where the overlay exists and has remaining balance in the entry’s period (requires a helper to query overlay and usage balance).
    - Otherwise, return null and require explicit selection.
  - `getEligibleBillingPlansForUI`: include an overlay marker and, if feasible, a quick “has_balance” boolean for UX hints.
- Update unit tests to reflect new rules.

Acceptance
- Disambiguation chooses the overlay plan when sensible; tests cover tie cases.

Rollout
- N/A

---

## Phase 7 — Time Entry Plan Selector Defaults

Goals
- Keep the selector optional, default intelligently to overlay-backed plan with balance.

Tasks
- In `server/src/components/time-management/time-entry/time-sheet/TimeEntryEditForm.tsx`:
  - When multiple plans are eligible, default to the one with an active overlay and remaining balance (fall back to previous heuristics if unknown).
  - Show the selector only when >1 eligible plans.
  - Tooltip copy to reflect overlay preference.

Acceptance
- UI defaults match disambiguation; selector appears only when needed.

Rollout
- N/A

---

## Phase 8 — Integration & Invoice Tests

Goals
- Ensure end-to-end generation with overlays passes and equals legacy behavior for equivalent configurations.

Tasks
- Update integration/infrastructure tests to use `createBucketOverlayForPlan`:
  - `usageBucketAndFinalization.test.ts`
  - `prepaymentInvoice.test.ts`
  - Invoice subtotal/tax/consistency/edge-case tests that reference bucket tables.
- Keep assertions on `bucket_usage` updates and invoice line item correctness (included vs overage).

Acceptance
- Tests pass with overlays; no regressions in totals or tax.

Rollout
- N/A

---

## Phase 9 — Cleanup

Goals
- Remove `Bucket` from `plan_type` usages in code after the feature is stable and migration is complete.

Tasks
- Remove bucket plan special-cases across code and tests so overlays are the sole bucket source:
  - `server/src/lib/actions/contractWizardActions.ts`: collapse the “bucket plan” branch so the wizard only ever mutates the fixed/usage line and appends overlay configuration rows; purge any fallbacks that still create a `contract_line_type = 'Bucket'`.
  - `ee/server/src/__tests__/integration/contract-wizard-happy-path.playwright.test.ts`: update assertions/helper flows to verify overlay persistence (`contract_line_service_configuration` + `contract_line_service_bucket_config`) and stop expecting a bucket contract line or plan type string.
  - Reporting and analytics callers (`server/src/lib/actions/client-portal-actions/client-billing-metrics.ts`, `server/src/lib/actions/report-actions/getRemainingBucketUnits.ts`, CSV/BI exports) need to replace `contract_line_type === 'Bucket'` filters with overlay joins and expose overlay metadata to the UI.
  - Billing/usage helpers (`server/src/lib/utils/contractLineDisambiguation.ts`, `server/src/lib/services/contractLineServiceConfigurationService.ts`, `server/src/lib/billing/billingEngine.ts`, `server/src/lib/services/bucketUsageService.ts`) should look up overlays via `contract_line_service_configuration` and never branch on plan type.
  - Test + seed utilities (`server/test-utils/billingTestHelpers.ts`, infrastructure invoice suites, nightly cleanup scripts) must drop `bucket_contract_lines` cleanup and rely solely on overlay tables.
- Update enum/documentation to drop `Bucket` as a plan type once callers above run overlay-only logic (e.g. `server/src/interfaces/billing.interfaces.ts`, `server/src/constants/billing.ts`, API surface types).
- Delete legacy UI step/components that are no longer referenced (wizard bucket step, bucket-plan filters, etc.).
- Add guardrails: introduce an automated check (`rg "contract_line_type.*Bucket"`) in CI or lint scripts so new plan-type branches cannot be reintroduced.

Acceptance
- Repository compiles and tests green without any `plan_type='Bucket'` coupling.

Rollout
- N/A

Verification
- `rg "contract_line_type.*'Bucket'"` and `rg "plan_type.*Bucket"` return no results in TypeScript/TSX sources (excluding documentation).
- Playwright contract wizard regression passes while persisting overlays.
- Billing smoke tests confirm bucket invoices derive from overlay rows only.

---

## Phase 10 — Documentation & Rollout

Goals
- Provide clear internal and customer-facing docs on overlays.

Tasks
- Internal docs:
  - Architecture overview (overlay model, configs, usage ledger).
  - Disambiguation rules, defaulting, and reporting by `billing_plan_id`.
- Customer-facing docs (if applicable):
  - “Bucket of Hours/Units” under Hourly/Usage lines; effective rate; overage; rollover.
- Release notes and upgrade guidance for existing tenants.

Acceptance
- Docs published and linked in relevant UI tooltips.

Rollout
- N/A

---

## Appendix — Implementation Notes

- Table references (no schema changes required for MVP):
  - `plan_service_configuration` (ensure per plan/service `configuration_type = 'Bucket'` exists)
  - `plan_service_bucket_config` (uses `total_minutes` for hours or usage units; `overage_rate`, `allow_rollover`, `billing_period`)
  - `bucket_usage` (minutes/units used + overage per period)
- Monthly fee for buckets:
  - MVP: bill via Fixed Fee lines (per plan or per service allocation); engine already handles fixed charges.
  - Future: add `monthly_fee` to `plan_service_bucket_config` if we want it stored alongside the overlay.
- Plan disambiguation/UI:
  - `getEligibleBillingPlansForUI` may include overlay hints: `{ has_overlay: boolean, has_balance?: boolean }` to aid defaults and UX copy.
- Reporting:
  - Keep `billing_plan_id` on time entries/usage for plan-level reporting; overlays don’t change that contract.

---

## Success Criteria

- Users configure bucket hours/units directly on Hourly/Usage lines.
- Time entries/usage automatically consume included amounts; overage is billed correctly.
- Default plan selection prefers overlay-backed lines when appropriate.
- All tests pass; migrations maintain historical accuracy; no regressions in invoice totals or taxes.