PSA/ee/docs/plans/2026-03-18-service-driven-invoicing-cutover/RUNBOOK.md

# Service-Driven Invoicing Cutover Runbook

## Purpose

Use this runbook when recurring due rows are missing or stale during the service-driven invoicing cutover.

It is the operator/developer entrypoint for:

- confirming whether a ready-to-invoice gap is caused by missing persisted `recurring_service_periods`
- assessing whether future service periods reach the required generation horizon
- running backfill or regeneration planning with the shared cutover helpers
- validating that repaired rows reappear in the due-work reader

Until a dedicated billing-maintenance UI exists, the supported maintenance entrypoint is the shared helper layer under `shared/billingClients/*.ts`, invoked through `pnpm exec tsx`.

## Cutover Sequence

Use this sequence when moving a tenant from billing-cycle-driven recurring operations to service-driven recurring invoicing.

1. Confirm cadence-owner authoring and selector-input recurring generation are already deployed in the target environment.
2. Validate that the tenant has the new `billing.recurring_service_periods` permissions granted to the operator roles that will inspect or repair service periods.
3. Run the coverage assessment for active recurring schedules to identify any obligation whose persisted future rows do not reach the target horizon.
4. Backfill active obligations that have no persisted future rows yet.
5. Regenerate untouched future rows for obligations whose cadence-owner, billing timing, or activity-window rules changed during rollout.
6. Re-run the due-work reader and confirm canonical persisted rows now appear for the expected invoice windows instead of compatibility billing-cycle fallbacks.
7. Exercise one contract-cadence invoice end-to-end through preview, generate, and invoiced history.
8. Exercise one post-drop client-cadence invoice end-to-end to confirm the compatibility obligation label still resolves through the surviving client-owned contract structure during cutover.
9. Only after both paths validate should billing ops rely on the `AutomaticInvoices` due-work table as the primary recurring surface for that tenant.

## Quick Diagnosis

### 1. Confirm the gap is materialization-related

If the due-work reader returns a row with:

- `reason: missing_service_period_materialization`
- a `billing_cycle_id`
- no persisted canonical row for the same execution identity

then the system is falling back to a compatibility billing-cycle row because the recurring service-period ledger is missing for that window.

The first validation step is the focused due-work reader harness:

```bash
cd server
pnpm exec vitest run src/test/unit/billing/recurringDueWorkReader.integration.test.ts -t "T075" --coverage.enabled=false
```

### 2. Inspect persisted rows directly

```sql
select
  record_id,
  schedule_key,
  period_key,
  lifecycle_state,
  cadence_owner,
  charge_family,
  service_period_start,
  service_period_end,
  invoice_window_start,
  invoice_window_end,
  invoice_id,
  invoice_charge_detail_id
from recurring_service_periods
where tenant = :tenant
  and client_id = :client_id
order by invoice_window_end desc, service_period_start desc;
```

### 3. Inspect the recurring obligations that should have generated rows

```sql
select
  cc.client_id,
  cl.contract_line_id,
  cl.cadence_owner,
  cl.billing_frequency,
  cl.billing_timing,
  cc.start_date,
  cc.end_date
from client_contracts cc
join contracts ct
  on ct.contract_id = coalesce(cc.template_contract_id, cc.contract_id)
 and ct.tenant = cc.tenant
join contract_lines cl
  on cl.contract_id = ct.contract_id
 and cl.tenant = ct.tenant
where cc.tenant = :tenant
  and cc.client_id = :client_id
  and cc.is_active = true
  and cl.is_recurring = true
order by cl.contract_line_id;
```

If the recurring obligation exists but the matching `recurring_service_periods` row does not, continue with coverage assessment and backfill/regeneration planning.

## Migration Checklist

Run this checklist before declaring the tenant ready for service-driven recurring invoicing:

- backfill active recurring obligations that still have zero future persisted rows
- replenish schedules whose generated horizon no longer reaches the target billing-ops window
- regenerate untouched future rows after cadence-owner, timing, or activity-window changes
- validate due-work selection for both contract-cadence and post-drop client-cadence windows
- validate preview/generate behavior for one unbridged contract-cadence window
- validate reverse/delete repair for one billed recurring invoice so reopened service periods reappear correctly
- keep compatibility billing-cycle monitoring in place until the due-work reader no longer reports missing-materialization fallbacks for the tenant

## Coverage Assessment Entry Point

Use this when you already have the current future records for one schedule or contract line and need to know whether the generated horizon is sufficient.

Prepare a JSON file with the current future rows, for example `tmp/recurring-service-period-existing.json`.

```bash
pnpm exec tsx <<'TS'
import fs from 'node:fs';
import { assessRecurringServicePeriodGenerationCoverage } from './shared/billingClients/recurringServicePeriodGenerationHorizon.ts';

const existingRecords = JSON.parse(
  fs.readFileSync('./tmp/recurring-service-period-existing.json', 'utf8'),
);

const coverage = assessRecurringServicePeriodGenerationCoverage({
  existingRecords,
  generatedThrough: '2026-03-18T00:00:00Z',
});

console.log(JSON.stringify(coverage, null, 2));
TS
```

Interpretation:

- `meetsTargetHorizon: false` means the schedule does not extend far enough ahead
- `needsReplenishment: true` means the remaining future horizon has dropped below the operational threshold
- continuity issues mean fix the stored rows before running generation again

## Backfill Entry Point

Use this when active recurring obligations have no persisted future rows yet and the system is falling back to compatibility billing-cycle rows.

Prepare:

- `tmp/recurring-service-period-candidates.json` with the candidate rows to materialize
- `tmp/recurring-service-period-existing.json` with any already-persisted rows for the same schedule

```bash
pnpm exec tsx <<'TS'
import fs from 'node:fs';
import { backfillRecurringServicePeriods } from './shared/billingClients/backfillRecurringServicePeriods.ts';

const candidateRecords = JSON.parse(
  fs.readFileSync('./tmp/recurring-service-period-candidates.json', 'utf8'),
);
const existingRecords = JSON.parse(
  fs.readFileSync('./tmp/recurring-service-period-existing.json', 'utf8'),
);

const plan = backfillRecurringServicePeriods({
  candidateRecords,
  existingRecords,
  backfilledAt: '2026-03-18T00:00:00Z',
  sourceRuleVersion: 'service-driven-invoicing-cutover-v1',
  sourceRunKey: 'manual-backfill-2026-03-18',
});

console.log(JSON.stringify(plan, null, 2));
TS
```

What to look for:

- `backfilledRecords` are new future rows to persist
- `realignedRecords` are regenerated replacements for mismatched untouched rows
- `skippedHistoricalCandidates` confirms the helper did not rewrite billed history
- any overlap error means the candidate set crosses the billed-history boundary and must be corrected before persistence

## Regeneration Entry Point

Use this when rows already exist but cadence-owner, timing, anchor, or activity-window changes mean untouched future rows must be refreshed.

Prepare:

- `tmp/recurring-service-period-existing.json`
- `tmp/recurring-service-period-candidates.json`

```bash
pnpm exec tsx <<'TS'
import fs from 'node:fs';
import { regenerateRecurringServicePeriods } from './shared/billingClients/regenerateRecurringServicePeriods.ts';

const existingRecords = JSON.parse(
  fs.readFileSync('./tmp/recurring-service-period-existing.json', 'utf8'),
);
const candidateRecords = JSON.parse(
  fs.readFileSync('./tmp/recurring-service-period-candidates.json', 'utf8'),
);

const plan = regenerateRecurringServicePeriods({
  existingRecords,
  candidateRecords,
  regeneratedAt: '2026-03-18T00:00:00Z',
  sourceRuleVersion: 'service-driven-invoicing-cutover-v1',
  sourceRunKey: 'manual-regeneration-2026-03-18',
});

console.log(JSON.stringify(plan, null, 2));
TS
```

What to look for:

- `preservedRecords` remain authoritative because they were billed, locked, or user-edited
- `regeneratedRecords` replace untouched future rows
- `supersededRecords` are the rows that should no longer be considered active
- `conflicts` must be reviewed before persisting because they mean a preserved override no longer matches regenerated candidates

## Persistence Rules

When applying a backfill or regeneration plan to the database:

- do not mutate `billed`, `edited`, or `locked` rows in place
- insert new generated/regenerated rows first
- only supersede untouched future rows that the plan explicitly replaced
- keep `invoice_id`, `invoice_charge_id`, and `invoice_charge_detail_id` intact for billed history
- if a deleted recurring invoice has already reopened linked rows to `locked`, treat those rows as preserved history-aware records, not disposable generated rows

## Post-Repair Validation

### Due-work reader validation

Run the focused due-work reader check after persisting repaired rows:

```bash
cd server
pnpm exec vitest run src/test/unit/billing/recurringDueWorkReader.integration.test.ts --coverage.enabled=false
```

The repaired execution window should either:

- appear as a canonical persisted due-work row, or
- disappear entirely because the row is not actually due

It should not continue surfacing only as a compatibility billing-cycle fallback unless materialization is still missing.

### DB-backed recurring linkage validation

Use the local Postgres listener on `127.0.0.1:57433` for focused recurring cutover checks:

```bash
cd server
DB_HOST=127.0.0.1 DB_PORT=57433 DB_NAME=server DB_NAME_SERVER=server DB_USER_SERVER=app_user DB_USER_ADMIN=postgres DB_PASSWORD_SERVER=postpass123 DB_PASSWORD_ADMIN=postpass123 pnpm exec vitest run --coverage.enabled=false src/test/integration/billingInvoiceTiming.integration.test.ts -t "T019|T020|T021"
```

What this proves:

- billed recurring service periods link back to canonical invoice detail rows
- unbridged contract-cadence invoices still link correctly
- deleting a linked recurring invoice reopens the affected service period instead of leaving it permanently hidden

## Reverse/Delete Repair Notes

If a recurring invoice was deleted during cutover testing and the due row did not come back:

1. query `recurring_service_periods` for the original `invoice_id`
2. confirm `invoice_id`, `invoice_charge_id`, and `invoice_charge_detail_id` were cleared
3. confirm `lifecycle_state = 'locked'`
4. rerun due-work validation

For reverse/delete validation during tenant rollout:

1. capture the recurring row's `schedule_key`, `period_key`, `invoice_id`, and `executionIdentityKey`
2. perform the reverse or hard-delete action from the billing UI
3. confirm invoiced history reflects the new invoice state
4. confirm the linked `recurring_service_periods` rows either reopen to `locked` or remain correctly linked for post-drop client-cadence behavior
5. rerun the due-work reader to confirm the same execution window is visible again when it should be invoiceable
6. if the row does not reappear, use the linkage and coverage checks above before retrying generation

`locked` is the expected restored state after delete-repair because the system cannot safely infer whether the prior mutable state was exactly `generated` or `edited`.

## Escalate Instead Of Forcing A Repair

Stop and investigate before persisting a plan if any of the following are true:

- regeneration conflicts mention `missing_candidate`, `service_period_mismatch`, `invoice_window_mismatch`, or `activity_window_mismatch`
- continuity issues show gaps or overlaps inside the active future ledger
- a billed row would need to move service-period boundaries
- the only available history for a disputed invoice exists in `invoice_charge_details` and not in persisted `recurring_service_periods`

In those cases, preserve the existing records, capture the conflicting rows, and repair the source cadence/timing inputs first.