PSA/ee/docs/plans/2025-10-29-extension-runtime-metadata-plan.md

Extension Runtime Metadata & Secrets Delivery Plan (Componentize-JS First)

Overview

• Deliver tenant/install-scoped configuration and secrets (e.g., Alga API tokens) to Wasm extensions while keeping values encrypted at rest and never logging plaintext.
• Standardize extension builds on the Wasmtime component model and componentize-js so developers ship typed components instead of raw Wasm modules.
• Reuse the existing composite secret infrastructure (shared/core/secretProvider, Vault, Docker secrets per docs/secrets_management.md) for both control plane storage and Runner decryption.
• Treat host functionality (secrets, outbound HTTP, UI proxy calls) as declarative “capability providers” exposed via WIT, inspired by wasmCloud’s provider model.
• Provide an opinionated SDK pipeline in ./sdk that wraps generated bindings so TypeScript developers can consume secrets/config without writing Rust or manual ABI glue, mirroring wasmCloud’s wash developer experience.

Goals

• Define WIT worlds for Runner ↔ guest interactions (context, secrets, HTTP, storage, logging, UI proxy) and instantiate components via Wasmtime’s component APIs.
• Shape host functionality as modular capability providers with explicit manifests so extensions declare the providers they need at publish time.
• Integrate componentize-js (currently v0.19.3, released 2025-10-27 per CHANGELOG.md) into the build pipeline so JS/TS projects emit .wasm components with stable metadata.
• Publish SDK packages (initially JS/TS) generated from WIT so developers retrieve secrets via await ctx.secrets.get(...) with minimal boilerplate.
• Keep rotation fast: cache decrypted material briefly, respect version stamps, and rely on the composite secret provider as the system of record.

Non-Goals

• Replacing the secret provider stack—Vault/Docker secrets remain authoritative; the Runner consumes its outputs.
• Shipping a JS VM in the Runner; extensions still compile to Wasm components using componentize-js.
• Solving deep observability/runbook requirements in this pass (tracked separately once the foundation ships).
• Providing backward compatibility adapters for legacy module-based extensions; the focus is the new component-based pipeline.

Current State (2025-10-29)

• Runner request model (ee/runner/src/models.rs) lacks config/secret fields; host context in ee/runner/src/engine/loader.rs tracks IDs only.
• Host imports in ee/runner/src/engine/host_api.rs are hand-wired functions (logging, http, storage) built for module-style Wasmtime; no alga.secrets surface exists.
• Gateway proxy (ee/server/src/app/api/ext/[extensionId]/[...path]/route.ts) forwards to /v1/execute without fetching install metadata or secrets.
• Control-plane services already depend on shared/core/secretProvider.ts to source secrets from env/filesystem/Vault (documented in docs/secrets_management.md); Runner has no integration.
• ./sdk contains CLI tooling and iframe helpers but no generated runtime bindings or component build pipeline.

Status update (2025-11-21):

• Gateway now forwards config, providers, and secretEnvelope in execute payloads (server/src/app/api/ext/[extensionId]/[[...path]]/route.ts).
• Runner host implements secrets and storage capability providers and calls POST /api/internal/ext-storage/install/{installId} with RUNNER_STORAGE_API_TOKEN (ee/runner/src/engine/host_api.rs).
• Runtime uses Wasmtime Component Model; wasm-js@1 is the enforced runtime and manifests are validated via manifest-v2.schema.ts.
• Componentized SDK template exists (sdk/alga-client-sdk/templates/component-basic) and buildExtUiSrc/iframe bootstrap are stable; still need generated bindings packaging and schema/docs alignment (JSON vs zod).

Tooling Status Snapshot (Oct 2025)

• componentize-js latest tag v0.19.3 (2025-10-27) fixes duplicate export naming, updates dependencies (orca_wasm → wirm), and keeps StarlingMonkey aligned—ensuring the jco toolchain is active and maintained.
• Recent releases (v0.19.2, v0.19.1) focus on Windows CI stability and StarlingMonkey updates, confirming cross-platform support and ongoing maintenance cadence.
• The toolchain now emits full component metadata compatible with Wasmtime 19.x+; we should plan to pin to componentize-js ≥ 0.19.3 and track upstream changelog for breaking changes.

Requirements & Constraints

• Component Model First: All new host APIs are defined in WIT and surfaced to guests through Wasmtime component instantiation. No new raw func_wrap imports.
• Secret Provenance: The control plane packages secrets using secretProvider as Vault transit ciphertext. Runner decrypts using Vault tokens mounted as Docker secrets.
• Capability Enforcement: Access to secrets/config is gated by manifest-declared scopes (secrets.get, config.read) and enforced in the host implementation.
• Componentize-JS Pipeline: JS/TS extensions must build via componentize-js (jco) with a standardized project template in sdk/. The pipeline outputs .wasm + metadata for publishing.
• Rotation & Caching: Envelopes include version/expires_at; Runner caches decrypted material in an LRU keyed by (tenant, install, version) with short TTL.
• DX: Generated bindings (JS/TS) expose ergonomic helpers (ctx.secrets.get, ctx.http.fetch) and hide low-level ABI concerns.
• Security: Plaintext secrets never hit logs, idempotency caches, or panic traces. Structures holding secrets must not implement Debug output. Browser/iframe surfaces never receive secrets; UI flows rely on host proxy APIs instead.

Proposed Architecture

1. WIT Worlds & Capability Provider Surface

• Author wit/extension-runner.wit describing:
  • interface alga.context (IDs, config map, request metadata).
  • interface alga.secrets (get, optional list, structured errors).
  • interface alga.http, alga.storage, alga.log, and alga.ui_proxy (for host-mediated UI actions).
• Encode capability requirements using custom metadata (e.g., @requires("cap:secrets.get")) so registry validation, host enforcement, and generated bindings stay in sync.
• Version the WIT world and publish alongside the SDK to support additive evolution.

2. Capability Provider Catalog

• Treat each host feature as a provider, similar to wasmCloud:
  • Core providers: secrets, outbound HTTP, storage, logging.
  • UI proxy provider: exposes predefined host endpoints extensions can call from the browser via gateway.
  • Future providers: messaging, scheduler hooks, custom domain integrations.
• Maintain provider definitions (WIT imports + manifest identifiers) so the registry knows which providers an extension needs and operators can enable/disable them per tenant.
• Document provider lifecycle (enable/disable, version bump) and align on naming (cap.alga.secrets, cap.alga.ui_proxy, etc.).

3. Control Plane, Gateway & Declarative Metadata

• Extend control-plane schema (tenant_extension_install_config, tenant_extension_install_secrets) storing Vault-transit ciphertext via secretProvider.
• Capture provider requirements in install metadata: manifest declares required providers; registry enforces availability before activation.
• Create service endpoint (POST /internal/runner/install-config) returning {config, secret_envelope, provider_flags, version}.
• Update Gateway to:
  • Fetch install config/secrets before invoking Runner.
  • Attach provider flags + version headers (x-ext-config-version, x-ext-secrets-version).
  • Expose host proxy routes (UI → gateway → runner handler) mapped to the alga.ui_proxy provider so browser clients never see secrets.

4. Runner Component Host Runtime

• Upgrade Wasmtime to a component-ready release and instantiate components via componentize-js metadata.
• Extend host context to store config maps, provider flags, and secret envelopes.
• Implement provider dispatch: the runner mounts each capability provider implementation (e.g., secrets provider decrypts Vault ciphertext mounted at /run/secrets/...).
• Secret redemption: decrypt Vault transit ciphertext in-process using mounted tokens/keys, populate short-lived cache keyed by (tenant, install, version).
• Ensure provider calls zeroize buffers, apply capability checks, and emit structured errors.

5. Developer Workflow & CLI (Componentize-JS Pipeline)

• Ship sdk/packages/component-runtime-template mirroring wasmCloud’s wash flow:
  1. alga-cli new component scaffolds TS project with tests and provider manifests.
  2. alga-cli dev builds via componentize-js, spins up a local runner shim, and exercises provider routes (including UI proxy).
  3. alga-cli publish bundles .wasm, .wit, provider manifest, and metadata.json for registry upload.
• Validate toolchain versions (componentize-js ≥ 0.19.3, Wasmtime version) and fail fast if mismatched.
• Expose extension install automation via POST /api/v1/extensions/install so CLI workflows can bypass the admin UI once API keys are provisioned.
• Provide smoke tests that run components against mocked capability providers to catch ABI drift before upload.

7. UI & Configuration Experience

• Publish @alga-psa/extension-runtime (JS/TS) wrapping generated bindings with helpers (createHandler, ctx.secrets.get, ctx.uiProxy.call).
• Ship UI-side helpers (@alga/extension-ui) that call gateway proxy endpoints with tenant/install context.
• Document workflows in sdk/docs: local dev loop, invoking provider APIs, using UI proxy without handling secrets.
• Provide runnable samples mirroring wasmCloud's language examples (TypeScript initially, add Rust/TinyGo later via wit-bindgen).

8. Extension Settings UI

• Extension Settings Page: A dedicated configuration UI at /msp/settings/extensions/[id]/settings reachable from the extension management list.
• Dynamic Form Generation: Render configuration inputs (text, number, boolean, select) based on the extension's manifest settings schema.
• Secret Management:
  • Distinct UI section for encrypted values (secrets) separate from plain configuration.
  • Write-only inputs for secrets (never echo back values).
  • Version tracking (secretsVersion) to indicate if a secret is set and when it was last updated.
• Actions:
  • "Save Changes": Persist both config (to tenant_extension_install_config) and secrets (to Vault via tenant_extension_install_secrets).
  • "Reset to Defaults": Revert configuration to manifest defaults and clear secrets.
• RBAC: Ensure only admins with appropriate permissions can view/edit these settings.
• Entry Point: Connect the "Settings" button in the Extension Management table (SettingsPage.tsx -> Extensions.tsx) to this new page.
• Publish @alga-psa/extension-runtime (JS/TS) wrapping generated bindings with helpers (createHandler, ctx.secrets.get, ctx.uiProxy.call).
• Ship UI-side helpers (@alga/extension-ui) that call gateway proxy endpoints with tenant/install context.
• Document workflows in sdk/docs: local dev loop, invoking provider APIs, using UI proxy without handling secrets.
• Provide runnable samples mirroring wasmCloud's language examples (TypeScript initially, add Rust/TinyGo later via wit-bindgen).

Implementation Phases

Phase 0 — Design & Toolchain Alignment

• Finalize WIT interface set and capability annotations.
• Pin toolchain versions (componentize-js ≥ 0.19.3, Wasmtime ≥ component-ready release) and capture upgrade strategy.
• Define secret envelope format (Vault transit ciphertext) and required Vault roles/tokens to mount into Runner containers. (Implemented base64 fallback plus Vault transit support with token file + mount configuration.)
• Design the capability provider catalog (IDs, manifests, lifecycle) and document how providers map to WIT imports. (Initial catalog lives in ee/runner/src/providers with normalization + validation helpers.)

Phase 1 — Control Plane & Gateway

• Implement schema migrations and persistence for config + secrets (ciphertext via secretProvider).
  • Added tenant_extension_install_config and tenant_extension_install_secrets via 20251031130000_create_install_config_tables.cjs, including UUID PKs, tenant/ install indexes, and Vault-friendly metadata (algorithm, transit_key, version, expires_at).
  • Introduced writer APIs in ee/server/src/lib/extensions/installConfig.ts (upsertInstallConfigRecord, upsertInstallSecretsRecord, deleteInstallSecretsRecord) that normalize capability sets, emit deterministic versions, and attempt Vault transit encryption with inline/base64 fallback.
  • Install flows (ExtensionRegistryServiceV2.install, installExtensionForCurrentTenantV2) now upsert config/secrets metadata and persist granted provider scopes alongside install records.
• Build the install-config service endpoint returning config + envelopes + version metadata.
  • /api/internal/ext-runner/install-config returns merged config/providers plus Vault envelope data, secured by x-runner-auth; re-used by runner and internal tooling.
  • Service hydrates bundle hash, manifest capabilities, install overrides, and exposes configVersion / secretsVersion with ISO timestamps for cache auditing.
• Update Gateway to call the endpoint, enrich execute payloads, and emit version headers.
  • Gateway routes obtain install state through loadInstallConfigCached (new helper under ee/server/src/lib/extensions/lib/install-config-cache.ts) before dispatching to /v1/execute.
  • Execute payload now carries { context.config, providers[], secret_envelope } plus headers x-ext-config-version / x-ext-secrets-version; runner errors are surfaced as 502 with request-id tagged logs.
  • Added a configurable (default 5s) per-install in-memory cache that keys on tenant+extension and refreshes on expiry or missing rows.
• Implement host-side proxy routes (UI → gateway → runner handler) so iframe code can trigger backend actions without receiving secrets directly.
  • /api/ext-proxy/[extensionId]/[...path] mirrors gateway auth, invokes runner ui-proxy:* handlers, and streams responses after header filtering; shares install-config cache to avoid duplicate lookups.
  • Maintains tenant RBAC checks and logs missing bundle/config scenarios for audit.
• Validate manifest/provider declarations during publish and record provider enablement per install.
  • Version publish continues to reject unknown capabilities; install paths normalize requested scopes against the known provider catalog before persisting.
  • Default provider set (cap:context.read, cap:log.emit) is enforced server-side so installs inherit baseline capabilities even if manifests omit them.

Phase 2 — Runner Component Host

• Detailed execution steps live in extension-runtime-wasmtime-plan.md; keep both documents aligned as milestones land.
• Upgrade Runner to Wasmtime component APIs and instantiate components for /v1/execute.
• Implement host functions per WIT (context, secrets, http, storage, logging) with capability checks.
• Add secret envelope redemption via Vault transit decryption (token mounted as Docker secret) plus short-lived cache; ensure no secret leakage in logs/traces.
• Wire provider registry inside the runner so each capability (secrets, ui_proxy, etc.) can be swapped/extended without touching core execute logic.

Phase 3 — SDK & Tooling (Componentize-JS Pipeline)

• Build the componentize-js project template and integrate it into alga-cli.
• Generate JS/TS bindings from WIT, publish @alga-psa/extension-runtime, and document usage. (Manual first-pass bindings plus proxy helpers shipped under sdk/extension-runtime.)
• Provide sample extension + automated test to validate secrets retrieval through the new host interface. (See sdk/samples/component/secrets-demo with Vitest example.)
• Enforce component artifact validation in the publishing pipeline (reject raw Wasm uploads). (Pack step now requires dist/component.wasm + metadata before producing bundles.)
• Add UI SDK helpers demonstrating the proxy pattern (UI calling gateway endpoints backed by runner handlers). (Available via callProxyJson in @alga-psa/extension-runtime.)
• Deliver local dev commands (alga-cli dev) that spin up mocked capability providers to mirror wasmCloud’s wash workflow. (New alga component dev task rebuilds components on file changes.)

Phase 4 — Rollout

• Ship an internal extension end-to-end (control plane → gateway → runner → component) to validate secrets delivery.
• Open beta to selected partners once SDK and tooling stabilize; iterate on developer feedback.
• Track follow-up work for observability, telemetry, and runbooks separately.

Phase 5 — UI Delivery

• Activate the "Settings" button in the Extension Management UI.
• Implement/Connect the ExtensionSettings component to the /msp/settings/extensions/[id]/settings route.
• Ensure proper loading of Enterprise vs. OSS components (graceful degradation or feature stub).
• Verify "Save Changes" correctly persists to the new configuration and secrets tables.
• Verify "Reset to Defaults" clears overrides.

Dependencies & Coordination

• Runner/Wasmtime specialists to handle component host migration.
• Platform/infra teams for Vault policies and Docker-secret delivery of runner tokens.
• Secrets platform owners maintaining secretProvider APIs and Vault transit setup.
• DX/docs teams for SDK publishing, tutorials, and developer onboarding.
• Registry/publishing pipeline owners to enforce component artifact requirements and provider declarations.
• Future capability provider owners (e.g., messaging, analytics) to contribute host implementations once the catalog is seeded.

Open Questions

• Additional guest languages: after JS/TS, should we generate Rust/TinyGo bindings via wit-bindgen to broaden support?
• WIT versioning: how do we communicate additive changes to developers and keep bindings in sync (semver for WIT packages)?
• How do we expand the host proxy catalog over time (e.g., standardized UI → gateway endpoints) to cover common extension UI use cases?
• What guardrails are needed in alga-cli to detect mismatched componentize-js versions or incompatible Wasmtime features?

Alternatives Considered

1. Handwritten Wasmtime host imports (status quo)
   Pros: minimal change to existing Runner code.
   Cons: every capability requires manual glue; no generated SDK; developers must handle ABI details. Rejected.

2. Embed QuickJS or another JS runtime in Runner
   Pros: lets developers ship plain JS without compilation.
   Cons: larger attack surface, dual runtime maintenance, and still requires secret plumbing; diverges from Wasm-component-first direction.

3. Expose secrets via HTTP endpoints or storage APIs
   Pros: simple to implement.
   Cons: weak capability enforcement, higher leakage risk, and no alignment with future component-based contracts.