Operations

The runbook — the queue, a failing collector, migrations, token rotation, the public page, the demo tenant, limits and scanners — and the collector how-to.

What to do when something stops. Each section names the symptom, where to look, what it usually is, and the command that fixes it. Every command runs against the roles' environment (.env.example); in the compose stack, prefix with docker compose -f deploy/docker-compose.yml exec serve or use make.

The order of investigation is always the same: is the data fresh? (the admin matrix), is work moving? (the queue), is anything failing? (the runs' error text and the logs). Alerts and the public page read from the same tables, so a stale matrix explains a quiet public page.

Reading the admin matrix

Admin → Collector matrix, or GET /admin/collector-runs/matrix. One cell per enabled account × configured region × module (per-account modules such as the cost pulls have one cell, at global). Each cell is the newest run's verdict:

The issues panel lists every account that is not ok, worst first. "Stored 0 of 12 attempted" is the collector's own count: it saw twelve instances and wrote none, which points at the ingest API rather than at AWS.

Staleness is judged against the cadence in TIER_*_INTERVAL; serve and schedule read the same values, so a cell cannot be stale by one process's clock and fresh by the other's.

The queue backs up

Symptom. no-data spreading across the matrix; panorama_queue_depth climbing on the scheduler's metrics listener (METRICS_LISTEN; the PanoramaQueueBehind alert in a source checkout's deploy/monitoring/alerts.rules.yml), and the oldest pending row's run_after receding into the past.

Look.

SELECT state, tier, count(*), min(run_after) FROM jobs GROUP BY 1, 2 ORDER BY 1, 2;
SELECT id, tenant_id, account_id, region, module, attempts, locked_at, lease_expires_at
  FROM jobs WHERE state = 'running' ORDER BY locked_at;

Usually.

• No workers. pending grows, nothing is running. Check work's logs and that it started at all: it refuses to start with zero or two credentials (WORKER_TOKEN, INGEST_TOKEN, INGEST_AUTH_DISABLED=true — exactly one), or with JOB_LEASE shorter than MODULE_TIMEOUT + HTTP_TIMEOUT.
• Workers that cannot reach the API. Runs register and then every batch fails; work logs 5xx or connection errors against API_URL. Fix the network path; the worker retries 5xx and the writes are replayable, so nothing was lost or doubled.
• Leases outliving dead workers. running rows with lease_expires_at in the past and no worker log lines. The scheduler's reaper frees them on its next tick; if it does not, schedule is down — it is leader-elected, so exactly one instance must hold the advisory lock. A reaped job is retried and its half-finished run is refused on every endpoint; the inventory is unchanged throughout (that is the design, and a test).
• One tenant's broken row. Enqueue failures for one tenant do not stop the others; find it in schedule's logs by tenant_id.
• One tenant behind, the fleet not. panorama_queue_tenant_oldest_pending_age_seconds{tenant_id} is the same age per tenant with an overdue job (no series means nothing overdue). Tenants take turns at the dequeue: the one with the fewest jobs running goes first, ties by lot, and its longest-due job is taken — so one tenant climbing while panorama_queue_oldest_pending_age_seconds stays flat is not its backlog's size, but its jobs not finishing (look at its running rows and its runs in the matrix). Map the id to a tenant with SELECT slug FROM tenants WHERE id = '<tenant_id>'.
• A job that kills its worker. A job is tried at most five times, and a claim that expires unreported counts as one. After the fifth the reaper leaves it failed with last_error "attempts exhausted without a report" — the failed depth counts it and the matrix shows it — and the dequeue never hands it out again. The scheduler enqueues the target afresh beside it on its next cadence; if that one dies the same way, the worker's logs for its job_id (an out-of-memory kill, a crash) say why.

Drill. scripts/queue-drill.sh (from this module's root, on a disposable database with the integration environment) enqueues 10,000 jobs for 100 accounts across three tenants of unequal size, largest first, and runs four workers against a fake cloud; it prints each tenant's time to its first and last collection and the dequeue's median and p99, and fails unless the smallest tenant's first collection came within twice the largest's.

Fix. Start or scale work; nothing needs draining by hand. To stop collecting for one account meanwhile, disable it in Admin → Accounts; the scheduler respects enabled on its next tick.

A collector fails

Symptom. Red cells for one account, or for one module across accounts.

Look. Estate → Collectors draws one block per cloud with its collectors grouped by category, each category showing its worst cell until opened: open the red one for the collector's own cell, whose tooltip carries the error. Then the account's run list (its name in the matrix opens its recent runs; GET /admin/collector-runs?accountId=), and work's logs by run_id.

Usually.

• AccessDenied on assume role. The reader role is missing, its trust policy does not name this deployment's account, or the ExternalId does not match. Press Verify on the account: it assumes the role and probes every module the tenant is entitled to, and its error is the real one. The external id is per tenant and shown once by panorama admin add-management-account; the stack parameter must carry that value. The other cause is the role's name. The platform's own role may assume only the names its stack's ReaderRoleName lists. A reader role deployed under any other name, or an account still on a name that has been dropped from that list, is refused. To rename a reader role, see deploy/cloudformation/platform-identity.md §5. Both names stay allowed until every account has moved.
• AccessDenied on a describe. The role assumes but the policy is not the shipped one. Redeploy deploy/cloudformation/reader-role.yaml; it is an exact allowlist and the collector needs all of it.
• Throttling / timeouts in one region. The run is incomplete: its snapshots were stored and nothing was tombstoned. That is correct — an enumeration that did not finish must not decide what is missing. It clears itself on the next run; if it persists, raise the tier's interval rather than MODULE_TIMEOUT (whose default is also its maximum).
• Every run of one module fails with no collector for this module. The scheduler enqueues a module this build does not carry — a worker image older than the scheduler's. Roll the workers.

Never delete rows to "clear" a failed run. A failed run tombstones nothing and the next successful one reconciles; cleanup expires stale running rows after CLEANUP_STALE_RUN_EXPIRY.

Migrations fail

Symptom. panorama migrate up exits non-zero; serve refuses readiness with schema is behind this binary: database is at N, this binary needs at least M — the migration has not run — or this binary is too old for the schema, when a breaking migration has outrun it (see "What a migration does to the running pods" below).

Look.

panorama migrate version        # what the database is at, and whether it is dirty

Usually.

• Dirty version. A migration failed half-way. golang-migrate records the target version with dirty = true and refuses to continue. Read the failed migration's SQL against the database, finish or undo it by hand, then make migrate-force VERSION_TO_FORCE=<the last good version> and run up again. Migrations are immutable once merged; the fix is never to edit one.
• An index build or drop failed. An index on a table that already holds rows is built and dropped CONCURRENTLY, alone in its migration (migrations/README.md rule 6). A drop that fails is simply run again — it says IF EXISTS and finishes the job. A concurrent build that fails — a duplicate under UNIQUE, a cancelled statement — leaves the index behind marked invalid, and the version dirty. Find it with SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid, DROP INDEX CONCURRENTLY it, force the version back to the one before and run up again. The retry fails on the name until the invalid index is gone, which is deliberate: an invalid index is written on every insert and read by nothing.
• 0179 refused (refusing to migrate past 0178: migration 0179 could not make a provider's subject one account). Single sign-on's first version bound one provider subject to a second account when a person's email changed, and 0179's unique index cannot be built over that. The migrator has already dropped the invalid index and set the version back to 0178, and a later run that finds 0179 dirty does the same first. panorama admin sso-duplicates lists each subject with its accounts; keep the one that should hold it, run panorama admin sso-release --tenant-slug <slug> --email <email> for each other (audited; that account binds again at its next sign-in), and run up again. 0180, right after it, is breaking: it takes binaries built before it out of service, because they create a first sign-in's account and bind it in two steps.
• Rolling 0252 back refused (refusing to roll back 0252: N self-approved Blueprint run(s) and M self-approved request(s) are not finished). 0252 lets a sole approver approve their own request, marked; it is breaking, because the binaries before it refuse an approval by the requester at their claim. Its down would leave a self-approved run that is running, cancelling or failed unable to move — not report, finish, resume or be cancelled — and have the older binary fail a queued, running or cancelling self-approved request at its next claim, so it stops before it changes anything. The migrator puts the version back to 0252, clean, and the pods keep serving — only for an error of the file itself, with 0252's self_approved columns and its schema_compat row still there. A failure writing the version after the file committed is left dirty at 0251: the columns are gone, the schema is 0251's, and make migrate-force VERSION_TO_FORCE=251 says so. Let the work finish, or cancel it, and roll back again. A finished run or request does not stop it, nor a failed request: a Retry of that under the older binary is refused at the claim, and it is submitted again for somebody else to approve. The marks go with their columns; the audit log keeps who approved what (selfApproved).
• Roles missing. Migration 0002 refuses to run until panorama_app and panorama_control exist: run deploy/bootstrap-roles.sql as the owner first (compose does this in its bootstrap service).
• Wrong role. MIGRATE_DATABASE_URL must be the schema owner. The application role has no CREATE on schema public by design; it cannot create schema_migrations, let alone a table.
• A partition missing. An insert into resource_snapshots fails with SQLSTATE 23514 around the turn of a month: the scheduler's maintenance pass did not run. Start schedule; it creates the current month, last month when an observation could still land there, and two months ahead on every tick that needs it.

A new serve will not report ready until the schema has reached its db.TargetVersion, so a rolling deploy runs migrate to completion before the new serve takes traffic; the previous release's pods keep serving through that when the migration is additive.

Backing up and restoring the database

One PostgreSQL database holds everything the platform knows: inventory, the permanent status history, cost, users, sessions, licences, the job queue and provisioning requests. The roles keep no state of their own, and Terraform state lives in each customer's bucket (D3), never here. Two things are not in a dump: the database roles panorama_app and panorama_control, which are cluster-level and come from deploy/bootstrap-roles.sql, and the secrets the deployment passes in (the database URLs and WORKER_TOKEN).

The policy is the operator's. Choose these and write them down:

• How. A logical dump (pg_dump -Fc as the schema owner), or the provider's snapshots and point-in-time recovery for a managed database. Either restores the whole database, and neither restores the roles.
• How often. The interval is the recovery point: everything after the last backup is lost. A daily dump is one day.
• How long. Keep enough backups to reach back past a mistake noticed late. Seven daily and four weekly is a reasonable start.
• Encryption. A dump holds password and token hashes, the inventory of every connected account and each customer's spend. Encrypt it at rest, move it only over TLS or SSH, and keep it off the host it backs up.
• Who. The operator runs the backups and tests them; the platform cannot see them. A backup never restored is a hope, so restore one into a scratch database every quarter and run the checks below.

One way to schedule it off the host. deploy/backup/ holds what the project's own deployment runs, for a compose stack on a systemd host: a nightly timer that dumps the database (pg_dump -Fc inside the postgres container), checks the dump with pg_restore --list, uploads it to S3 with its SHA-256 (S3 refuses an object that arrived different) and reads the stored size and ETag back; and deploy/cloudformation/platform-backups.yaml, a versioned, encrypted bucket whose lifecycle expires every version and whose policy lets the platform's own role write and list one prefix and nothing else — the role on an internet-facing host cannot read a dump or delete one. A failure leaves the unit failed and a FAILED file beside the last result; check both after each deploy. deploy/backup/README.md has the install and the checks.

The restore drill (2026-10-19). Run it from the operator's machine, never on the host it backs up: deploy/backup/restore-drill.sh OUT_DIR with the operator's AWS identity.

1. The newest dump is downloaded with S3's checksum validated, and its SHA-256 compared with the object's.
2. A disposable postgres:16 starts with the live database's owner and database names (step 2 of "Restoring", below; the script reads them from the live container's POSTGRES_USER and POSTGRES_DB, and DB_USER and DB_NAME override them), and deploy/bootstrap-roles.sql creates the service roles with role_names=panorama and throwaway passwords.
3. pg_restore --exit-on-error, timed: that is the restore half of the RTO.
4. Every table is counted in the restored database and, in a READ ONLY transaction, in the live one. A table the platform writes to after the dump (collection runs, snapshots) has more rows live; a table with fewer, or one missing on either side, fails the drill.
5. The container goes with its volume (docker rm -fv), and the dump is deleted: it holds every tenant's data.

Write the date and the result down where the operator keeps the recurring duties. The project's own deployment ran it first on 2026-10-19: a 33.5 MB dump restored in 6 seconds, 203 tables, 195 equal, 8 grown live, none fewer.

Restoring. The order matters: roles before data, data before the roles start, and the guard before anything runs.

1. Stop serve, work, schedule, notify and cleanup.

2. Start a fresh, empty database, then create the roles with deploy/bootstrap-roles.sql as the owner. Compose does this in its bootstrap service. Two sets of names must match the dump:

   • The owner and the database. A dump grants to its owner by name, so it restores only under the same owner and database names, with --no-owner or without. An install from before release 174 kept its own (docs/panorama-rename.md); one installed since has panorama_owner and panorama. The commands below say <database> for that reason: read the names off the live database (SELECT current_database(), current_user as the owner) rather than from this page. An install that kept its old owner and database names can still have the service roles panorama_app and panorama_control, which is the next bullet's case.
   • The service roles. A dump taken at schema 174 or later grants to panorama_app and panorama_control, which bootstrap does not create on a fresh cluster unless told to. Give it -v role_names=panorama (compose: BOOTSTRAP_ROLE_NAMES=panorama). Without it, pg_restore stops at the first grant with role "panorama_app" does not exist.

3. pg_restore --exit-on-error -d <database> <dump> as the schema owner.

4. panorama migrate up. The dump is at the backup's schema version, and a newer image migrates it forward. An image older than the backup's schema refuses with this binary is too old for the schema, so restore with the same image or a newer one.

5. panorama admin after-restore --backup '<the file, snapshot or time>', before work and schedule start. A provisioning request the backup holds as queued, running or cancelling may have run since the backup, and the backup cannot say. Each one fails with the class restored and a message naming the backup, and the command prints it with its state location. Nothing of it runs again until someone retries it. Check each one against its account first:

   • a create that ran after the backup has resources and state;
   • one destroyed since, by a request the restore forgot, has neither, and a retry would create them again.

   The platform cannot notice a restore, since a snapshot looks exactly like the live database, so this step is the operator's. Skipped, the scheduler's first tick requeues the running work and an agent runs it again.

6. Start the roles, and wait for /readyz.

7. Check the restore:

   • panorama migrate version is the target, not dirty;
   • an administrator can log in;
   • panorama admin show-license --tenant-slug <slug> shows the licence;
   • the timeline reaches the backup's time;
   • the provisioning requests after-restore listed show Failed (restored).

What a restore loses, and what comes back. Everything written after the backup is gone:

• History. Collection records one transition from the backup's state to the present on its next run; the transitions in between are lost.
• Spend. The next ingest pulls the missing days again.
• People. Users created and passwords changed after the backup are gone, and so are sessions issued after it: everyone signs in again.

Two things come back that should not:

• Revoked credentials. An ingest, worker or agent token revoked after the backup, or a user disabled since, is live again. Revoke or disable it again before the roles start.
• Notifications. A notification delivered after the backup is undelivered in it, and is sent again.

Measured (2026-09-12). The drill restored a 0.5 MB dump, the demo estate, into an empty cluster in 2 seconds. serve answered /readyz 4 seconds after the empty cluster started, and every check above had passed at 5 seconds. A real estate takes longer in proportion to its size, so time a restore of your own backup. That time is your recovery time objective (RTO); your backup interval is your recovery point objective (RPO).

The provisioning owner record

Every resource that a provisioning state holds has one owner record: its key, the state it is in, and the request whose apply that state is. It is what an Import will be refused against ("already managed") and what a request leases while its task runs, so two actions never act on one resource at once. Ingest keeps it as it collects; the table starts empty on the release that brings it, and a migration may not fill it. So, once after upgrading to that release, and safely at any time after:

panorama admin rebuild-state-owners --dry-run    # what it would add and remove, per tenant
panorama admin rebuild-state-owners              # every tenant; --tenant-slug <slug> for one

Compose: sudo docker compose run --rm --no-deps -T serve admin rebuild-state-owners. It adds a row for every resource the inventory sees carrying the provenance stamp of an apply that has not been destroyed, and removes the rows of applies that are gone. Run again, it adds and removes nothing.

Not running it is not unsafe — a claim and an Import evaluate the stamps live whatever the table holds — but until it runs, a destroy leases only what has been collected since the upgrade.

"N recorded for one apply and stamped by another." A resource whose owner record names one apply while it carries another apply's stamp: somebody copied a tag, or two states each hold it. Nothing is replaced; the first record stands. Find both requests' state (the request pages name where each is), check which one holds the resource with tofu state list, and remove it from the other state before anything else acts on it.

"… which ended without reporting what its state holds." A resource carrying the stamp of a create that has no provisioned resource and ended without its agent's report: its agent stopped heartbeating and the platform ended the task — the last attempt timed out, a cancel was reaped, or another action took over its lease — or the restore guard failed it. What that apply made is unknown, so the resource stays owned by it and an Import of it is refused. A create that timed out is settled by Retry: the next attempt runs in the same state, and its report records what the state holds. A reaped cancel cannot be retried: list its state (keyed by the request's id under the agent's backend, unless the request page names another location) with tofu state list. Destroy what should not exist from that state; a resource you mean to keep and import must first leave the state (tofu state rm) and lose its panorama:request_id tag, because the owner record reads the stamp.

Rotating an ingest token

Per-tenant collector credentials live in control.ingest_tokens as hashes; the plaintext is shown once at creation.

panorama admin create-ingest-token --tenant-slug acme --name 'prod collector 2026-09'

Deploy the new value to the collector, confirm new runs arrive (the account's run list, or work's logs), then revoke the old one through the command, never with an UPDATE by name:

panorama admin revoke-ingest-token --tenant-slug acme --name 'prod collector 2026-08'

Token names are not unique — another tenant may well have a "prod collector" too — and the command scopes the revocation to the tenant named. If that tenant holds more than one live token of the name it refuses, and --all says every one of them is meant. Revocation is immediate: the next request with the old token is a 401.

The fleet credential rotates the same way:

panorama admin create-worker-token --name 'eu-west fleet 2026-09'
panorama admin revoke-worker-token --name 'eu-west fleet 2026-08'

Revoking it stops every worker at their next request without touching per-tenant collectors. In the compose stack the fleet's credential is WORKER_TOKEN in deploy/.env, which make compose-up issued on the stack's first run: issue the new one (docker compose -f deploy/docker-compose.yml run --rm -T serve admin create-worker-token --name '…' --raw), replace the value, docker compose -f deploy/docker-compose.yml up -d work, then revoke the old.

INGEST_TOKEN, the single-tenant shortcut, is a configured value rather than a row: change it on serve and work together and restart both.

Watching the roles

Every role is its own process with its own registry: serve answers /metrics on its API listener; work, schedule, notify and cleanup each answer /metrics and /healthz on METRICS_LISTEN. The listener is fail-closed — a taken port stops the role at startup, and a listener that dies later stops the role rather than leaving it working unobserved — and it carries no authentication: bind it where only the scraper reaches it. A source checkout's deploy/monitoring/ — platform 0.1.2's /deploy does not carry it — holds a scrape configuration for the compose ports, the five alerts and a dashboard, and the drill that proves the alerts fire.

serve's /metrics is on the listener the public reaches, so the edge must refuse it: deploy/proxy/nginx.conf.example answers it 404 (location = /metrics), and an Ingress needs the same — a rule sending /metrics nowhere, or a server-snippet with that location. Scrape serve directly, on its pod or container address, never through the edge.

Liveness is not progress. The probe on /healthz says the process runs; panorama_worker_last_job_completed_timestamp_seconds, panorama_scheduler_last_tick_timestamp_seconds (meaningful while panorama_scheduler_leader == 1) and panorama_notify_last_cycle_timestamp_seconds say its work moves, and PanoramaRoleStalled alerts on the gap. A standby scheduler reports panorama_scheduler_leader 0 and publishes no queue numbers; the leader does, every tick.

Releasing to Kubernetes

deploy/k8s/deploy.sh with your overlay (a copy of deploy/overlays/example, its image pinned by digest; the base names no usable tag) and the namespace is the release, and "deployed" at its end means every role runs the new image and the current configuration. It renders the kustomization once and fingerprints the configuration: every ConfigMap it rendered, and the resourceVersion of every Secret a pod template reads — the values themselves never pass through the script. That digest replaces the placeholder annotation (panorama.dev/config-hash) in each Deployment's pod template, so a release that only changes a ConfigMap key, or only rotates the Secret, rolls the pods that read it, and a release that changes nothing rolls nothing. Then the two phases: the migrate Job alone, waited for; then the roles, and a rollout wait on every Deployment the apply returned (ROLLOUT_TIMEOUT, default 6m, above the manifests' progressDeadlineSeconds of 300 so the Deployment's own verdict arrives first).

When a role does not roll out the script still waits for the others, then prints the pods, the failed Deployments' events and their last log lines, names the roles, and exits 1. The boundary at that point: the migration is applied and stays. A role that failed still serves its previous ReplicaSet where one exists — a rolling update keeps it until the new pods are Ready — so on an upgrade this is a stuck release, not an outage; on a first install there is no previous ReplicaSet and the role is down. Fix the cause and run the script again (the migration is a no-op the second time), or kubectl rollout undo each failed Deployment — which brings the previous binaries back only when every migration since them is additive; the next section says how to tell.

One role's rollback also brings back a known gap: a cleanup older than f563864 (2026-09-11) drops expired cost_data months without holding cost_data's lock or deleting the derived rows again first, so a rollup rebuild that commits between its tenant pass and its drop leaves a rollup whose facts are gone, until the next night's pass deletes it by date. Rolling the other roles back while cleanup stays on the newer release avoids it; the newer cleanup excludes writers of either release.

What a migration does to the running pods

The migration runs first and the roles roll after it, so for the length of the rollout the pods still serving are the previous release's, against the new schema. Whether that is fine is the migration's to say, on its first line, and every migration from 0154 on says it (the migration tests refuse one that does not):

• -- compatibility: additive — a new table, an index, a column that is nullable or has a default. Nothing an older binary reads or writes changes. The previous release's serve pods stay ready while the migration commits and the new pods roll in, so the API does not go down, and kubectl rollout undo brings them back: they become ready against the newer schema. The tests hold additive to a short list of statements an older binary cannot notice and refuse the rest by name — any data change, UNIQUE on an existing table, a function call, a DO block, a replaced definition, a drop or a revoke (migrations/README.md rule 4 has the list).
• -- compatibility: breaking; oldest binary N — the migration records, in schema_compat, that binaries built before N cannot serve it. Every serve pod older than N fails readiness the moment the migration commits, so the Service has no endpoints until the new pods are ready: a breaking release is an outage the length of one pod start. And kubectl rollout undo to a binary older than N stalls — its pods never become ready, and the rollout leaves the current ones serving. To roll back across a breaking migration, roll the migration back first (make migrate-down removes its schema_compat row with it), then undo.

Readiness is the rule: a pod built for target T is ready when the schema's version is at least T and no applied migration's oldest supported binary is above T — db.Compatible, read by /readyz. Both refusals are real: a pod below the floor may read a column that is gone, and a pod ahead of the schema needs tables the migration has not made. /readyz says which.

To keep a change from being breaking, split it: expand (add the new shape, additive), release, then contract (drop the old shape) in a later release whose oldest binary is the expand's release — the pods it outruns are the ones that no longer exist. The release that introduces this rule, 0154, is itself not seamless: the pods it replaces demand an exact schema version and leave the Service when it commits, once.

A new notification kind is never additive in an outbox that already carries several kinds — provision_request_events or blueprint_run_events — because a notifier from in earlier releases claims every row there whatever its kind and sends one it does not know to the tenant's default channels (the migration tests refuse it, reviewed exception or not). Give the kind a table of its own, or declare the migration breaking. Since then the notifier claims only the kinds it knows: a row of a newer kind, written while an older notify is still running or after notify alone was rolled back, waits unsent and unmarked. panorama_notify_outbox_depth counts it and panorama_notify_outbox_oldest_age_seconds grows, so PanoramaNotificationBacklog fires after fifteen minutes; rolling notify forward to the writer's release sends it. Nothing ages such a row out — cleanup deletes only notified rows.

Rotating the Secret is a release: update panorama-secrets, run the script, and every role rolls onto the new credentials in one pass. The roles do not re-read a Secret while they run, so a Secret updated without a release changes nothing until something else rolls the pods.

KubernetesConfigRotationDrill (owner-run)

On a disposable cluster (kind), with a Postgres the cluster reaches, the roles from deploy/bootstrap-roles.sql created, and the Secret in place:

1. Run the release and record the digest it prints and the pod names.
2. Run it again unchanged: the digest and the pod names are the same.
3. Set LOG_LEVEL to DEBUG in the ConfigMap and run it: every role's pods are replaced and the digest differs.
4. Rotate the Secret (apply it again with a new WORKER_TOKEN) and run it: every role's pods are replaced.
5. Break one role in an overlay (an argument notify does not accept) and run it: the script exits 1, names that Deployment, prints its events and logs, and the other roles rolled. kubectl rollout undo on it restores the previous pods.

Record the digests, the pod lists and the script's output in deploy/k8s/evidence-<date>.txt.

Releasing the platform image

A release is one image digest, ghcr.io/cloudpanorama/panorama/panorama@sha256:…, named :X.Y.Z once and never moved, signed with cosign keyless through the release workflow's GitHub identity, with a CycloneDX SBOM attested to it. The workflows derive the image's path and the signing identity from the repository's own name, so they did not change when the repository moved to github.com/cloudpanorama/panorama on 2026-10-18; the documents that name them did.

• The version is the VERSION file beside this module's go.mod, a bare X.Y.Z. Every image CI builds reports X.Y.Z+<12-char commit> (panorama version), and the compose default (deploy/docker-compose.yml) and the README's Quickstart name the same X.Y.Z — internal/deploy/platform_release_test.go fails when they drift.
• The candidate is what CI's image job pushed as :<commit> from main, after every other job passed. A commit pushed with [skip ci] has none and cannot be released.
• The image is one multi-platform index of linux/amd64 and linux/arm64. CI builds both on its own runner by cross-compilation — every build stage of the Dockerfile runs on the builder's platform, so no emulation is set up — into a registry of the job's own, checks and scans each platform there (deploy/platform-release.sh platforms and check), and copies the very index that passed to GHCR. The release refuses a :<commit> that is not exactly that index, so no commit from before the change can be released.
• The release is the tag platform/v<X.Y.Z> on that commit, equal to the file. ../.github/workflows/platform-release.yml runs only on it: it refuses a tag that differs from the file and a commit main never published, checks each platform's image again (contents, its binaries' architecture, and on the runner's platform the version, commit and licence key; the compose default), scans each, signs the index digest recursively (the index and both platforms' images), attests each platform's SBOM to its own digest and to the index's, runs the install page's cosign verify itself on the index and on both platforms, and only then names the digest :X.Y.Z — a registry-side copy, so the digest cannot change, refused when :X.Y.Z already names another. Its artifact panorama-release-X.Y.Z holds the release record (the index digest, each platform's digest under platforms, the pull reference, the signing identity) and both SBOMs.

To release, on a green main:

# 1. a commit that bumps observability/VERSION, the compose default and the
#    README's version together, pushed WITHOUT [skip ci]; wait for its CI run
#    to be green (the image job pushed :<commit>)
git tag -a platform/vX.Y.Z -m "platform X.Y.Z" <commit> && git push origin platform/vX.Y.Z
# 2. the workflow's release record names the digest
gh run download <run-id> -n panorama-release-X.Y.Z && jq -r .pull panorama.release.json

Then keep the record in deploy/releases/, as X.Y.Z.json; put its pull into the README's IMAGE= line, the compose default (deploy/docker-compose.yml) and deploy/.env.example's PANORAMA_IMAGE (and the digest into a Kubernetes overlay's digest:) — internal/deploy TestTheReleasedVersionIsPinnedByItsDigest fails until all three name it — regenerate the docs (npm run docs:generate), and check the verification from a machine that is not the runner, with cosign 3 or later (the release signs in cosign 3's bundle format; an older cosign does not read it and reports "no signatures found" for a correctly signed image):

cosign verify ghcr.io/cloudpanorama/panorama/panorama@sha256:<digest> \
  --certificate-identity https://github.com/cloudpanorama/panorama/.github/workflows/platform-release.yml@refs/tags/platform/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The last release is 0.1.2, released on 2026-10-24 by run 40089605086 from commit bd9645a6, the first whose /deploy carries every file the install and connect pages name — the proxy example, the reader templates, the platform-identity stack and the Kubernetes manifests; .dockerignore is that list, and internal/deploy holds it to the pages (released on the owner's decision of 2026-10-22): ghcr.io/cloudpanorama/panorama/panorama:0.1.2@sha256:f487a34623502b37bbed14b3a2dfbde9f3d773e3c5715a94a99e2fd294c446c2 (deploy/releases/0.1.2.json, kept verbatim). The index names linux/amd64 sha256:0b9d532d5df14bb39598c347120f3a281ef61e1771691969bace34e9613943a5 and linux/arm64 sha256:ad4b08d3f788e9ee3740e6545012a7b5b9183f0baf5c91ffe5eacb792b6133ed. The session verified it from the Mac, signed out of GHCR, with cosign 3.1.3 on 2026-10-24: cosign verify accepts the index and both platforms, cosign verify-attestation --type cyclonedx accepts the index, and 0.1.1's identity is refused. A docker cp of /deploy from each platform's image, pulled signed out, gave the 44 files .dockerignore lets in, byte for byte the release commit's. The README, the compose default and deploy/.env.example pin it.

The one before, 0.1.1, was released on 2026-10-19 by run 39247827260 from commit eb4beb0a, the first from github.com/cloudpanorama/panorama and the first multi-platform one: ghcr.io/cloudpanorama/panorama/panorama:0.1.1@sha256:2e5a11ce87d37bafc9344272ed5906aae324fb7b3898aea7fa3139f1ea75e26c (deploy/releases/0.1.1.json, kept verbatim). The index names linux/amd64 sha256:993a498a8f269c256870b02f86ae56a5801c24adcf9f760744ae475559fd9bc0 and linux/arm64 sha256:78030ddbee6ad524843885cb387bf50a07bc7abc4a095b2c0647c01833788156; it was verified the same way on 2026-10-19. Its image's /deploy carries the three compose files only. The package is public since 2026-10-20 (the owner's step; a signed-out pull and cosign verify from a fresh host passed the same day).

The one before, 0.1.0, was released on 2026-10-14 by run 38640260588 from commit b6e4be08 (a SHA from before the history rewrite; ../../platform/history/2026-10-18-sha-map.txt gives its new one), under the repository's former name: ghcr.io/dawidper/ccc/panorama:0.1.0@sha256:3002531a0bf74a752cc4baaf19ec6f7e08914f67805c6963535dc6d0bad2d610 (deploy/releases/0.1.0.json, kept verbatim; one linux/amd64 image, from before the index, so its record names no platforms). That package stays private where it was published and is superseded by 0.1.1.

A mistake is a new version: a published :X.Y.Z is never moved, and the workflow refuses to. A rerun of the same tag is safe — it signs again and finds :X.Y.Z already naming its digest.

The panorama rename

Release 174 gives everything that runs the product's name: the binary, the images, the settings, the service roles, the metrics and alerts, the session cookie, the token prefixes, and the compose and Kubernetes objects. Migration 0174 is breaking (oldest binary 174): it renames the two service roles, so the image, the connection strings and the settings change in one step. Everyone signs in once more, and tokens issued before it are issued again. docs/panorama-rename.md is the upgrade: what each name was, what it costs, the steps for compose and Kubernetes, and rolling back.

The Admin area

Since then an administrator's portal has an Admin section of its own, which a member never sees: its pages refuse a member by name, and every read behind them is administrator-only in the contract. It holds the organisation's pages — Users, Teams & access, Single sign-on (when the licence carries it), Notification channels and Licence — and two of its own:

• Overview, one read (GET /admin/overview): the licence and its accounts, the users (active, deactivated, administrators, break-glass), the AWS accounts and their collection as the collector matrix rolls it up, single sign-on's state, and the latest audit entries.
• Audit log (GET /admin/audit): the tenant's own audit_log rows, newest first, read under row-level security and filtered by who acted, an action prefix, the subject and the days, a page at a time by cursor. Before and after are shown as the operation recorded them.

The old /settings/users, /settings/sso, /settings/org and /settings/license addresses lead an administrator to the new pages. A member's /settings/org leads to their own teams, and their licence page stays where it was. The modules' own setup (AWS accounts and collectors, Provisioning's templates and agents, Spend's attribution and budgets) stays in the modules.

Your own settings

Since then Settings is each person's own, an administrator's too, and every route behind it acts on the session's account and nothing a body or a path could name:

• Profile: a display name, shown where the portal names people — the users list, an alert's acknowledgement, the audit log and provisioning — with the email beside it. The email is the account's, and an administrator's to change.
• Password, as before.
• Sessions (GET/DELETE /auth/sessions): where the person is signed in — when each session began and was last used, the address and browser it began from, the current one marked — and signing out one or all the others. A session is named by an id derived from its token's hash, which ends that session and looks nothing else up. The address and browser are recorded; an older session shows neither.
• Inbox and Notifications: the person's own notifications, in the portal only, with a count in the top bar, and which kinds reach them — status alerts and their own provisioning requests unless they turn them off, their teams' budgets and cost anomalies when they turn them on. Each kind reaches a person only for what they may see: an administrator hears about every account and team, a member about their teams and their teams' accounts, and an account no team owns reaches the administrators alone. The inbox is filled by the notifier in the transaction that marks a batch, so while a channel's sends fail the batch is retried and its entries wait with it. Cleanup keeps an entry ninety days, read or not (the notifications category of panorama_cleanup_deleted_total).
• Preferences: a time zone, a date and a number format and the theme, applied across the portal as soon as they are saved; a choice left at the browser's own is not stored. The first save, whatever its values, is recorded with the server's time and ticks the Dashboard's first-run checklist step; no later save moves it. Dismissing the checklist is recorded beside it, for that person alone.
• Your teams: the person's memberships and their role in each, read-only.

/settings/notifications was the channels page and is now each person's choices; the organisation's channels are in Admin, under Notification channels.

Onboarding and removing people

Every user operation is a supported one: nothing below needs SQL, and each leaves an audit_log row (org.user.*, auth.password.*) naming the actor, the person and what changed — never a password or a token.

Which tenant a login is for. The request host first, when its first label is a tenant's slug (acme.panorama.example is acme's); then tenantSlug in the body; then, in a deployment with exactly one tenant, that one. The deployment's own hostname is a subdomain too — platform.<domain> — and its first label is nobody's slug, so it falls through to the body and the sole tenant. Set DEPLOYMENT_HOST to that hostname: then its label is never read as a tenant, whatever tenants exist, and bootstrap and seed-demo refuse a tenant named after it (review round 2). The portal's form sends only the email and the password, which is why a single-tenant install signs in from any host and a multi-tenant one needs either a tenant host or the slug.

A person's credential works only at its own tenant's host. The host never decides which tenant a request acts for — the credential does — but a session or a read token sent to a host that names another tenant is refused 401 unauthorized, the answer an unknown credential gets, and the server logs a person's credential presented at another tenant's host with both tenants' ids. A host names a tenant exactly as a sign-in reads it: a first label that is a tenant's slug, never DEPLOYMENT_HOST. So the platform's own host, a self-hosted deployment's hostname, localhost and an address keep accepting every tenant's credentials. A spreadsheet or a script that answers 401 with a token that works elsewhere is calling the wrong tenant's address. Machine credentials (collector, agent, marker, reviewer) are not checked against the host. Logout holds to it too: a live session sent to log out at another tenant's host is that 401 and is not ended (a cookie that is no live session logs out anywhere); and a single sign-on step-up returned at another tenant's host is not recorded.

In the portal, an administrator's Users page adds a person with a platform role and shows their temporary password once, moves the role, issues a password reset (a token shown once, good for 24 hours and one use, redeemed at /reset-password), ends every session they hold, and deactivates or reactivates them. Everyone changes their own password under Password; doing so ends their other sessions and keeps the one they typed into. The same operations from the command line, for the operator holding the database credentials:

panorama admin create-user     --tenant-slug acme --email eng@acme.test --role engineer
panorama admin reset-password  --tenant-slug acme --email eng@acme.test
panorama admin set-role        --tenant-slug acme --email eng@acme.test --role viewer
panorama admin disable-user    --tenant-slug acme --email eng@acme.test
panorama admin enable-user     --tenant-slug acme --email eng@acme.test
panorama admin revoke-sessions --tenant-slug acme --email eng@acme.test

--raw on create-user and reset-password prints the credential alone, for a pipe. What each change does to sessions is the rule, not a side effect: a role change, a deactivation, a password change and a redeemed reset all end the sessions that predate them, so no open tab keeps a privilege or a credential that has moved. Deactivation refuses the login with the same answer as a wrong password.

The last administrator. The tenant's last enabled administrator cannot be demoted or deactivated (409 last_admin); two administrators changing each other at once are serialised on the row and one is refused. If a tenant nonetheless has no administrator who can sign in — every one deactivated by the operator, or the only one gone — the break-glass is panorama admin bootstrap with a new email, which adds an administrator to the existing tenant, or panorama admin enable-user on one of the deactivated ones. Neither overwrites an existing account.

Single sign-on

Teams from the directory. A team bound to a group of the provider (Admin area → Teams → Directory, or PUT /org/teams/{teamId}/directory-group) takes its membership from the ID token's groups claim at each sign-in: a person carrying the group joins, one no longer carrying it leaves; the platform role and every hand-kept team stay as they are, and each change is an audit entry with no actor. The claim is groups unless the registration names another (Okta and Keycloak let a customer name it; Entra ID and Google emit groups). The licence must carry the scim capability — Enterprise does — or bindings are refused and sign-ins sync nothing. Unbinding keeps the members and makes the team hand-kept again.

A tenant whose licence carries the sso capability signs in through its own OpenID Connect provider: Entra ID, Okta, Google or IAM Identity Center. An administrator registers it under Admin → Single sign-on: the issuer URL; a client id and secret created at the provider for a web application using the authorization-code flow; the redirect URI the page shows, https://<the portal's host>/auth/sso/callback, registered at the provider; the role a first sign-in gets; and the email domains a first sign-in may come from. Entra ID does not say whether an address is verified, so it needs its domains listed. The secret is kept on the server and never shown again, and saving or removing the registration asks for the password again.

A first sign-in binds the account that has the address, or creates one with the chosen role. From then on the account is the provider's subject, and a different subject with the same address is refused. The session is a password session in every respect: the per-user cap, the twelve-hour limit, revocation. Two first sign-ins of one person at once, or a provider that later changes the person's email, still make one account: the subject is looked up first, and it is unique within the tenant and the issuer.

Changing the provider. An account is its provider's subject, and a subject belongs to one issuer, so a new issuer is refused (409) while any account is bound to another. The page says how many are bound and offers to release them — an administrator, with the password asked again, audited as auth.identity_provider.release. Released, each account binds again, by its email, at its first sign-in through the provider then registered: the same account, under enforced mode too, and break-glass passwords work throughout. Removing the registration releases nothing: registering another issuer afterwards is refused the same way. A sign-in that was still at the old provider when the new issuer was saved is refused when it comes back, and makes or binds no account; one that finished first is counted, and the new issuer stays refused until it is released too.

A sign-in finishes only in the browser that began it: the start sets a short-lived cookie scoped to /auth/sso, and a callback without it — the same link opened in another browser — is refused and signs nobody in. Every request the server makes for a sign-in (the discovery document, the token exchange, the provider's keys) is held to the destination policy the notification channels use, below: https on port 443, no credentials in the URL, every address the host resolves to public, checked again on the address actually dialled, and no redirect followed. An issuer that fails it cannot be saved.

An account that signs in only through the provider has no password anyone knows, so it confirms a destructive action at the provider instead: the confirmation dialog offers Confirm with your identity provider, the provider asks for a fresh sign-in, and the browser comes back to the page with the session confirmed for ten minutes, as a password confirms it. The fresh sign-in must be the same account's and at most five minutes old.

Saving or removing the registration leaves an audit_log row (auth.identity_provider.create, .update or .delete) naming the actor, with the registration before and after; of the secret it keeps only whether one is set and whether the change replaced it.

Enforced stops passwords for everyone but break-glass administrators, who are marked on the Users page. Mark one before enforcing. When the provider is down and nobody was marked, from the host:

panorama admin break-glass    --tenant-slug acme --email admin@acme.test
panorama admin reset-password --tenant-slug acme --email admin@acme.test

The administrator signs in with the reset password, then clears Enforced or corrects the registration; break-glass --off takes the mark away again. Each change leaves an audit_log row naming the actor.

Notification channels

A tenant's destinations — a Slack incoming webhook, a Microsoft Teams workflow, a PagerDuty service, a list of email addresses or a webhook of its own (and the operator's AI reviewer, below) — are managed on the portal's Notifications page or through /org/notification-channels; nothing is inserted by hand any more. An administrator creates a channel with a name, a kind and a target, scopes it to a team or leaves it as the tenant default, tests it, rotates its target, disables it or removes it. Every change is audited (notify.channel.*) with the redacted view of the target and never the target itself.

The destination policy. A target is a URL this process will POST to from inside the deployment's network, so it is checked before it is stored and again on the address actually dialled: https only, no credentials in the URL, a host that resolves, and every address it resolves to public — loopback, link-local (the cloud metadata endpoint), RFC 1918, unique-local, carrier-grade NAT and the reserved ranges are refused, as is a host with one public and one private answer (a rebinding setup). Redirects are never followed. A refusal is 400 invalid_request with the reason and never the URL. There is deliberately no opt-out for an internal receiver: a destination inside the network is one an operator with the database credentials can still insert by hand, and that row is theirs to answer for.

What a read shows. The target is never read back: the API and the page show its host, a hint (the last four characters of its path) and a fingerprint (sha256), so two channels can be told apart and a rotation shown to have changed something. Rotate by giving a new target; the last test result is cleared with it.

Testing a channel. "Test" sends one fixed message through the same sender the outbox uses and reports delivered, refused (the destination answered with an error or a redirect; the status is shown), unreachable (no answer: dns, tls, timeout, network) or blocked (the policy refused it before a byte was sent: private_address, invalid_target). The destination's own words are never repeated. The result stays on the channel until the next test or a rotation.

Each kind

slack, webhook and teams targets are tenant-selected HTTPS destinations and stay under the destination policy above. A PagerDuty routing key is not a URL and is checked for its shape alone. Every kind takes every message routed to it except PagerDuty, which takes those that carry an incident (estate transitions, acknowledgements, budget thresholds); an acknowledgement reaches PagerDuty only. Triage (below) decides which events reach channels at all — there is no second, per-channel filter.

Email is the operator's transport. The tenant names addresses; the relay is the deployment's, set in SMTP_URL, SMTP_USERNAME, SMTP_PASSWORD and SMTP_FROM (deploy/.env.example) and read by notify and serve. Without SMTP_URL the kind is refused at creation. The relay is deliberately not under the tenant destination policy: no tenant input reaches it, so an operator may use a relay inside their own network. What holds instead: smtp:// on 587 with STARTTLS or smtps:// on 465 with implicit TLS and nothing else; the certificate checked against the system roots and the relay's name; a relay on 587 that does not offer STARTTLS refused before the password or any address is sent (the test result says unreachable / tls); one deadline (NOTIFY_TIMEOUT for the notifier, ten seconds for a test) over the whole conversation; and a relay's refusal reported by its reply code (refused, 550), never its words, its name or the credential. An address list with a line break, a display name or a URL is refused at creation, and the sender refuses it again.

Each email carries its own Message-ID on SMTP_FROM's domain — receivers refuse mail without one, and a relay need not add it — and Auto-Submitted: auto-generated, so out-of-office replies do not answer an alert. The subject is the message's header shortened to 200 characters (the body has it whole), and every header line is folded under the line limit; an address is at most 254 characters. make test-mail-relay sends through a real SMTP server (Mailpit, in Docker) over both TLS modes; it checks the sender, not your relay — after setting SMTP_URL, test one email channel.

The weekly team digest is on when PORTAL_URL names the portal's origin and the organisation has Spend. In the first notify cycle of each ISO week (Monday 00:00 UTC), each team's digest — month to date and forecast per currency, budgets, open anomalies, open alerts, spend missing a required tag, links back to the Spend pages, and, for an organisation with Estate, the count of the team's open findings with the top three by exposure, then importance, then cost (the findings the team owns in its own accounts, as its members see them on the Findings page) — goes to that team's own channels and to the inbox of each member who chose it in Settings; never to the organisation's default channels, and nothing for a team without a channel of its own except those inboxes. Membership is read when it is sent. A digest is recorded as sent in notification_suppression (kind cost.digest, scope <team>/<week>) in the transaction that sends it, so delivery is at least once: a send that succeeded at a destination whose transaction then failed is sent again. Until then it is owed as a cost.digest.pending row, written before any send, so a failed send — across a restart or the week's turn — is tried again a minute later, after every other team due; a team whose channel keeps failing holds back nobody else. A pending row that does not go away is a channel that keeps failing. Ten digests per organisation per cycle, each in its own transaction; the rest follow on the next.

Scheduled reports run beside the digest, on the same two conditions (PORTAL_URL set, the organisation has Spend). A team's lead or an administrator schedules the team's monthly report to one of the team's own email, Slack, Teams or webhook channels on Spend's Settings page. A month's report is owed from the third of the next month (two days for the month's last spend to arrive) and is written to cost_report_deliveries, one row per (schedule, month, channel), unique, so it is owed once however often the notifier restarts; Send now owes the month to date once per day. Email attaches the breakdown and the tag violations as CSV files, each cut at 3 MiB with a last line saying how much it holds and where the whole export is; the other channels get the links. Delivery is at least once, the row marked in the transaction that sends it. A failed send is tried again a minute later, behind every other report due, so a failing channel holds back nobody else. Before sending, the destination is checked again: a schedule or channel switched off, or a channel moved to another team, sends nothing, and the row says why (dropped_reason). That check holds until the send is done: the schedule and channel are locked while the report goes out, so an edit or delete of either waits for the send — at most NOTIFY_TIMEOUT — and once it returns, nothing more of that report reaches the old destination. A report whose schedule or channel is being edited at that moment is tried again a minute later. A row with no notified_at and a growing attempts is a channel that keeps failing.

Rotating a channel's credential

Every target is a credential — a Slack or Teams URL carries its signature, a PagerDuty key opens incidents, an address list says where alerts leak. Rotate the same way for each kind: create the new credential at the far end (a new Slack webhook, a new Workflows trigger, a new integration key on the PagerDuty service), give it as the channel's new target (Rotate on the Notifications page, or PATCH /org/notification-channels/{id} with target), press Test and see delivered, then revoke the old one at the far end. The fingerprint changes and the last test result is cleared, so the audit log (notify.channel.update) shows the rotation without the secret. For PagerDuty, an incident opened under the old key is the old integration's: resolve it there if the service changed.

The operator's own credentials rotate outside the product. The SMTP password: issue a new one at the relay, put it in SMTP_PASSWORD (deploy/.env or the panorama-secrets Secret's optional SMTP_PASSWORD key) and restart notify and serve — both read it at start; test one email channel; then revoke the old password at the relay. Changing the relay itself (SMTP_URL) is the same restart. The AI reviewer queue is the section below.

The home cloud

A deployment signs in to every cloud it reads from one identity (the owner, 2026-10-10): its home, PLATFORM_HOME_CLOUD, set on serve, work and notify. Each cloud works alone; when a deployment reads two or three, the others are reached by federating from the home, with a token the home issues — never with a second credential, and never with whatever else the host happens to hold.

• aws, or unset. The platform's own identity is the AWS default chain — IAM Roles Anywhere on the platform host. Google Cloud is reached through workload identity federation with an AWS provider; Azure through outbound federation (option A), or with AZURE_CERTIFICATE_FILE (option B).
• gcp. The identity is the host's certificate, GCP_CERTIFICATE_FILE and GCP_PRIVATE_KEY_FILE, presented to an X.509 provider. AWS is reached with AssumeRoleWithWebIdentity on AWS_PLATFORM_ROLE_ARN, Azure with a federated credential — both with GCP_PLATFORM_SERVICE_ACCOUNT's ID token.
• azure. The identity is the host's certificate, AZURE_CERTIFICATE_FILE, on the platform's own app (AZURE_PLATFORM_TENANT_ID, AZURE_PLATFORM_CLIENT_ID). AWS is reached with AssumeRoleWithWebIdentity on AWS_PLATFORM_ROLE_ARN, Google Cloud through an OIDC provider for the directory's issuer — both with that app's Entra ID token.

Unset is aws, exactly as every deployment ran in earlier releases. Admin → Accounts shows the home and how each other cloud is reached, read-only (GET /admin/cloud-identity); changing it is a change to the deployment's settings and a restart.

The settings are held to their home when a role starts: the home's own are required, and one that belongs to another home is refused (for example GCP_CERTIFICATE_FILE under aws, or AZURE_CERTIFICATE_FILE under gcp), so a deployment cannot hold two identities by accident. A certificate that cannot sign — a missing file, a key that is not the certificate's, a certificate out of date (expired, or not valid yet) — stops the role before it listens, naming the setting. With PLATFORM_HOME_CLOUD=aws named outright, a role whose AWS default chain finds no credentials stops too; unset keeps the start that logs the missing identity and carries on. serve and work build the identity at every start; notify only when it sends to the AI reviewer's queue (AI_REVIEWER_QUEUE_URL), the one thing it reaches a cloud for. The certificate files are read again, and held to the same rules, at each sign-in, so a certificate renewed on the host is used without a restart, and one that runs out while the role runs stops signing rather than being presented past its date. Under Azure's certificate — the azure home, or option B under aws — each assertion takes its key, its certificate and the thumbprint it names from one read of AZURE_CERTIFICATE_FILE, and the onboarding's thumbprint is the certificate in the file now: upload the new certificate to the app (--append keeps the old one) before replacing the file, and replace the file whole — key and certificate together, by rename — so no sign-in reads half of it. A cloud the home does not reach is not offered: GET /auth/me lists only the clouds the home reaches, and AWS is among them under another home only when AWS_PLATFORM_ROLE_ARN is set.

AWS from another home. One role in the operator's AWS account, AWS_PLATFORM_ROLE_ARN, assumed with the home's token — the role every customer's reader role trusts, as it trusts the Roles Anywhere role under an AWS home. From a Google home, the role trusts AWS's built-in Google provider:

{"Effect":"Allow","Principal":{"Federated":"accounts.google.com"},
 "Action":"sts:AssumeRoleWithWebIdentity",
 "Condition":{"StringEquals":{"accounts.google.com:oaud":"sts.amazonaws.com",
   "accounts.google.com:aud":"UNIQUE_ID","accounts.google.com:sub":"UNIQUE_ID"}}}

where UNIQUE_ID is GCP_PLATFORM_SERVICE_ACCOUNT's numeric id (gcloud iam service-accounts describe ... --format='value(uniqueId)'), and the pool's principal holds the Service Account OpenID Connect Identity Token Creator role on that account, and nothing else:

gcloud iam service-accounts add-iam-policy-binding PLATFORM_SA --project=POOL_PROJECT \
  --role=roles/iam.serviceAccountOpenIdTokenCreator \
  --member="principal://iam.googleapis.com/projects/POOL_PROJECT_NUMBER/locations/global/workloadIdentityPools/panorama-x509/subject/HOST_CN"

From an Azure home, the role trusts an IAM OIDC provider for the directory's issuer (https://sts.windows.net/TENANT_ID/) with the client id api://CLIENT_ID; the platform's app sets that application ID URI under "Expose an API". The session is named cloudpanorama-federated.

Connecting a Google Cloud project

A Google Cloud project is an account of its own cloud. Estate reads its BigQuery datasets, service accounts, Firebase apps and Cloud Functions as the project's reader service account. The platform holds no key file: its own AWS credentials, IAM Roles Anywhere on the platform host, are exchanged at Google's Security Token Service through workload identity federation, and the federated token impersonates the reader. For an operator who knows AWS: a project is an account, a service account is a role, a custom role is a customer-managed policy, and the binding that lets the platform impersonate the reader is the trust policy.

Once per deployment, in a project the operator holds: a workload identity pool whose provider accepts only the platform's home identity, and GCP_WORKLOAD_IDENTITY_PROVIDER set on work and serve to the provider's name, //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Under an AWS home (the default) it is an AWS provider that accepts only the platform's AWS role, its attribute.aws_role mapped to the role's name — the binding below names it so (Google's default AWS mapping would make it the assumed-role ARN instead):

gcloud iam workload-identity-pools create panorama --project=POOL_PROJECT --location=global
gcloud iam workload-identity-pools providers create-aws panorama-aws --project=POOL_PROJECT \
  --location=global --workload-identity-pool=panorama --account-id=PLATFORM_ACCOUNT_ID \
  --attribute-mapping="google.subject=assertion.arn,attribute.aws_role=assertion.arn.extract('assumed-role/{role}/')" \
  --attribute-condition="attribute.aws_role=='PLATFORM_ROLE'"

Under an Azure home it is an OIDC provider for the directory's issuer (--issuer-uri=https://sts.windows.net/TENANT_ID/ --allowed-audiences=api://CLIENT_ID). Under a Google home it is an X.509 provider, and no AWS is involved at all:

# trust_store.yaml: the CA that issued the host's certificate
#   trustStore:
#     trustAnchors:
#     - pemCertificate: "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
gcloud iam workload-identity-pools create panorama-x509 --project=POOL_PROJECT --location=global
gcloud iam workload-identity-pools providers create-x509 panorama-host --project=POOL_PROJECT \
  --location=global --workload-identity-pool=panorama-x509 \
  --trust-store-config-path=trust_store.yaml \
  --attribute-mapping="google.subject=assertion.subject.dn.cn" \
  --attribute-condition="assertion.subject.dn.cn=='HOST_CN'"

with PLATFORM_HOME_CLOUD=gcp, GCP_CERTIFICATE_FILE (the host's certificate, PEM, leaf first) and GCP_PRIVATE_KEY_FILE (its key). The exchange is made over mutual TLS with that certificate at sts.mtls.googleapis.com; the key never leaves the host. The certificate is at most 390 days, P-256 or RSA, issued by a CA kept off the platform host. Until serve has it, the portal does not offer Google Cloud: GET /auth/me lists the clouds this deployment reads, and Admin → Accounts has no project to connect. A project registered while it was set stays listed in its own block, which says "This deployment does not read Google Cloud." and counts none of its accounts as being read.

Per project, in the customer's project. The role file is in the deploy/ the install copied out of the image (from platform 0.1.2; the README's Quickstart), so run these from the directory that holds it:

gcloud iam service-accounts create panorama-reader --project=PROJECT_ID
gcloud iam roles create panoramaMetadataReader --project=PROJECT_ID \
  --file=deploy/gcp/reader-role.yaml
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=serviceAccount:panorama-reader@PROJECT_ID.iam.gserviceaccount.com \
  --role=projects/PROJECT_ID/roles/panoramaMetadataReader --condition=None
gcloud iam service-accounts add-iam-policy-binding \
  panorama-reader@PROJECT_ID.iam.gserviceaccount.com --project=PROJECT_ID \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/projects/POOL_PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL/attribute.aws_role/PLATFORM_ROLE"

That member is the AWS home's. Under a Google home it is the host's subject, principal://iam.googleapis.com/projects/POOL_PROJECT_NUMBER/locations/global/workloadIdentityPools/panorama-x509/subject/HOST_CN; under an Azure home, principal://.../workloadIdentityPools/POOL/subject/SERVICE_PRINCIPAL_OBJECT_ID.

Then register it, verify it, and enable it:

panorama admin add-gcp-project --tenant-slug acme --project-id PROJECT_ID \
  --name "Acme production" --reader panorama-reader@PROJECT_ID.iam.gserviceaccount.com

The registration is written to the audit log as estate.account.register, with the system as the actor; POST /admin/accounts/projects writes the same row with the administrator as the actor.

Google applies new IAM bindings within a few minutes: a verification run straight after the two bindings above can fail with Permission 'iam.serviceAccounts.getAccessToken' denied, and passes when run again a little later.

Verification (POST /admin/accounts/{accountId}/verify) signs in as the reader, reads the reader account and the project, then runs each Google collector's probe; each check names the permission it needs. It then discovers the project's regions once and adds the ones in use to the account, so the first collection is already regional, and lists them. A tenant without Estate is not probed and nothing is discovered: the answer lists those checks as skipped, with why, rather than as passed. Enabling (PUT /admin/accounts/{accountId}) is the same entitlement-checked step as an AWS account's, and a project counts toward the licence's account cap.

The same three steps from the command line, for an operator on the host or a deployment whose administrators have no portal session to hand, and one collection on a deployment that runs no schedule:

panorama admin verify-account --tenant-slug acme --account-id PROJECT_ID
panorama admin enable-account --tenant-slug acme --account-id PROJECT_ID
panorama admin collect-now    --tenant-slug acme --account-id PROJECT_ID

verify-account runs the API's verification and prints each check; it exits non-zero when one fails. It works for an AWS account too, without the cost report's checks, which are the API's alone. It reads entitlements as serve does: with ENTITLEMENTS_UNRESTRICTED=true a tenant with no entitlement runs every probe, and the flag beside an installed licence is refused. enable-account is the API's enable under the same account cap (--disable switches it off); the change is written to the audit log as estate.account.update, with the system as the actor. collect-now queues one job for each inventory module of the account's cloud, and a worker collects them; a module with a job already waiting keeps that one. It is written to the audit log as estate.account.collect, with the system as the actor, in the transaction that queues the jobs; the Admin area's Collect now is the same row with the administrator as the actor.

An AWS account that moves to a renamed reader role is repointed the same way:

panorama admin set-role-arn --tenant-slug acme --account-id 111122223333 \
  --role-arn arn:aws:iam::111122223333:role/panorama-reader

It is the API's roleArn update, under the API's rules. The ARN must be an IAM role in this account, in a partition AWS has. The external id does not change. The change is written to the audit log as estate.account.update, with the system as the actor. Run verify-account after it. deploy/cloudformation/platform-identity.md §5 gives the whole move.

An account's regions change the same way:

panorama admin set-regions --tenant-slug acme --account-id 111122223333 \
  --regions eu-west-1,eu-north-1

It is the API's regions update, under the API's rules: no region may be blank. The list replaces the old one, so name every region to keep. The change is written to the audit log as estate.account.update, with the system as the actor. Run collect-now after it. A new AWS region also wants ec2-lifecycle-events.yaml deployed there (deploy/cloudformation/README.md, "EC2 instances between two reads"); until it is, that region's lifecycle read reports that the queue is not set up, and the inventory is collected regardless.

What is read, and what is not: reader-role.yaml grants thirty-three permissions. None returns table rows, routine bodies, function configuration or environment, Firestore documents, messages or keys. Eleven would allow more than the product reads: compute.instances.list returns an instance's metadata items (startup scripts, SSH keys), container.clusters.list a cluster's master credentials and node metadata, run.services.list a Cloud Run service's containers' environment, arguments and command and its build's configuration, pubsub.topics.list a topic's message transforms (JavaScript source), pubsub.subscriptions.list the same transforms and a push subscription's endpoint, the backend services' list and get, global and regional, a backend service's custom header values and secrets' identifiers and hashes, and the URL maps' lists, global and regional, the header values a route adds and matches (the load balancers). Every request carries a field mask that keeps those off the wire, and an answer that carries them anyway is refused whole — the run is incomplete, with Google's answer in its message, and nothing of it is stored (MILESTONE_1 §13.3).

Load balancers. A forwarding rule is the row — what a person knows as the load balancer — with the chain behind it and its backends' health. gcp.compute.forwardingrule reads a region's rules, and gcp.compute.globalforwardingrule the project's global ones once under global; each reads the target proxies, URL maps and backend services beside its rules, and only when a rule names them, so a project with no load balancer costs one read per run. Health is backendServices.getHealth, a POST that reads, asked once per backend group Google checks: instance groups and zonal endpoint groups. A rule's state says what it came to: healthy, unhealthy-backends (some), no-healthy-backends (none, or a group with nothing in it), and three unknowns — no-backends (an HTTP-to-HTTPS redirect), health-not-checked (serverless, internet and bucket backends, which Google does not health-check) and backends-not-read (a target pool or instance, whose own health this does not read). chain-unread means a listing or health read behind the rule failed — a permission the role lacks shows here with Google's message under Not read — and the rule is still listed. A Private Service Connect endpoint takes its connection's state. Google reports each endpoint's IPv4 and IPv6 health apart: a passthrough rule (one naming its backend service) is judged in its own address family, so an IPv6 one by its backends' IPv6 health; a proxy rule, whatever its frontend's version, by the IPv4 health of the connections it opens to its backends. Cloud Functions are read from the Functions API's listing of every location, masked to each function's name, state, labels and times; an answer carrying its environment, secrets, build or trigger is refused whole. A function is listed under its own name whether or not it has ever run, carries its own labels, takes its status from its state (active, deploying, failed, …), and goes when the listing stops naming it. Its triggers come from the execution_count metrics of the last two weeks, matched by that name; when those cannot be read the function is still listed, Triggers not read. A second-generation function's Cloud Run service is listed as well, labelled Managed by cloudfunctions, under the service's own name, which Google lowercases and hyphenates. The reader role must carry cloudfunctions.functions.list: an existing project's role takes it with the gcloud iam roles update command at the head of deploy/gcp/reader-role.yaml; until then the function collection is denied.

Regions. Instances, GKE clusters, subnets, Cloud Run services and regional forwarding rules are keyed by the region their zone or location is in, and read one region at a time; datasets, service accounts, apps, functions, Cloud SQL instances, buckets, networks, topics, subscriptions and global forwarding rules are global in the resource key, their location — where they have one — a property the region filter reads. A project registers with no regions. Region discovery (gcp.discovery.region, daily per project, and once at verification) reads one names-only listing per regional service and adds the regions in use to the account; the scheduler then runs the regional collectors there. The subnets Google creates in every region for an auto-mode network (default) are not use: a region holding only those is not added, and they appear on their network's row, not as rows of their own. Cloud Run's listing of every location names Google's restricted regions as unreachable on every call: discovery asks each such location on its own, and one that refuses the project with LOCATION_POLICY_VIOLATED — a region Google has not opened to it — is not applicable and is skipped (a debug log line). A location that still cannot be read is recorded in the discovery's message, not as a failure, and fails verification's check with Google's answer. It only adds: an admin removes a region (PUT /admin/accounts/{accountId} with regions), and may add one by hand the same way. A resource in a region the project had never used appears after the next discovery; Estate → Accounts says when that last ran and how many regions it found. A discovery Google refused part of — an API the project has not enabled — is recorded as finished, with the refusal in its message, and adds what it could read; any other failure fails the run and is retried. A regional collector in a region where the project has not enabled that service's API fails its run with Google's message until the API is enabled.

Connecting an Azure subscription

An Azure subscription is an account of its own cloud, registered by its id and read as an app registration in the customer's directory. Seven collectors read an enabled subscription once per cycle: virtual machines, AKS clusters, SQL databases, PostgreSQL flexible servers, storage accounts, virtual networks and App Service apps, each from one subscription-wide list (the capability matrix, "The Azure reader"), and Performance's perf.azuremonitor reads Azure Monitor's metrics of the measured ones once an hour, under the role's one action that is no collector's: Azure Monitor's metrics read. A statistic Azure refuses for a metric (InvalidSamplingType: an aggregation that metric does not serve) costs that series alone: the run stores every other series, moves on, and is finalized with the refused series named in its message on the collector matrix. Nothing for an operator to change; it names a catalogue entry to correct. A resource Azure Monitor answers "Failed to find metric configuration" for — a Linux consumption function app, at times — is dropped for that run alone: nothing of it is stored, every other resource is, and the finalized run names it with Azure's words; it is read again on the next run (slice 5). Any other 400 still fails the run. Azure answers zeros for the time before a resource existed, so no point is kept from before the inventory's creation time (a type's own, or Resource Manager's systemData.createdAt); App Service gives none, so a site's first pass starts at the first interval in which any of its series is not zero. work signs in as serve does, from the same AZURE_CERTIFICATE_FILE or none. Admin → Accounts offers Azure where this deployment reads it: the Azure block's Connect asks for the subscription, the directory and the app registration by their ids, and answers with the onboarding below; the API and the CLI register the same way. Everything here is accepted on fakes; it is proved live, against the test subscription.

A list Azure throttles is waited out as its Retry-After says, up to three times and thirty seconds a wait; one still throttled, or one whose x-ms-ratelimit-remaining-*-reads headers say the subscription's read budget is spent before its last page, ends the run incomplete, and the next cycle reads it again — what was seen is kept, and nothing is tombstoned on its strength.

The platform holds no secret for Azure. Its own AWS identity asks STS for a web identity token (AWS outbound identity federation), and Entra ID exchanges it for a Resource Manager token because the app trusts that token's issuer, subject and audience as a federated credential. The issuer and subject are the platform's, the same for every organisation it serves; the audience is the organisation's own, api://AzureADTokenExchange/<organisation id>, and the platform asks for no other when it reads that organisation's subscriptions. That is what an AWS account's per-tenant external id is: without it, one organisation could register another's app and subscription and the platform would read them for it. For an operator who knows AWS: a subscription is an account, the app registration is a role, the federated credential is its trust policy, and the custom role is a customer-managed policy.

Once per deployment, in the platform's AWS account (the owner's step on ours; prepared, not run):

aws iam enable-outbound-web-identity-federation
aws iam get-outbound-web-identity-federation-info   # the issuer, https://<id>.tokens.sts.global.api.aws

and the platform's role allowed to ask for the token, for Entra ID's audiences only, for at most five minutes:

{"Effect": "Allow", "Action": "sts:GetWebIdentityToken", "Resource": "*",
 "Condition": {"ForAllValues:StringLike": {"sts:IdentityTokenAudience": "api://AzureADTokenExchange/*"},
               "NumericLessThanEquals": {"sts:DurationSeconds": 300}}}

A platform role made by deploy/cloudformation/platform-identity.yaml gets exactly this statement from the stack's AzureFederation=true, off by default:

aws cloudformation update-stack --stack-name cloudpanorama-platform-identity \
  --template-body file://deploy/cloudformation/platform-identity.yaml --capabilities CAPABILITY_NAMED_IAM \
  --parameters ParameterKey=AzureFederation,ParameterValue=true <every other parameter: UsePreviousValue=true>

The token is asked of the region's STS endpoint (AWS_REGION), because the global endpoint does not serve it. The issuer and the subject the customer's app must trust are the ones in the platform's own token, and the onboarding reads them from one: they are the account's issuer and the platform role's ARN.

Under a Google home the same federated credential trusts the platform service account instead: issuer https://accounts.google.com, subject GCP_PLATFORM_SERVICE_ACCOUNT's numeric id, audience the organisation's api://AzureADTokenExchange/<organisation> — the onboarding reads the issuer and subject from the platform's own token, as it does here. A deployment whose home is Azure signs with its certificate, below.

A deployment off AWS has no such issuer, and signs with a certificate instead (AZURE_CERTIFICATE_FILE, a PEM file with the key and its certificate). The customer uploads the certificate to the app; the key never leaves the deployment. It is not a shared secret, but it is not federation either, and is for that case only. Its assertion is addressed to Entra ID's token endpoint and cannot name the organisation, so an app that holds the certificate trusts every organisation on that deployment: option B suits a deployment off AWS that serves one organisation. No path takes a client secret.

Per subscription, in the customer's directory and subscription — the template does all of it. It and the role file are in deploy/azure/, which the install copied out of the image (from platform 0.1.2; the README's Quickstart):

cd deploy/azure
az login --tenant TENANT_ID
az account set --subscription SUBSCRIPTION_ID
az deployment sub create --location westeurope --template-file reader.bicep \
  --parameters issuer=ISSUER subject=SUBJECT audience=AUDIENCE

or the same as commands (APP_ID is what az ad app create prints):

az ad app create --display-name panorama-reader --query appId -o tsv
az ad sp create --id APP_ID
az ad app federated-credential create --id APP_ID --parameters \
  '{"name":"panorama-federation","issuer":"ISSUER","subject":"SUBJECT","audiences":["AUDIENCE"]}'
sed s/SUBSCRIPTION_ID/<the subscription id>/ reader-role.json > role.json
az role definition create --role-definition @role.json
az role assignment create --assignee APP_ID --role "Panorama metadata reader" \
  --scope /subscriptions/SUBSCRIPTION_ID

The role is deploy/azure/reader-role.json, a custom role holding exactly the actions the platform's reads use — never the built-in Reader, whose */read reads app settings and Key Vault secrets' names. Then register it:

panorama admin add-azure-subscription --tenant-slug acme --account-id SUBSCRIPTION_ID \
  --name "Acme production" --azure-tenant-id TENANT_ID --client-id APP_ID

The registration is written to the audit log as estate.account.register, with the system as the actor.

ISSUER, SUBJECT and AUDIENCE are the onboarding's (GET /admin/accounts/{accountId}/onboarding, or the answer of POST /admin/accounts/subscriptions, which registers the same way), whose steps are these with all three filled in. Verification (panorama admin verify-account, or POST /admin/accounts/{accountId}/verify) signs in as the app — a refused exchange is reported with Entra ID's AADSTS code — reads the subscription, then the app's own permissions at the subscription: it fails when an action the reads need is not granted there (a role assigned at a resource group grants nothing at the subscription), and when the app holds any action beyond the reader role's — a wildcard such as the built-in Reader's */read, any data action, any */listKeys/action or */listSecrets/action, anything else — naming what a known one would hand over (internal/onboarding's never-granted list). A NotAction counts only in the role assignment it is in. A token the platform holds is sent to Resource Manager alone, over https, and a redirect is never followed. The account id is the subscription id, lower-cased; an ARM id is case-insensitive, and the key holds it lower-cased by resourcekey.FromAzureID alone.

When the public page is wrong

It is read from the same tables as everything else, cached for thirty seconds per tenant and rate-limited per client address. "Nothing is being monitored yet" means no live resources pass the base filter: a tombstoned resource, or one whose bucket has not been collected successfully for a day, is absent here as everywhere. Check the matrix first.

The page can never carry a resource name, id, account id, region, IP or ARN; TestThePublicPageLeaksNothing seeds recognisable values of each and asserts their absence. If a change to internal/api/handlers/public_status.go is proposed, that is a security review.

The front door. A visitor with no session who opens the portal's root sees this page for the tenant the host shows — the tenant the host names by its first label, or, on a single-tenant deployment, the one tenant — with a link to sign in. On a single-tenant deployment exposed to the internet that means the bare host reveals the tenant's slug and the aggregate health this page shows to anyone, and nothing else: no name, no resource, and nothing about how many tenants there are (GET /public/tenant answers with the slug and the demo mark alone, or a 404). A deployment that wants the sign-in page as its door keeps the root behind its own front, or holds more than one tenant.

The demo tenant

panorama admin seed-demo creates a tenant flagged is_demo, which the scheduler excludes from collection and notification work. It populates inventory, organization, spend and simulated provisioning history without cloud calls. Its data ages like any other tenant's — retention applies, and the matrix will go no-data after three cadences because nothing collects — which is what make seed RESET=1 is for. A slug that names a real tenant is refused. There is no way to turn a demo tenant into a real one; create a real one with panorama admin bootstrap.

Re-running the seed upgrades an inventory-only demo without resetting its login. RESET=1 refreshes inventory and operational history; it preserves platform decisions and the original cost-history window. Seeded execution logs are simulated, the example agent is revoked and the webhook is disabled.

seed-demo --refresh is the reset plus the dated history moved to today — cost through yesterday, its anomalies, the AI reviews — and keeps users, sessions, budgets and decisions. It refuses a tenant that does not exist, so it never prints a login; it is what a demo left standing runs on a timer. deploy/demo/ holds the public demo's nightly unit and script (dump, refresh, read-back, a status file and a failed unit on error); ../platform/docs/demo-data.md says what each refresh writes.

The built-in library

The platform carries the provisioning templates it ships inside its own binary, and serve verifies them at start: every hash the library has released is present and hashes to its name. A binary that fails this refuses to start with the embedded library: … — a defect of the build, not of the deployment's configuration or data: roll back to the previous image, and report it. Nothing in the database can cause it.

A new library release reaches a deployment with a platform upgrade, and an account with its agent's upgrade: a request pinned to a library version waits, queued, until an agent that carries it claims it. The library read (GET /provisioning/library) says which accounts' agents can run each template and which agent version the rest need; GET /provisioning/agents shows the library release each agent last reported.

The AI reviewer

The ai-reviewer notification destination hands every alert and cost anomaly routed to it to the operator's queue as one small claim check per subject; the reviewer service (../reviewer/) on the operator's machine reads the subject through the platform and answers. What an operator runs and watches:

The queue and the identity. deploy/cloudformation/ai-reviewer.yaml makes the queue (cloudpanorama-ai-reviews), its dead-letter queue, an alarm on it, and the reviewer's IAM Roles Anywhere role under the platform's trust anchor, allowed to receive, delete and re-hide on that queue alone. Pass its QueueArn to platform-identity.yaml as AIReviewerQueueArn so the platform may send, and its QueueUrl to the platform as AI_REVIEWER_QUEUE_URL. The queue's retention is one day: a claim check's token answers for a day (reviews.TokenLife), so a reviewer down longer finds nothing it could answer and nothing waits to be retried in vain. The visibility timeout is fifteen minutes, the review's wall-clock budget three times over, extended by the service while a review runs; a reviewer that crashes mid-review sees the message again after that, and answers it idempotently. A message received seven times without an answer is dead-lettered — three delivery attempts plus the hides a review over its tenant's rate may need (each hide is a receive, and SQS hides for at most twelve hours) — and the alarm <stack>-dead-letters says so. A deferral is not a failure: panorama-reviewer status counts it as deferred for rate.

The audience. A review reads what the people who will read its answer may read, and no more: the subject account's team — tenant-wide only when no team holds the account — intersected with the asker's scope when a person asked, recorded in ai_reviews.audience_* when the review opens and only ever narrowed by a reopen. Access that changes afterwards narrows it further: the reviewer's reads and its answer are held to the recorded audience as the account's team and the requester are now, and a member reads the review only while their teams cover the recorded audience. So when an account moves team, its open and past reviews stay with the administrators — the new team is not handed the old team's answers — and a person who leaves a team stops seeing the reviews they asked for there — including the AI-review entries in their inbox, which carry the answer and are filtered on every read (migration 0203 names the review on each entry; an entry written before it is shown to administrators only). A notifier review of a team's alert therefore cannot read the tenant's or an account's forecast (a member's rule), and gets 403 there; that is the rule working, not a fault. A review opened by a binary older than migration 0202 has no audience and reads no team-scoped data until its token expires, within a day.

A storm. The destination sends claim checks ten to a batch and skips a subject reviewed inside the last hour: a storm of alerts on one resource is one review an hour.

The capability. The reviewer is sold as the licence capability ai-reviewer (the owner's decision, 2026-09-17): a tenant without it cannot create the destination, and GET /ai/reviews is refused. The Enterprise tier carries it.

Watching the service. On the operator's machine, panorama-reviewer status says when it last polled, what it last reviewed and with which verdict, and how many messages were delivered, left for redelivery or found expired; its log carries ids, never summaries. The dead-letter queue's depth is the number to watch on the platform's side.

The Triage sidecar

Triage (TRIAGE_SPEC.md) puts every event to a small decision model — reviewer, people, or nobody — before the notifier routes it (in shadow: nothing routes by it yet). By default the model is Open-Jev's 2B checkpoint in a sidecar beside the roles, built by us from pinned sources with the weights baked in (deploy/triage/README.md); TypeSafe's hosted Jev is the alternative. The roles that ask it, notify and serve, read the TRIAGE_* settings (deploy/.env.example) and refuse to start on one that fails its rules: an alias model, a key for the sidecar or none for the hosted provider, a public address for the sidecar.

What it does today: shadow only. The notifier puts every status transition, cost anomaly, budget crossing and provisioning event to the model and records the decision in triage_decisions — and routes every event exactly as before, whatever the model says or whether it answers. A kind is in shadow once a provider is configured and off without one. The decisions of one batch share one TRIAGE_TIMEOUT, so a slow or absent model delays a notification by at most that long; the events it did not reach are recorded as timed out. The notifier's event rate is rate(panorama_triage_events_total[1h]) on the notify role's metrics, by kind and mode — counted with or without a model, so read it before sizing a sidecar; panorama_triage_suggestions_total counts what the record suggested, and panorama_triage_errors_total the failures by class.

It has no authentication, so its isolation is the control. In compose it runs only with the triage profile, publishes no port, and sits on an internal network that only notify and serve share — no route out either: it starts offline. In Kubernetes, deploy/k8s/components/triage adds the Deployment, a ClusterIP Service and a NetworkPolicy admitting those two pods on port 8791 and nothing out, and sets TRIAGE_PROVIDER, TRIAGE_URL and TRIAGE_MODEL in panorama-config. Never give it an Ingress, a published port or a route from the proxy.

Turning it on. Compose: COMPOSE_PROFILES=triage and TRIAGE_PROVIDER=open-jev in deploy/.env, then docker compose up -d (the first up builds the ~10 GB image unless TRIAGE_IMAGE names the published ghcr.io/cloudpanorama/panorama/panorama-triage:<commit>-cpu). Kubernetes: add the component to your overlay and pin the image by digest beside panorama's. The model the image answers as is open-jev-2b@sha256:3076462e6356412082e79af909227b39b2863b90def79155ca0821aa506b7ded, the digest of its checkpoint; the image refuses to serve under any other name, and the platform refuses an answer from any other model, so TRIAGE_MODEL must be exactly that.

Checking it. The sidecar logs one serving line with the model and device once the checkpoint is loaded, and only then answers /health; compose's health check and Kubernetes' startup probe allow three minutes for that, against nine seconds measured. make test-triage-model (in this module, with TRIAGE_MODEL and TRIAGE_URL set) sends each kind's example state through the platform's own client and checks the answer: by hand or nightly, never in per-push CI, because it needs the model.

Resources, measured. On the platform host (2026-10-05, 12 AMD Genoa vCPUs, no GPU, the platform running beside it), the linux/amd64 CPU image (9.8 GB) built from the pinned sources, on an internal network with no route out, at a lower CPU weight than the platform's roles. Decisions were asked one at a time with each kind's example state (make test-triage-model's, 394 to 644 input tokens), then all four at once:

By kind, on 4 CPUs (p50 / p95): provisioning 2.9 / 3.1 s, budget 3.3 / 3.5 s, cost anomaly 3.7 / 3.8 s, status transition 4.6 / 4.9 s. The first decision after a start is up to 2 s slower (2 CPUs: 9.3 s, 4: 5.4 s, 8: 3.5 s). The sidecar answers one decision at a time: decisions asked together queue, so the last of four waits for the other three. Time grows with the state, about 7 ms per input token on 4 CPUs (Open-Jev counts the state once per choice, so three times): 2,060 tokens took 14 s, 5,336 took 36 s, and a state at the model's 4,096-token limit (11,771) 83 s. A larger state is refused with 422 and the decision goes to people.

Memory: 0.8 GiB once loaded, 3.4 to 3.7 GiB once answering — 2.7 GiB of that is the weights, mapped from the image as they are first used — and a peak of 4.9 GiB at the largest state. It loaded in 9.3 seconds with its files out of the page cache, 7 seconds with them in it.

So the shipped limits: 4 CPUs, which OMP_NUM_THREADS matches (torch takes every core it sees otherwise); memory 6 GiB in compose (6g), and in Kubernetes a 4 Gi request with a 6 Gi limit; a three-minute start-up allowance in the image's health check, compose and the startup probe. More CPUs help little (8 CPUs cut p95 by a third); fewer cost more (2 CPUs add half).

TRIAGE_TIMEOUT. The default 10 s holds for the shipped 4 CPUs asked one decision at a time, with twice the p95 to spare. Allow about twice the p95 times the decisions in flight: 15 s on 2 CPUs, 40 s if four are asked together on 4. A decision that times out goes to people, so a timeout too short loses nothing but a decision — but a kind that times out often has nothing to enforce, and its record shows it. On a development Mac's Docker (3 CPUs, arm64) one decision took 16 to 36 seconds: measure where the sidecar will run before a kind leaves shadow.

The licence

A tenant's modules come from a licence: an ed25519-signed JSON file the owner issues and an admin installs. Nothing phones home. The binary carries the verifying public key (LICENSE_PUBLIC_KEY at build time — the Makefile, the Dockerfile and CI all pass it as the same ldflag); the signing key never enters the product, this repository, or any machine the product runs on.

What the customer sees. GET /entitlements carries one word, state, and the portal turns it into a banner on every page: expiring (thirty days out, or for the last third of a licence shorter than ninety days — a thirty-day trial warns for its last ten; warning, nothing locked; a row with no issue date, manual or installed before migration 0137, keeps thirty), lapsed (past valid_until, destructive, every module locked, collection continues for fourteen days), paused (past that grace, collection stopped), none (no licence at all), unrestricted (the development exception). A valid licence has no banner. Nothing is ever deleted because of a lapse; a new licence reopens the modules the moment it is installed and the scheduler enqueues the tenant again on its next tick.

Two things the release exercise taught (2026-09-17). A renewal is a file issued after the one installed: the anti-rollback rule compares issued_at, so a renewal issued before a short-lived file that came later is refused as the older one — issue the renewal fresh. And a file in the licences mount is read by the container as another user: copy it at 644 (a 600 file is "permission denied" from inside).

Install, or replace.

panorama admin install-license acme.license            # slug comes from the file
panorama admin install-license acme.license --tenant-slug acme   # belt and braces
panorama admin show-license --tenant-slug acme

The path is read inside the container. In the compose stack, which has no licences mount, mount the file for the one run (at 644, above):

docker compose -f deploy/docker-compose.yml run --rm \
  -v "$PWD/acme.license:/licences/acme.license:ro" serve admin install-license /licences/acme.license

Install verifies the signature before it reads a byte of the payload, refuses an expired file, a file for another tenant, and a build with no public key, and is a no-op for a licence already installed. A newer licence (by issued_at) replaces the row; an older file, however valid, is refused with both licence ids and issue times, so re-installing a stale file cannot roll a renewal back. A licence installed before schema 137 recorded issue times is handled the same way with what the row still knows: re-installing its own file records the issue time ("recorded its issue time"), a different file installs only if it was issued after the row was written, and an older one is refused with the way out spelled out. After upgrading such a deployment, re-install the current file once. show-license says which license_id is live; a hand-typed grant with no expiry shows as "no expiry". The server refuses to start with ENTITLEMENTS_UNRESTRICTED=true once any licence is installed: the flag and the licence describe different deployments, so a stack that had the flag set (compose defaults it on) must drop it before the first install.

Which key does this build carry? panorama version prints it:

docker run --rm ghcr.io/cloudpanorama/panorama/panorama@sha256:<the digest you deploy> version
# panorama v1.4.0 (c0f91d6)
# licence key: 57023e96…

"licence key: none" is a build with an empty LICENSE_PUBLIC_KEY: it accepts no licence and runs only with ENTITLEMENTS_UNRESTRICTED. CI refuses to build the image unless the repository variable LICENSE_PUBLIC_KEY is exactly 64 lowercase hex characters, reads the line above off the image it built on every run, and pushes only the image that passed — main included. The Dockerfile itself refuses to finish a build whose binary did not take the key it was given, so a moved linker target fails every builder, not only CI. Setting the variable is gh variable set LICENSE_PUBLIC_KEY --body "$(cat license-signing.pub)".

Issue — the owner's side, on the owner's machine, never in the image:

go build -o cloudpanorama-license ./cmd/cloudpanorama-license   # from observability/
cloudpanorama-license keygen --out ~/keys                        # once; refuses a git tree
cloudpanorama-license issue --key ~/keys/license-signing.key \
    --tenant acme --licensee 'Acme GmbH' --tier professional \
    --max-accounts 50 --valid-until 2027-09-01 --out acme.license
cloudpanorama-license issue --key ~/keys/license-signing.key \
    --tenant acme --licensee 'Acme GmbH' --trial --out acme-trial.license
cloudpanorama-license inspect acme.license --pub ~/keys/license-signing.pub

Tiers are ids (essentials, professional, enterprise) with module and capability defaults (PLATFORM.md §7); --modules, --capabilities and --max-accounts override them. A trial is an ordinary licence: every module, ten accounts, thirty days, licensee suffixed "(trial)" — valid for twenty days, then expiring for its last ten. The private key lives in the owner's password manager as the hex seed keygen wrote; the .pub beside it is the value for LICENSE_PUBLIC_KEY.

Rotate the key when it may have leaked, or on a schedule if you prefer one. New keypair; the new public key goes into the next release; every live licence is re-issued with the new key (each has a license_id and an issued_at, so the re-issue is auditable against the old list) and re-installed by each customer as part of upgrading. Until a customer installs the re-issued file, the old row stays valid — the new build only refuses new installs signed by the old key, it does not re-verify what is already in the database.

A customer's licence lapsed and they renewed — install the new file; that is the whole procedure. If the tenant sat paused for longer than fourteen days, the first collection after the install is a normal reconciliation: resources that disappeared meanwhile are tombstoned with that run's run_id in the finalize log, as any run would.

Connecting a Cost & Usage Report

A report is read under a role of its own: the inventory reader role never reads object content, so each account that shares a report deploys deploy/cloudformation/cost-report-role.yaml, scoped to that report's bucket and prefix, and the platform assumes that role — with the tenant's external id — for the report and nothing else. The steps, per account:

1. In AWS Billing, define the report: CSV, GZIP, hourly or daily, with resource ids, delivered to a bucket and a report path prefix. AWS delivers the first files up to 24 hours later.
2. Deploy the template with the platform account, the external id, the bucket and the prefix followed by the report name (reports/ccc for a report named ccc under reports), and take its RoleArn output.
3. In the portal's cost settings, add the report: account, bucket, its region, prefix, report name and the role ARN. Save.
4. Verify the account (POST /admin/accounts/{accountId}/verify, or the portal's verify button). Beside the inventory checks, the report role is assumed and its two reads are tried where a pull would make them: s3:ListBucket on the report prefix, then s3:GetObject on the newest manifest. A failed check names the action and the ARN to grant on; a report with no role entered yet is reported as the missing step.
5. The pull runs on the slow tier. cost_ingestion_state.last_error for source cur carries the same named reason when it fails, and the settings page shows the last period ingested once it succeeds.

What it will not do: read a bucket other than the one configured, list a prefix the role's condition does not admit, or fall back to the inventory role — a report without a role of its own is refused before any request is made. A bucket encrypted with a customer-managed KMS key needs kms:Decrypt on that key added to the role; the template says where.

Rebuilding Cost & Usage Report periods

The CUR job re-reads a billing period only when AWS publishes a new assembly of it, so a correction to the parser — the amortization mapping of 2026-09-06 is the first — leaves every period already ingested as it was. panorama admin rebuild-cur re-ingests them:

panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08 --dry-run   # the before
panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08             # mark and enqueue
panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08 --dry-run   # the after, once a worker has run

--dry-run prints each stored period — account, month, assembly, whether it is current, its line-item and row counts, and the stored unblended and amortized totals per currency — and changes nothing. Without it the command forgets the current assembly of every period in the range, enqueues the cost.cur job for each account so the same reports are streamed again through the current parser, and requests the attribution rebuild. The re-ingest replaces each period in one transaction, the path a republished month takes: running the command twice marks nothing the second time, and a re-ingest never adds to a period. --account narrows it to one account. The reports must still be in the bucket: a period whose files AWS has removed fails its run and is recorded in cost_ingestion_state like any other missing report.

The same re-read records the tags of each resource line (cost_data.line_tags, migration 0258) for periods ingested before it. Until a period has them, a member of a team that holds spend there by tag reads the cost explanation at the service grain; a period AWS republishes records them on its own.

Google Cloud's and Azure's exports record them too: each pull writes, on every resource row it alone charges, the labels or tags of the export rows the row sums. Every pull re-reads the recent months (three for Google, two for Azure), so those months have them after the first pull of the release; older months keep none and read at the service grain for a team holding spend there by label or tag. A project or subscription two exports charge in one month (moved between billing accounts) records none for that month.

The rollups follow the facts, durably: a period's assembly records when its rollups were built (rollups_built_at, migrations 0138–0139), the record is written only after the build succeeds, and every run rebuilds the periods still owed before it reads anything new. After upgrading through 0139 every current period is owed once — the first cost.cur run of each account rebuilds its rollups and records it; --dry-run shows owed until then.

What the correction changed, so the before and after read right: a Savings Plan negation no longer subtracts the on-demand price from the effective cost a second time; an unused Savings Plan commitment and an unused reservation stay in the amortized view; an ordinary fee such as Support is money in both views; only an upfront reservation fee and a negation amortize to zero on their own rows. Expect amortized totals to rise for any month with Savings Plans, reservations or a Support plan.

How the tables grow

Measured, not estimated. A resource here is an EC2 instance with the details its collector emits and six tags.

resource_status_transitions is not partitioned, deliberately. It is never dropped by age, so a partition would buy no retention, and every reader is bounded by an index range rather than by the table — the timeline's page, the public page's seven days, the notifier's outbox and one resource's history. Measured as panorama_app at 300,000 and 3,000,000 rows: each plan was the same index range scan at both sizes, the timeline and the outbox at 0.1 ms, and the public page's cost followed the rows inside its seven days, not the table. Revisit at 10 million rows in any database, or when one of those plans leaves its index: partitioning then means a primary key of (id, occurred_at) and a rewrite of the table, a few GiB at that size.

SELECT relname, reltuples::bigint AS rows, pg_size_pretty(pg_total_relation_size(oid))
  FROM pg_class
 WHERE relname IN ('resources', 'resource_snapshots', 'resource_status_transitions');

(reltuples of a partitioned parent is -1; sum its partitions, or count.)

The cost levers

What the running bill is made of and which settings move it, so nobody has to re-derive the 2026-09-09 review's model (its §9: a pilot at roughly $205–400 a month, a small SaaS at $805–1,270, mid scale at $5,150–6,860, where the two lines that grow are Cost Explorer's request fees and snapshot storage). All of these are configuration.

Both levers the review found blocked by code are now configuration or built in: snapshots are change-only (no setting — the owner's decision) and cost history has its retention (2026-09-11).

Limits, headers and scanners

Every unauthenticated route has a per-address budget and every body-taking route a size limit, both derived from the contract by middleware.Policies (security: [] → 120 requests a minute per address; requestBody → 1 MiB unless the operation says x-max-body-bytes; x-rate-limit on any route) and applied by the guarded router before authentication. A client past its budget gets 429 rate_limited with Retry-After; a body past its limit 413. The budget is one count for every serve replica, kept in the database: each limited request is one statement on rate_limit_windows, an unlogged table that also deletes a few ended windows as it goes, so it needs no cleanup job and a crash only restarts the counts. When that statement fails or takes over 250 ms, each process counts on its own — the budget per replica — and logs the shared rate limit could not be charged once a minute per route; that line means the database is struggling, not the limiter. The address is the one CLIENT_ADDRESS_SOURCE declares — behind a proxy that is the trusted proxy's header, so set it, or every client is the proxy. Every response carries the security headers (middleware.SecurityHeaders; HSTS only when COOKIE_SECURE).

The session's CSRF boundary is the API's, not the cookie's. An unsafe request (anything but GET, HEAD, OPTIONS) on a session route — and on /auth/login and /auth/logout — is refused with 403 origin_not_trusted unless the browser's own context vouches for it: Sec-Fetch-Site: same-origin, or an Origin that is this server's host or one listed in CORS_ORIGINS. A body on such a route must be labelled application/json (415 unsupported_media_type). Two consequences for a deployment: a portal served from a second origin works exactly when that origin is in CORS_ORIGINS (the same list CORS answers for), and a proxy that rewrites the Host header must either preserve it (proxy_set_header Host $host in nginx) or list the public origin in CORS_ORIGINS, or browsers without Fetch Metadata will be refused. Collectors, workers and agents are not held to any of this: their credential is a header, and the check is by credential kind. ../web/e2e/README.md runs the real-browser proof against a same-site sibling.

Time and concurrency budgets

A byte limit caps a body's size, not how long a client may take to send it. Every route now has a time budget too, from the contract and applied by the guarded router before the first byte of the body is read: a route with a body waits 30 s for it unless the operation says x-body-timeout-seconds — 120 s for the snapshot batch (the worker gives itself a minute to send one, HTTP_TIMEOUT's ceiling), 60 s for agent logs and template sources — and a route without waits 10 s for whatever unread body a request carried. A request with no body at all gets no read deadline: the budget bounds reading the request, never the handler's work, which a deadline left on the connection would cancel. The response gets 60 s past that. The claim's long poll is the one route that holds a connection longer on purpose, and its handler extends its own deadline to the poll plus 30 s; PROVISIONING_CLAIM_POLL_TIMEOUT is bounded at 4 minutes so the server's write ceiling (5 minutes, cli.serve) clears it. The ceilings — 5 minutes to read, 5 to write — are not the budgets: they reset a connection's deadlines between requests and bound the routes the router does not guard (the portal's files, the probes).

Concurrency is bounded where a body is large enough to matter: x-max-inflight caps how many bodies a route reads at once — two snapshot batches (a batch buffer is up to 192 MiB, and the k8s manifest gives the process 512 MiB), sixteen agent log batches — and the next caller gets 429 rate_limited with Retry-After: 5 rather than a place in memory the process does not have. Collectors and agents already retry a 429 with backoff. panorama_http_requests_inflight counts every request being served, bodies still being read included, so a stall is visible while it stalls; a gauge sitting at a route's cap is the signal.

A stalled body holds no database connection: every write path decodes the body before it opens a transaction, and the budget ends the read before anything else happens. What is held is one goroutine, one socket and the bytes buffered so far, for the budget and no longer.

The edge. Nothing in this repository fronts the API by default — compose binds it to loopback and the k8s manifests leave the Ingress to you — so the application's budgets are the boundary, not a convenience. A proxy adds TLS, a per-connection ceiling before the application sees the socket and a total request rate, and it can break the two long paths if its own timeouts sit below the application's: deploy/proxy/nginx.conf.example is a configuration sized against these numbers (client_body_timeout and client_max_body_size at or above the batch route's, proxy_read_timeout above the claim poll plus the response budget, Host preserved), and it answers 404 for the ingest routes — /collection-runs and /ingest/ — because the workers reach serve directly and those routes can tombstone an inventory. The same numbers on an nginx Ingress are the nginx.ingress.kubernetes.io/proxy-read-timeout: "120", proxy-body-size: 192m and proxy-request-buffering: "off" annotations; an AWS ALB has an idle timeout (60 s by default) that must exceed the claim poll and has no body-read timeout at all, which is the argument for the application's.

Three scanners run in CI and can fail a build: govulncheck on both Go modules, npm audit on what the portal ships, Trivy on the image before it is pushed. What each fails on, and what to do when one does, is docs/security.md "The triage rule". The Go module pins its patch release (go 1.26.10 in go.mod, since 2026-10-21) because the standard library is what the scanner most often reports; bumping it is the routine answer.

Adding a collector

The M2 step-by-step, for extending the inventory to a new AWS service.

The step-by-step for the next module, written against the two that exist: internal/collectors/aws/ec2 (instances) and internal/collectors/k8s/eks (nodes and pods). Follow it in order; each step has a test that fails when it is skipped. Nothing in the frontend changes — a resource type the backend describes renders without a line of module-specific code, and noServiceSpecificCode.test.ts fails the build if one appears.

A module is one resource type in one service: aws.rds.instance, aws.s3.bucket. It is the identifier a run registers under, the unit the scheduler enqueues, and the column in the admin matrix. Pick it first; every step below names it.

0. Read these before writing anything

• MILESTONE_1_PLAN.md §3, the lifecycle — buckets, runs, generations, fences, tombstones. A collector that misunderstands incomplete either freezes reconciliation or deletes resources that are still running.
• internal/collectors/collector.go, the interface, and incomplete.go.
• internal/collectors/aws/ec2/, all four files. Copy its shape, not its code.

1. The collector: internal/collectors/<cloud>/<service>/collector.go

Implement collectors.Collector:

Rules the EC2 collector demonstrates:

• Never touch the database. Return snapshots; the worker posts them.
• AdditionalInfo is the deriver's input and the detail page's content. Put the raw provider state under a key the deriver reads (state, systemStatus, …) and the fields the descriptor will show. Keep it under the item size budget (ingest.MaxAdditionalInfoBytes); a large sub-object such as a security-group list is fine, a whole API response is not.
• Omit what the provider says is gone. EC2 keeps terminated instances visible for an hour; emitting one means "I observed it", so it is never tombstoned. Let reconciliation do deletion.
• Tags are a field, not a key in AdditionalInfo (Snapshot.Tags). Store exactly what the provider returned — no case folding.
• Take the SDK client through the aws.Config you are handed. It already carries the assumed role and the region; construct the service client from it and nothing else, so tests can substitute a fake.

Test it with a fake client (collector_test.go): a full page, a paginated list whose second page fails (must return Incomplete), an enrichment failure (must not), and the terminated/omitted case for your service's equivalent.

A Google Cloud collector implements collectors.GoogleCollector the same way, with the project reader's *http.Client in place of an aws.Config (gcptest.Client fakes it), and since then two more rules hold (MILESTONE_1 §13.3, §13.4):

• Every request goes through gcpclient.GetMasked with a mask that names the fields the collector shows and nothing else. Where the listing permission also returns content (an instance's metadata, a cluster's credentials, a Run service's environment), the mask's Excluded names those paths, and an answer carrying one is refused whole. The tests hold every request the collector can make to its mask and prove such an answer is dropped; onboarding.TestTheContentBearingGrantsAreExactlyTheMaskedOnes names the permission.
• Regional or global is the identifier's: a type whose identifier names a zone or region is regional — its buckets are the job's region, it reads only that region's locations (gcpclient.ZonesOf for a region's zones), and region discovery's sources gains its aggregated, names-only read; one whose identifier names none is PerAccount under global, with its location in additionalInfo.region for the region filter. The key derives from the identifier the bill carries (resourcekey.FromGoogleBilling), and the registry test holds the region rule. A type whose identifiers name a region for some resources and global for others — a forwarding rule — is two modules over one type, one per placement, and resourcekey.PlacedByIdentifier names it so the registry test holds each module to its own.
• A read Google makes a POST — backendServices.getHealth — goes through gcpclient.PostMasked, which admits only the methods ReadingPosts names and masks the answer as GetMasked does. A new one joins that list with the reason it reads, and TestOnlyTheReadingPostsAreSent holds the list.

2. The status deriver: status.go

func init() {
	status.Register(system, service, resourceType, derive)
	rendermeta.Register(descriptor())
}

func derive(info map[string]any) (status.Status, string) { … }

One function from AdditionalInfo to (Status, raw state). Status is derived once, at ingest, and stored; nothing re-derives it in SQL or TypeScript. An unknown state is Degraded (something you did not anticipate deserves a look, not a green tick). Do not return Operational for "no information": the EC2 deriver treats missing status checks as Operational only when the collector marked them unavailable explicitly, so a missing key is a bug you see, not one you hide.

Registering happens in init() because the deriver must exist in every process that stores a snapshot. That is why internal/cli/schedule.go and work.go import the package — and why step 5 matters: a binary that schedules a module without importing its deriver stores every resource of that type as Degraded, loudly, on purpose.

Test the table: every provider state you know of, the unknown state, and the enrichment-unavailable case (status_test.go).

3. The render descriptor: rendermeta.go

The detail page and the list columns come from here, not from the frontend.

rendermeta.Descriptor{
	System: "aws", Service: "rds", ResourceType: "instance",
	Label: "RDS Instance", PluralLabel: "RDS Instances", Icon: "database",
	ListColumns: []string{"resourceName", "state", "additionalInfo.engine", "region"},
	Groups: []rendermeta.Group{{Title: "Overview", Fields: []rendermeta.Field{
		{Key: "state", Label: "State", Format: rendermeta.Badge, BadgeMap: …},
		{Key: "additionalInfo.engine", Label: "Engine", Format: rendermeta.Text},
	}}},
}

Keys are paths into the resource (resourceName, state, region, additionalInfo.<field>); formats are the shared vocabulary in internal/rendermeta (Text, Badge, DateTime, Boolean, Bytes, Table with Columns, JSON). PluralLabel is what the public page says ("3 instances became degraded"). tags is rendered by the registry for every type; do not describe it.

rendermeta's tests validate every registered descriptor at build time: a key that does not exist on the snapshot type, a badge map naming a state the deriver never returns, a format without its required options — each fails the suite, with the module named. GET /resource-types serves the registry with an ETag, and the portal renders whatever it says.

4. Register it once: internal/modules

Buckets are step 1's Buckets(); this step is about who asks for them, and since then the answer is one table. Add one entry to modules.All():

{New: func() collectors.Collector { return rds.New() }, Tier: queue.TierStandard, Actions: rds.Actions},

• Tier decides the cadence: critical (5 m), standard (1 h), slow (24 h). The scheduler's list, the admin matrix's cells, the worker's registry and Verify's probes all derive from this entry, so a module scheduled is one a worker carries and serve can verify — the three used to be listed separately and drifted.
• Actions is a package-level var beside your API interface: exactly the read calls Collect and Probe make, nothing content-bearing (no GetObject, no GetQueryResults, no user data). The reader-role test takes its allowlist from the table, so the template deploy/cloudformation/reader-role.yaml must grant the same calls in a statement of its own — that is the one thing still written twice, because the template is what a customer deploys and the test is what keeps it honest. The test also holds a denylist; if your call is on it, the call is wrong for this product, and the module cannot be built (Lambda's ListFunctions returns environment variables).
• TofuTypes are the OpenTofu resource types that create what your module collects, one for one — aws_db_instance for RDS instances; aws_security_group and aws_default_security_group for security groups. A type whose one resource your collector returns as several rows may be named too, when each row carries its tags: an ElastiCache replication group is its member clusters, and the diff's per-kind cap counts it present while any of them stands. The drift diff compares a create's stamped resources of these types, and of no others: leave one out and its deletions go unseen, name one your collector does not return with its tags and a clean create reads as missing. A module whose Tags are TagsNotRead still names its types, so the portal can say why they are not compared. Name only the types whose resources your collector returns as rows of its kind; a type stays in one module (provisioned_test.go holds both, and the first wave's live placements). Your module's own collection runs are what the diff waits for: a type is compared once a run of the module that started after the create has finished, in every region it runs in for the account, or under global when the module is PerAccount — so a run that fails leaves new creates unjudged rather than missing.

5. What the table checks at build time

internal/modules imports your package, which runs its init() in every process — schedule, work and serve alike — so the deriver exists wherever a snapshot is stored. TestEveryModuleHasADeriverAndADescriptor then walks every module's buckets and fails if a deriver or a descriptor is missing: the loud all-Degraded failure of step 2 is caught before a commit, not on an estate. (It caught the Kubernetes node and pod types on its first run: derivers, no descriptors.) There is no separate import to wire.

6. Tests that must exist before the module ships

Then run the whole thing against the compose stack with one real account enabled, watch the matrix cell go ok, open a resource, and read the detail page. If a field is missing there, it is missing from AdditionalInfo or from the descriptor — never from the frontend.

What writing this document said about the abstractions

The task asked for this to be treated as a finding, so here it is.

The interface holds. Collector's four methods and the incomplete distinction were enough for EKS — a second cloud, two resource types from one read, a presigned token instead of an SDK call — without a change to the interface, the worker, ingest or the frontend. That is the result Phase C was for.

Registration was spread across five places — the package's own init() (deriver and descriptor), scheduledModules, worker.NewRegistry, the verify list in serve.go, and the IAM allowlist with its mirror test — and a fifteen-module M2 would have forgotten one of the five at least once per module. Done, before the third collector as this said: internal/modules is the one value carrying the collector, its tier and its IAM actions, and the scheduler's list, the worker's registry, the verify list and the policy test derive from it (step 4). What stays written twice is the CloudFormation template, on purpose.

The IAM policy is the step most likely to be wrong, because it is the one that cannot be tested without an AWS account. The mirror test keeps the document and the code in step; it cannot say whether AWS agrees. The first real run of a new module against a fresh account is the test, and the runbook says what its failure looks like.