Operations

The runbook — the queue, a failing collector, migrations, token rotation, the public page, the demo tenant, limits and scanners — and the collector how-to.

What to do when something stops. Each section names the symptom, where to look, what it usually is, and the command that fixes it. Every command runs against the roles' environment (.env.example); in the compose stack, prefix with docker compose -f deploy/docker-compose.yml exec serve or use make.

The order of investigation is always the same: is the data fresh? (the admin matrix), is work moving? (the queue), is anything failing? (the runs' error text and the logs). Alerts and the public page read from the same tables, so a stale matrix explains a quiet public page.

Reading the admin matrix

Admin → Collector matrix, or GET /admin/collector-runs/matrix. One cell per enabled account × configured region × module (per-account modules such as the cost pulls have one cell, at global). Each cell is the newest run's verdict:

The issues panel lists every account that is not ok, worst first. "Stored 0 of 12 attempted" is the collector's own count: it saw twelve instances and wrote none, which points at the ingest API rather than at AWS.

Staleness is judged against the cadence in TIER_*_INTERVAL; serve and schedule read the same values, so a cell cannot be stale by one process's clock and fresh by the other's.

The queue backs up

Symptom. no-data spreading across the matrix; panorama_queue_depth climbing on the scheduler's metrics listener (METRICS_LISTEN; the PanoramaQueueBehind alert in deploy/monitoring/alerts.rules.yml), and the oldest pending row's run_after receding into the past.

Look.

SELECT state, tier, count(*), min(run_after) FROM jobs GROUP BY 1, 2 ORDER BY 1, 2;
SELECT id, tenant_id, account_id, region, module, attempts, locked_at, lease_expires_at
  FROM jobs WHERE state = 'running' ORDER BY locked_at;

Usually.

• No workers. pending grows, nothing is running. Check work's logs and that it started at all: it refuses to start with zero or two credentials (WORKER_TOKEN, INGEST_TOKEN, INGEST_AUTH_DISABLED=true — exactly one), or with JOB_LEASE shorter than MODULE_TIMEOUT + HTTP_TIMEOUT.
• Workers that cannot reach the API. Runs register and then every batch fails; work logs 5xx or connection errors against API_URL. Fix the network path; the worker retries 5xx and the writes are replayable, so nothing was lost or doubled.
• Leases outliving dead workers. running rows with lease_expires_at in the past and no worker log lines. The scheduler's reaper frees them on its next tick; if it does not, schedule is down — it is leader-elected, so exactly one instance must hold the advisory lock. A reaped job is retried and its half-finished run is refused on every endpoint; the inventory is unchanged throughout (that is the design, and a test).
• One tenant's broken row. Enqueue failures for one tenant do not stop the others; find it in schedule's logs by tenant_id.

Fix. Start or scale work; nothing needs draining by hand. To stop collecting for one account meanwhile, disable it in Admin → Accounts; the scheduler respects enabled on its next tick.

A collector fails

Symptom. Red cells for one account, or for one module across accounts.

Look. The cell's tooltip, the account's run list (Admin → Accounts → the account → recent runs; GET /admin/collector-runs?accountId=), and work's logs by run_id.

Usually.

• AccessDenied on assume role. The reader role is missing, its trust policy does not name this deployment's account, or the ExternalId does not match. Press Verify on the account: it assumes the role and probes every module the tenant is entitled to, and its error is the real one. The external id is per tenant and shown once by panorama admin add-management-account; the stack parameter must carry that value.
• AccessDenied on a describe. The role assumes but the policy is not the shipped one. Redeploy deploy/cloudformation/reader-role.yaml; it is an exact allowlist and the collector needs all of it.
• Throttling / timeouts in one region. The run is incomplete: its snapshots were stored and nothing was tombstoned. That is correct — an enumeration that did not finish must not decide what is missing. It clears itself on the next run; if it persists, raise the tier's interval rather than MODULE_TIMEOUT (whose default is also its maximum).
• Every run of one module fails with no collector for this module. The scheduler enqueues a module this build does not carry — a worker image older than the scheduler's. Roll the workers.

Never delete rows to "clear" a failed run. A failed run tombstones nothing and the next successful one reconciles; cleanup expires stale running rows after CLEANUP_STALE_RUN_EXPIRY.

Migrations fail

Symptom. panorama migrate up exits non-zero; serve refuses readiness with schema is behind this binary: database is at N, this binary needs at least M — the migration has not run — or this binary is too old for the schema, when a breaking migration has outrun it (see "What a migration does to the running pods" below).

Look.

panorama migrate version        # what the database is at, and whether it is dirty

Usually.

• Dirty version. A migration failed half-way. golang-migrate records the target version with dirty = true and refuses to continue. Read the failed migration's SQL against the database, finish or undo it by hand, then make migrate-force VERSION_TO_FORCE=<the last good version> and run up again. Migrations are immutable once merged; the fix is never to edit one.
• An index build or drop failed. An index on a table that already holds rows is built and dropped CONCURRENTLY, alone in its migration (migrations/README.md rule 6). A drop that fails is simply run again — it says IF EXISTS and finishes the job. A concurrent build that fails — a duplicate under UNIQUE, a cancelled statement — leaves the index behind marked invalid, and the version dirty. Find it with SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid, DROP INDEX CONCURRENTLY it, force the version back to the one before and run up again. The retry fails on the name until the invalid index is gone, which is deliberate: an invalid index is written on every insert and read by nothing.
• 0179 refused (refusing to migrate past 0178: migration 0179 could not make a provider's subject one account). Single sign-on's first version bound one provider subject to a second account when a person's email changed, and 0179's unique index cannot be built over that. The migrator has already dropped the invalid index and set the version back to 0178, and a later run that finds 0179 dirty does the same first. panorama admin sso-duplicates lists each subject with its accounts; keep the one that should hold it, run panorama admin sso-release --tenant-slug <slug> --email <email> for each other (audited; that account binds again at its next sign-in), and run up again. 0180, right after it, is breaking: it takes binaries built before it out of service, because they create a first sign-in's account and bind it in two steps.
• Roles missing. Migration 0002 refuses to run until panorama_app and panorama_control exist: run deploy/bootstrap-roles.sql as the owner first (compose does this in its bootstrap service).
• Wrong role. MIGRATE_DATABASE_URL must be the schema owner. The application role has no CREATE on schema public by design; it cannot create schema_migrations, let alone a table.
• A partition missing. An insert into resource_snapshots fails with SQLSTATE 23514 around the turn of a month: the scheduler's maintenance pass did not run. Start schedule; it creates the current month, last month when an observation could still land there, and two months ahead on every tick that needs it.

A new serve will not report ready until the schema has reached its db.TargetVersion, so a rolling deploy runs migrate to completion before the new serve takes traffic; the previous release's pods keep serving through that when the migration is additive.

Backing up and restoring the database

One PostgreSQL database holds everything the platform knows: inventory, the permanent status history, cost, users, sessions, licences, the job queue and provisioning requests. The roles keep no state of their own, and Terraform state lives in each customer's bucket (D3), never here. Two things are not in a dump: the database roles panorama_app and panorama_control, which are cluster-level and come from deploy/bootstrap-roles.sql, and the secrets the deployment passes in (the database URLs and WORKER_TOKEN).

The policy is the operator's (p7_62). Choose these and write them down:

• How. A logical dump (pg_dump -Fc as the schema owner), or the provider's snapshots and point-in-time recovery for a managed database. Either restores the whole database, and neither restores the roles.
• How often. The interval is the recovery point: everything after the last backup is lost. A daily dump is one day.
• How long. Keep enough backups to reach back past a mistake noticed late. Seven daily and four weekly is a reasonable start.
• Encryption. A dump holds password and token hashes, the inventory of every connected account and each customer's spend. Encrypt it at rest, move it only over TLS or SSH, and keep it off the host it backs up.
• Who. The operator runs the backups and tests them; the platform cannot see them. A backup never restored is a hope, so restore one into a scratch database every quarter and run the checks below.

Restoring. The order matters: roles before data, data before the roles start, and the guard before anything runs.

1. Stop serve, work, schedule, notify and cleanup.

2. Start a fresh, empty database, then create the roles with deploy/bootstrap-roles.sql as the owner. Compose does this in its bootstrap service. Two sets of names must match the dump:

   • The owner and the database. A dump grants to its owner by name, so it restores only under the same owner and database names, with --no-owner or without. An install from before release 174 kept its own (docs/panorama-rename.md); one installed since has panorama_owner and panorama.
   • The service roles. A dump taken at schema 174 or later grants to panorama_app and panorama_control, which bootstrap does not create on a fresh cluster unless told to. Give it -v role_names=panorama (compose: BOOTSTRAP_ROLE_NAMES=panorama). Without it, pg_restore stops at the first grant with role "panorama_app" does not exist.

3. pg_restore --exit-on-error -d <database> <dump> as the schema owner.

4. panorama migrate up. The dump is at the backup's schema version, and a newer image migrates it forward. An image older than the backup's schema refuses with this binary is too old for the schema, so restore with the same image or a newer one.

5. panorama admin after-restore --backup '<the file, snapshot or time>', before work and schedule start. A provisioning request the backup holds as queued, running or cancelling may have run since the backup, and the backup cannot say. Each one fails with the class restored and a message naming the backup, and the command prints it with its state location. Nothing of it runs again until someone retries it. Check each one against its account first:

   • a create that ran after the backup has resources and state;
   • one destroyed since, by a request the restore forgot, has neither, and a retry would create them again.

   The platform cannot notice a restore, since a snapshot looks exactly like the live database, so this step is the operator's. Skipped, the scheduler's first tick requeues the running work and an agent runs it again.

6. Start the roles, and wait for /readyz.

7. Check the restore:

   • panorama migrate version is the target, not dirty;
   • an administrator can log in;
   • panorama admin show-license --tenant-slug <slug> shows the licence;
   • the timeline reaches the backup's time;
   • the provisioning requests after-restore listed show Failed (restored).

What a restore loses, and what comes back. Everything written after the backup is gone:

• History. Collection records one transition from the backup's state to the present on its next run; the transitions in between are lost.
• Spend. The next ingest pulls the missing days again.
• People. Users created and passwords changed after the backup are gone, and so are sessions issued after it: everyone signs in again.

Two things come back that should not:

• Revoked credentials. An ingest, worker or agent token revoked after the backup, or a user disabled since, is live again. Revoke or disable it again before the roles start.
• Notifications. A notification delivered after the backup is undelivered in it, and is sent again.

Measured (2026-09-12, p7_62). The drill restored a 0.5 MB dump, the demo estate, into an empty cluster in 2 seconds. serve answered /readyz 4 seconds after the empty cluster started, and every check above had passed at 5 seconds. A real estate takes longer in proportion to its size, so time a restore of your own backup. That time is your recovery time objective (RTO); your backup interval is your recovery point objective (RPO).

Rotating an ingest token

Per-tenant collector credentials live in control.ingest_tokens as hashes; the plaintext is shown once at creation.

panorama admin create-ingest-token --tenant-slug acme --name 'prod collector 2026-09'

Deploy the new value to the collector, confirm new runs arrive (the account's run list, or work's logs), then revoke the old one through the command, never with an UPDATE by name:

panorama admin revoke-ingest-token --tenant-slug acme --name 'prod collector 2026-08'

Token names are not unique — another tenant may well have a "prod collector" too — and the command scopes the revocation to the tenant named. If that tenant holds more than one live token of the name it refuses, and --all says every one of them is meant. Revocation is immediate: the next request with the old token is a 401.

The fleet credential rotates the same way:

panorama admin create-worker-token --name 'eu-west fleet 2026-09'
panorama admin revoke-worker-token --name 'eu-west fleet 2026-08'

Revoking it stops every worker at their next request without touching per-tenant collectors.

INGEST_TOKEN, the single-tenant shortcut, is a configured value rather than a row: change it on serve and work together and restart both.

Watching the roles

Every role is its own process with its own registry (p7_16): serve answers /metrics on its API listener; work, schedule, notify and cleanup each answer /metrics and /healthz on METRICS_LISTEN. The listener is fail-closed — a taken port stops the role at startup, and a listener that dies later stops the role rather than leaving it working unobserved — and it carries no authentication: bind it where only the scraper reaches it. deploy/monitoring/ holds a scrape configuration for the compose ports, the five alerts and a dashboard, and the drill that proves the alerts fire.

serve's /metrics is on the listener the public reaches, so the edge must refuse it: deploy/proxy/nginx.conf.example answers it 404 (location = /metrics), and an Ingress needs the same — a rule sending /metrics nowhere, or a server-snippet with that location. Scrape serve directly, on its pod or container address, never through the edge (p7_48).

Liveness is not progress. The probe on /healthz says the process runs; panorama_worker_last_job_completed_timestamp_seconds, panorama_scheduler_last_tick_timestamp_seconds (meaningful while panorama_scheduler_leader == 1) and panorama_notify_last_cycle_timestamp_seconds say its work moves, and PanoramaRoleStalled alerts on the gap. A standby scheduler reports panorama_scheduler_leader 0 and publishes no queue numbers; the leader does, every tick.

Releasing to Kubernetes

deploy/k8s/deploy.sh with your overlay (a copy of deploy/overlays/example, its image pinned by digest; the base names no usable tag) and the namespace is the release, and "deployed" at its end means every role runs the new image and the current configuration (p7_17). It renders the kustomization once and fingerprints the configuration: every ConfigMap it rendered, and the resourceVersion of every Secret a pod template reads — the values themselves never pass through the script. That digest replaces the placeholder annotation (panorama.dev/config-hash) in each Deployment's pod template, so a release that only changes a ConfigMap key, or only rotates the Secret, rolls the pods that read it, and a release that changes nothing rolls nothing. Then the two phases: the migrate Job alone, waited for; then the roles, and a rollout wait on every Deployment the apply returned (ROLLOUT_TIMEOUT, default 6m, above the manifests' progressDeadlineSeconds of 300 so the Deployment's own verdict arrives first).

When a role does not roll out the script still waits for the others, then prints the pods, the failed Deployments' events and their last log lines, names the roles, and exits 1. The boundary at that point: the migration is applied and stays. A role that failed still serves its previous ReplicaSet where one exists — a rolling update keeps it until the new pods are Ready — so on an upgrade this is a stuck release, not an outage; on a first install there is no previous ReplicaSet and the role is down. Fix the cause and run the script again (the migration is a no-op the second time), or kubectl rollout undo each failed Deployment — which brings the previous binaries back only when every migration since them is additive; the next section says how to tell.

One role's rollback also brings back a known gap: a cleanup older than f563864 (p7_49, 2026-09-11) drops expired cost_data months without holding cost_data's lock or deleting the derived rows again first, so a rollup rebuild that commits between its tenant pass and its drop leaves a rollup whose facts are gone, until the next night's pass deletes it by date. Rolling the other roles back while cleanup stays on the newer release avoids it; the newer cleanup excludes writers of either release.

What a migration does to the running pods (p7_42)

The migration runs first and the roles roll after it, so for the length of the rollout the pods still serving are the previous release's, against the new schema. Whether that is fine is the migration's to say, on its first line, and every migration from 0154 on says it (the migration tests refuse one that does not):

• -- compatibility: additive — a new table, an index, a column that is nullable or has a default. Nothing an older binary reads or writes changes. The previous release's serve pods stay ready while the migration commits and the new pods roll in, so the API does not go down, and kubectl rollout undo brings them back: they become ready against the newer schema. The tests hold additive to a short list of statements an older binary cannot notice and refuse the rest by name — any data change, UNIQUE on an existing table, a function call, a DO block, a replaced definition, a drop or a revoke (migrations/README.md rule 4 has the list).
• -- compatibility: breaking; oldest binary N — the migration records, in schema_compat, that binaries built before N cannot serve it. Every serve pod older than N fails readiness the moment the migration commits, so the Service has no endpoints until the new pods are ready: a breaking release is an outage the length of one pod start. And kubectl rollout undo to a binary older than N stalls — its pods never become ready, and the rollout leaves the current ones serving. To roll back across a breaking migration, roll the migration back first (make migrate-down removes its schema_compat row with it), then undo.

Readiness is the rule: a pod built for target T is ready when the schema's version is at least T and no applied migration's oldest supported binary is above T — db.Compatible, read by /readyz. Both refusals are real: a pod below the floor may read a column that is gone, and a pod ahead of the schema needs tables the migration has not made. /readyz says which.

To keep a change from being breaking, split it: expand (add the new shape, additive), release, then contract (drop the old shape) in a later release whose oldest binary is the expand's release — the pods it outruns are the ones that no longer exist. The release that introduces this rule, 0154, is itself not seamless: the pods it replaces demand an exact schema version and leave the Service when it commits, once.

Rotating the Secret is a release: update panorama-secrets, run the script, and every role rolls onto the new credentials in one pass. The roles do not re-read a Secret while they run, so a Secret updated without a release changes nothing until something else rolls the pods.

KubernetesConfigRotationDrill (owner-run)

On a disposable cluster (kind), with a Postgres the cluster reaches, the roles from deploy/bootstrap-roles.sql created, and the Secret in place:

1. Run the release and record the digest it prints and the pod names.
2. Run it again unchanged: the digest and the pod names are the same.
3. Set LOG_LEVEL to DEBUG in the ConfigMap and run it: every role's pods are replaced and the digest differs.
4. Rotate the Secret (apply it again with a new WORKER_TOKEN) and run it: every role's pods are replaced.
5. Break one role in an overlay (an argument notify does not accept) and run it: the script exits 1, names that Deployment, prints its events and logs, and the other roles rolled. kubectl rollout undo on it restores the previous pods.

Record the digests, the pod lists and the script's output in deploy/k8s/evidence-<date>.txt.

The panorama rename

Release 174 (p7_69) gives everything that runs the product's name: the binary, the images, the settings, the service roles, the metrics and alerts, the session cookie, the token prefixes, and the compose and Kubernetes objects. Migration 0174 is breaking (oldest binary 174): it renames the two service roles, so the image, the connection strings and the settings change in one step. Everyone signs in once more, and tokens issued before it are issued again. docs/panorama-rename.md is the upgrade: what each name was, what it costs, the steps for compose and Kubernetes, and rolling back.

The Admin area

Since p7_88 an administrator's portal has an Admin section of its own, which a member never sees: its pages refuse a member by name, and every read behind them is administrator-only in the contract. It holds the organisation's pages — Users, Teams & access, Single sign-on (when the licence carries it), Notification channels and Licence — and two of its own:

• Overview, one read (GET /admin/overview): the licence and its accounts, the users (active, deactivated, administrators, break-glass), the AWS accounts and their collection as the collector matrix rolls it up, single sign-on's state, and the latest audit entries.
• Audit log (GET /admin/audit): the tenant's own audit_log rows, newest first, read under row-level security and filtered by who acted, an action prefix, the subject and the days, a page at a time by cursor. Before and after are shown as the operation recorded them.

The old /settings/users, /settings/sso, /settings/org and /settings/license addresses lead an administrator to the new pages. A member's /settings/org leads to their own teams, and their licence page stays where it was. The modules' own setup (AWS accounts and collectors, Provisioning's templates and agents, Spend's attribution and budgets) stays in the modules.

Your own settings

Since p7_89 Settings is each person's own, an administrator's too, and every route behind it acts on the session's account and nothing a body or a path could name:

• Profile: a display name, shown where the portal names people — the users list, an alert's acknowledgement, the audit log and provisioning — with the email beside it. The email is the account's, and an administrator's to change.
• Password, as before (p7_20).
• Sessions (GET/DELETE /auth/sessions): where the person is signed in — when each session began and was last used, the address and browser it began from, the current one marked — and signing out one or all the others. A session is named by an id derived from its token's hash, which ends that session and looks nothing else up. The address and browser are recorded from p7_89 on; an older session shows neither.
• Inbox and Notifications: the person's own notifications, in the portal only, with a count in the top bar, and which kinds reach them — status alerts and their own provisioning requests unless they turn them off, their teams' budgets and cost anomalies when they turn them on. Each kind reaches a person only for what they may see: an administrator hears about every account and team, a member about their teams and their teams' accounts, and an account no team owns reaches the administrators alone. The inbox is filled by the notifier in the transaction that marks a batch, so while a channel's sends fail the batch is retried and its entries wait with it. Cleanup keeps an entry ninety days, read or not (the notifications category of panorama_cleanup_deleted_total).
• Preferences: a time zone, a date and a number format and the theme, applied across the portal as soon as they are saved; a choice left at the browser's own is not stored.
• Your teams: the person's memberships and their role in each, read-only.

/settings/notifications was the channels page and is now each person's choices; the organisation's channels are in Admin, under Notification channels.

Onboarding and removing people

Every user operation is a supported one (p7_20): nothing below needs SQL, and each leaves an audit_log row (org.user.*, auth.password.*) naming the actor, the person and what changed — never a password or a token.

Which tenant a login is for. The request host first, when its first label is a tenant's slug (acme.panorama.example is acme's); then tenantSlug in the body; then, in a deployment with exactly one tenant, that one. The deployment's own hostname is a subdomain too — platform.<domain> — and its first label is nobody's slug, so it falls through to the body and the sole tenant (p7_90). Set DEPLOYMENT_HOST to that hostname: then its label is never read as a tenant, whatever tenants exist, and bootstrap and seed-demo refuse a tenant named after it (review round 2). The portal's form sends only the email and the password, which is why a single-tenant install signs in from any host and a multi-tenant one needs either a tenant host or the slug.

In the portal, an administrator's Users page adds a person with a platform role and shows their temporary password once, moves the role, issues a password reset (a token shown once, good for 24 hours and one use, redeemed at /reset-password), ends every session they hold, and deactivates or reactivates them. Everyone changes their own password under Password; doing so ends their other sessions and keeps the one they typed into. The same operations from the command line, for the operator holding the database credentials:

panorama admin create-user     --tenant-slug acme --email eng@acme.test --role engineer
panorama admin reset-password  --tenant-slug acme --email eng@acme.test
panorama admin set-role        --tenant-slug acme --email eng@acme.test --role viewer
panorama admin disable-user    --tenant-slug acme --email eng@acme.test
panorama admin enable-user     --tenant-slug acme --email eng@acme.test
panorama admin revoke-sessions --tenant-slug acme --email eng@acme.test

--raw on create-user and reset-password prints the credential alone, for a pipe. What each change does to sessions is the rule, not a side effect: a role change, a deactivation, a password change and a redeemed reset all end the sessions that predate them, so no open tab keeps a privilege or a credential that has moved. Deactivation refuses the login with the same answer as a wrong password.

The last administrator. The tenant's last enabled administrator cannot be demoted or deactivated (409 last_admin); two administrators changing each other at once are serialised on the row and one is refused. If a tenant nonetheless has no administrator who can sign in — every one deactivated by the operator, or the only one gone — the break-glass is panorama admin bootstrap with a new email, which adds an administrator to the existing tenant, or panorama admin enable-user on one of the deactivated ones. Neither overwrites an existing account.

Single sign-on

Teams from the directory (p6_6). A team bound to a group of the provider (Admin area → Teams → Directory, or PUT /org/teams/{teamId}/directory-group) takes its membership from the ID token's groups claim at each sign-in: a person carrying the group joins, one no longer carrying it leaves; the platform role and every hand-kept team stay as they are, and each change is an audit entry with no actor. The claim is groups unless the registration names another (Okta and Keycloak let a customer name it; Entra ID and Google emit groups). The licence must carry the scim capability — Enterprise does — or bindings are refused and sign-ins sync nothing. Unbinding keeps the members and makes the team hand-kept again.

A tenant whose licence carries the sso capability signs in through its own OpenID Connect provider: Entra ID, Okta, Google or IAM Identity Center (p5_5). An administrator registers it under Admin → Single sign-on: the issuer URL; a client id and secret created at the provider for a web application using the authorization-code flow; the redirect URI the page shows, https://<the portal's host>/auth/sso/callback, registered at the provider; the role a first sign-in gets; and the email domains a first sign-in may come from. Entra ID does not say whether an address is verified, so it needs its domains listed. The secret is kept on the server and never shown again, and saving or removing the registration asks for the password again.

A first sign-in binds the account that has the address, or creates one with the chosen role. From then on the account is the provider's subject, and a different subject with the same address is refused. The session is a password session in every respect: the per-user cap, the twelve-hour limit, revocation. Two first sign-ins of one person at once, or a provider that later changes the person's email, still make one account: the subject is looked up first, and it is unique within the tenant and the issuer.

Changing the provider. An account is its provider's subject, and a subject belongs to one issuer, so a new issuer is refused (409) while any account is bound to another. The page says how many are bound and offers to release them — an administrator, with the password asked again, audited as auth.identity_provider.release. Released, each account binds again, by its email, at its first sign-in through the provider then registered: the same account, under enforced mode too, and break-glass passwords work throughout. Removing the registration releases nothing: registering another issuer afterwards is refused the same way. A sign-in that was still at the old provider when the new issuer was saved is refused when it comes back, and makes or binds no account; one that finished first is counted, and the new issuer stays refused until it is released too.

A sign-in finishes only in the browser that began it: the start sets a short-lived cookie scoped to /auth/sso, and a callback without it — the same link opened in another browser — is refused and signs nobody in. Every request the server makes for a sign-in (the discovery document, the token exchange, the provider's keys) is held to the destination policy the notification channels use, below: https on port 443, no credentials in the URL, every address the host resolves to public, checked again on the address actually dialled, and no redirect followed. An issuer that fails it cannot be saved.

An account that signs in only through the provider has no password anyone knows, so it confirms a destructive action at the provider instead: the confirmation dialog offers Confirm with your identity provider, the provider asks for a fresh sign-in, and the browser comes back to the page with the session confirmed for ten minutes, as a password confirms it. The fresh sign-in must be the same account's and at most five minutes old.

Saving or removing the registration leaves an audit_log row (auth.identity_provider.create, .update or .delete) naming the actor, with the registration before and after; of the secret it keeps only whether one is set and whether the change replaced it.

Enforced stops passwords for everyone but break-glass administrators, who are marked on the Users page. Mark one before enforcing. When the provider is down and nobody was marked, from the host:

panorama admin break-glass    --tenant-slug acme --email admin@acme.test
panorama admin reset-password --tenant-slug acme --email admin@acme.test

The administrator signs in with the reset password, then clears Enforced or corrects the registration; break-glass --off takes the mark away again. Each change leaves an audit_log row naming the actor.

Notification channels

A tenant's destinations — a Slack incoming webhook or a webhook of its own — are managed on the portal's Notifications page or through /org/notification-channels (p7_21); nothing is inserted by hand any more. An administrator creates a channel with a name, a kind and a target, scopes it to a team or leaves it as the tenant default, tests it, rotates its target, disables it or removes it. Every change is audited (notify.channel.*) with the redacted view of the target and never the target itself.

The destination policy. A target is a URL this process will POST to from inside the deployment's network, so it is checked before it is stored and again on the address actually dialled: https only, no credentials in the URL, a host that resolves, and every address it resolves to public — loopback, link-local (the cloud metadata endpoint), RFC 1918, unique-local, carrier-grade NAT and the reserved ranges are refused, as is a host with one public and one private answer (a rebinding setup). Redirects are never followed. A refusal is 400 invalid_request with the reason and never the URL. There is deliberately no opt-out for an internal receiver: a destination inside the network is one an operator with the database credentials can still insert by hand, and that row is theirs to answer for.

What a read shows. The target is never read back: the API and the page show its host, a hint (the last four characters of its path) and a fingerprint (sha256), so two channels can be told apart and a rotation shown to have changed something. Rotate by giving a new target; the last test result is cleared with it.

Testing a channel. "Test" sends one fixed message through the same sender the outbox uses and reports delivered, refused (the destination answered with an error or a redirect; the status is shown), unreachable (no answer: dns, tls, timeout, network) or blocked (the policy refused it before a byte was sent: private_address, invalid_target). The destination's own words are never repeated. The result stays on the channel until the next test or a rotation.

Connecting a Google Cloud project

A Google Cloud project is an account of its own cloud (p7_82). Estate reads its BigQuery datasets, service accounts, Firebase apps and Cloud Functions as the project's reader service account. The platform holds no key file: its own AWS credentials, IAM Roles Anywhere on the platform host, are exchanged at Google's Security Token Service through workload identity federation, and the federated token impersonates the reader. For an operator who knows AWS: a project is an account, a service account is a role, a custom role is a customer-managed policy, and the binding that lets the platform impersonate the reader is the trust policy.

Once per deployment, in a project the operator holds: a workload identity pool with an AWS provider that accepts only the platform's AWS role, and GCP_WORKLOAD_IDENTITY_PROVIDER set on work and serve to the provider's name, //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>.

Per project, in the customer's project:

gcloud iam service-accounts create panorama-reader --project=PROJECT_ID
gcloud iam roles create panoramaMetadataReader --project=PROJECT_ID \
  --file=observability/deploy/gcp/reader-role.yaml
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=serviceAccount:panorama-reader@PROJECT_ID.iam.gserviceaccount.com \
  --role=projects/PROJECT_ID/roles/panoramaMetadataReader --condition=None
gcloud iam service-accounts add-iam-policy-binding \
  panorama-reader@PROJECT_ID.iam.gserviceaccount.com --project=PROJECT_ID \
  --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/projects/POOL_PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL/attribute.aws_role/PLATFORM_ROLE"

Then register it, verify it, and enable it:

panorama admin add-gcp-project --tenant-slug acme --project-id PROJECT_ID \
  --name "Acme production" --reader panorama-reader@PROJECT_ID.iam.gserviceaccount.com

Verification (POST /admin/accounts/{accountId}/verify) signs in as the reader, reads the reader account and the project, then runs each Google collector's probe; each check names the permission it needs. Enabling (PUT /admin/accounts/{accountId}) is the same entitlement-checked step as an AWS account's, and a project counts toward the licence's account cap.

The same three steps from the command line, for an operator on the host or a deployment whose administrators have no portal session to hand, and one collection on a deployment that runs no schedule:

panorama admin verify-account --tenant-slug acme --account-id PROJECT_ID
panorama admin enable-account --tenant-slug acme --account-id PROJECT_ID
panorama admin collect-now    --tenant-slug acme --account-id PROJECT_ID

verify-account runs the API's verification and prints each check; it exits non-zero when one fails. It works for an AWS account too, without the cost report's checks, which are the API's alone. enable-account is the API's enable under the same account cap (--disable switches it off). collect-now queues one job for each inventory module of the account's cloud, and a worker collects them; a module with a job already waiting keeps that one.

What is read, and what is not: reader-role.yaml grants seven permissions and none that returns content — no table rows, no routine bodies, no function configuration or environment, no Firestore documents, no keys. Cloud Functions are found from their execution_count metrics, so a function that has not run in two weeks is not listed, and functions deployed as Cloud Run services are not collected. Every Google Cloud type is global in the resource key, its location a property.

When the public page is wrong

It is read from the same tables as everything else, cached for thirty seconds per tenant and rate-limited per client address. "Nothing is being monitored yet" means no live resources pass the base filter: a tombstoned resource, or one whose bucket has not been collected successfully for a day, is absent here as everywhere. Check the matrix first.

The page can never carry a resource name, id, account id, region, IP or ARN; TestThePublicPageLeaksNothing seeds recognisable values of each and asserts their absence. If a change to internal/api/handlers/public_status.go is proposed, that is a security review.

The front door (p7_92). A visitor with no session who opens the portal's root sees this page for the tenant the host shows — the tenant the host names by its first label, or, on a single-tenant deployment, the one tenant — with a link to sign in. On a single-tenant deployment exposed to the internet that means the bare host reveals the tenant's slug and the aggregate health this page shows to anyone, and nothing else: no name, no resource, and nothing about how many tenants there are (GET /public/tenant answers with the slug and the demo mark alone, or a 404). A deployment that wants the sign-in page as its door keeps the root behind its own front, or holds more than one tenant.

The demo tenant

panorama admin seed-demo creates a tenant flagged is_demo, which the scheduler excludes from collection and notification work. It populates inventory, organization, spend and simulated provisioning history without cloud calls. Its data ages like any other tenant's — retention applies, and the matrix will go no-data after three cadences because nothing collects — which is what make seed RESET=1 is for. A slug that names a real tenant is refused. There is no way to turn a demo tenant into a real one; create a real one with panorama admin bootstrap.

Re-running the seed upgrades an inventory-only demo without resetting its login. RESET=1 refreshes inventory and operational history; it preserves platform decisions and the original cost-history window. Seeded execution logs are simulated, the example agent is revoked and the webhook is disabled.

The licence

A tenant's modules come from a licence (p5_1): an ed25519-signed JSON file the owner issues and an admin installs. Nothing phones home. The binary carries the verifying public key (LICENSE_PUBLIC_KEY at build time — the Makefile, the Dockerfile and CI all pass it as the same ldflag); the signing key never enters the product, this repository, or any machine the product runs on.

What the customer sees. GET /entitlements carries one word, state, and the portal turns it into a banner on every page: expiring (thirty days out, warning, nothing locked), lapsed (past valid_until, destructive, every module locked, collection continues for fourteen days), paused (past that grace, collection stopped), none (no licence at all), unrestricted (the development exception). A valid licence has no banner. Nothing is ever deleted because of a lapse; a new licence reopens the modules the moment it is installed and the scheduler enqueues the tenant again on its next tick.

Two things the release exercise taught (p5_1, 2026-09-17). A renewal is a file issued after the one installed: the anti-rollback rule compares issued_at, so a renewal issued before a short-lived file that came later is refused as the older one — issue the renewal fresh. And a file in the licences mount is read by the container as another user: copy it at 644 (a 600 file is "permission denied" from inside).

Install, or replace.

panorama admin install-license acme.license            # slug comes from the file
panorama admin install-license acme.license --tenant-slug acme   # belt and braces
panorama admin show-license --tenant-slug acme

Install verifies the signature before it reads a byte of the payload, refuses an expired file, a file for another tenant, and a build with no public key, and is a no-op for a licence already installed. A newer licence (by issued_at) replaces the row; an older file, however valid, is refused with both licence ids and issue times, so re-installing a stale file cannot roll a renewal back. A licence installed before schema 137 recorded issue times is handled the same way with what the row still knows: re-installing its own file records the issue time ("recorded its issue time"), a different file installs only if it was issued after the row was written, and an older one is refused with the way out spelled out. After upgrading such a deployment, re-install the current file once. show-license says which license_id is live; a hand-typed grant with no expiry shows as "no expiry". The server refuses to start with ENTITLEMENTS_UNRESTRICTED=true once any licence is installed: the flag and the licence describe different deployments, so a stack that had the flag set (compose defaults it on) must drop it before the first install.

Which key does this build carry? panorama version prints it:

docker run --rm ghcr.io/dawidper/ccc/panorama@sha256:<the digest you deploy> version
# panorama v1.4.0 (c0f91d6)
# licence key: 57023e96…

"licence key: none" is a build with an empty LICENSE_PUBLIC_KEY: it accepts no licence and runs only with ENTITLEMENTS_UNRESTRICTED. CI refuses to build the image unless the repository variable LICENSE_PUBLIC_KEY is exactly 64 lowercase hex characters, reads the line above off the image it built on every run, and pushes only the image that passed — main included. The Dockerfile itself refuses to finish a build whose binary did not take the key it was given, so a moved linker target fails every builder, not only CI. Setting the variable is gh variable set LICENSE_PUBLIC_KEY --body "$(cat license-signing.pub)".

Issue — the owner's side, on the owner's machine, never in the image:

go build -o cloudpanorama-license ./cmd/cloudpanorama-license   # from observability/
cloudpanorama-license keygen --out ~/keys                        # once; refuses a git tree
cloudpanorama-license issue --key ~/keys/license-signing.key \
    --tenant acme --licensee 'Acme GmbH' --tier professional \
    --max-accounts 50 --valid-until 2027-09-01 --out acme.license
cloudpanorama-license issue --key ~/keys/license-signing.key \
    --tenant acme --licensee 'Acme GmbH' --trial --out acme-trial.license
cloudpanorama-license inspect acme.license --pub ~/keys/license-signing.pub

Tiers are ids (essentials, professional, enterprise) with module and capability defaults (PLATFORM.md §7); --modules, --capabilities and --max-accounts override them. A trial is an ordinary licence: every module, ten accounts, thirty days, licensee suffixed "(trial)". The private key lives in the owner's password manager as the hex seed keygen wrote; the .pub beside it is the value for LICENSE_PUBLIC_KEY.

Rotate the key when it may have leaked, or on a schedule if you prefer one. New keypair; the new public key goes into the next release; every live licence is re-issued with the new key (each has a license_id and an issued_at, so the re-issue is auditable against the old list) and re-installed by each customer as part of upgrading. Until a customer installs the re-issued file, the old row stays valid — the new build only refuses new installs signed by the old key, it does not re-verify what is already in the database.

A customer's licence lapsed and they renewed — install the new file; that is the whole procedure. If the tenant sat paused for longer than fourteen days, the first collection after the install is a normal reconciliation: resources that disappeared meanwhile are tombstoned with that run's run_id in the finalize log, as any run would.

Connecting a Cost & Usage Report

A report is read under a role of its own (p7_22): the inventory reader role never reads object content, so each account that shares a report deploys deploy/cloudformation/cost-report-role.yaml, scoped to that report's bucket and prefix, and the platform assumes that role — with the tenant's external id — for the report and nothing else. The steps, per account:

1. In AWS Billing, define the report: CSV, GZIP, hourly or daily, with resource ids, delivered to a bucket and a report path prefix. AWS delivers the first files up to 24 hours later.
2. Deploy the template with the platform account, the external id, the bucket and the prefix followed by the report name (reports/ccc for a report named ccc under reports), and take its RoleArn output.
3. In the portal's cost settings, add the report: account, bucket, its region, prefix, report name and the role ARN. Save.
4. Verify the account (POST /admin/accounts/{accountId}/verify, or the portal's verify button). Beside the inventory checks, the report role is assumed and its two reads are tried where a pull would make them: s3:ListBucket on the report prefix, then s3:GetObject on the newest manifest. A failed check names the action and the ARN to grant on; a report with no role entered yet is reported as the missing step.
5. The pull runs on the slow tier. cost_ingestion_state.last_error for source cur carries the same named reason when it fails, and the settings page shows the last period ingested once it succeeds.

What it will not do: read a bucket other than the one configured, list a prefix the role's condition does not admit, or fall back to the inventory role — a report without a role of its own is refused before any request is made. A bucket encrypted with a customer-managed KMS key needs kms:Decrypt on that key added to the role; the template says where.

Rebuilding Cost & Usage Report periods

The CUR job re-reads a billing period only when AWS publishes a new assembly of it, so a correction to the parser — the amortization mapping of 2026-09-06 (p7_9) is the first — leaves every period already ingested as it was. panorama admin rebuild-cur re-ingests them:

panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08 --dry-run   # the before
panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08             # mark and enqueue
panorama admin rebuild-cur --tenant-slug acme --from 2026-01 --to 2026-08 --dry-run   # the after, once a worker has run

--dry-run prints each stored period — account, month, assembly, whether it is current, its line-item and row counts, and the stored unblended and amortized totals per currency — and changes nothing. Without it the command forgets the current assembly of every period in the range, enqueues the cost.cur job for each account so the same reports are streamed again through the current parser, and requests the attribution rebuild. The re-ingest replaces each period in one transaction, the path a republished month takes: running the command twice marks nothing the second time, and a re-ingest never adds to a period. --account narrows it to one account. The reports must still be in the bucket: a period whose files AWS has removed fails its run and is recorded in cost_ingestion_state like any other missing report.

The rollups follow the facts, durably: a period's assembly records when its rollups were built (rollups_built_at, migrations 0138–0139), the record is written only after the build succeeds, and every run rebuilds the periods still owed before it reads anything new. After upgrading through 0139 every current period is owed once — the first cost.cur run of each account rebuilds its rollups and records it; --dry-run shows owed until then.

What the correction changed, so the before and after read right: a Savings Plan negation no longer subtracts the on-demand price from the effective cost a second time; an unused Savings Plan commitment and an unused reservation stay in the amortized view; an ordinary fee such as Support is money in both views; only an upfront reservation fee and a negation amortize to zero on their own rows. Expect amortized totals to rise for any month with Savings Plans, reservations or a Support plan.

How the tables grow

Measured, not estimated (p7_49; the numbers and how to take them again are in ../platform/tasks/p7_49.md). A resource here is an EC2 instance with the details its collector emits and six tags.

resource_status_transitions is not partitioned, deliberately. It is never dropped by age, so a partition would buy no retention, and every reader is bounded by an index range rather than by the table — the timeline's page, the public page's seven days, the notifier's outbox and one resource's history. Measured as panorama_app at 300,000 and 3,000,000 rows: each plan was the same index range scan at both sizes, the timeline and the outbox at 0.1 ms, and the public page's cost followed the rows inside its seven days, not the table. Revisit at 10 million rows in any database, or when one of those plans leaves its index: partitioning then means a primary key of (id, occurred_at) and a rewrite of the table, a few GiB at that size.

SELECT relname, reltuples::bigint AS rows, pg_size_pretty(pg_total_relation_size(oid))
  FROM pg_class
 WHERE relname IN ('resources', 'resource_snapshots', 'resource_status_transitions');

(reltuples of a partitioned parent is -1; sum its partitions, or count.)

The cost levers

What the running bill is made of and which settings move it, so nobody has to re-derive the 2026-09-09 review's model (its §9: a pilot at roughly $205–400 a month, a small SaaS at $805–1,270, mid scale at $5,150–6,860, where the two lines that grow are Cost Explorer's request fees and snapshot storage). All of these are configuration since p7_50.

Both levers the review found blocked by code are now configuration or built in: snapshots are change-only (no setting — the owner's decision) and cost history has its retention (p7_49, 2026-09-11).

Limits, headers and scanners

Every unauthenticated route has a per-address budget and every body-taking route a size limit, both derived from the contract by middleware.Policies (security: [] → 120 requests a minute per address; requestBody → 1 MiB unless the operation says x-max-body-bytes; x-rate-limit on any route) and applied by the guarded router before authentication. A client past its budget gets 429 rate_limited with Retry-After; a body past its limit 413. The address is the one CLIENT_ADDRESS_SOURCE declares — behind a proxy that is the trusted proxy's header, so set it, or every client is the proxy. Every response carries the security headers (middleware.SecurityHeaders; HSTS only when COOKIE_SECURE).

The session's CSRF boundary is the API's, not the cookie's (p7_18). An unsafe request (anything but GET, HEAD, OPTIONS) on a session route — and on /auth/login and /auth/logout — is refused with 403 origin_not_trusted unless the browser's own context vouches for it: Sec-Fetch-Site: same-origin, or an Origin that is this server's host or one listed in CORS_ORIGINS. A body on such a route must be labelled application/json (415 unsupported_media_type). Two consequences for a deployment: a portal served from a second origin works exactly when that origin is in CORS_ORIGINS (the same list CORS answers for), and a proxy that rewrites the Host header must either preserve it (proxy_set_header Host $host in nginx) or list the public origin in CORS_ORIGINS, or browsers without Fetch Metadata will be refused. Collectors, workers and agents are not held to any of this: their credential is a header, and the check is by credential kind. ../web/e2e/README.md runs the real-browser proof against a same-site sibling.

Time and concurrency budgets

A byte limit caps a body's size, not how long a client may take to send it (p7_19). Every route now has a time budget too, from the contract and applied by the guarded router before the first byte of the body is read: a route with a body waits 30 s for it unless the operation says x-body-timeout-seconds — 120 s for the snapshot batch (the worker gives itself a minute to send one, HTTP_TIMEOUT's ceiling), 60 s for agent logs and template sources — and a route without waits 10 s for whatever unread body a request carried. The response gets 60 s past that. The claim's long poll is the one route that holds a connection longer on purpose, and its handler extends its own deadline to the poll plus 30 s; PROVISIONING_CLAIM_POLL_TIMEOUT is bounded at 4 minutes so the server's write ceiling (5 minutes, cli.serve) clears it. The ceilings — 5 minutes to read, 5 to write — are not the budgets: they reset a connection's deadlines between requests and bound the routes the router does not guard (the portal's files, the probes).

Concurrency is bounded where a body is large enough to matter: x-max-inflight caps how many bodies a route reads at once — two snapshot batches (a batch buffer is up to 192 MiB, and the k8s manifest gives the process 512 MiB), sixteen agent log batches — and the next caller gets 429 rate_limited with Retry-After: 5 rather than a place in memory the process does not have. Collectors and agents already retry a 429 with backoff. panorama_http_requests_inflight counts every request being served, bodies still being read included, so a stall is visible while it stalls; a gauge sitting at a route's cap is the signal.

A stalled body holds no database connection: every write path decodes the body before it opens a transaction, and the budget ends the read before anything else happens. What is held is one goroutine, one socket and the bytes buffered so far, for the budget and no longer.

The edge. Nothing in this repository fronts the API by default — compose binds it to loopback and the k8s manifests leave the Ingress to you — so the application's budgets are the boundary, not a convenience. A proxy adds TLS, a per-connection ceiling before the application sees the socket and a total request rate, and it can break the two long paths if its own timeouts sit below the application's: deploy/proxy/nginx.conf.example is a configuration sized against these numbers (client_body_timeout and client_max_body_size at or above the batch route's, proxy_read_timeout above the claim poll plus the response budget, proxy_request_buffering off on the batch route, Host preserved). The same numbers on an nginx Ingress are the nginx.ingress.kubernetes.io/proxy-read-timeout: "120", proxy-body-size: 192m and proxy-request-buffering: "off" annotations; an AWS ALB has an idle timeout (60 s by default) that must exceed the claim poll and has no body-read timeout at all, which is the argument for the application's.

Three scanners run in CI and can fail a build: govulncheck on both Go modules, npm audit on what the portal ships, Trivy on the image before it is pushed. What each fails on, and what to do when one does, is docs/security.md "The triage rule". The Go module pins its patch release (go 1.26.8 in go.mod, since the 2026-09-11 minor move, p7_59) because the standard library is what the scanner most often reports; bumping it is the routine answer.

Adding a collector

The M2 step-by-step, for extending the inventory to a new AWS service.

The step-by-step for the next module, written against the two that exist: internal/collectors/aws/ec2 (instances) and internal/collectors/k8s/eks (nodes and pods). Follow it in order; each step has a test that fails when it is skipped. Nothing in the frontend changes — a resource type the backend describes renders without a line of module-specific code, and noServiceSpecificCode.test.ts fails the build if one appears.

A module is one resource type in one service: aws.rds.instance, aws.s3.bucket. It is the identifier a run registers under, the unit the scheduler enqueues, and the column in the admin matrix. Pick it first; every step below names it.

0. Read these before writing anything

• MILESTONE_1_PLAN.md §3, the lifecycle — buckets, runs, generations, fences, tombstones. A collector that misunderstands incomplete either freezes reconciliation or deletes resources that are still running.
• internal/collectors/collector.go, the interface, and incomplete.go.
• internal/collectors/aws/ec2/, all four files. Copy its shape, not its code.

1. The collector: internal/collectors/<cloud>/<service>/collector.go

Implement collectors.Collector:

Rules the EC2 collector demonstrates:

• Never touch the database. Return snapshots; the worker posts them.
• AdditionalInfo is the deriver's input and the detail page's content. Put the raw provider state under a key the deriver reads (state, systemStatus, …) and the fields the descriptor will show. Keep it under the item size budget (ingest.MaxAdditionalInfoBytes); a large sub-object such as a security-group list is fine, a whole API response is not.
• Omit what the provider says is gone. EC2 keeps terminated instances visible for an hour; emitting one means "I observed it", so it is never tombstoned. Let reconciliation do deletion.
• Tags are a field, not a key in AdditionalInfo (Snapshot.Tags). Store exactly what the provider returned — no case folding.
• Take the SDK client through the aws.Config you are handed. It already carries the assumed role and the region; construct the service client from it and nothing else, so tests can substitute a fake.

Test it with a fake client (collector_test.go): a full page, a paginated list whose second page fails (must return Incomplete), an enrichment failure (must not), and the terminated/omitted case for your service's equivalent.

2. The status deriver: status.go

func init() {
	status.Register(system, service, resourceType, derive)
	rendermeta.Register(descriptor())
}

func derive(info map[string]any) (status.Status, string) { … }

One function from AdditionalInfo to (Status, raw state). Status is derived once, at ingest, and stored; nothing re-derives it in SQL or TypeScript. An unknown state is Degraded (something you did not anticipate deserves a look, not a green tick). Do not return Operational for "no information": the EC2 deriver treats missing status checks as Operational only when the collector marked them unavailable explicitly, so a missing key is a bug you see, not one you hide.

Registering happens in init() because the deriver must exist in every process that stores a snapshot. That is why internal/cli/schedule.go and work.go import the package — and why step 5 matters: a binary that schedules a module without importing its deriver stores every resource of that type as Degraded, loudly, on purpose.

Test the table: every provider state you know of, the unknown state, and the enrichment-unavailable case (status_test.go).

3. The render descriptor: rendermeta.go

The detail page and the list columns come from here, not from the frontend.

rendermeta.Descriptor{
	System: "aws", Service: "rds", ResourceType: "instance",
	Label: "RDS Instance", PluralLabel: "RDS Instances", Icon: "database",
	ListColumns: []string{"resourceName", "state", "additionalInfo.engine", "region"},
	Groups: []rendermeta.Group{{Title: "Overview", Fields: []rendermeta.Field{
		{Key: "state", Label: "State", Format: rendermeta.Badge, BadgeMap: …},
		{Key: "additionalInfo.engine", Label: "Engine", Format: rendermeta.Text},
	}}},
}

Keys are paths into the resource (resourceName, state, region, additionalInfo.<field>); formats are the shared vocabulary in internal/rendermeta (Text, Badge, DateTime, Boolean, Bytes, Table with Columns, JSON). PluralLabel is what the public page says ("3 instances became degraded"). tags is rendered by the registry for every type; do not describe it.

rendermeta's tests validate every registered descriptor at build time: a key that does not exist on the snapshot type, a badge map naming a state the deriver never returns, a format without its required options — each fails the suite, with the module named. GET /resource-types serves the registry with an ETag, and the portal renders whatever it says.

4. Register it once: internal/modules

Buckets are step 1's Buckets(); this step is about who asks for them, and since p6_1 the answer is one table. Add one entry to modules.All():

{New: func() collectors.Collector { return rds.New() }, Tier: queue.TierStandard, Actions: rds.Actions},

• Tier decides the cadence: critical (5 m), standard (1 h), slow (24 h). The scheduler's list, the admin matrix's cells, the worker's registry and Verify's probes all derive from this entry, so a module scheduled is one a worker carries and serve can verify — the three used to be listed separately and drifted.
• Actions is a package-level var beside your API interface: exactly the read calls Collect and Probe make, nothing content-bearing (no GetObject, no GetQueryResults, no user data). The reader-role test takes its allowlist from the table, so the template deploy/cloudformation/reader-role.yaml must grant the same calls in a statement of its own — that is the one thing still written twice, because the template is what a customer deploys and the test is what keeps it honest. The test also holds a denylist; if your call is on it, the call is wrong for this product, and the module cannot be built (Lambda's ListFunctions returns environment variables: p6_1 rule 4).

5. What the table checks at build time

internal/modules imports your package, which runs its init() in every process — schedule, work and serve alike — so the deriver exists wherever a snapshot is stored. TestEveryModuleHasADeriverAndADescriptor then walks every module's buckets and fails if a deriver or a descriptor is missing: the loud all-Degraded failure of step 2 is caught before a commit, not on an estate. (It caught the Kubernetes node and pod types on its first run: derivers, no descriptors.) There is no separate import to wire.

6. Tests that must exist before the module ships

Then run the whole thing against the compose stack with one real account enabled, watch the matrix cell go ok, open a resource, and read the detail page. If a field is missing there, it is missing from AdditionalInfo or from the descriptor — never from the frontend.

What writing this document said about the abstractions

The task asked for this to be treated as a finding, so here it is.

The interface holds. Collector's four methods and the incomplete distinction were enough for EKS — a second cloud, two resource types from one read, a presigned token instead of an SDK call — without a change to the interface, the worker, ingest or the frontend. That is the result Phase C was for.

Registration was spread across five places — the package's own init() (deriver and descriptor), scheduledModules, worker.NewRegistry, the verify list in serve.go, and the IAM allowlist with its mirror test — and a fifteen-module M2 would have forgotten one of the five at least once per module. Done, in p6_1 batch 1, before the third collector as this said: internal/modules is the one value carrying the collector, its tier and its IAM actions, and the scheduler's list, the worker's registry, the verify list and the policy test derive from it (step 4). What stays written twice is the CloudFormation template, on purpose.

The IAM policy is the step most likely to be wrong, because it is the one that cannot be tested without an AWS account. The mirror test keeps the document and the code in step; it cannot say whether AWS agrees. The first real run of a new module against a fresh account is the test, and the runbook says what its failure looks like.