Provisioning agent

Install, configure, verify, upgrade and revoke the agent that runs OpenTofu in your account — and what it can never do.

Audience: whoever installs the reference agent — agent/ in this repository — inside an AWS account or a Google Cloud project. This is the reference implementation's own manual: what it reads, what it writes, and how it is shipped. The account-level operations — issuing a token, verifying, upgrading, revoking — are in agent-operations.md and apply to any conforming agent, this one included.

Status: written on 2026-09-03, with the binary.

---

1. Get it

Two forms, both built from agent/:

The agent is two roles of one binary and one image (D18, platform/docs/agent-execution-boundary.md): panorama-agent broker holds the platform token, claims, heartbeats, streams logs and reports, writes the generated files, and runs no customer code; panorama-agent worker — another user, no token — checks the source out, runs OpenTofu and reads state, and keeps its process space to itself between attempts (it makes itself unreadable to its own user, kills everything else of its user before and after every attempt, and refuses to begin one unless it is alone). The two meet over a Unix socket in the shared work directory, one connection per attempt. The image carries two users, broker (uid 1000) and worker (uid 1001), in one group (panorama, gid 2000), git for git-sourced templates, and exposes /healthz and /readyz on :8091 from the broker (§5). panorama-agent version prints the build's version, which is what register reports.

panorama-agent unisolated runs both halves in one process. It is for development and the acceptance tests only, refuses to start unless PANORAMA_DEVELOPMENT_UNISOLATED=1 names it, and is never a supported deployment: in it, customer code runs in the process that holds the token.

1.1 Pull and verify the published image

A release is a version and the digest it names, kept as its release record (agent/deploy/releases/X.Y.Z.json, the pull field). This page installs 0.3.47, the version on main, which no release run has published yet (0.3.36, published 2026-10-25, is the last published release, agent/deploy/releases/0.3.36.json; 0.3.34 was the first); until 0.3.47's release record is kept here, replace <digest> with the one that record gives.

To install the last published release now, use 0.3.36: one index, sha256:3fefb868e49cab2d3ef97967378a88c598cb5903c55c6bcb45b7594c807e963b, whose linux/amd64 image is sha256:a98c411f22fba2cb40071883dd87d2ca206ad3c738ad6c1d5140e7aa6ab1f75f and whose linux/arm64 image is sha256:05c8a6f0ba15594b9f7e8b8035f5598d034a9291e8a08b5831b659303c244be9; set the image to ghcr.io/cloudpanorama/panorama/panorama-agent:0.3.36@ and that index digest, and verify as agent/v0.3.36 in place of agent/v0.3.47 below.

The image is public: docker pull needs no registry login.

IMAGE=ghcr.io/cloudpanorama/panorama/panorama-agent:0.3.47@sha256:<digest>   # the release's version and digest
docker pull "$IMAGE"
cosign verify "$IMAGE" \
  --certificate-identity https://github.com/cloudpanorama/panorama/.github/workflows/agent-release.yml@refs/tags/agent/v0.3.47 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

cosign verify accepts the digest only if the repository's agent release workflow signed it, on the tag of that version, through GitHub's own identity — nothing else in the repository can sign as that workflow. The digest is the index of both platforms; the release signs the index and each platform's image under it, so a verification of the digest a runtime resolved for its host passes as well. cosign verify-attestation --type cyclonedx with the same two flags returns each platform's SBOM. It needs cosign 3 or later: the release signs in cosign 3's bundle format, and an older cosign (2.4.1 was tried) reports "no signatures found" for a correctly signed image.

The commands in §3 and the reference compose file take the image from $IMAGE and PANORAMA_AGENT_IMAGE; a locally built image (make image) works in either the same way.

2. Configure

Everything is environment variables. The broker reads all of them; the worker reads only PANORAMA_WORK_DIR, PANORAMA_TOFU_BIN, PANORAMA_STATE_READ_TIMEOUT, PANORAMA_BROKER_UID, PANORAMA_GIT_CONFIG and PANORAMA_ALLOWED_GIT_HOSTS, and refuses to start if PANORAMA_AGENT_TOKEN or PANORAMA_PLATFORM_URL is set in its environment. Three are mandatory for the broker; the S3 backend settings are mandatory for production, and everything else has a default.

Cloud credentials come from the environment the agent runs in — instance profile, IRSA, task role, or the AWS_* variables your operators set — and from nowhere else. There is no setting for one, and the platform never sends one.

2.1 What OpenTofu is allowed to see

The agent builds OpenTofu's environment from nothing and passes through only (PROVISIONING_SPEC.md §7.6):

PATH HOME TMPDIR TZ
AWS_REGION AWS_DEFAULT_REGION AWS_PROFILE
AWS_CONFIG_FILE AWS_SHARED_CREDENTIALS_FILE
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI AWS_CONTAINER_CREDENTIALS_FULL_URI
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
AWS_WEB_IDENTITY_TOKEN_FILE AWS_ROLE_ARN AWS_ROLE_SESSION_NAME
GOOGLE_APPLICATION_CREDENTIALS GOOGLE_CLOUD_PROJECT GOOGLE_REGION GOOGLE_ZONE
GOOGLE_IMPERSONATE_SERVICE_ACCOUNT
ARM_USE_OIDC ARM_OIDC_TOKEN_FILE_PATH ARM_TENANT_ID ARM_CLIENT_ID
ARM_USE_AKS_WORKLOAD_IDENTITY AZURE_FEDERATED_TOKEN_FILE AZURE_TENANT_ID AZURE_CLIENT_ID
ARM_USE_CLI ARM_USE_MSI
TF_IN_AUTOMATION TF_INPUT TF_DATA_DIR

So a template's local-exec never sees PANORAMA_AGENT_TOKEN, PANORAMA_PLATFORM_URL, or anything else you set for the agent — and it also never sees static AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pairs, which are not on the list on purpose. Give the agent a role, not a key pair. If your setup can only provide static keys, put them in a shared credentials file and point AWS_SHARED_CREDENTIALS_FILE at it; the SDK inside OpenTofu reads the file, and the values are never in the environment a template could dump.

Nothing for Kubernetes is on the list, and nothing needs to be: a template that manages a cluster configures the kubernetes and helm providers from the cluster's own data sources, so there is no kubeconfig and no exec plugin to pass through.

2.1.1 An agent on Google Cloud

An agent serves one account, and on Google that account is one project: its token is issued for the project id, as an AWS agent's is for the account's twelve digits, once the project is connected and enabled (agent-operations.md §2).

The documented homes are the ones that carry an identity without a key: a Compute Engine VM with an attached service account, a GKE workload using Workload Identity, or a Cloud Run job. In all three the provider finds the credential by itself and you set nothing.

GOOGLE_CLOUD_PROJECT, GOOGLE_REGION and GOOGLE_ZONE are the provider's defaults, so a template need not hard-code where it is running.

GOOGLE_APPLICATION_CREDENTIALS is on the list for the case where the ambient identity is not the one the templates should use: it names a credential configuration — an external-account file for workload identity federation, or an impersonation file naming the service account to assume.

It must name one of two shapes, and nothing else. The agent reads the file at startup and refuses to start otherwise, saying what it found:

GOOGLE_APPLICATION_CREDENTIALS does not name a credential configuration this agent
accepts (/etc/panorama/creds.json): it carries refresh_token; use workload identity
or an attached service account

Accepted: external_account and impersonated_service_account, whose source_credentials is checked the same way — impersonation backed by a key file is a key file. Refused: a service_account key; an authorized_user file, which is what gcloud auth application-default login writes and holds a plaintext OAuth refresh token and your client secret; a type nobody recognises; a document with no type at all; and any file carrying private_key, private_key_id, refresh_token or client_secret at any depth, whatever it calls itself.

This is not a style preference. The variable is passed through to OpenTofu, so the file it names is in reach of any template that can run local-exec — and anything long-lived there can be copied out and reused after the attempt ends. A federated or impersonation configuration fetches a short-lived token and is worth far less to whoever copies it. The same rule is why there is no GOOGLE_CREDENTIALS on the list at all: that variable holds the key's contents.

The IAM the agent's identity needs is the same shape as the AWS reader-plus- provisioner split: permission to create what its templates create in its own project, and nothing in any other — or, with PANORAMA_RUNNER_SERVICE_ACCOUNT_PROJECT set (the table above), only permission to impersonate each template's runner account, which holds that template's permissions alone.

GOOGLE_IMPERSONATE_SERVICE_ACCOUNT is on the list for that setting: the worker sets it to the request's runner account, and the providers and the gcs backend impersonate it with the agent's own identity. It names an account and holds no credential.

On Azure (Import's generated path, the storage account) the agent signs in through federation only. Give it a token file the environment keeps fresh, and the identity that trusts it: ARM_USE_OIDC=true, ARM_OIDC_TOKEN_FILE_PATH, ARM_TENANT_ID and ARM_CLIENT_ID. On AKS, set ARM_USE_AKS_WORKLOAD_IDENTITY=true with the workload identity webhook's AZURE_FEDERATED_TOKEN_FILE, AZURE_TENANT_ID and AZURE_CLIENT_ID. On AWS, the file is a web identity token that STS gives the agent's role (aws sts get-web-identity-token, outbound web identity federation), written again before it expires by a sidecar or timer (agent-operations.md §3.1.2 has the command, the interval and the file's mode). With Azure runners, the token source writes one token per runner into a directory only the broker reads, PANORAMA_RUNNER_AZURE_TOKEN_DIR, and the worker holds no token file of its own: the broker hands it the attempt's runner's token alone. With an Azure runner configured, a request without exactly one whole federation mode fails before OpenTofu runs. The agent writes use_cli = false and use_msi = false into every azurerm provider configuration a template declares, and sets both variables to false for OpenTofu, so the host's Azure CLI login or managed identity is never used (agent-operations.md §3.1.2). A client secret or certificate is not on the list, and the agent refuses to start beside one (ARM_CLIENT_SECRET, for example), naming the variable. State may stay in the backend above, or move to Azure Blob Storage with PANORAMA_STATE_BACKEND=azurerm (from 0.3.35), signed in through this same federation and nothing else: the backend block says use_azuread_auth = true, use_cli = false and use_msi = false, and with state on Azure every attempt needs one whole federation, runners or not. An account key, a SAS token or a connection string (ARM_ACCESS_KEY, ARM_SAS_TOKEN, AZURE_STORAGE_KEY, AZURE_STORAGE_SAS_TOKEN, AZURE_STORAGE_CONNECTION_STRING) is not on the list either, and the agent refuses to start beside one, whatever the backend.

2.2 What git is allowed to see

Every attempt gets a HOME and a TMPDIR of its own, inside its directory under PANORAMA_WORK_DIR, and both go with it. That is what a template's local-exec, git and OpenTofu see as home for the attempt: whatever a template writes there — a .gitconfig naming a hook directory, a credential helper, a provider plugin cache — is gone before the next attempt begins, and the worker's real home directory is root's and read-only. Git inside an attempt runs with no global configuration except the file named by PANORAMA_GIT_CONFIG, and is told, per invocation, that exactly the attempt's directory is a repository it may work in although the broker's user created it (safe.directory); nothing a template can write changes either.

Private template repositories therefore get their credentials through PANORAMA_GIT_CONFIG and nowhere else: mount a git configuration file read-only into the worker and name it there. A credential helper it names runs as the worker, so the secret it reads must be readable by the worker's user — and by nothing a template can reach: a mounted secret file, not a file under the work directory.

Resolving a template's ref to a commit (git ls-remote, when a request names a branch or a tag) runs in the worker for the same reason: the broker never runs git against a template's URL.

What git may fetch. The platform checks a template's URL when it is published; the agent checks it again before any git process starts, because a stored row can change after that. Refused, whatever the host: any scheme but https:// and ssh:// (and git's user@host:path form) — so no git://, http://, file://, local path or ext:: helper — a credential inside the URL (before the @, or in a query or fragment), and a URL that begins with -. The log line naming the URL shows only its scheme, host and path. Refused by address: a host that is, or resolves to, link-local (where the cloud metadata service lives), loopback, the unspecified address or EC2's IPv6 metadata address — every address it resolves to, not the first. A private address is not refused: the agent runs in your network, and a git server on 10.0.0.0/8 is a normal thing to have. For the strict answer, name your hosts in PANORAMA_ALLOWED_GIT_HOSTS and nothing else is fetched. Git itself is also told to use only the https and ssh transports (protocol.allow=never with those two allowed), so a URL that got past the check would still stop at git.

The check resolves the host and git resolves it again, so a name you do not control can answer differently the second time. PANORAMA_ALLOWED_GIT_HOSTS is the setting for that; the worker's IAM role is the ceiling either way.

Where a destroy may point. A create writes state at exactly <PANORAMA_STATE_PREFIX>/<requestId>.tfstate in PANORAMA_STATE_BUCKET (with the template's slug before the request id when runner roles are set; a destroy of a create an agent before 0.3.32 wrote without it is accepted at that key) — on Google Cloud Storage, under the prefix <PANORAMA_STATE_PREFIX>/<requestId>, where the backend keeps its own default.tfstate — on Azure Blob Storage, at the blob of the S3 key's shape — and reports that location: s3://bucket/key, gcs://bucket/prefix, azurerm://account/container/key or file:///path. A destroy runs against the location its create reported, never against the agent's current setting. The agent refuses a destroy whose location is not exactly the one its own create wrote, refuses a scheme it does not know, and refuses one that does not say which create it undoes: another request's state beside it, another bucket or prefix, a nested key, a traversal, a local file. It is a configuration failure before anything is fetched, so a wrong or tampered location cannot tear down another request's state. The worker's IAM role is still the real limit: grant it state access to that bucket and prefix only.

That exactness has a consequence when PANORAMA_STATE_BACKEND changes: a create made under the old backend reported the old location, and this agent would now write a different one, so its destroy is refused as configuration — "is not in Google Cloud Storage, where this agent keeps state", or the same sentence about S3 or Azure Blob Storage. It is the right refusal (the state is where it is, and the new backend's credentials cannot reach the old bucket anyway), but it means a switch is not retroactive. Destroy what the old backend holds before switching, or keep an agent on the old setting until those requests are gone.

3. Run

Two processes, two users, one shared work directory — whatever runs them. The work directory belongs to the broker's user and the shared group, mode 2770, so the broker writes an attempt's files and the worker reads and executes them; a template's local-exec runs as the worker and cannot see the broker's process, environment or home. The worker must have a user of its own: it sweeps every other process of that user before and after every attempt, and refuses to start if any other process of its user exists.

Run both with a read-only root filesystem: the only writable place is the work directory, and everything git or OpenTofu later trusts — the binaries, the home directories, /etc — stays what the image shipped.

Run both with every capability dropped and no new privileges, and with a process and a memory ceiling on each. Neither runs as root, so neither needs a capability; the ceilings keep a template's fork bomb or runaway allocation inside the worker's own container, refused or killed there, rather than taking the host's process table or memory. Give the memory ceiling no swap behind it, so a blow-up stops at the limit. The reference values: the worker 1024 processes (threads count) and 4 GiB, the broker 256 and 1 GiB; raise the worker's memory for a plan over large providers. The one step that runs as root, setting the volume's owner and mode, needs CHOWN, FOWNER and FSETID and nothing else — without FSETID the kernel silently drops the setgid bit the shared group depends on.

Docker Compose — the reference layout, agent/deploy/compose.yaml: a volume-init service that sets the volume's owner and mode once, a broker service (uid 1000) with the platform URL and token, a worker service (uid 1001) with neither, both read_only: true, cap_drop: [ALL], no-new-privileges, a pids_limit and a mem_limit equal to its memswap_limit (PANORAMA_WORKER_MEMORY and PANORAMA_BROKER_MEMORY set the memory), restart: always and stop_grace_period: 10m (below). Fill in the environment — PANORAMA_AGENT_IMAGE to the $IMAGE you verified (§1.1) — and docker compose up -d.

Give both processes ten minutes to stop. A stop in the middle of an apply is a drain, not a kill: OpenTofu gets an interrupt and up to 20 s to finish its in-flight calls and write state, the broker then reads the state (up to PANORAMA_STATE_READ_TIMEOUT, 5 minutes by default) to say whether anything was created, and sends the report (up to 2 minutes). Heartbeats continue throughout, so the platform does not reap the task. A runtime that kills sooner — Docker's default is 10 s — loses that report: resources created outside state, the state lock still held, the retry failing on it. Set it everywhere the agent runs: stop_grace_period: 10m (Compose), --stop-timeout 600 (docker run), terminationGracePeriodSeconds: 600 (Kubernetes), TimeoutStopSec=600 on both units (systemd). ECS caps stopTimeout at 120 s, less than a full drain: on ECS, stop the agent only when the Agents page shows no running task.

Plain docker run — the same three steps by hand, with IMAGE set as in §1.1:

docker volume create panorama-work
docker run --rm --user 0:0 --read-only --cap-drop ALL --cap-add CHOWN --cap-add FOWNER --cap-add FSETID \
  --security-opt no-new-privileges:true \
  -v panorama-work:/var/lib/panorama-agent/work --entrypoint /bin/sh "$IMAGE" \
  -c 'chown 1000:2000 /var/lib/panorama-agent/work && chmod 2770 /var/lib/panorama-agent/work'
docker run -d --name panorama-worker --restart always --stop-timeout 600 --user 1001:2000 --read-only \
  --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 1024 --memory 4g --memory-swap 4g \
  -v panorama-work:/var/lib/panorama-agent/work -e PANORAMA_BROKER_UID=1000 -e AWS_REGION=eu-west-1 \
  "$IMAGE" worker
docker run -d --name panorama-broker --restart always --stop-timeout 600 --user 1000:2000 --read-only \
  --cap-drop ALL --security-opt no-new-privileges:true --pids-limit 256 --memory 1g --memory-swap 1g \
  -v panorama-work:/var/lib/panorama-agent/work \
  -e PANORAMA_PLATFORM_URL=https://acme.ccc.example -e PANORAMA_AGENT_TOKEN=... -e PANORAMA_WORKER_UID=1001 \
  -e PANORAMA_STATE_BUCKET=acme-terraform-state -e PANORAMA_STATE_REGION=eu-west-1 \
  -e PANORAMA_STATE_LOCK_TABLE=acme-terraform-locks \
  -p 127.0.0.1:8091:8091 \
  "$IMAGE" broker

Cloud credentials are the worker's: under an ECS task role or IRSA the worker container picks them up; on EC2 the instance profile does the same, and on Google Cloud the attached service account or a workload-identity configuration does. The broker needs none and, where the runtime cannot withhold them (a task role, an instance profile), holds credentials it does not use. With PANORAMA_STATE_BACKEND=gcs the worker is also the process that proves the bucket, so give it the Google identity even where the broker has none — the broker waits for the worker's first word before it registers, and a worker that cannot prove the bucket stops the agent there.

ECS (EC2 and Fargate): one task definition, two containers from the image with user set to 1000:2000 and 1001:2000, one shared volume mounted at /var/lib/panorama-agent/work in both, the worker with a container restart policy. Leave pidMode unset: the default keeps the two PID namespaces apart. In each container, readonlyRootFilesystem, linuxParameters.capabilities.drop ["ALL"] and a hard memory limit.

Kubernetes: one pod, two containers, an emptyDir at the work path in both, securityContext.runAsUser 1000 and 1001, fsGroup 2000, shareProcessNamespace left false, restartPolicy: Always, terminationGracePeriodSeconds: 600; in each container's securityContext, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false and capabilities.drop: [ALL], and a memory limit under resources.limits (the pod's process limit is the kubelet's podPidsLimit). Project the IRSA service-account token into the worker container only — and on GKE, bind the Kubernetes service account to the Google one for the worker container the same way. Probe the broker container only: liveness on /healthz, readiness on /readyz, never the other way round (§5). With Azure runners the worker holds no token at all: set automountServiceAccountToken: false on the pod, project the runners' tokens into the broker's container only, and keep the pod off an instance metadata service with an identity. The worker refuses to start otherwise (agent-operations.md §3.1.2).

Bare binaries, systemd — two units, two users, panorama-broker and panorama-worker, one group panorama, the work directory chown panorama-broker:panorama and chmod 2770:

# panorama-broker.service
[Service]
User=panorama-broker
Group=panorama
EnvironmentFile=/etc/panorama-agent/broker.env      # PANORAMA_PLATFORM_URL, PANORAMA_AGENT_TOKEN, PANORAMA_WORK_DIR, PANORAMA_WORKER_UID, the state settings
ExecStart=/usr/local/bin/panorama-agent broker
Restart=always
RestartSec=5
KillSignal=SIGTERM
TimeoutStopSec=600
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/panorama-agent/work
NoNewPrivileges=yes
CapabilityBoundingSet=
TasksMax=256
MemoryMax=1G
MemorySwapMax=0

# panorama-worker.service
[Service]
User=panorama-worker
Group=panorama
EnvironmentFile=/etc/panorama-agent/worker.env      # PANORAMA_WORK_DIR, PANORAMA_BROKER_UID, AWS_REGION; never the token
ExecStart=/usr/local/bin/panorama-agent worker
Restart=always
RestartSec=5
KillMode=control-group
TimeoutStopSec=600
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/panorama-agent/work
NoNewPrivileges=yes
CapabilityBoundingSet=
TasksMax=1024
MemoryMax=4G
MemorySwapMax=0

panorama-worker must be the only process of its user, ever: the worker sweeps that user before and after every attempt. KillMode=control-group makes a restart the same sweep, performed by systemd.

With Azure runners on a host whose metadata service has an identity, add IPAddressDeny=169.254.169.254 to the worker's unit (and fd00:ec2::254 on EC2 with IPv6 metadata), and run the token minting as a unit of its own user. A worker that can reach a minting source refuses to start (agent-operations.md §3.1.2).

SIGTERM to the broker interrupts the task in flight at its next safe point, reports it as cancelled by the agent (the platform records that as a failed request of class cancelled; nothing is destroyed), wipes the workspace, and exits — heartbeating until the report is sent. TimeoutStopSec=600 covers the drain (above). SIGTERM to the worker ends the attempt on its side; the broker reports it as an infrastructure failure and the runtime restarts the worker, which sweeps before it listens again.

3.1 ECS Fargate in one stack

deploy/cloudformation/agent-runtime.yaml is the layout above as one CloudFormation stack — what Install an agent opens (agent-operations.md §2.1). It holds the agent on Fargate (0.5 vCPU, 1 GB, a public address, AssignPublicIp), the pinned image, a volume-init container, a security group with one rule, HTTPS out, the token in Secrets Manager for the broker alone (a NoEcho parameter), a versioned, encrypted state bucket kept when the stack is deleted, a DynamoDB lock table, 30 days of logs, and a task role with the managed policy you choose — PowerUserAccess by default, which creates no IAM roles: a template that needs one needs a policy that grants it; it may pass or assume only roles it created under the stack's RolePath output, /cloudpanorama-agent/<the stack's id>/.

What protects the broker's token from the worker. The task role carries a permissions boundary the stack creates, which caps whatever policy you choose: it denies every Secrets Manager action on the token's secret, every IAM action on the agent's own two roles and on the boundary (under /cloudpanorama-agent-runtime/), removing any principal's boundary, creating or re-policying a role or user without the same boundary, passing, assuming or re-trusting any role outside the stack's RolePath (a path holding the stack's id, so no role can predate it there), and minting credentials for an IAM user. The role's own policies deny the same about the token, the roles, the agent's ECS service and its stack, and the secret's policy lets only the execution role read it.

So the one-step install is for a dedicated account only (the owner, 2026-10-03): Install an agent asks the admin to confirm that the account holds nothing but what the agent creates ("dedicatedAccount": true, recorded in the audit log). For a shared account, use the manual install above with a policy you scope. This is not a guarantee in a shared account. A worker whose policy lets it change compute that already runs in the account under a stronger identity — the code of an existing Lambda function, a command on an existing EC2 instance, a pipeline's build, an existing stack's service role — acts through that identity, and no boundary on the worker bounds it. PowerUserAccess reaches the first two. The token is protected only where the worker's policy cannot reach such compute: an account that holds none, or a policy that excludes it.

Fargate caps stopTimeout at 120 s (above), so update or delete the stack only while the Agents page shows no running task.

4. What it does with a task

agent-protocol.md §3 is the loop; this is what it leaves on disk, which is what you will see if you look inside PANORAMA_WORK_DIR mid-task:

worker.sock                       the worker's rendezvous, present between attempts and gone the moment the broker has connected
task-<taskId>-<random>/           mode 2770: the broker's, readable and writable by the worker's group
  main.tf …                       the template — checked out at its SHA by the worker, or the inline bytes after verification
  backend_override.tf             your S3 (or local) backend; wins over any backend the template declares
  ccc_provenance_override.tf      provider "aws" default_tags carrying panorama:request_id and friends (§9): an override, merged into your template's own provider blocks — default and every alias — with your default_tags kept and the stamp merged last
  generated.auto.tfvars.json      the approved variable values, through a JSON encoder
  panorama_imports.tf             an Import's import step only: one import block per address the template declares, identifiers from the platform; removed before the retag. A normal file: OpenTofu refuses import blocks in an override file
  panorama.tfplan                      the saved plan apply consumes
  .terraform/                     providers and modules, private to this task

After a create applies, the worker reads the outputs the task was handed (tofu output -json) and keeps every sensitive value to itself: only the names of the ones it withholds reach the broker and the platform.

The directory is removed when the task ends — success, failure, cancel, supersede, or the agent stopping. To recover a task's state later, panorama-agent recovery-workspace writes the three generated files again beside the source, with this same code (agent-operations.md §7.2). With PANORAMA_STATE_BACKEND=local, state/ beside it holds the state files and is not removed.

A process that dies mid-task cannot remove it, so both roles sweep. The worker, when it starts, removes whatever its own user left in any task-* or resolve-* directory — the attempt's .home and .tmp, which are the worker's alone and which the broker cannot enter; the broker, when it starts and before and after each task, removes every task-* and resolve-* directory, since it runs one task at a time and any it finds are a previous one's. Nothing else under PANORAMA_WORK_DIR is touched. What the broker cannot remove is logged and goes after the first task the restarted worker runs (from 0.3.33; before it, after the second).

.terraform/ is per task on purpose: every attempt downloads its providers again. A plugin cache shared between attempts would let one template leave a provider binary for the next to run, which is what the execution boundary exists to prevent. The cost is the download: the AWS provider, 6.64.0 for linux/amd64, is 183 MB (2026-09-10), fetched from registry.opentofu.org and GitHub on every create and destroy. A read-only mirror you provide would remove it; it is not supported yet.

An output line is carried from the worker to the broker whole up to about 672 KiB and cut beyond that, before the broker scrubs it and cuts it to the platform's line budget; a larger line once broke the channel and was reported as the worker going away.

Logs are scrubbed before they leave the host (PROVISIONING_SPEC.md §7.5): the exact values of every variable the template's schema marks sensitive, AWS access key ids, password=/secret=/token=-shaped assignments, and PEM blocks. Resource ids, ARNs and hashes are left alone.

5. Health

Two probes on PANORAMA_HEALTH_ADDR, both answering the same document:

{"registered": true, "agentId": "…", "accountId": "111122223333",
 "uptimeSeconds": 8123,
 "current": {"taskId": "…", "action": "create", "stage": "apply",
             "startedAt": "…", "stageDurationsMs": {"prepare": 812, "init": 4210, "plan": 9021}},
 "last": {"taskId": "…", "action": "create", "stage": "apply", "outcome": "succeeded",
          "stageDurationsMs": {"prepare": 640, "init": 3890, "plan": 8770, "apply": 61220}}}

The agent takes no inbound traffic, so what readiness gates is a rollout: a new version whose registration is refused is never reported ready. Both are unauthenticated and carry ids only, so bind them to localhost or the pod network, not to the world. Only the broker serves them; a worker that fails exits, and its restart policy brings it back.

Changed 2026-09-28. /healthz answered 503 until the agent registered, and this section said to point a liveness probe at it. An agent waiting out a refused registration was then restarted a few probes into every wait, for as long as the refusal lasted. /healthz is liveness now and the old answer is /readyz's, so a liveness probe set up by the old advice needs no change.

6. Exit codes

A 403 (account disabled, module not entitled) is not an exit, at start or later: the agent registers or claims again every one to three minutes until somebody fixes it on the platform side, as agent-protocol.md §7 says it should, and logs the refusal each time. Refused at start, it is not registered: /readyz answers 503 until it is, while /healthz stays 200, since the process is alive and doing what it should (§5).

7. Testing your installation without touching the cloud

make test-acceptance in agent/ runs the tests that need a real OpenTofu — the environment-allowlist test among them — against the null provider with a local state backend: no AWS account, no cost. It needs tofu (or PANORAMA_TOFU_BIN=terraform) and the network for the provider download.

Operating an agent

Audience: whoever installs and operates the agent inside your AWS account or Google Cloud project. If you are writing an agent, you want agent-protocol.md instead — that is the contract, and it is deliberately complete without this file.

Status: written on 2026-09-01, alongside the protocol.

---

1. Install

One agent per account. That is not a recommendation — the token is issued for exactly one (tenant, account) pair, so an agent physically cannot claim work for another account, and running two agents for one account is supported (see agent-sequences.md §5.4) but buys nothing. On Google Cloud the account is a project: its token names the project id (§2), and everything below reads the same.

Where it runs. Anywhere in that account with outbound HTTPS and an identity:

A service-account key is refused, at startup, by name. If GOOGLE_APPLICATION_CREDENTIALS names one, the agent will not start and says so. Use one of the three shapes above, or point that variable at a credential configuration — workload identity federation or impersonation. agent-install.md §2.1.1 says why.

On a self-hosted deployment: its own host, or the platform's for a small install. The recommendation is an instance, task or pod of the agent's own in the account it provisions — one of the shapes above — so the agent holds its identity and nothing else. A small install may run it on the platform's own host instead, beside the compose stack, with this trade-off: a template's local-exec runs as the worker, and on the platform's instance it can reach the instance metadata endpoint and with it the platform's own role (the readers it assumes, Cost Explorer, the federation to Google and Azure), whatever identity you give the agent — so anyone who may publish or approve a template may act as the platform. Choose it only where the people who approve templates already administer the platform, and give the agent its own role (an AWS config with credential_source = Ec2InstanceMetadata and role_arn mounted into the worker, agent-install.md §2.1) rather than the platform's. The clean-install drill ran both: round 1 on the platform's host, round 2 on an instance of its own.

No inbound rule is ever needed. The agent polls us; we never call it. If somebody asks you to open a port for this, they are describing a different product.

What the role needs. Whatever your templates create, plus read and write on the S3 bucket and DynamoDB table you use for Terraform state — or, with the GCS backend, on the state bucket alone, the permissions in §7.1. We do not publish a policy for the first half because we do not know what your templates do, and a policy we invented would either be too small to work or too large to defend.

Two image tags. panorama-agent:<version> is the small one (146 MB): every attempt downloads the providers it needs, which measured 14–42 seconds for a template using only the AWS provider. panorama-agent:<version>-mirror (370 MB) carries those providers already, at pinned versions, read-only, and the same init takes 6–8 seconds with no network at all. Take it if you run blueprints, or if the host has no route to the OpenTofu registry. Nothing else differs; a template needing a provider the mirror does not carry still fetches it, and the workspace still holds the unpacked providers either way.

2. Issue a token

A tenant admin, in the portal: Provisioning → Agents → Issue agent token, choosing the account from the enabled ones the registry holds.

The account is the registry's id for it: an AWS account's twelve digits, or a Google Cloud project's project id — not its number and not its display name, the same id it was connected under.

POST /provisioning/agents      {"name": "prod-eu-west-1", "accountId": "111122223333"}
POST /provisioning/agents      {"name": "gke-europe-west1", "accountId": "acme-prod-42"}
201  {"agent": {...}, "token": "..."}

The token is shown once. We store only sha256(token); there is no endpoint that reads it back, and losing it means issuing a new one and revoking the old.

The account must already be in the registry and enabled, or this is 409 account_not_managed — an agent bound to an account nobody has enabled would poll forever for work that cannot be created. An account another organisation has connected is not in yours, and gets the same answer. An id of neither shape is 400 invalid_request, and nothing is issued in either case.

2.1 Or install it in one step (AWS)

For a dedicated account only (the owner, 2026-10-03): an AWS account that holds nothing but what the agent creates. The dialog asks you to confirm it, the request carries "dedicatedAccount": true, and the confirmation is written to the audit log. In a shared account the agent could act through compute already there under a stronger identity, which the stack cannot bound; for a shared account, use the manual install (§1–§3) with a policy you scope.

For an AWS account, Provisioning → Agents → Install an agent (account, region) answers with an AWS CloudFormation quick-create link instead of a bare token: open it signed in to that account, pick a VPC and subnets, and create the stack (cloudpanorama-agent; contents in agent-install.md §3.1). Your session creates it on your credentials; the platform assumes nothing in the account. POST /provisioning/agents/install {"name", "accountId", "region", "dedicatedAccount": true} → 201 {"agent", "launchUrl", "expiresAt"}. The link is a secret, shown once, and its token lapses an hour after issue unless an agent has registered with it; then it is an ordinary token (§6). A deployment without the three AGENT_INSTALL_* settings answers 409 agent_install_unavailable.

3. Configure and start

Three settings for the broker, and nothing else is mandatory; the worker gets the work directory and no token (agent-install.md §3 has the two- process layouts):

PANORAMA_PLATFORM_URL=https://<your-tenant>.ccc.example
PANORAMA_AGENT_TOKEN=<the token from 2.2>
PANORAMA_WORK_DIR=/var/lib/panorama-agent/work

The work directory must exist before the agent starts, and the agent's user must be able to write to it. If either is not so, the agent stops at start-up with exit code 2 and a message naming the directory.

The agent registers on start and logs the policy it was given. If it logs versionStatus: unsupported, it will register and then be refused work; upgrade it before doing anything else.

3.1 One role per template (AWS)

By default a request runs with the agent's own AWS identity, which must then hold every template's permissions. Set PANORAMA_RUNNER_ROLE_ARN_PREFIX on the worker and each request runs as <prefix><template slug> instead, a role granted only that template's documented policy and the state bucket's objects beneath <PANORAMA_STATE_PREFIX>/<template slug>/, trusting the agent's role (agent-install.md §2 has the setting). From 0.3.32 the agent keeps that template's state there, <PANORAMA_STATE_PREFIX>/<template slug>/<request id>.tfstate, so one template's role cannot read or overwrite another's (set the setting on the broker too: it lays the state out). A destroy of a create an earlier agent wrote at <PANORAMA_STATE_PREFIX>/<request id>.tfstate is still accepted at that key, if its role may reach it. With it set, a request whose role is missing fails, naming the role; the agent never falls back to its own identity. The role bounds what the template's providers may do. It does not stop code in a template that reaches the agent's own identity directly, such as a local-exec.

Who signs in is the agent's (from 0.3.36). A provider argument beats the profile the agent sets, and every runner role trusts the agent's own identity, so a template could otherwise have run as the agent or as another template's runner. The agent therefore refuses before init a template whose aws (or awscc) provider block, under any name and in any module, sets profile, assume_role, assume_role_with_web_identity, access_key, secret_key, token, shared_config_files or shared_credentials_files, whatever the value, and role_arn in awscc. A role to assume is a runner role here, never the template's. Region, default tags, endpoints and the rest stay the template's. On Google the same holds for impersonate_service_account, its _delegates, credentials, access_token and external_credentials (§3.1.1). A template override file may not configure any of these providers at all. A module the template calls is checked after init, before anything plans.

A template reads no OpenTofu state (from 0.3.36). The terraform_remote_state data source reads a backend with the agent's credentials, or with a profile, a role or an account its own config names, so a template could read any state the agent reaches, another template's included. The agent refuses a template that uses it, whatever its backend, before init; in a module the template calls, after init. A template that needs another's outputs takes them as variables.

deploy/cloudformation/library-runner-roles.yaml makes one role per Library template, and one per type Import generates for. It is in the deploy/ the platform's install copies out of its image (from platform 0.1.2). Deploy it in the agent's account from an administrator identity, naming the agent's role and its PANORAMA_STATE_PREFIX, and again whenever a platform release changes it:

aws cloudformation deploy --stack-name cloudpanorama-library-runners \
  --template-file deploy/cloudformation/library-runner-roles.yaml \
  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameter-overrides AgentRoleName=AGENT_ROLE_NAME StatePrefix=STATE_PREFIX

The prefix to set is then arn:aws:iam::ACCOUNT:role/cloudpanorama/library-runner/.

Runners for templates of your own. A git or inline template runs as the role its slug names, and the stack makes roles only for the Library's templates and Import's types. deploy/aws/tenant-runners/main.tf.json, in the same deploy/, makes the others: give tenant_runners each template's slug and the IAM actions its role holds, and it makes one role per slug the way the stack makes the Library's. Each is under the same path, so the prefix above names it, trusts the agent's role alone, holds exactly those actions, and reads and writes state only beneath <PANORAMA_STATE_PREFIX>/<slug>/. Apply it with OpenTofu 1.9 or later, from an administrator identity, beside the stack. Its own state goes in the state bucket outside every runner's prefix:

tofu -chdir=deploy/aws/tenant-runners init \
  -backend-config=bucket=STATE_BUCKET -backend-config=key=tenant-runners.tfstate \
  -backend-config=region=REGION
tofu -chdir=deploy/aws/tenant-runners apply \
  -var agent_role_name=AGENT_ROLE_NAME -var state_prefix=STATE_PREFIX \
  -var 'tenant_runners={"our-queue":["sqs:CreateQueue","sqs:TagQueue"]}'

A slug that names a Library runner is refused at plan, before anything is made: a slug the Library has released, a name beginning generated-, negative-control, or a slug whose Google service account is one of theirs (below). So are an action with a wildcard and a template with no actions. Adding a template is a new apply with the longer map. Making the role by hand, as the stack makes one, stays an alternative.

3.1.1 One account per template (Google Cloud)

The same on Google, with a service account for a role. Set PANORAMA_RUNNER_SERVICE_ACCOUNT_PROJECT on the worker to the project the runner accounts are in, and each request impersonates its template's account there: the template's slug, or for a slug over 30 characters its first 21, a hyphen and 8 hex characters of its SHA-256 (agent-install.md §2). Each account holds a custom role of only that template's documented policy, and the state bucket's objects beneath <PANORAMA_STATE_PREFIX>/<template slug>/, where the agent then keeps that template's state (set the setting on the broker too: it lays the state out); the agent's identity may impersonate it (iam.serviceAccounts.getAccessToken) and needs nothing else to run the template. deploy/gcp/library-runners/main.tf.json, in the same deploy/, makes them for the library's templates. Apply it with OpenTofu in the runners' project from an administrator identity; its own state goes in the state bucket under a prefix no runner can reach, and STATE_PREFIX is the agent's PANORAMA_STATE_PREFIX, or empty when it has none:

tofu -chdir=deploy/gcp/library-runners init \
  -backend-config=bucket=STATE_BUCKET -backend-config=prefix=library-runners
tofu -chdir=deploy/gcp/library-runners apply \
  -var project=PROJECT_ID -var state_bucket=STATE_BUCKET -var state_prefix=STATE_PREFIX \
  -var agent_member=serviceAccount:AGENT_SERVICE_ACCOUNT_EMAIL

The module makes the runners for templates of your own too: give tenant_runners each template's slug and the IAM permissions its runner holds, as -var 'tenant_runners={"our-topic":["pubsub.topics.create"]}'. Each is one more account like the Library's, named the same way, and the agent finds it by the slug. A slug must begin with a letter and be at most 50 characters, so that its account and custom role ids exist; two slugs that would be one account, a Library runner's name (as on AWS) and a permission that is not one are refused at plan. The module needs OpenTofu 1.9 or later.

Google limits how many service accounts one apply may create per minute; if the first apply stops at a quota error, run it again a minute later. A missing account, or one the agent may not impersonate, fails the request naming the account; the agent never falls back. A template whose google or google-beta provider block names an account to impersonate, a delegation chain, a credential, a token or external credentials is refused before init (from 0.3.36, §3.1).

A generated template's runner (from 0.3.30). A template Import generates for a resource (agent-protocol.md §3.9) has the slug generated-<import id>, which names no runner. Its tasks carry runnerSlug, the runner of the resource's type, generated-<type> (generated-aws-s3-bucket, generated-gcp-storage-bucket), and the agent runs them as that role or account, its Google state beneath that name. Each holds the documented policies of every Library template that adopts the type, combined, so it can import, retag and destroy the resource and the companions Import lists. Both runner files make one per type Import generates for; re-apply them (the CloudFormation stack, and the OpenTofu configuration) before adopting through a generated template with runners set. Without the runner, the request fails naming it, as for any template.

A template of your own never runs as a Library runner (from the first agent after 0.3.41). A git or inline template runs as the runner its slug names, and every runner above trusts the agent. So a slug that names one — any slug the Library has released, any generated- name, negative-control, or a slug whose Google service account is one of theirs — is refused by the platform when a template is created, cloned, copied or enabled under another slug, and a template made before that is refused when its request is claimed. The agent refuses such a task too, before anything is laid out and whether or not runners are set, checked against the Library its own image carries: this template's slug is reserved: a Library template's slug, a slug beginning generated- and negative-control name the identities the Library's templates run as, and a git or inline template never runs as one; nothing was run. A Library template under its own slug and a generated template's tasks run as before. Copy a refused template to another slug.

3.1.2 One identity per runner (Azure)

On Azure (from 0.3.31), a runner is a user-assigned managed identity. It holds a custom role of the runner's actions, assigned at the subscription, and has a federated credential that trusts the agent's own token. No secret is involved. Azure cannot name an identity the way a role ARN or an account email can, so the agent is given each runner's client id:

1. As the subscription's Owner, apply deploy/azure/library-runners/main.tf.json (in the same deploy/) with subscription_id, an existing resource_group and location for the identities, and the agent token's agent_issuer and agent_subject. For an agent on AWS, these are the account's outbound web identity issuer and the agent role's ARN. On AKS they are the cluster's OIDC issuer and system:serviceaccount:<namespace>:<name>. Each runner trusts the audience <agent_audience>/<runner> alone; agent_audience defaults to api://AzureADTokenExchange.
2. Write tofu output -json client_ids to a file the worker can read, and set PANORAMA_RUNNER_AZURE_CLIENT_IDS to its path (agent-install.md §2).
3. Mint one token per runner into a directory only the broker reads, and set PANORAMA_RUNNER_AZURE_TOKEN_DIR to it on the broker ("One token per runner", below). Azure runners require it.

It makes one runner per Azure template of the Library, named by its slug and holding exactly the template's documented actions (from library release 28, azure-storage-account-private), and one per Azure type Import generates for, generated-<type> for each of the seven Azure types Estate collects (from library release 34), whose actions are what reading, importing, retagging and destroying that type's resource take — a storage account's with its lifecycle policy. The generated cluster runner lists clusters' user credentials, and Import takes only a cluster whose local accounts are off; none holds a cluster's administrator credentials. One more, negative-control, holds the storage account template's actions without Microsoft.Storage/storageAccounts/write: a live run maps the template's slug to it, with PANORAMA_STATE_PREFIX set to <state_prefix>/negative-control, and its create must be refused. None holds listKeys: the provider reads an account without its keys. A runner the file does not name fails the request, naming it. So does one Entra ID refuses. The agent never falls back to its own identity.

The subscription is the request's (from 0.3.36). An agent serves one account, and for Azure that account is a subscription. The agent writes it as subscription_id into every azurerm provider configuration a template declares, over whatever the template says, so no template reaches another subscription; a request for an account that is not a subscription id fails in prepare when its template configures azurerm. Who signs in is the agent's too: a template that sets client_id, tenant_id, a token or a token file in an azurerm provider block is refused before init, since every runner trusts the agent's own issuer and subject.

Scope the runners to resource groups. By default each runner's role is assigned at the subscription, so it reaches every resource there. Some runners' actions read other resources' secrets wherever they reach: the function app runners hold Microsoft.Web/sites/config/list/action, which lists every web app's settings in their scope. To keep the runners out of the rest of the subscription, set runner_resource_groups to the existing groups the agent's requests may act in when you apply the module. Each runner's role is then assigned at each of those groups and not at the subscription. Then write tofu output -json resource_groups to a file the broker can read, and set PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS to its path on the broker (agent-install.md §2). The broker reads it for every request and fails a request in prepare, before OpenTofu runs, when it names a group outside its runner's list. That covers the resource_group_name variable, a resource id in any variable (a machine's subnet_id), an adopt's import ids and a resource_group_name the template writes literally. A resource id in another subscription is refused, and so is a request that names no group at all. The check is the early refusal, not the boundary: a group a template reaches through an expression the agent does not evaluate is refused by Azure, because the runner holds no role there. Choose groups set aside for provisioned resources. Every listed group is open to every runner at all times. A subscription dedicated to the agent gives the same separation with no list. Adding a group is a re-apply and a new file, and Azure can take up to 10 minutes to honour a new assignment.

Your own Azure templates need the scope (the owner 2026-10-27). An agent whose runners keep the subscription runs Library templates and the templates Import generates, and nothing else on Azure. A request for one of your own templates, from git or inline, fails in prepare as configuration, before its source is fetched, saying to set PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS. A destroy of one fails the same way, so keep the setting while such resources exist. An Azure account served without runners at all is refused the same way, naming PANORAMA_RUNNER_AZURE_CLIENT_IDS too. The agent tells a generated template by the runner the platform names for it (generated-<type>), never by the template's own slug. With the setting, the agent advertises the capability azure-resource-groups at register. Scoping a Library-only agent changes nothing it runs, as long as each request's group is in the list.

Library-only agents too, from 2027-01-31 (the owner 2026-10-27). Every Azure agent's runners are to be scoped by then, whatever it runs. Until then, an agent whose runners keep the subscription (PANORAMA_RUNNER_AZURE_CLIENT_IDS set, PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS not) logs a warning each time its broker starts, naming the date, this section and the setting. The first agent release on or after 2027-01-31 refuses to start in that state: the broker, and the development single-process mode, log the reason as configuration, naming PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS, and exit 2. The date that counts is the release's, never the clock's: each build carries the date of the commit it was built from, and a release is never rebuilt. An agent already running, or restarted on the image it has, never starts refusing on that day by itself; the refusal arrives with the upgrade that carries it. To be ready, apply the runners module with runner_resource_groups and set the file on the broker, as above, before that upgrade.

Your own Azure templates. A template of your own, from git or inline, runs in Azure only on an agent whose runners are scoped to resource groups: the agent advertises it as azure-resource-groups when PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS is set, and the platform offers such a request to no other agent. Until the account's agent is scoped, the request waits, queued, and its page names the setting; a Blueprint's slot does not take the account, and says the same. Library templates and the templates Import generates run as before. Like every template of your own, it runs as the runner named by its slug. Give the module tenant_runners: each template's slug and the Azure actions its runner holds, as -var 'tenant_runners={"our-store":["Microsoft.Storage/storageAccounts/read","Microsoft.Storage/storageAccounts/write"]}' beside runner_resource_groups, which it requires. The module makes one more identity per slug, as it makes the Library's: a custom role of exactly those actions assigned at each of the groups, and a federated credential trusting <agent_audience>/<slug>. Its client_ids and resource_groups outputs name it, so write both files again, and mint its token into the token directory as for every runner. A slug that is a Library or generated runner's name is refused at plan, as on AWS; so are a slug under 3 characters and an action with a wildcard. The module needs OpenTofu 1.9 or later. Making the runner by hand stays an alternative: the identity, the role assigned at the groups, the credential and the three entries above. A runner made by hand under a slug you then give the module is made twice and the apply fails on the name; delete the hand-made identity and role first, or tofu import them. An inline source is checked when it is published: an azurerm provider block that chooses its own sign-in, a renamed held provider, provider registration other than "none", a resource group made and OpenTofu state read are refused there, as the agent refuses them; a subscription_id is said to be overwritten. A git source the agent checks when it runs.

Federation only, checked before each request (from 0.3.31). With PANORAMA_RUNNER_AZURE_CLIENT_IDS set, a request is signed in as its runner only through exactly one federation mode, set on the worker:

• ARM_USE_OIDC=true, ARM_OIDC_TOKEN_FILE_PATH (absolute) and ARM_TENANT_ID; or
• ARM_USE_AKS_WORKLOAD_IDENTITY=true, AZURE_FEDERATED_TOKEN_FILE and AZURE_TENANT_ID.

The token file is the runner's own, which the worker writes for each attempt ("One token per runner", below); set no token file of your own. If neither mode is on, or both are, or the directory is missing, the request fails in prepare, before OpenTofu runs, naming what is missing. Beyond that check:

• The agent writes use_cli = false and use_msi = false into every azurerm provider configuration a template declares, in an override file of its own (ccc_azure_signin_override.tf). Import's provider block already says so.
• OpenTofu runs with ARM_USE_CLI=false and ARM_USE_MSI=false, whatever you set. The variables alone are not enough: azurerm 5.9.0 reads ARM_USE_CLI=false as its default, which is true.
• A template whose azurerm provider sets use_cli or use_msi to anything but false, in any file, or names a client secret or certificate, is refused before init.
• These rules hold azurerm under any name (from 0.3.36). A template must require azurerm (and aws, google, google-beta) under its own name; alternate = { source = "hashicorp/azurerm" } is refused before init. A module the template calls may not configure azurerm itself. It takes the root module's configuration, and the agent refuses one that does after init, before anything is planned. These checks read .tofu and .tofu.json files as OpenTofu does. A template may not ship a .tofu file with the name of one the agent writes, such as backend_override.tofu.
• A template's own override file (*_override.tf, override.tf, and the .tf.json, .tofu and .tofu.json forms) may not configure azurerm. Such a file is merged after the agent's and would replace the subscription. An azurerm provider block may not set environment, metadata_host, msi_endpoint, msi_api_version or ado_pipeline_service_connection_id.

So an Azure CLI login or a managed identity on the agent's host is never used; neither is the runner.

One token per runner. Entra ID tells one runner's federated credential from another's by the token's issuer, subject and audience alone. A template is code: a local-exec, an external data source or another provider can read any token file its environment names and exchange it for any runner that trusts it. So each runner trusts a token minted for it alone, and the worker never holds one that another runner trusts:

• Whatever mints the agent's tokens writes one per runner, for the audience <agent_audience>/<runner>, into a directory, named by the runner: $TOKEN_DIR/<runner>. The runners are the keys of the client ids file.
• Set PANORAMA_RUNNER_AZURE_TOKEN_DIR to that directory on the broker. The broker reads the attempt's runner's token and hands the worker that one alone, afresh before each request that runs OpenTofu. The worker writes it into the attempt's own home and points ARM_OIDC_TOKEN_FILE_PATH (or AZURE_FEDERATED_TOKEN_FILE) at it.
• The worker keeps its federation switch and directory (ARM_USE_OIDC=true and ARM_TENANT_ID, or the AKS pair) and no token file of its own. A worker that can read one refuses to start, and fails a request in prepare, since that is the token a template would exchange. A worker given PANORAMA_RUNNER_AZURE_TOKEN_DIR refuses to start.
• The directory is the broker's: mode 0700, or 0750 with a group the worker is not in, and not owned by the worker's user. The broker refuses at start a directory the worker could reach. It fails a request in prepare whose runner has no token there, or whose token names an audience other than <agent_audience>/<runner> (another runner's, two runners', or the bare agent_audience), naming the file.

Required. Azure runners take per-runner tokens or nothing. The shared token is not accepted:

• The broker, and the development single-process mode, refuse to start with PANORAMA_RUNNER_AZURE_CLIENT_IDS and without PANORAMA_RUNNER_AZURE_TOKEN_DIR, exit 2, naming the setting.
• A worker serving Azure runners says so when the broker connects. A broker without the directory stops there, exit 2, before it registers or claims. This covers a broker that is not given the client ids file.
• A request that reaches the worker without its runner's token fails in prepare, before OpenTofu runs, whatever sent it.

To move a deployment that signs its runners in with one shared token, keep the order: the old runners module trusts only the bare audience, and the new one only <agent_audience>/<runner>.

1. Set up the minting of one token per runner into the broker's directory (below), beside the shared token, and check that each file appears.
2. Stop the agent. Set PANORAMA_RUNNER_AZURE_TOKEN_DIR on the broker, and remove the shared token file and its variable from the worker (ARM_OIDC_TOKEN_FILE_PATH, or the AKS projection), keeping the switch and the directory id.
3. Apply the runners module again, so each runner trusts its own audience.
4. Start the agent. Requests queue on the platform while it is stopped and none is lost.

An upgraded agent left with the shared token does not start, and its log names what is missing. Nothing signs in with the shared token in between.

The minting identity must be out of the worker's reach. Whatever mints the tokens can mint one for any runner. A template that can reach that identity mints its own token for any runner, and no setting of the agent prevents it. So a worker serving Azure runners probes at start for what could mint them, and refuses to start, exit 2, naming each source it found. There is no setting that turns the probe off. It looks for:

The metadata probes ask for an identity, so an answer alone is not a source. A metadata service with no identity is not one, and neither is OpenStack's, which serves EC2's paths without IAM. EC2 with IMDSv2 required and a hop limit of 1 is not one either: it answers a container's connection, but its session token never reaches the container. A connection that fails or times out is not a source. Each request waits at most a second, so on a plain host the worker starts within a few seconds.

What the probe cannot see is a credential you give the worker yourself: a key, a credentials file, or GOOGLE_APPLICATION_CREDENTIALS for state in another cloud. Its identity must not be one that can mint the runners' tokens.

The layouts where the boundary holds:

• Kubernetes. Project the runners' tokens into the broker's container only: a projected serviceAccountToken per runner, with audience: <agent_audience>/<runner> and path: <runner>. Set automountServiceAccountToken: false on the pod, so the worker holds no service account token. Without a workload identity webhook in the pod, the worker holds no AWS, Google or AKS token either. Where the node's metadata service has an identity (EKS, GKE, AKS nodes), deny the pod's egress to 169.254.169.254/32 with a network policy. The broker needs no metadata service.
• Docker Compose or docker run. Run the minting outside the worker's container: a host timer, or a sidecar on the host's network, writing into a volume that only the broker mounts. On EC2, require IMDSv2 with a hop limit of 1. On a Google or Azure VM with an identity, drop the containers' traffic to 169.254.169.254 (the DOCKER-USER chain).
• systemd. Run the minting as a unit of its own user, writing into the broker's directory. Give the worker's unit IPAddressDeny=169.254.169.254 (and fd00:ec2::254 on EC2 with IPv6 metadata) when the host's metadata service has an identity.
• ECS cannot separate them, since every container of a task reaches its task role. The worker refuses to start there with Azure runners. Run the agent for an Azure subscription on Kubernetes, Compose or systemd.

The development single-process mode is not probed. That one process reads every runner's token itself, so there is no boundary to keep.

Keeping the tokens fresh on AWS. The agent holds no AWS signer, so an agent on AWS gets its tokens from a sidecar, or from a cron or systemd timer, running as the agent's AWS identity and out of the worker's reach (above). That identity needs sts:GetWebIdentityToken, and the account needs outbound web identity federation (aws iam enable-outbound-web-identity-federation). Every 5 minutes it writes:

umask 077
for runner in $(jq -r 'keys[]' "$CLIENT_IDS"); do
  aws sts get-web-identity-token --region "$AWS_REGION" \
    --audience "$AGENT_AUDIENCE/$runner" --signing-algorithm RS256 --duration-seconds 900 \
    --query WebIdentityToken --output text > "$TOKEN_DIR/$runner.new" \
    && mv "$TOKEN_DIR/$runner.new" "$TOKEN_DIR/$runner"
done

• $AGENT_AUDIENCE is the runners module's agent_audience (its default is api://AzureADTokenExchange), and $CLIENT_IDS the file PANORAMA_RUNNER_AZURE_CLIENT_IDS names.
• $TOKEN_DIR is PANORAMA_RUNNER_AZURE_TOKEN_DIR, on a volume the broker reads and the worker does not mount. The sidecar runs as the broker's user, or as one whose group only the broker shares.
• Each token lives 900 seconds (STS allows 60 to 3600) and is rewritten every 300. The broker reads it again before each stage, so a stage that starts just before a rewrite still holds a token valid for at least 10 more minutes. Entra ID needs the token only at sign-in.
• The rename makes each write atomic, so a reader never sees half a token.

If the sidecar stops, requests fail at sign-in once the last token expires. They do not fall back to anything else.

State on Azure (from 0.3.35). With PANORAMA_STATE_BACKEND=azurerm (§7.1 makes the account), apply the same module with state_storage_account_id (the state account's resource id) and state_container as well, and state_prefix if PANORAMA_STATE_PREFIX is not agents. Each runner then holds one more role, Library runner state: blob read and write, no delete, granted at the account's resource group under a condition that admits only blobs in that container beneath <state_prefix>/<runner>/, and listings only there. The agent keeps that runner's state at <PANORAMA_STATE_PREFIX>/<runner>/<request id>.tfstate. So one runner cannot read or overwrite another's state, as on AWS (§3.1) and Google (§3.1.1). The grant is at the group, not at the container, because the account carries a ReadOnly lock and Azure refuses a role assignment beneath one.

Set one of the three settings, the one for the agent's cloud: two at once are refused at start.

3.2 The reference agent

agent/ in this repository is the reference implementation; its own manual — every setting, the container image, what it leaves on disk, the exit codes — is agent-install.md. The end-to-end smoke that runs it with real OpenTofu against a disposable account is one command: observability/scripts/smoke-provisioning.sh (settings at the top of the file). Nightly or by hand; never per PR.

4. Verify

The agent is working when Provisioning → Agents shows a lastSeenAt inside the last minute and the version you expect. Submit and approve a request; it should move queued → running within one poll interval.

If a request sits in queued: the agent for that account is not running. Requests are account-scoped structurally, so an agent for a different account will never pick it up no matter how healthy it looks. A request for a Library template also waits while that agent does not carry the Library release the template's version needs, or runs with PANORAMA_ALLOW_LIBRARY_SOURCES=false: upgrade the agent (§5), or turn the Library back on.

5. Upgrade

Report and warn; refuse only below a declared minimum.

• At or above minAgentVersion — the agent works. If it is behind the current release, the Agents page marks it outdated and nothing else happens.
• Below minAgentVersion — register still succeeds, deliberately, so that the agent appears in the portal with its version and you can see what to upgrade. claim answers 426 Upgrade Required naming the required version. The agent logs it and keeps polling; nothing is dropped, and the queue is picked up the moment it is upgraded.

Upgrading is: stop the agent, replace the image, start it — and the stop is a drain, not a formality. A task in flight when you stop the agent is interrupted, and OpenTofu, the state read and the report take up to ten minutes between them; the agent heartbeats throughout so the platform does not reap it, and reports the task cancelled, with whether anything was created. A runtime that kills the agent before that loses the report: resources created outside state, the state lock still held, and the re-queued attempt failing on the lock. So give both processes a ten-minute stop (agent-install.md §3 has the setting for each runtime), or, where the runtime cannot (ECS caps it at 120 s), wait for the Agents page to show no running task before stopping.

minAgentVersion moves rarely and never without notice. It exists for changes that cannot be made compatible, not for tidiness. That is the whole of the protocol's versioning: there is no separate protocol number to negotiate. An agent reports its version (panorama-agent version: X.Y.Z+<commit> — every build carries its commit after a +, which never ranks) and the platform admits it or names the minimum.

The move to 0.2.0 came without a notice period. No customer install existed, and the owner decided on 2026-09-12 to switch at once.

Plan inspection arrives by capability, not by version (2026-09-28). An agent that advertises plan-inspection at register is handed what each approved plan may do, checks its plan against it before applying, and applies nothing that does more — a create template that would import something already existing is refused before apply, as configuration. The platform acts on the capability only for the version that registered it, so a rolling upgrade on one token needs no care: until the new version has registered, the old one runs as it always has. minAgentVersion did not move (the owner, 2026-09-20: 0.2.x agents keep serving create and destroy). Every agent that inspects also confirms with the platform, one heartbeat answered, immediately before it applies; one that cannot reach the platform at that moment fails the task as infrastructure with nothing applied, and the request can be retried once the platform is back.

The Library arrives with the image (2026-09-28). The Library's templates ship inside the agent image, so an account gets a newer Library release by upgrading its agent; the Library names, for each template, the agent version it needs. The agent reports the release it carries on every claim, and a request pinned to a later release waits, queued, for an agent that carries it — the platform never sends the code instead. A request pinned to an earlier release runs on every later agent: the Library only grows, so a destroy still runs the code that created what it destroys. minAgentVersion did not move: an agent that reports no release is offered no Library template and runs everything else as before.

Import and outputs arrive with 0.3.0, by capability (2026-09-28). Agent 0.3.0 advertises adopt — it runs Import's "Make fully managed": the existing resource imported into a new state, with nothing else changed, then stamped — and outputs: after a create it reports the outputs the template declares, never one marked sensitive, for a later step to be wired from. A request that needs either is offered only to an agent that advertises it; one the account's agent cannot run waits, queued, and its page names the agent version that can. minAgentVersion did not move (the owner, 2026-09-20): 0.2.x agents keep serving create and destroy, and are never offered an import or a step whose outputs another consumes. Raising it is a decision of its own once the pilot's agents are on 0.3.0.

Drift judges what 0.3.0 creates (2026-09-28). Agent 0.3.0 also reports how many resources it tagged of each type, which is how the drift diff compares only what the inventory can see — the types it reads with their tags — and leaves out the roles, log groups and listeners it cannot. A create an older agent ran is recorded with its total alone: drift does not judge it, and the Provisioned resources page says so. A create keeps the answer it was recorded with; upgrading the agent changes what the next ones report. minAgentVersion did not move.

The Library's release 7 arrives with 0.3.6 (2026-10-03): a public certificate that ACM issues, validated through records the template writes in a Route 53 zone of the account, and a new version of the imported certificate. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.5's. A request for a release-7 template waits, queued, for an agent on 0.3.6 or later.

The Library's release 8 arrives with 0.3.7 (2026-10-04): its first Google templates — two Cloud Storage buckets, two Pub/Sub topics and a Pub/Sub subscription that writes to Cloud Storage — for an agent serving a Google Cloud project (agent-install.md §2.1.1). The identity the agent runs as needs the permissions each template's iam lists in the project. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.6's. A request for a release-8 template waits, queued, for an agent on 0.3.7 or later.

The Library's release 9 arrives with 0.3.8 (2026-10-05): a new version of the EC2 instance reached through Session Manager, which may reach PostgreSQL on port 5432 in one address range when postgres_cidr_block names one — what the small network, database and instance Blueprint wires from its network. Without it the instance allows HTTPS out alone, as before, and its iam is unchanged. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.7's. A request for the release-9 version waits, queued, for an agent on 0.3.8 or later.

The Library's release 10 arrives with 0.3.9 (2026-10-05): Google networking and identity — two VPC networks with private subnetworks, one of them behind Cloud NAT, two service accounts with no key, a website served from a Cloud Storage bucket through a load balancer, and an internal passthrough load balancer. The identity the agent runs as needs the permissions each template's iam lists in the project; the Compute Engine ones include the operations the provider waits on. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.7's. A request for a release-10 template waits, queued, for an agent on 0.3.9 or later.

The Library's release 11 arrives with 0.3.10 (2026-10-05): Google compute and serverless — a private Compute Engine instance and a Spot one, a private Cloud Run service and a public one, an HTTP function callers need IAM to invoke and a function run for each message on a Pub/Sub topic. The identity the agent runs as needs the permissions each template's iam lists in the project, including iam.serviceAccounts.actAs on the accounts a service, a function or an instance runs as. The two functions are built by Cloud Build and need its API, and Artifact Registry's, enabled in the project. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.7's. A request for a release-11 template waits, queued, for an agent on 0.3.10 or later.

The Library's release 12 arrives with 0.3.11 (2026-10-05): the Kubernetes mesh Blueprint's templates — peering requests, peering routes, a mesh certificate authority, Istio's control plane, Istio's remote clusters and a sample that checks the mesh — and new versions of the NAT network (five more outputs) and of the development EKS cluster (public_access_ips). The Istio templates install charts the agent fetches from istio-release.storage.googleapis.com at apply, and reach the cluster's Kubernetes API: the agent's address must be among the cluster's public_access_cidrs, and its role among the cluster's administrators. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.10's. A request for a release-12 template waits, queued, for an agent on 0.3.11 or later.

The Library's release 13 arrives with 0.3.12 (2026-10-05): new versions of the two Cloud Storage buckets, the bucket website and the Pub/Sub subscription writing to Cloud Storage, whose iam adds compute.projects.get (the provider reads the project through Compute Engine after creating a bucket); and of the two Pub/Sub templates that grant Pub/Sub's service agent a role, which now make the agent exist first through the google-beta provider's google_project_service_identity, so a project's first apply no longer fails on an agent Google has not created yet. The image's provider mirror carries google-beta 6.50.0 beside google. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.11's. A request for a release-13 version waits, queued, for an agent on 0.3.12 or later.

The Library's release 14 arrives with 0.3.13 (2026-10-05): new versions of the mesh's certificate authority and Istio control plane templates, whose iam adds kms:PutKeyPolicy. KMS refuses to create a key whose policy would not let the caller change it later, so without it the runner role could not create either key. The Kubernetes mesh Blueprint pins the new versions. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.12's. A request for a release-14 version waits, queued, for an agent on 0.3.13 or later.

The Library's release 15 arrives with 0.3.14 (2026-10-06): new versions of the same two templates, whose iam adds secretsmanager:ListSecretVersionIds. A secret value written write-only is never read back, so the provider finds its version by listing the secret's versions, and without it the runner role could not create, read or destroy one. The Kubernetes mesh Blueprint pins the new versions. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.13's. A request for a release-15 version waits, queued, for an agent on 0.3.14 or later.

The Library's release 16 arrives with 0.3.15 (2026-10-06): new versions of the NAT network and both peering templates. The network's routes are routes of their own rather than inline in its route tables, so a route a peering adds to one of them survives the network's next plan; its iam adds ec2:DeleteRoute. A peering within one account and region is accepted by its requester, whose iam adds ec2:AcceptVpcPeeringConnection, and the routes template accepts only another account's or region's connection, so an acceptance never rewrites the tags the requester gave the connection. The Kubernetes mesh Blueprint pins the new versions. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.14's. A request for a release-16 version waits, queued, for an agent on 0.3.15 or later.

The Library's release 17 arrives with 0.3.16 (2026-10-06): four Google templates, the last of the types Estate collects on Google that the Library owed. Two Cloud SQL for PostgreSQL instances, one for development and one highly available, with no public address and reached through a Private Service Connect endpoint in the request's subnetwork; and two private GKE clusters, Autopilot and Standard, with private nodes and the Kubernetes API at its private endpoint alone. An agent on Google needs, beside the permissions their iam lists, the Cloud SQL Admin, Kubernetes Engine and Compute Engine APIs enabled in its project. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.15's. A request for a release-17 version waits, queued, for an agent on 0.3.16 or later.

The Library's release 18 arrives with 0.3.17 (2026-10-06): a new version of the private Compute Engine instance template, which can import an existing instance. It takes the instance's exact name as instance_name, and an import fills in the name, zone, machine type, subnetwork, boot disk size and service account from Estate; the boot disk's type and the account's scopes are kept as they are. Its iam is unchanged. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.16's. A request for a release-18 version waits, queued, for an agent on 0.3.17 or later.

The Library's release 19 arrives with 0.3.18 (2026-10-06): new versions of nine Google templates, from what the live run saw. Both Cloud Run services no longer plan a change right after they are made: Cloud Run reads back a service-level scaling block they never write, which a plan no longer sets back. Every Google template that makes a service account adds iam.serviceAccounts.list to its iam (Google authorises reading the account with it), both networks add compute.subnetworks.setPrivateIpGoogleAccess (repairing a subnetwork's Private Google Access), and the internal load balancer adds compute.regionBackendServices.use. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.17's. A request for a release-19 version waits, queued, for an agent on 0.3.18 or later.

The Library's release 21 arrives with 0.3.20 (2026-10-08): new versions of both SQS queue templates, standard and FIFO. Each declares max_message_size, the largest message either of its queues accepts, from 1024 to 1048576 bytes and 1048576 (AWS's own default) unless set; the earlier versions left it to the provider's 262144. An import fills it in from Estate, so importing a queue SQS made with its default no longer plans to shrink both queues. Their iam is unchanged. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.19's. A request for a release-21 version waits, queued, for an agent on 0.3.20 or later.

The Library's release 22 arrives with 0.3.21 (2026-10-08): new versions of both Cloud SQL for PostgreSQL templates, development and production. Their iam adds compute.addresses.use, which Google checks when the private endpoint's forwarding rule takes its address: without it, creating an instance with an endpoint is refused. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.20's. A request for a release-22 version waits, queued, for an agent on 0.3.21 or later.

The Library's release 23 arrives with 0.3.22 (2026-10-09): new versions of both Cloud SQL for PostgreSQL templates, development and production. Their iam adds servicedirectory.namespaces.create and servicedirectory.services.create: Google registers the private endpoint in Service Directory, and without them creating an instance with an endpoint is refused. Re-apply the Google runner accounts (§3.1.1) before running these versions under them. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.21's. A request for a release-23 version waits, queued, for an agent on 0.3.22 or later.

The Library's release 24 arrives with 0.3.24 (2026-10-09): new versions of both Cloud SQL for PostgreSQL templates, development and production. Their iam adds servicedirectory.services.delete: deleting the private endpoint removes its Service Directory registration, and without it destroying an instance with an endpoint is refused. Re-apply the Google runner accounts (§3.1.1) before running these versions under them. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.22's. A request for a release-24 version waits, queued, for an agent on 0.3.24 or later.

The Library's release 25 arrives with 0.3.25 (2026-10-09): a new version of the private Cloud Run service template. OpenTofu's deletion protection becomes deletion_protection, off by default as before, and an import turns it on, as the import reads it. Until now an import of a Cloud Run service planned to turn it off, which updates the service, so the import step refused it. Destroying an imported service now takes an update that turns it off first. Its iam is unchanged. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.24's. A request for a release-25 version waits, queued, for an agent on 0.3.25 or later.

The Library's release 26 arrives with 0.3.26 (2026-10-09): a new version of the two-zone VPC template. Given a database's instance class in database_instance_class, it puts its subnets in the first two zones where RDS offers that class for PostgreSQL on gp3 storage, not the region's first two: RDS no longer offered db.t4g.micro on gp3 in eu-west-2a or eu-west-2b. Without a class it plans exactly as before. Its iam adds rds:DescribeOrderableDBInstanceOptions: update the network's runner role (§3.1) before running it under one. The network, database and service Blueprint pins it and takes a run variable, database_instance_class (db.t4g.micro unless the run names another class), which it gives both the network and the database. Nothing else changes: the protocol, the capabilities and minAgentVersion are 0.3.25's. A request for a release-26 version waits, queued, for an agent on 0.3.26 or later.

The Library's release 27 arrives with 0.3.27 (2026-10-10): a new version of the private Cloud Run service template, and the agent's half of a destroy that turns protection off first (agent-protocol.md §3.8). The template takes its revisions' labels and minimum instances as variables, and an import fills them in, with the maximum, as Estate read the service, so adopting a service gcloud run deploy made with labels or its own scaling changes nothing. Its destroy declares the protection it turns off: a destroy of a release-27 service, approved, applies deletion_protection = false and then destroys, both in the destroy's log. An agent before 0.3.27 holds no release-27 hash, so it is never handed one. A refused plan now names the settings it would have changed, on every agent from 0.3.27. Its iam is unchanged. Nothing else changes: the capabilities and minAgentVersion are 0.3.26's. A request for a release-27 version waits, queued, for an agent on 0.3.27 or later.

0.3.28 can generate a template for an import (2026-10-11): it advertises the capability generate and runs an adopt's generate step (agent-protocol.md §3.9). OpenTofu writes the configuration for the resource and its companions in your environment, with the template's runner credentials. A companion the provider finds absent is dropped and named. Every attribute the provider marks sensitive becomes a variable with no default, so its value is never in the configuration. The result is planned again and must import and do nothing else; nothing is applied and no state is written. The step reads the cloud only, so the adopt runner role is enough. The skeleton it starts from is an inline source, and the template it produces is one too, so the agent runs them only with inline sources allowed (PANORAMA_ALLOW_INLINE_SOURCES=true, agent-install.md). The platform hands the step to an agent that advertises generate, from the release that carries migration 0251; an agent without it keeps every other adopt. The Library and minAgentVersion are 0.3.27's.

0.3.29 opens the secret values an import needs (2026-10-12): a resource whose secret the provider reads back — a secret version's value — is imported with that value, which the requester types on the request page. The page encrypts it in the browser to each agent of the account (agent-protocol.md §3.10), and the platform keeps only the encrypted form, for one attempt and at most 24 hours. The agent advertises sealed-values beside generate (so only with inline sources allowed) and keeps its key in sealing.key under PANORAMA_WORK_DIR, the broker's own, mode 0600, encrypted under its token: it survives a restart, and a new token makes a new key. Nothing needs configuring and no volume is added. A value reaches OpenTofu in a 0600 variables file of the attempt, written by the worker and removed after the last apply; it is redacted from every log line, and a plan made with it reports no digest. OpenTofu keeps the value in the resource's state, in your own state storage, as it does for any secret it manages. The platform hands such an import only to an agent that advertises sealed-values, from the release that carries migration 0253. The Library and minAgentVersion are 0.3.27's.

0.3.30 generates an S3 bucket's configuration, and runs a generated template as its type's runner (2026-10-14). OpenTofu writes a bucket's bucket_prefix beside bucket, because the provider reads it back, and the two conflict, so the generating plan failed for every bucket. When the plan fails on conflicting arguments alone, the agent drops the argument the provider computes. Where both are computed, it drops the one that does not hold the import identifier. It names each in the log and plans the result again under the import rules. Any other conflict still fails the step as OpenTofu said. With runner roles or accounts set, a generated template's task runs as the runner the platform names in runnerSlug (§3.1.1). An older agent uses the template's slug, which names no runner, so the request fails naming it. The Library, the capabilities and minAgentVersion are unchanged.

0.3.31 generates an Azure storage account's configuration (2026-10-14).

• The image's provider mirror carries azurerm 5.9.0.
• OpenTofu's environment admits azurerm's federation variables (§3.1.2, agent-install.md §2). The agent refuses to start beside a client secret.
• The provenance stamp lands on each azurerm resource's own tags, since azurerm has no default tags.
• Generation leaves out an argument that the provider read back as a value its own validation refuses: an unset "", 0 or -1, such as a lifecycle policy's unused tiering days.
• Where the provider plans a value for an argument left out unless its deprecated predecessor is set (azurerm's public_network_access), the agent writes the predecessor at the value the import read. It names each one in the log.
• With PANORAMA_RUNNER_AZURE_CLIENT_IDS set, it signs in as the runner, through exactly one federation mode, checked before OpenTofu runs. Every azurerm provider configuration is held to use_cli = false and use_msi = false (§3.1.2).

The Library, the capabilities and minAgentVersion are unchanged.

0.3.32 keeps an AWS runner role's state beneath its template's slug (2026-10-14). With PANORAMA_RUNNER_ROLE_ARN_PREFIX set, a create writes <PANORAMA_STATE_PREFIX>/<template slug>/<request id>.tfstate, the prefix library-runner-roles.yaml holds each role to, as Google's runner accounts have been since then (§3.1). A destroy of a create an earlier agent wrote without the slug is accepted at that key. Without the setting the layout is unchanged. The Library, the capabilities and minAgentVersion are unchanged.

0.3.33 sweeps after each task as well as before it (2026-10-15). A worker that dies mid-task leaves the attempt's directory holding its own HOME and TMPDIR, which only the restarted worker can remove. When the broker claimed its next task before the worker was back, the sweep before that task ran first and the empty directory stayed until the task after. The sweep after the task now removes it. The Library, the capabilities and minAgentVersion are unchanged.

0.3.34 stops when told to while it holds a request (2026-10-15). A worker that received SIGTERM or SIGINT just as it read the broker's next request, before serving it, could wait forever instead of exiting, until the container runtime killed it, skipping its own clean-up of the attempt's directory (the restarted worker's sweep still removed it). It now exits as it does at any other moment. The Library, the capabilities and minAgentVersion are unchanged.

0.3.35 keeps state in Azure Blob Storage (2026-10-19), with PANORAMA_STATE_BACKEND=azurerm, PANORAMA_STATE_STORAGE_ACCOUNT and the container in PANORAMA_STATE_BUCKET. It locks with a lease on the state blob, and signs in through Entra ID alone (§7.1). Beside that:

• It refuses to start beside an Azure storage key, SAS token or connection string, whatever the backend.
• A template override file that declares a backend or the cloud block is refused before init, on every backend. Override files apply in name order, so one named after backend_override.tf would have replaced the agent's backend.
• A run that meets a held lock, on any backend, fails with a message naming the lock and §7.4, not the cloud account's general one.

The Library, the capabilities and minAgentVersion are unchanged.

0.3.36 holds who a template signs in as and what it reaches (2026-10-25), and the Library's release 28 arrives with it: the first Azure template, azure-storage-account-private (§3.1.2). It refuses templates that 0.3.35 ran. A refused request fails as configuration before OpenTofu plans anything: in prepare, or after init for a module the template calls. Its page shows the agent's message, which names the file and what it sets, for example:

[prepare/configuration] template main.tf sets profile in provider "aws"; who signs in to AWS is the agent's: the template's runner, or the agent itself where it has none: source identity mismatch

Refused from 0.3.36, on every cloud (§3.1, §3.1.1, §3.1.2):

• An aws or awscc provider block that sets profile, assume_role, assume_role_with_web_identity, access_key, secret_key, token, shared_config_files or shared_credentials_files, and role_arn in awscc; a google or google-beta one that sets credentials, access_token, impersonate_service_account, its _delegates or external_credentials. Any value counts, an empty one included.
• An azurerm provider block that sets client_id or its file, tenant_id, auxiliary_tenant_ids, use_oidc, an OIDC token, its file or its request settings, use_aks_workload_identity, ado_pipeline_service_connection_id, environment, metadata_host, msi_endpoint or msi_api_version. 0.3.35 refused only a secret, a certificate, and use_cli or use_msi other than false.
• A template override file that configures aws, awscc, google, google-beta or azurerm at all.
• aws, google, google-beta or azurerm required under another local name, and a .tofu file named after one the agent writes, such as backend_override.tofu.
• The terraform_remote_state data source, whatever its backend. A template that needs another's outputs takes them as variables.
• In a module the template calls, any of these, and an azurerm provider configuration of its own.

On Azure, every azurerm provider configuration also runs in the request's subscription, over whatever the template sets. A request for an account that is not a subscription id fails in prepare when its template configures azurerm.

With Azure runners, prepare before you upgrade. 0.3.36 does not start with PANORAMA_RUNNER_AZURE_CLIENT_IDS set and without PANORAMA_RUNNER_AZURE_TOKEN_DIR on the broker: it exits 2, and its configuration log line begins missing required setting: PANORAMA_RUNNER_AZURE_TOKEN_DIR. The one shared token 0.3.35 signed every runner in with is no longer accepted. A worker serving Azure runners also refuses to start, exit 2, when it can reach what mints the runners' tokens, naming each source it found (the table in §3.1.2). The shared token file is one, and so is every container on ECS. So, before the new image starts:

1. Move an agent on ECS that serves Azure runners to Kubernetes, Docker Compose or systemd, laid out as §3.1.2 says.
2. Mint one token per runner into a directory only the broker reads, from an identity out of the worker's reach, and check that each file appears.
3. Move to the new image in the order "One token per runner" gives: stop the agent, set the directory on the broker, take the shared token file and its variable off the worker, apply the runners module again, then start. Requests queue on the platform while it is stopped.

Nothing changes for an agent on AWS or Google that runs only Library templates, or templates Import generates. None of them sets anything the rules above refuse, and an agent without Azure runners is not probed. Only your own templates, and the modules they call, can be refused. Read them for the arguments above before upgrading.

A request for the release-28 template waits, queued, for an agent on 0.3.36 or later. The protocol, the capabilities and minAgentVersion are unchanged.

0.3.47 carries OpenTofu 1.12.9 (2026-10-28), in place of 1.12.6: 1.12.6 was built with Go 1.26.6, and the image's vulnerability gate refused it for CVE-2026-56857, a standard-library defect fixed in Go 1.26.9. 1.12.9 is built with Go 1.26.10. A patch release of the same line: every Library template requires OpenTofu 1.8.0 or 1.11.0 and later, and the providers the -mirror image carries are unchanged. If you run a bare binary with your own tofu (§1), nothing changes for you. The protocol, the capabilities and minAgentVersion are unchanged.

Pin the image by version and digest (panorama-agent:X.Y.Z@sha256:…, the pull of the version's release record) and verify it before the upgrade, as agent-install.md §1.1 does; published images are never rebuilt or retagged, and there is no latest. One digest serves linux/amd64 and linux/arm64 alike. Rolling back is the same act as upgrading, with the previous version — which keeps working as long as it is at or above the minimum.

Your own Azure templates need the runners scoped to resource groups (2026-10-27). From the release that carries it, an agent with Azure runners at the subscription, or on an Azure account without runners, refuses a request for a template of your own (git or inline) in prepare, as configuration, before fetching it, and names PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS (§3.1.2). Library templates and the templates Import generates run as before. Nothing is refused today that ran before: the platform offers no template of a tenant's own to an Azure account yet. Before you add your own Azure templates, apply the runners module with runner_resource_groups and set the file on the broker. Library-only agents are to be scoped from 2027-01-31, announced in the next note. The protocol and minAgentVersion are unchanged. A scoped agent sends one more capability, azure-resource-groups, which a platform that does not act on it ignores.

Library-only Azure agents are to be scoped from 2027-01-31 (the owner 2026-10-27). From the release that carries this note, an agent with Azure runners at the subscription (PANORAMA_RUNNER_AZURE_CLIENT_IDS set, PANORAMA_RUNNER_AZURE_RESOURCE_GROUPS not) logs a warning each time its broker starts, naming 2027-01-31 and §3.1.2; it runs as before. The first agent release on or after 2027-01-31 refuses to start in that state, exit 2, naming the setting. The date is the release's and never the clock's, so no agent stops on that day by itself: it stops at the upgrade, if its runners are still subscription-wide. Before you upgrade to a release from that date on, apply the runners module with runner_resource_groups and set the file on the broker (§3.1.2). Agents without Azure runners, and scoped ones, see nothing. The protocol, the capabilities and minAgentVersion are unchanged.

0.3.46 generates the configuration of the other six Azure types Import adopts (2026-10-28): a virtual network, a Linux machine, a Linux function app, a PostgreSQL flexible server, a SQL database and an AKS cluster. azurerm reads an argument the resource never set back as its zero ("", 0, false). OpenTofu writes that zero into the configuration it generates, and azurerm's own validation then refuses it, alone or beside the argument it conflicts with. 0.3.45 failed the generate step for every one of the six.

• Generation resolves each conflict, each group of which only one argument may be set, and each argument set without the rest of its group, by what the resource set. An argument at its zero is left out. One the provider computes, beside one it does not, is left to the provider. A false beside an argument that is set is left out.
• A value refused where the error names no argument, such as an id that does not parse, is found at the error's position in the configuration and left out by the same rules. Inside an attribute that holds objects, such as a network's subnets, an empty member and one only the provider sets become null.
• The agent validates the result offline, and normalises again what that uncovers, up to four times. It names each change in the log. Anything else still fails the step as OpenTofu said, and the result is planned under the import rules, as before.

Apply the runners module again (§3.1.2): the generated SQL database runner now reads the database operation's status, which its update and delete poll. The Library, the protocol, the capabilities and minAgentVersion are unchanged.

Across the panorama rename (platform release 174). From that release the platform no longer parses a token issued before it, so issue the agent a new one (§2) when the platform upgrades. An agent already running takes it in the setting it has and carries on at its version. Its image and settings take their new names with its next image: panorama-agent, and settings named PANORAMA_*. The upgrade note (observability/docs/panorama-rename.md) lists what each was. Rename them in the same step as the image: the new image does not read the old names.

6. Revoke

Provisioning → Agents → Revoke, or DELETE /provisioning/agents/{agentId}.

Effective on the agent's next call: every subsequent request from that token is 401. The row is kept rather than deleted, because past tasks refer to it.

A task the revoked agent still holds is not finished by it — its writes are refused, and the scheduler reaps the task after the heartbeat timeout, which returns the request to queued. If you revoked because the host was compromised, note that revoking does not stop an apply already running on that host; it only stops the results being accepted. Terminate the host.

7. Recover state

Terraform state is yours (D3): it lives in your bucket, and the platform never reads it. Each create writes to s3://<PANORAMA_STATE_BUCKET>/<PANORAMA_STATE_PREFIX>/<request id>.tfstate — or, with PANORAMA_STATE_BACKEND=gcs, gcs://<PANORAMA_STATE_BUCKET>/<PANORAMA_STATE_PREFIX>/<request id>, where the backend keeps its own default.tfstate, or, with azurerm (from 0.3.35), azurerm://<PANORAMA_STATE_STORAGE_ACCOUNT>/<container>/<PANORAMA_STATE_PREFIX>/<request id>.tfstate, with the template's slug before the request id when the agent runs AWS runner roles (§3.1, from 0.3.32) or Google runner accounts (§3.1.1), and the runner's with Azure runners (§3.1.2) — and every request reports the location its agent used; the portal shows it on the request. This section is what to do when that state is lost, locked or half written. The drill ran every procedure here, and they are written from what worked.

7.1 Before you need it

• Versioning on the state bucket. A previous version is the only way back from a lost or overwritten state. Turn it on before the first apply. On Google Cloud Storage that is object versioning, and the agent refuses to start without it (gcloud storage buckets update gs://<bucket> --versioning).

• Keep noncurrent versions long enough to notice a mistake: a lifecycle rule that expires them after 30 days or more. Google's equivalent condition is daysSinceNoncurrentTime.

• Encryption at rest (SSE-S3 or SSE-KMS) and no public access. Google encrypts at rest unconditionally; there the two settings to check are uniform bucket-level access and public-access prevention, and the agent refuses to start without either (--uniform-bucket-level-access, --public-access-prevention).

• The lock table: DynamoDB, on demand, with a string hash key named LockID, named in PANORAMA_STATE_LOCK_TABLE. On Google Cloud Storage there is none: the backend locks with an object beside the state, under a generation precondition, so there is nothing to provision and nothing to name. PANORAMA_STATE_LOCK_TABLE is refused alongside PANORAMA_STATE_BACKEND=gcs rather than ignored.

• On Azure Blob Storage (PANORAMA_STATE_BACKEND=azurerm, from 0.3.35) there is no lock table either: the backend leases the state blob, with no expiry, while it writes. The agent signs in to the account through Entra ID alone, as the identity of §3.1.2's federation, and never with an account key or a SAS token. It refuses to start beside ARM_ACCESS_KEY, ARM_SAS_TOKEN, AZURE_STORAGE_KEY, AZURE_STORAGE_SAS_TOKEN or AZURE_STORAGE_CONNECTION_STRING, naming the variable. Make the account once, as the subscription's Owner, and then lock it:

  az storage account create -n <account> -g <group> -l <location> --sku Standard_LRS --kind StorageV2 \
    --allow-shared-key-access false --allow-blob-public-access false --min-tls-version TLS1_2 --https-only true
  az storage account blob-service-properties update -n <account> -g <group> --enable-versioning true \
    --enable-delete-retention true --delete-retention-days 30 \
    --enable-container-delete-retention true --container-delete-retention-days 30
  az storage container-rm create --storage-account <account> -g <group> -n <container> --public-access off
  # grants (§3.1.2's runners module, or the agent's own role below) before the lock
  az lock create --name state-read-only --lock-type ReadOnly -g <group> \
    --resource-type Microsoft.Storage/storageAccounts --resource <account>
  

  • Shared-key access off means a key or a SAS token is refused by the account itself, not just by the agent.
  • Versioning and blob soft delete are what §7.3 restores from. The agent does not check them at start, as it does on Google Cloud Storage, so check them here.
  • The ReadOnly lock is there because a runner such as generated-azure-storage-account holds storageAccounts/write and delete across the subscription, which would let it change or delete the state account. The lock refuses both to everyone, until someone with Microsoft.Authorization/locks/delete, which no runner holds, removes it. It does not touch the data plane, which is all the backend uses. While it is on, the container and any role assignment beneath the account cannot be changed, so make grants at the account's resource group.
  • Without runners, the agent's own identity needs Storage Blob Data Contributor on the container. With them, each runner needs only §3.1.2's state role; the agent's identity needs none.

• Whoever recovers needs more than the agent's role. On AWS: s3:ListBucketVersions, s3:GetObjectVersion and s3:PutObject on the bucket, and dynamodb:GetItem, dynamodb:PutItem and dynamodb:DeleteItem on the table. On Google: roles/storage.objectAdmin on the state bucket — it lists and reads noncurrent versions, writes a restored state and deletes the lock object. The agent's own service account needs that same object access plus one bucket permission, storage.buckets.get, which is what the four startup checks read (roles/storage.legacyBucketReader grants it on its own); it never needs roles/storage.admin. That identity belongs to the worker, not the broker: the worker is the process that runs the checks and writes the state, and the broker needs no cloud credential at all (agent-execution-boundary.md §6). A worker that cannot prove the bucket refuses the broker's first word, so the agent stops before it claims anything and the reason is in the worker's log.

7.2 A recovery workspace

Every procedure below runs OpenTofu the way the agent did, against the same state and the same configuration. Beside the template's source, the agent writes three files: the backend (with encrypt = true), the provenance stamp merged into every aws provider block, and the approved variables. A workspace rebuilt without them either cannot plan (The root module input variable "…" is not set) or plans against different provider tags. A clean plan there proves nothing. So rebuild it with the agent's own code:

1. An empty directory with the template's source at the request's version: the commit the request ran (its version's git SHA), or the inline source, or — for a Library template — the version's main.tf, which its page shows read-only: the same bytes the agent's image carries under the version's hash, and the only one of the template's files OpenTofu reads.

2. The approved variable values, as a JSON object in a file. The non-sensitive ones are on the request's page. The sensitive ones come from wherever the requester took them: the platform shows them as [redacted] and never returns them, and [redacted] is not a value.

3. The generated files, written by the agent's own code:

   panorama-agent recovery-workspace --dir "$PWD/recovery" \
     --request-id <id> --template <slug> --template-version <n> --team <slug> \
     --variables vars.json --state-location <the request's state location> \
     --state-region <region> --lock-table <table>
   

   --request-id is the request that created the resources: a create's own id, or for a destroy the create it destroys. The stamp names the creator. --team is the request's team slug, from Org → Teams (the request's page shows the team's name). Every create and destroy has a team, and the stamp carries it. The source may be .tf or .tf.json files. The command contacts nothing; it only writes the three files.

   The location decides the backend, so a gcs://bucket/prefix location takes neither --state-region nor --lock-table — leave both off; they are S3's, and a GCS backend block has no place to put them.

4. tofu init with the same OpenTofu release. It reads the state without locking it.

7.3 A lost or overwritten state version

Symptom. tofu plan in the recovery workspace wants to create what already exists. A retry in that state would create it a second time, so do not retry yet.

1. List the key's versions and find the last good one: aws s3api list-object-versions --bucket <bucket> --prefix <key>.

2. Download it, look at it, and upload it back as the latest version:

   aws s3api get-object --bucket <bucket> --key <key> --version-id <id> restored.tfstate
   aws s3 cp restored.tfstate s3://<bucket>/<key>
   

   Looking first is the point of downloading. The file is JSON, and its resources should be what the account holds. AWS also restores a version by copying it onto the same key (copy-object with a versionId), but that puts back a file nobody has read.

3. tofu plan then refuses: state data in S3 does not have the expected content … update the Digest value stored in the DynamoDB table to the following value: <md5>. The lock table keeps the MD5 of the last state OpenTofu wrote, which is the bad one. Check that the value it names is your restored file's (md5 -q restored.tfstate on macOS, md5sum on Linux), then set it:

   aws dynamodb put-item --table-name <table> \
     --item '{"LockID":{"S":"<bucket>/<key>-md5"},"Digest":{"S":"<md5>"}}'
   

4. tofu plan -detailed-exitcode must exit 0, meaning no changes. Only then retry or destroy anything.

On Google Cloud Storage the state is <prefix>/default.tfstate and every write leaves a noncurrent generation, so the same three steps read:

gcloud storage ls --all-versions --long "gs://<bucket>/<prefix>/default.tfstate"
gcloud storage cp "gs://<bucket>/<prefix>/default.tfstate#<generation>" restored.tfstate
gcloud storage cp restored.tfstate "gs://<bucket>/<prefix>/default.tfstate"

Step 3 has no equivalent and is skipped: the digest OpenTofu compares on S3 is kept in the lock table, and the GCS backend keeps no digest anywhere — it reads the object it finds. Step 4 is unchanged and still the thing that decides whether the restore worked.

On Azure Blob Storage (from 0.3.35) the state is the blob <prefix>/<request id>.tfstate (the runner before the request id with Azure runners), and each write leaves a version. As someone with Storage Blob Data Contributor on the container, signed in with az login:

az storage blob list --account-name <account> -c <container> --prefix <key> --include v --auth-mode login \
  --query "[].{version:versionId,current:isCurrentVersion,modified:properties.lastModified}" -o table
az storage blob download --account-name <account> -c <container> -n <key> --version-id <version> -f restored.tfstate --auth-mode login
az storage blob upload --account-name <account> -c <container> -n <key> -f restored.tfstate --overwrite --auth-mode login

The upload is refused while the blob is leased: §7.4 first. Step 3 is skipped, as on Google: the azurerm backend keeps no digest. Step 4 is unchanged.

7.4 A stale lock

Symptom. A retry fails as infrastructure, and from 0.3.35 the request's message says so: "the state is locked (lock <ID>, taken for <operation> at <time>): another run holds it, or a run stopped outright left it behind. The agent never breaks a lock". Before 0.3.35 the message was only "the cloud account refused or failed; check IAM, quotas and the state backend". Either way, the attempt's log, on the request's task in the portal, has Error acquiring the state lock, then Lock Info: with the lock's ID, who took it, the operation and when. The agent never runs force-unlock itself: it cannot tell a lock a live run holds from one a killed run left, and breaking the first lets two runs write one state. A lock is left behind when OpenTofu is killed outright: a crash, kill -9, or a runtime that stops the agent before its drain (§5).

1. Make sure nothing holds it. The task that took the lock is finished in the portal, and no tofu process runs on the agent's host. A lock a live apply holds is protecting your state; removing it lets two applies write at once.

2. Look for resources outside state. OpenTofu does not write state after every resource, so an apply killed outright may have created resources its state never recorded, and the agent then reports no partial apply. In the drill, an apply killed 25 seconds in left an empty state. Look in two passes:

   • The stamp, first. aws resourcegroupstaggingapi get-resources --tag-filters Key=panorama:request_id,Values=<request id> lists what carries it. This is not an inventory. A resource type that cannot carry tags does not appear, and neither does one an apply created but had not yet tagged.
   • Then what the template makes. Read the template for the resource types it creates, and list each of them in the account and region. Check CloudTrail for the agent role's calls in the request's time window.

   Import what belongs in state, or remove it, before retrying: a retry creates it again.

3. In the recovery workspace: tofu force-unlock -force <lock ID>.

4. Retry the request. OpenTofu applies what the state does not hold.

On Google Cloud Storage the shape is the same and two details differ. There is no lock table: the lock is an object beside the state, <prefix>/default.tflock, written under a generation precondition — which is why nothing had to be provisioned for it. And the stamp is a label, so step 2's first pass reads:

gcloud asset search-all-resources \
  --scope=projects/<project> --query='labels.panorama_request_id:<request id>'

with the same caveat: it is not an inventory. A resource type that cannot carry labels does not appear, and neither does one an apply created but had not yet labelled. The second pass is the same reading of the template, with Cloud Audit Logs in place of CloudTrail. tofu force-unlock in step 3 is unchanged — it removes the lock object — and the permissions it needs are the Google ones in §7.1, not the S3 and DynamoDB ones.

One detail worth knowing before you type it: the lock ID this backend wants is the lock object's generation, the number the error's Lock Info prints as ID. The object's own contents name a different, UUID-shaped id, and force-unlock refuses that one — "Lock ID should be numerical value". Copy the ID from the error, or read it from the object's metadata (gcloud storage ls --long --all-versions), never from inside the file.

On Azure Blob Storage (from 0.3.35) the lock is a lease on the state blob itself, with no expiry: a killed apply leaves the blob leased until someone releases it. Every later attempt fails with the same lock ID, the lease's, which the request's message gives. The lock's details are in the blob's metadata (terraformlockid), and az storage blob show --auth-mode login shows properties.lease.state as leased. Step 2's first pass is az resource list --tag panorama:request_id=<request id> (the stamp is a tag, on each resource's own tags), with the Activity Log in place of CloudTrail. Step 3 is tofu force-unlock -force <lock ID> in §7.2's workspace, signed in as the identity that holds the state grant: the runner's client id in ARM_CLIENT_ID, with the agent's token file. It releases the lease with that ID. Measured on 2026-10-19: an apply killed outright left the lease, two retries were each refused naming the same lease ID, and after force-unlock the retry ran.

7.5 An interrupted apply

An apply stopped part-way leaves resources behind. The agent reports the task with partialApply: true and the state location, and the portal shows the request failed as partial. Nothing is destroyed automatically (D10). Either:

• retry, which applies against the same state and converges on what the template describes; or
• destroy what was made, when it should not stay. The request's Request destroy asks for a destroy of what the create left: a create that stopped part-way has a provisioned resource, status partial, naming the state it reported. The destroy goes through approval like any other and runs against that state. While it is requested or running, the create cannot be retried, and while a retry is queued or running, a destroy is refused. Without the portal, §7.2's workspace and tofu destroy still work, followed by §7.4's two passes, which must find nothing.

In the drill, the agent stopped 3 seconds after SIGTERM and OpenTofu released its lock. The request showed cancelled with partialApply: true, and a retry with the agent restarted converged on what the template describes.

If the agent itself was killed, the report may be lost too. The platform then reaps the task after the heartbeat timeout, and the next attempt can meet the stale lock of §7.4.

7.6 After a platform restore

The platform's own database restore has its own procedure (the runbook's "Backing up and restoring the database"). Its panorama admin after-restore fails every request that was in flight when the backup was taken, with the class restored, and lists each one's state location. Check each one against its account with §7.2's workspace before retrying it.

Measured (2026-09-12, on LocalStack).

• A lost version: back, with a clean plan, 14 seconds after the restore's first command.
• A stale lock: 102 seconds from the kill to a converged retry, including one retry that met the lock.
• An interrupted apply: 91 seconds from the agent's stop to a converged retry, including restarting the agent.

---