Provisioning agent

Install, configure, verify, upgrade and revoke the agent that runs OpenTofu in your account — and what it can never do.

Audience: whoever installs the reference agent — agent/ in this repository — inside an AWS account. This is the reference implementation's own manual: what it reads, what it writes, and how it is shipped. The account-level operations — issuing a token, verifying, upgrading, revoking — are in agent-operations.md and apply to any conforming agent, this one included.

Status: written by p4_5 on 2026-09-03, with the binary.

---

1. Get it

Two forms, both built from agent/:

The agent is two roles of one binary and one image (D18, platform/docs/agent-execution-boundary.md): panorama-agent broker holds the platform token, claims, heartbeats, streams logs and reports, writes the generated files, and runs no customer code; panorama-agent worker — another user, no token — checks the source out, runs OpenTofu and reads state, and keeps its process space to itself between attempts (it makes itself unreadable to its own user, kills everything else of its user before and after every attempt, and refuses to begin one unless it is alone). The two meet over a Unix socket in the shared work directory, one connection per attempt. The image carries two users, broker (uid 1000) and worker (uid 1001), in one group (panorama, gid 2000), git for git-sourced templates, and exposes /healthz on :8091 from the broker. panorama-agent version prints the build's version, which is what register reports.

panorama-agent unisolated runs both halves in one process. It is for development and the acceptance tests only, refuses to start unless PANORAMA_DEVELOPMENT_UNISOLATED=1 names it, and is never a supported deployment: in it, customer code runs in the process that holds the token.

2. Configure

Everything is environment variables. The broker reads all of them; the worker reads only PANORAMA_WORK_DIR, PANORAMA_TOFU_BIN, PANORAMA_STATE_READ_TIMEOUT, PANORAMA_BROKER_UID, PANORAMA_GIT_CONFIG and PANORAMA_ALLOWED_GIT_HOSTS, and refuses to start if PANORAMA_AGENT_TOKEN or PANORAMA_PLATFORM_URL is set in its environment. Three are mandatory for the broker; the S3 backend settings are mandatory for production, and everything else has a default.

Cloud credentials come from the environment the agent runs in — instance profile, IRSA, task role, or the AWS_* variables your operators set — and from nowhere else. There is no setting for one, and the platform never sends one.

2.1 What OpenTofu is allowed to see

The agent builds OpenTofu's environment from nothing and passes through only (PROVISIONING_SPEC.md §7.6):

PATH HOME TMPDIR TZ
AWS_REGION AWS_DEFAULT_REGION AWS_PROFILE
AWS_CONFIG_FILE AWS_SHARED_CREDENTIALS_FILE
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI AWS_CONTAINER_CREDENTIALS_FULL_URI
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
AWS_WEB_IDENTITY_TOKEN_FILE AWS_ROLE_ARN AWS_ROLE_SESSION_NAME
TF_IN_AUTOMATION TF_INPUT TF_DATA_DIR

So a template's local-exec never sees PANORAMA_AGENT_TOKEN, PANORAMA_PLATFORM_URL, or anything else you set for the agent — and it also never sees static AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pairs, which are not on the list on purpose. Give the agent a role, not a key pair. If your setup can only provide static keys, put them in a shared credentials file and point AWS_SHARED_CREDENTIALS_FILE at it; the SDK inside OpenTofu reads the file, and the values are never in the environment a template could dump.

2.2 What git is allowed to see

Every attempt gets a HOME and a TMPDIR of its own, inside its directory under PANORAMA_WORK_DIR, and both go with it. That is what a template's local-exec, git and OpenTofu see as home for the attempt: whatever a template writes there — a .gitconfig naming a hook directory, a credential helper, a provider plugin cache — is gone before the next attempt begins, and the worker's real home directory is root's and read-only. Git inside an attempt runs with no global configuration except the file named by PANORAMA_GIT_CONFIG, and is told, per invocation, that exactly the attempt's directory is a repository it may work in although the broker's user created it (safe.directory); nothing a template can write changes either.

Private template repositories therefore get their credentials through PANORAMA_GIT_CONFIG and nowhere else: mount a git configuration file read-only into the worker and name it there. A credential helper it names runs as the worker, so the secret it reads must be readable by the worker's user — and by nothing a template can reach: a mounted secret file, not a file under the work directory.

Resolving a template's ref to a commit (git ls-remote, when a request names a branch or a tag) runs in the worker for the same reason: the broker never runs git against a template's URL.

What git may fetch (p7_46). The platform checks a template's URL when it is published; the agent checks it again before any git process starts, because a stored row can change after that. Refused, whatever the host: any scheme but https:// and ssh:// (and git's user@host:path form) — so no git://, http://, file://, local path or ext:: helper — a credential inside the URL (before the @, or in a query or fragment), and a URL that begins with -. The log line naming the URL shows only its scheme, host and path. Refused by address: a host that is, or resolves to, link-local (where the cloud metadata service lives), loopback, the unspecified address or EC2's IPv6 metadata address — every address it resolves to, not the first. A private address is not refused: the agent runs in your network, and a git server on 10.0.0.0/8 is a normal thing to have. For the strict answer, name your hosts in PANORAMA_ALLOWED_GIT_HOSTS and nothing else is fetched. Git itself is also told to use only the https and ssh transports (protocol.allow=never with those two allowed), so a URL that got past the check would still stop at git.

The check resolves the host and git resolves it again, so a name you do not control can answer differently the second time. PANORAMA_ALLOWED_GIT_HOSTS is the setting for that; the worker's IAM role is the ceiling either way.

Where a destroy may point (p7_46). A create writes state at exactly <PANORAMA_STATE_PREFIX>/<requestId>.tfstate in PANORAMA_STATE_BUCKET, and reports that location; a destroy runs against the location its create reported. The agent refuses a destroy whose location is not exactly the one its own create wrote — <PANORAMA_STATE_PREFIX>/<createRequestId>.tfstate in PANORAMA_STATE_BUCKET, or <PANORAMA_WORK_DIR>/state/<createRequestId>.tfstate for local state — and one that does not say which create it undoes: another request's state beside it, another bucket or prefix, a nested key, a traversal, a local file. It is a configuration failure before anything is fetched, so a wrong or tampered location cannot tear down another request's state. The worker's IAM role is still the real limit: grant it state access to that bucket and prefix only.

3. Run

Two processes, two users, one shared work directory — whatever runs them. The work directory belongs to the broker's user and the shared group, mode 2770, so the broker writes an attempt's files and the worker reads and executes them; a template's local-exec runs as the worker and cannot see the broker's process, environment or home. The worker must have a user of its own: it sweeps every other process of that user before and after every attempt, and refuses to start if any other process of its user exists.

Run both with a read-only root filesystem: the only writable place is the work directory, and everything git or OpenTofu later trusts — the binaries, the home directories, /etc — stays what the image shipped.

Docker Compose — the reference layout, agent/deploy/compose.yaml: a volume-init service that sets the volume's owner and mode once, a broker service (uid 1000) with the platform URL and token, a worker service (uid 1001) with neither, both read_only: true, restart: always and stop_grace_period: 10m (below). Fill in the environment and docker compose up -d.

Give both processes ten minutes to stop (p7_46). A stop in the middle of an apply is a drain, not a kill: OpenTofu gets an interrupt and up to 20 s to finish its in-flight calls and write state, the broker then reads the state (up to PANORAMA_STATE_READ_TIMEOUT, 5 minutes by default) to say whether anything was created, and sends the report (up to 2 minutes). Heartbeats continue throughout, so the platform does not reap the task. A runtime that kills sooner — Docker's default is 10 s — loses that report: resources created outside state, the state lock still held, the retry failing on it. Set it everywhere the agent runs: stop_grace_period: 10m (Compose), --stop-timeout 600 (docker run), terminationGracePeriodSeconds: 600 (Kubernetes), TimeoutStopSec=600 on both units (systemd). ECS caps stopTimeout at 120 s, less than a full drain: on ECS, stop the agent only when the Agents page shows no running task.

Plain docker run — the same three steps by hand:

docker volume create panorama-work
docker run --rm --user 0:0 -v panorama-work:/var/lib/panorama-agent/work --entrypoint /bin/sh panorama-agent:0.2.0 \
  -c 'chown 1000:2000 /var/lib/panorama-agent/work && chmod 2770 /var/lib/panorama-agent/work'
docker run -d --name panorama-worker --restart always --stop-timeout 600 --user 1001:2000 --read-only \
  -v panorama-work:/var/lib/panorama-agent/work -e PANORAMA_BROKER_UID=1000 -e AWS_REGION=eu-west-1 \
  panorama-agent:0.2.0 worker
docker run -d --name panorama-broker --restart always --stop-timeout 600 --user 1000:2000 --read-only \
  -v panorama-work:/var/lib/panorama-agent/work \
  -e PANORAMA_PLATFORM_URL=https://acme.ccc.example -e PANORAMA_AGENT_TOKEN=... -e PANORAMA_WORKER_UID=1001 \
  -e PANORAMA_STATE_BUCKET=acme-terraform-state -e PANORAMA_STATE_REGION=eu-west-1 \
  -e PANORAMA_STATE_LOCK_TABLE=acme-terraform-locks \
  -p 127.0.0.1:8091:8091 \
  panorama-agent:0.2.0 broker

Cloud credentials are the worker's: under an ECS task role or IRSA the worker container picks them up; on EC2 the instance profile does the same. The broker needs none and, where the runtime cannot withhold them (a task role, an instance profile), holds credentials it does not use.

ECS (EC2 and Fargate): one task definition, two containers from the image with user set to 1000:2000 and 1001:2000, one shared volume mounted at /var/lib/panorama-agent/work in both, the worker with a container restart policy. Leave pidMode unset: the default keeps the two PID namespaces apart.

Kubernetes: one pod, two containers, an emptyDir at the work path in both, securityContext.runAsUser 1000 and 1001, fsGroup 2000, shareProcessNamespace left false, restartPolicy: Always, terminationGracePeriodSeconds: 600. Project the IRSA service-account token into the worker container only.

Bare binaries, systemd — two units, two users, panorama-broker and panorama-worker, one group panorama, the work directory chown panorama-broker:panorama and chmod 2770:

# panorama-broker.service
[Service]
User=panorama-broker
Group=panorama
EnvironmentFile=/etc/panorama-agent/broker.env      # PANORAMA_PLATFORM_URL, PANORAMA_AGENT_TOKEN, PANORAMA_WORK_DIR, PANORAMA_WORKER_UID, the state settings
ExecStart=/usr/local/bin/panorama-agent broker
Restart=always
RestartSec=5
KillSignal=SIGTERM
TimeoutStopSec=600
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/panorama-agent/work

# panorama-worker.service
[Service]
User=panorama-worker
Group=panorama
EnvironmentFile=/etc/panorama-agent/worker.env      # PANORAMA_WORK_DIR, PANORAMA_BROKER_UID, AWS_REGION; never the token
ExecStart=/usr/local/bin/panorama-agent worker
Restart=always
RestartSec=5
KillMode=control-group
TimeoutStopSec=600
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/panorama-agent/work

panorama-worker must be the only process of its user, ever: the worker sweeps that user before and after every attempt. KillMode=control-group makes a restart the same sweep, performed by systemd.

SIGTERM to the broker interrupts the task in flight at its next safe point, reports it as cancelled by the agent (the platform records that as a failed request of class cancelled; nothing is destroyed), wipes the workspace, and exits — heartbeating until the report is sent. TimeoutStopSec=600 covers the drain (above). SIGTERM to the worker ends the attempt on its side; the broker reports it as an infrastructure failure and the runtime restarts the worker, which sweeps before it listens again.

4. What it does with a task

agent-protocol.md §3 is the loop; this is what it leaves on disk, which is what you will see if you look inside PANORAMA_WORK_DIR mid-task:

worker.sock                       the worker's rendezvous, present between attempts and gone the moment the broker has connected
task-<taskId>-<random>/           mode 2770: the broker's, readable and writable by the worker's group
  main.tf …                       the template — checked out at its SHA by the worker, or the inline bytes after verification
  backend_override.tf             your S3 (or local) backend; wins over any backend the template declares
  ccc_provenance_override.tf      provider "aws" default_tags carrying panorama:request_id and friends (§9): an override, merged into your template's own provider blocks — default and every alias — with your default_tags kept and the stamp merged last
  generated.auto.tfvars.json      the approved variable values, through a JSON encoder
  panorama.tfplan                      the saved plan apply consumes
  .terraform/                     providers and modules, private to this task

The directory is removed when the task ends — success, failure, cancel, supersede, or the agent stopping. To recover a task's state later, panorama-agent recovery-workspace writes the three generated files again beside the source, with this same code (agent-operations.md §7.2). With PANORAMA_STATE_BACKEND=local, state/ beside it holds the state files and is not removed.

A process that dies mid-task cannot remove it, so both roles sweep (p7_53). The worker, when it starts, removes whatever its own user left in any task-* or resolve-* directory — the attempt's .home and .tmp, which are the worker's alone and which the broker cannot enter; the broker, when it starts and before each task, removes every task-* and resolve-* directory, since it runs one task at a time and any it finds are a previous one's. Nothing else under PANORAMA_WORK_DIR is touched. What the broker cannot remove is logged and goes once the worker has restarted.

.terraform/ is per task on purpose: every attempt downloads its providers again. A plugin cache shared between attempts would let one template leave a provider binary for the next to run, which is what the execution boundary exists to prevent. The cost is the download: the AWS provider, 6.64.0 for linux/amd64, is 183 MB (2026-09-10), fetched from registry.opentofu.org and GitHub on every create and destroy. A read-only mirror you provide would remove it; it is not supported yet (p7_53 records the decision).

An output line is carried from the worker to the broker whole up to about 672 KiB and cut beyond that, before the broker scrubs it and cuts it to the platform's line budget; a larger line once broke the channel and was reported as the worker going away.

Logs are scrubbed before they leave the host (PROVISIONING_SPEC.md §7.5): the exact values of every variable the template's schema marks sensitive, AWS access key ids, password=/secret=/token=-shaped assignments, and PEM blocks. Resource ids, ARNs and hashes are left alone.

5. Health

GET /healthz on PANORAMA_HEALTH_ADDR:

{"registered": true, "agentId": "…", "accountId": "111122223333",
 "uptimeSeconds": 8123,
 "current": {"taskId": "…", "action": "create", "stage": "apply",
             "startedAt": "…", "stageDurationsMs": {"prepare": 812, "init": 4210, "plan": 9021}},
 "last": {"taskId": "…", "action": "create", "stage": "apply", "outcome": "succeeded",
          "stageDurationsMs": {"prepare": 640, "init": 3890, "plan": 8770, "apply": 61220}}}

503 until registered; 200 after, whether or not a task is running. Point a liveness probe at it. It is unauthenticated and carries ids only, so bind it to localhost or the pod network, not to the world.

6. Exit codes

A 403 (account disabled, module not entitled) is not an exit: the agent polls slowly until somebody fixes it on the platform side, as agent-protocol.md §7 says it should.

7. Testing your installation without touching the cloud

make test-acceptance in agent/ runs the tests that need a real OpenTofu — the environment-allowlist test among them — against the null provider with a local state backend: no AWS account, no cost. It needs tofu (or PANORAMA_TOFU_BIN=terraform) and the network for the provider download.

Operating an agent

Audience: whoever installs and operates the agent inside your AWS account. If you are writing an agent, you want agent-protocol.md instead — that is the contract, and it is deliberately complete without this file.

Status: written by p4_1 on 2026-09-01, alongside the protocol.

---

1. Install

One agent per AWS account. That is not a recommendation — the token is issued for exactly one (tenant, account) pair, so an agent physically cannot claim work for another account, and running two agents for one account is supported (see agent-protocol.md §5.4) but buys nothing.

Where it runs. Anywhere in that account with outbound HTTPS and a role:

No inbound rule is ever needed. The agent polls us; we never call it. If somebody asks you to open a port for this, they are describing a different product.

What the role needs. Whatever your templates create, plus read and write on the S3 bucket and DynamoDB table you use for Terraform state. We do not publish a policy for the first half because we do not know what your templates do, and a policy we invented would either be too small to work or too large to defend.

2. Issue a token

A tenant admin, in the portal: Provisioning → Agents → Issue token, naming the account.

POST /provisioning/agents      {"name": "prod-eu-west-1", "accountId": "111122223333"}
201  {"agent": {...}, "token": "..."}

The token is shown once. We store only sha256(token); there is no endpoint that reads it back, and losing it means issuing a new one and revoking the old.

The account must already be in the registry and enabled, or this is 409 account_not_managed — an agent bound to an account nobody has enabled would poll forever for work that cannot be created.

3. Configure and start

Three settings for the broker, and nothing else is mandatory; the worker gets the work directory and no token (agent-install.md §3 has the two- process layouts):

PANORAMA_PLATFORM_URL=https://<your-tenant>.ccc.example
PANORAMA_AGENT_TOKEN=<the token from 2.2>
PANORAMA_WORK_DIR=/var/lib/panorama-agent/work

The work directory must exist before the agent starts, and the agent's user must be able to write to it. If either is not so, the agent stops at start-up with exit code 2 and a message naming the directory (p7_66).

The agent registers on start and logs the policy it was given. If it logs versionStatus: unsupported, it will register and then be refused work; upgrade it before doing anything else.

3.1 The reference agent

agent/ in this repository is the reference implementation; its own manual — every setting, the container image, what it leaves on disk, the exit codes — is agent-install.md. The end-to-end smoke that runs it with real OpenTofu against a disposable account is one command: observability/scripts/smoke-provisioning.sh (settings at the top of the file). Nightly or by hand; never per PR.

4. Verify

The agent is working when Provisioning → Agents shows a lastSeenAt inside the last minute and the version you expect. Submit and approve a request; it should move queued → running within one poll interval.

If a request sits in queued: the agent for that account is not running. Requests are account-scoped structurally, so an agent for a different account will never pick it up no matter how healthy it looks.

5. Upgrade

Report and warn; refuse only below a declared minimum.

• At or above minAgentVersion — the agent works. If it is behind the current release, the Agents page marks it outdated and nothing else happens.
• Below minAgentVersion — register still succeeds, deliberately, so that the agent appears in the portal with its version and you can see what to upgrade. claim answers 426 Upgrade Required naming the required version. The agent logs it and keeps polling; nothing is dropped, and the queue is picked up the moment it is upgraded.

Upgrading is: stop the agent, replace the image, start it — and the stop is a drain, not a formality. A task in flight when you stop the agent is interrupted, and OpenTofu, the state read and the report take up to ten minutes between them; the agent heartbeats throughout so the platform does not reap it, and reports the task cancelled, with whether anything was created. A runtime that kills the agent before that loses the report: resources created outside state, the state lock still held, and the re-queued attempt failing on the lock. So give both processes a ten-minute stop (agent-install.md §3 has the setting for each runtime), or, where the runtime cannot (ECS caps it at 120 s), wait for the Agents page to show no running task before stopping.

minAgentVersion moves rarely and never without notice. It exists for changes that cannot be made compatible, not for tidiness. That is the whole of the protocol's versioning: there is no separate protocol number to negotiate. An agent reports its version (panorama-agent version: X.Y.Z+<commit> — every build carries its commit after a +, which never ranks) and the platform admits it or names the minimum.

The move to 0.2.0 came without a notice period. No customer install existed, and the owner decided on 2026-09-12 to switch at once.

Pin the image by version (panorama-agent:X.Y.Z) or by digest; published images are never rebuilt or retagged, and there is no latest. Rolling back is the same act as upgrading, with the previous version — which keeps working as long as it is at or above the minimum.

Across the panorama rename (platform release 174, p7_69). From that release the platform no longer parses a token issued before it, so issue the agent a new one (§2) when the platform upgrades. An agent already running takes it in the setting it has and carries on at its version. Its image and settings take their new names with its next image: panorama-agent, and settings named PANORAMA_*. The upgrade note (observability/docs/panorama-rename.md) lists what each was. Rename them in the same step as the image: the new image does not read the old names.

6. Revoke

Provisioning → Agents → Revoke, or DELETE /provisioning/agents/{agentId}.

Effective on the agent's next call: every subsequent request from that token is 401. The row is kept rather than deleted, because past tasks refer to it.

A task the revoked agent still holds is not finished by it — its writes are refused, and the scheduler reaps the task after the heartbeat timeout, which returns the request to queued. If you revoked because the host was compromised, note that revoking does not stop an apply already running on that host; it only stops the results being accepted. Terminate the host.

7. Recover state

Terraform state is yours (D3): it lives in your bucket, and the platform never reads it. Each create writes to s3://<PANORAMA_STATE_BUCKET>/<PANORAMA_STATE_PREFIX>/<request id>.tfstate, and every request reports the location its agent used; the portal shows it on the request. This section is what to do when that state is lost, locked or half written. p7_63's drill ran every procedure here, and they are written from what worked.

7.1 Before you need it

• Versioning on the state bucket. A previous version is the only way back from a lost or overwritten state. Turn it on before the first apply.
• Keep noncurrent versions long enough to notice a mistake: a lifecycle rule that expires them after 30 days or more.
• Encryption at rest (SSE-S3 or SSE-KMS) and no public access.
• The lock table: DynamoDB, on demand, with a string hash key named LockID, named in PANORAMA_STATE_LOCK_TABLE.
• Whoever recovers needs more than the agent's role: s3:ListBucketVersions, s3:GetObjectVersion and s3:PutObject on the bucket, and dynamodb:GetItem, dynamodb:PutItem and dynamodb:DeleteItem on the table.

7.2 A recovery workspace

Every procedure below runs OpenTofu the way the agent did, against the same state and the same configuration. Beside the template's source, the agent writes three files: the backend (with encrypt = true), the provenance stamp merged into every aws provider block, and the approved variables. A workspace rebuilt without them either cannot plan (The root module input variable "…" is not set) or plans against different provider tags. A clean plan there proves nothing. So rebuild it with the agent's own code:

1. An empty directory with the template's source at the request's version: the commit the request ran (its version's git SHA), or the inline source.

2. The approved variable values, as a JSON object in a file. The non-sensitive ones are on the request's page. The sensitive ones come from wherever the requester took them: the platform shows them as [redacted] and never returns them, and [redacted] is not a value.

3. The generated files, written by the agent's own code:

   panorama-agent recovery-workspace --dir "$PWD/recovery" \
     --request-id <id> --template <slug> --template-version <n> --team <slug> \
     --variables vars.json --state-location <the request's state location> \
     --state-region <region> --lock-table <table>
   

   --request-id is the request that created the resources: a create's own id, or for a destroy the create it destroys. The stamp names the creator. --team is the request's team slug, from Org → Teams (the request's page shows the team's name). Every create and destroy has a team, and the stamp carries it. The source may be .tf or .tf.json files. The command contacts nothing; it only writes the three files.

4. tofu init with the same OpenTofu release. It reads the state without locking it.

7.3 A lost or overwritten state version

Symptom. tofu plan in the recovery workspace wants to create what already exists. A retry in that state would create it a second time, so do not retry yet.

1. List the key's versions and find the last good one: aws s3api list-object-versions --bucket <bucket> --prefix <key>.

2. Download it, look at it, and upload it back as the latest version:

   aws s3api get-object --bucket <bucket> --key <key> --version-id <id> restored.tfstate
   aws s3 cp restored.tfstate s3://<bucket>/<key>
   

   Looking first is the point of downloading. The file is JSON, and its resources should be what the account holds. AWS also restores a version by copying it onto the same key (copy-object with a versionId), but that puts back a file nobody has read.

3. tofu plan then refuses: state data in S3 does not have the expected content … update the Digest value stored in the DynamoDB table to the following value: <md5>. The lock table keeps the MD5 of the last state OpenTofu wrote, which is the bad one. Check that the value it names is your restored file's (md5 -q restored.tfstate on macOS, md5sum on Linux), then set it:

   aws dynamodb put-item --table-name <table> \
     --item '{"LockID":{"S":"<bucket>/<key>-md5"},"Digest":{"S":"<md5>"}}'
   

4. tofu plan -detailed-exitcode must exit 0, meaning no changes. Only then retry or destroy anything.

7.4 A stale lock

Symptom. A retry fails, and the request shows only the class infrastructure ("the cloud account refused or failed; check IAM, quotas and the state backend"). The cause is in that attempt's log, on the request's task in the portal: Error acquiring the state lock, then Lock Info: with the lock's ID, who took it, the operation and when. A lock is left behind when OpenTofu is killed outright: a crash, kill -9, or a runtime that stops the agent before its drain (§5).

1. Make sure nothing holds it. The task that took the lock is finished in the portal, and no tofu process runs on the agent's host. A lock a live apply holds is protecting your state; removing it lets two applies write at once.

2. Look for resources outside state. OpenTofu does not write state after every resource, so an apply killed outright may have created resources its state never recorded, and the agent then reports no partial apply. In p7_63's drill, an apply killed 25 seconds in left an empty state. Look in two passes:

   • The stamp, first. aws resourcegroupstaggingapi get-resources --tag-filters Key=panorama:request_id,Values=<request id> lists what carries it. This is not an inventory. A resource type that cannot carry tags does not appear, and neither does one an apply created but had not yet tagged.
   • Then what the template makes. Read the template for the resource types it creates, and list each of them in the account and region. Check CloudTrail for the agent role's calls in the request's time window.

   Import what belongs in state, or remove it, before retrying: a retry creates it again.

3. In the recovery workspace: tofu force-unlock -force <lock ID>.

4. Retry the request. OpenTofu applies what the state does not hold.

7.5 An interrupted apply

An apply stopped part-way leaves resources behind. The agent reports the task with partialApply: true and the state location, and the portal shows the request failed as partial. Nothing is destroyed automatically (D10). Either:

• retry, which applies against the same state and converges on what the template describes; or
• destroy what was made, when it should not stay. The request's Request destroy asks for a destroy of what the create left: a create that stopped part-way has a provisioned resource, status partial, naming the state it reported (p7_65). The destroy goes through approval like any other and runs against that state. While it is requested or running, the create cannot be retried, and while a retry is queued or running, a destroy is refused. Without the portal, §7.2's workspace and tofu destroy still work, followed by §7.4's two passes, which must find nothing.

In p7_63's drill, the agent stopped 3 seconds after SIGTERM and OpenTofu released its lock. The request showed cancelled with partialApply: true, and a retry with the agent restarted converged on what the template describes.

If the agent itself was killed, the report may be lost too. The platform then reaps the task after the heartbeat timeout, and the next attempt can meet the stale lock of §7.4.

7.6 After a platform restore

The platform's own database restore has its own procedure (the runbook's "Backing up and restoring the database"). Its panorama admin after-restore fails every request that was in flight when the backup was taken, with the class restored, and lists each one's state location. Check each one against its account with §7.2's workspace before retrying it.

Measured (2026-09-12, p7_63, on LocalStack).

• A lost version: back, with a clean plan, 14 seconds after the restore's first command.
• A stale lock: 102 seconds from the kill to a converged retry, including one retry that met the lock.
• An interrupted apply: 91 seconds from the agent's stop to a converged retry, including restarting the agent.

---