Running deploy
Page not available in that version
The current page Running deploy doesn't exist in version 1.6.2 of the documentation for this product.
Overview
deploy is on the installation ISO, at /mnt/esb3027/deploy. Run it as root on the first server in the inventory, with the ISO mounted at /mnt/esb3027. It reaches every other machine over SSH. For its commands, options, files and exit status, see the deploy reference.
To change what is installed, edit the files in /etc/acd/deploy and run deploy. Do not change the release with helm directly: the next deploy replaces it with what the configuration says.
What a Run Does
/mnt/esb3027/deploy
The first run installs; every later run re-applies your settings and changes only what differs. Running it again is safe, including after a failed run. deploy checks each machine after changing its network, and on a running cluster changes them one at a time, but a wrong network setting can still cut a machine off, and the run then stops with [ACD-NODE-UNREACHABLE].
Each run goes through fixed stages:
| Stage | What it does | Changes anything? |
|---|---|---|
| Gather inputs | Reads every machine and works out the settings | No |
| Preflight | Checks the machines, the settings and, on a running cluster, the locked settings | No |
| Backup | Archives each machine’s configuration on the first server | Writes the backup only |
| Media | Copies what the other machines need from the ISO | Yes |
| Values | Writes values.yaml | Yes |
| Host network | Configures firewall zones, routing rules and kernel network settings, then verifies them | Yes |
| Cluster | Installs or upgrades k3s: the first server, then the other servers one at a time, then the agents | Yes |
| Platform | Installs or upgrades the cdn-platform release, loads the MaxMind databases, and runs the health check | Yes |
deploy backup runs only the first and third stages. --skip-tags preflight skips the second; use it only when you are sure a check is wrong for your site.
When deploy stops on a problem it recognizes, its message starts with a code such as [ACD-TLS-HOSTNAME]. See Installation Troubleshooting.
Backups
Important: These backups hold only the deployment’s configuration, each machine’s configuration and an etcd snapshot of the Kubernetes state. They do not include the data stored in the cluster: not the PostgreSQL databases, not Kafka’s messages, not the contents of any persistent volume, and not the metrics. Restoring one of these backups puts a machine’s configuration back; it cannot recover lost data. Back up the data separately: see Backup Procedures.
Every run of deploy takes a backup first, and deploy backup takes one without changing anything. Each backup is a folder under /var/lib/acd/backups on the first server: see the deploy reference for its files. deploy never deletes a backup: remove old folders yourself. Keep your own copies of the TLS certificate and key and the MaxMind databases, which the backup leaves out.
Important: Treat backups as secret material. A backup of a running cluster holds an etcd snapshot of the whole cluster, which contains every Kubernetes Secret: database passwords, the Zitadel master key and TLS keys among them. Every machine’s archive, servers’ and agents’, also holds the cluster token, and the servers’ hold the cluster-admin kubeconfig,
/etc/rancher/k3s/k3s.yaml. Keep backups, and any copies you make, readable only by root, and delete the ones you no longer need.
The etcd snapshot is in two places, and deploy prunes neither:
the backup folder under
/var/lib/acd/backupson the first server, inside one server’s<machine>.tar.gz. Remove an old backup folder with:rm -r /var/lib/acd/backups/<folder>/var/lib/rancher/k3s/server/db/snapshotson that server, namedacd-backup-<UTC time>-<machine>-<number>. List and delete old ones on that server with:k3s etcd-snapshot list k3s etcd-snapshot delete <name>
k3s also takes its own etcd snapshots, in the same folder on every server, named etcd-snapshot-<machine>-<time>: one every 12 hours, keeping the latest five, which k3s prunes itself. They hold every Kubernetes Secret too, so treat them, and any copy of them, as secret material.
The backup folder on the first server also holds a copy of /etc/acd/deploy, except the certs/ and maxmind/ folders and deploy.log, which it leaves out. Keep a copy of the latest backup folder off the first server, so that losing it does not lose the configuration deploy needs. Protect the copy as the warning above describes.
Before relying on a backup, check it:
cd /var/lib/acd/backups/<folder>
sha256sum -c SHA256SUMS
Restoring a Machine’s Files
Copy the machine’s archive and checksum file to it, then, as root:
tar --selinux --xattrs --acls -C / -xpzf <machine>.tar.gz
sha256sum -c <machine>.files.sha256
Keep the --selinux --xattrs --acls options: plain tar restores some k3s files with the wrong SELinux label. Then reboot the machine, or restart NetworkManager and k3s.
Rolling back the cluster itself also needs the etcd snapshot, restored on one server with the k3s cluster-reset procedure: stop k3s on every server, restore that server’s archive, run k3s server --cluster-reset --cluster-reset-restore-path=<snapshot path> on it, start k3s there, and rejoin the other servers. See the k3s etcd snapshot documentation for the full procedure.
What to Send Support
When the final health check fails, deploy writes /etc/acd/deploy/support-bundle-acd.tar.gz: send it to AgileTV support. When a run stops before the health check, send deploy.log, decision-record.json and the probe/ folder from /etc/acd/deploy instead.