acd-admin

Synopsis

acd-admin list
acd-admin status [<target>]
acd-admin volumes [<name> [--avg]]
acd-admin logs <target> [-f] [-n <lines>] [-p] [-c <container>] [--since <time>] [--until <time>]
acd-admin restart <target> [--timeout <duration> | --no-wait | --node <node>]
acd-admin shell <target> [--pod <pod>] [--node <node>] [-l <selector>] [-c <container>]
acd-admin exec <target> [--pod <pod>] [--node <node>] [-l <selector>] [-c <container>] -- <command> [<argument>...]
acd-admin exec <target> --all [-l <selector>] [-c <container>] -- <command> [<argument>...]
acd-admin cordon <node>
acd-admin uncordon <node>
acd-admin drain <node> [--force]
acd-admin help [<command>]
acd-admin [<command>] -h | --help
acd-admin -V | --version

Description

acd-admin manages the running cluster by admin target: a short name such as manager, kafka or postgresql. It works out which Kubernetes objects, labels and namespaces a target means, and handles each kind the safe way for it. acd-admin list prints the valid names. A name that is not a target is an error that lists them.

It is installed on every machine at /usr/local/bin/acd-admin, and on every server /usr/bin/acd-admin links to it, so sudo acd-admin works without changing the sudo configuration. Run it on a server, as root or with sudo, after deploy has installed the cluster. Anywhere else it stops at once with:

Error: run acd-admin on a server node as root, or set KUBECONFIG.

Before deploy has installed the cdn-platform release, it stops with:

Error: the cdn-platform chart is not installed; run deploy first.

Each command runs in a short-lived pod in the acd-system namespace, started from the release that is installed, so the tool always matches it. The pod is removed when the command ends, Ctrl-C included, and when the session is hung up or terminated. A pod left behind by a dropped session can be removed: see Leftover acd-admin Pods.

Each session also has a deadline, so that one whose terminal is gone still ends. At the deadline the session stops, whether or not anyone is still attached:

acd-admin never changes a chart setting. A later deploy leaves its changes alone, and does not undo a drain.

restart, cordon, uncordon, drain, and opening a shell or exec session each record a Kubernetes Event on the affected object. The Event’s message holds the command line, quoted as a shell would need, the user and the server’s hostname:

acd-admin exec manager -- sh -c 'exit 7' run by alice on manager-1

The user is the one who logged in to the server, which sudo and su keep: alice when she logged in as herself and ran sudo acd-admin or sudo -i, and root only after a root login. This is attribution, not proof: it is only as reliable as the server’s logins. acd-admin status <target> shows the Events of restart, which are on the target’s workloads, and of shell and exec, which are on the pod the session ran in. A shell or exec Event is shown only while its pod exists. Once the pod is replaced, by a restart for example, read it with kubectl:

kubectl get events -n default --field-selector reason=Exec
kubectl get events -n default --field-selector reason=Shell

The Events of cordon, uncordon and drain are on the node, in the default namespace:

kubectl get events -n default --field-selector involvedObject.kind=Node

The command line includes every argument, exec’s command among them, and anyone who can read Events in the namespace can see it until the Event expires, after about an hour by default. Read or save an Event you need before then. Never pass a password or other secret as an argument: pass it on standard input (not with --all, which gives the command no standard input).

Bash completion for commands, options, targets and nodes is installed on every machine.

Commands

list

acd-admin list

Prints every admin target and its kind, including targets with no pods.

TARGET      KIND

status

acd-admin status [<target>]

With no target, prints a table of admin targets, then a table of nodes:

TARGET                     READY  RESTARTS/1H  HEALTH  CHANGED
alertmanager               1/1    0            ok      103m ago
confd                      1/1    0            ok      2h ago
confsej                    2/2    0            ok      104m ago
director                   0/0    0            ok      -
gateway                    2/2    0            ok      104m ago
geoip-loader               1/1    0            ok      104m ago
grafana                    2/2    0            ok      104m ago
kafka                      3/3    0            ok      103m ago
kube-state-metrics         1/1    0            ok      104m ago
manager                    3/3    0            ok      57m ago
metrics-aggregator         1/1    0            ok      104m ago
mib-frontend               2/2    0            ok      104m ago
node-exporter              4/4    0            ok      2h ago
postgresql                 3/3    0            ok      55m ago
redis                      1/1    0            ok      2h ago
selection-input            1/1    0            ok      2h ago
vector                     1/1    0            ok      2h ago
victoria-metrics           1/1    0            ok      103m ago
victoria-metrics-longterm  1/1    0            ok      103m ago
vmagent                    1/1    0            ok      103m ago
vmalert                    1/1    0            ok      118m ago
zitadel                    6/6    0            ok      104m ago

NODE       READY  ROLES               SCHEDULING   DIRECTOR
manager-1  yes    control-plane,etcd  schedulable  no
manager-2  yes    control-plane,etcd  schedulable  no
manager-3  yes    control-plane,etcd  schedulable  no
manager-4  yes    -                   schedulable  no

The admin target table:

The node table:

status exits 0 whatever the health it shows. To check health, read HEALTH: every admin target should show ok.

With a target, prints one line per pod (POD, NODE, READY, RESTARTS, AGE, LAST TERMINATION). For kafka it adds the Strimzi conditions, and for postgresql the CloudNativePG phase and the current primary. Then it lists the Volumes the target’s pods mount, if any. It ends with the target’s recent Warning events and the audit Events of acd-admin itself (WHEN, KIND, OBJECT, REASON, MESSAGE).

The Volume table of acd-admin status redis looks like this:

VOLUME      SIZE  USED     HEALTH    REPLICAS
maxmind     1Gi   unknown  healthy   1/1
redis-data  8Gi   unknown  degraded  1/2

The output is for reading; there is no machine-readable format.

volumes

acd-admin volumes [<name> [--avg]]

A Volume is the persistent storage that a pod of an admin target mounts, named by its claim (PersistentVolumeClaim). volumes lists every claim in the platform’s namespace. It only reads and changes nothing.

With no name, it prints one row per Volume:

VOLUME        TARGET                      SIZE  USED             HEALTH      REPLICAS
maxmind       alertmanager,confd +3 more  1Gi   unknown          healthy     3/3
data-kafka-0  kafka                       20Gi  ━━╸───────  25%  healthy     1/1
redis-data    redis                       8Gi   ━━━━━━━━╸─  88%  rebuilding  1/3
scratch       -                           2Gi   unknown          -           -

On a narrow terminal, the less important columns are left out, but VOLUME and HEALTH always show. Piped or redirected output always has every column.

One Volume

With a Volume name, volumes prints that Volume in detail. The detail adds the Longhorn volume name (as Longhorn’s events and support dumps name it), the access mode, the pods that mount the Volume and their nodes, and the node and state of each replica.

Volume:        data
Target:        kafka
Longhorn:      pvc-data
Size:          20Gi
Used:          ━━╸───────  25%  5.0 GiB of 20.0 GiB
Health:        healthy
Mode:          ReadWriteOnce
Locality:      local (pod and a running replica on node-1)
IO (last 1s):  read 1.50 MiB/s 120 IOPS 0.80 ms, write 4.00 KiB/s 6 IOPS 1.05 ms

POD      NODE
kafka-0  node-1

REPLICA NODE  STATE
node-1        running

Locality says whether a running replica is on the node where the Volume’s pod runs: local or remote. It is - when there is no single pod node, as for a detached or ReadWriteMany Volume.

For a Volume with one replica, Locality decides all of its IO: when it is remote, every read and write crosses the network to another node. For a Volume with several replicas, every write goes to every replica, so write latency includes the network even when Locality is local. A remote multi-replica Volume is not, by itself, the cause of slow writes.

IO (last 1s) is the Volume’s read and write load over the last second: throughput, IOPS and latency.

Only --avg uses VictoriaMetrics. When VictoriaMetrics does not answer, volumes --avg prints the rest of the detail, then exits 1 with an error.

Partial Data

When part of the storage information cannot be read, such as a node’s kubelet that does not answer, only the affected cells show unknown, a warning goes to standard error, and volumes still exits 0. Only failing to list the claims is an error. For example:

warning: reading the kubelet on node node-2 failed; fill of its Volumes is unknown: HTTP 503

logs

acd-admin logs <target> [-f] [-n <lines>] [-p] [-c <container>] [--since <time>] [--until <time>]

Shows the logs of every pod and container of the target, init containers excluded. Each line starts with its pod, and its container when the pod has more than one: <pod>[/<container>] <line>. A bounded window is merged in time order across pods.

<time> is a duration before now, such as 2m, '1h 30m' or '10m ago', or a timestamp such as '2026-10-16 12:00:00' or 2026-10-16T12:00:00Z. Quote a value that contains a space, as shown: unquoted, --since 1h 30m is refused with unexpected argument '30m'. Timestamps are always UTC, never the server’s local time, unlike journalctl. Only the logs the cluster still holds are available: those of the current and the previous instance of each container.

restart

acd-admin restart <target> [--timeout <duration> | --no-wait | --node <node>]

Restarts every pod of the target, the safe way for its kind: a rolling restart for ordinary workloads, a broker-by-broker rolling update through Strimzi for kafka, and for postgresql a CloudNativePG restart of the replicas first, then a switchover of the primary.

By default it waits, showing progress such as 2/3 pods updated, until the restart is complete. Ctrl-C stops the waiting only: the restart carries on, and acd-admin says so. A target with nothing running is left alone, and acd-admin says so.

A workload that a restart would not roll is refused before anything changes. The message says why, and what to do where there is an alternative. For example, a workload with the OnDelete update strategy is restarted one node at a time with --node.

Most admin targets are one workload. zitadel is two, the Deployments cdn-platform-zitadel and cdn-platform-zitadel-login, so restart zitadel --node <node> deletes the pods of both on that node.

shell

acd-admin shell <target> [--pod <pod>] [--node <node>] [-l <selector>] [-c <container>]

Opens an interactive session in one Ready pod of the target:

For example, to list the Kafka topics, run this in acd-admin shell kafka:

kafka-topics.sh --bootstrap-server "$BOOTSTRAP_SERVER" --list

Caution: shell postgresql opens psql as the database superuser. Every statement runs with full rights over every database of the manager, and nothing asks for confirmation. Prefer a replica (-l cnpg.io/instanceRole=replica) for reading, and run changes on the primary only when you mean to.

Without --pod, --node or --selector, it uses the first Ready pod. For postgresql it uses the primary, and refuses if the primary is not Ready, even when replicas are. With only --selector on postgresql, it uses the primary if the selector matches it, and otherwise the first Ready pod that matches. To use a replica, pass -l cnpg.io/instanceRole=replica. If no Ready pod matches, it refuses and lists the pods.

The options combine, and each only narrows the target’s own pods: a selector never reaches a pod outside the target. <selector> takes the label selector syntax of kubectl -l, such as key=value, key!=value, key in (a,b) or !key. The cluster checks it: one it cannot parse is refused with the cluster’s error before anything runs in a pod.

exec

acd-admin exec <target> [--pod <pod>] [--node <node>] [-l <selector>] [-c <container>] -- <command> [<argument>...]
acd-admin exec <target> --all [-l <selector>] [-c <container>] -- <command> [<argument>...]

Runs a command, exactly as given after --, in one pod of the target, chosen as for shell. It gets a terminal only when both standard input and standard output are terminals, so output can be redirected or piped.

On kafka the command runs in the same environment as shell kafka: the Kafka command line tools on the PATH and the bootstrap address in BOOTSTRAP_SERVER. No shell parses the command in the pod, though. Each argument arrives as your own shell on the server left it, and that shell has already replaced $BOOTSTRAP_SERVER with its own value, which is empty. To use the pod’s variables, run the command through sh -c, in single quotes:

acd-admin exec kafka -- sh -c 'kafka-topics.sh --bootstrap-server "$BOOTSTRAP_SERVER" --list'

A command that cannot start, such as one not found in the container, prints the reason on standard error. On kafka the pod’s shell reports it, and exits 127.

The command and its arguments are recorded in the audit Event on each pod, where anyone who can read Events in the namespace can see them. Pass a password or other secret on standard input (not with --all), never as an argument.

cordon

acd-admin cordon <node>

Stops new pods from being scheduled on the node. Pods already there keep running.

cordon refuses, changing nothing, when no other node would stay schedulable, because no pod that is moved, restarted or recreated could then start anywhere. A node counts as unschedulable when it is cordoned or carries a NoSchedule or NoExecute taint, such as CriticalAddonsOnly on dedicated servers. Cordoning a node that is already unschedulable (cordoned or tainted) is never refused. drain refuses the same way, even with --force, but without that exemption: it refuses a node that is already unschedulable too.

Error: cordon manager-3

Caused by:
    cordoning manager-3 would leave no schedulable node, so no pod moved, restarted or recreated could start anywhere; make another node schedulable first, or run `k3s kubectl cordon manager-3` to cordon it anyway

kubectl adds a line such as pod acd-system/acd-admin-... terminated (Error) after this and some other errors. It names the session pod ending, not a second failure.

A refused drain says:

draining manager-3 would leave no schedulable node, so no pod moved, restarted or recreated could start anywhere; make another node schedulable first, or run `k3s kubectl drain --ignore-daemonsets --delete-emptydir-data manager-3` to drain it anyway

To cordon the last schedulable node anyway, run k3s kubectl cordon <node>, and to drain it anyway, k3s kubectl drain --ignore-daemonsets --delete-emptydir-data <node>. acd-admin keeps working with every node cordoned, so acd-admin uncordon can still return a node to service. On a cluster with dedicated servers, which carry the CriticalAddonsOnly taint, the acd-admin session cannot start once every agent is cordoned: return a node to service with k3s kubectl uncordon <node> instead.

uncordon

acd-admin uncordon <node>

Returns a cordoned or drained node to service: it clears both the unschedulable mark and the drained label.

drain

acd-admin drain <node> [--force]

Takes the node out of service for maintenance. It cordons the node, marks it as drained, and moves its workloads to other nodes through the eviction API, respecting their disruption budgets. Metrics and ingress forwarding keep running on the node, so it can still be observed. Longhorn’s node components keep running too, but its storage replicas on the node stop. acd-admin session pods are never moved either, whether the drain’s own, another operator’s or one left behind: a session on the node runs until it ends.

While it waits, it reports what it is waiting for. A disruption budget can refuse an eviction for now, such as for a storage volume’s last healthy replica. The drain names that pod and keeps retrying. It also names any pod that is terminating but not yet gone, with when its deletion started and the finalizer holding it, if any. On a node that is not Ready, such a pod can stay until the node comes back.

The drain has no timeout of its own, but its session ends after 12 hours: see the deadlines under Description. Ctrl-C, or the end of the session, stops the waiting, not the drain: the node stays drained, and pods already moved stay where they went. Pods still waiting to move are not retried until drain runs again, which carries on where it stopped; uncordon returns the node to service instead.

Before it changes anything, drain refuses, and exits non-zero, when:

• no other node would stay schedulable, as for cordon. --force does not override this.
• the node runs a pod that no controller recreates, such as one started by hand, because evicting it deletes it for good. The message names each such pod as <namespace>/<name>. A pod that has finished does not count, and neither does an acd-admin session pod.
• the node runs every Ready CDN Director pod, because draining it would stop CDN routing.

Drained Nodes

A drained node carries two marks, which drain sets:

• the node label acd.agiletv.com/drained=true, which status shows as drained under SCHEDULING, and which keeps the CDN Director off the node;
• the unschedulable mark that cordon also sets (spec.unschedulable, shown by kubectl as the node.kubernetes.io/unschedulable taint).

A drained node stays drained through reboots and later deploy runs, until acd-admin uncordon clears both marks.

kubectl uncordon clears only the unschedulable mark. The node then takes new pods again and is back in service, but it keeps the drained label: status still shows it as drained, and the CDN Director stays off it. Run acd-admin uncordon on it to clear the label.

Safety

Read this before you drain, cordon or restart anything on a production cluster.

• One server at a time. The servers hold the cluster datastore (etcd), which needs a majority of them. With three servers, two down at once stops the Kubernetes API: nothing can be scheduled, moved or changed until one returns. Drain, reboot or restart one server, and return it to service, before you start on the next.

• Wait for the cluster to recover before the next drain. After you return a drained node to service, wait until acd-admin status shows every admin target ok, and acd-admin volumes shows no Volume degraded, rebuilding or faulted. A drain stops the drained node’s storage replicas, and Volumes stay degraded or rebuilding until they recover: see Longhorn Storage. Plain acd-admin status does not show this.

• Kafka tolerates one broker down, because every topic but one keeps three replicas. The exception is selection_input, which has a single replica: selection input pauses while the broker that holds it restarts or moves, and is lost with that broker’s volume. To see which broker holds it, run this in acd-admin shell kafka:

  kafka-topics.sh --bootstrap-server "$BOOTSTRAP_SERVER" --describe --topic selection_input
  

  The Leader number N is the pod cdn-platform-kafka-N, and acd-admin status kafka shows its node.

• Cordoning or draining the PostgreSQL primary’s node switches the primary. CloudNativePG moves the primary to an instance on another node, which briefly interrupts database writes. acd-admin status postgresql shows the current primary.

• Some admin targets run a single pod. In the production preset these are alertmanager, confd, geoip-loader (with MaxMind databases), kube-state-metrics, metrics-aggregator, redis, selection-input, vector, victoria-metrics, victoria-metrics-longterm, vmagent and vmalert. Restarting one, or draining its node, stops it until its new pod is Ready.

• restart --node skips disruption budgets. It deletes the target’s pods on the node rather than evicting them, so nothing holds it back, even when they are the target’s only Ready pods.

Options

Every command accepts:

acd-admin help prints the same as acd-admin --help, and acd-admin help <command> the same as acd-admin <command> --help.

acd-admin alone also accepts:

Environment

Exit Status

An interrupted, hung-up or terminated restart carries on in the cluster, and such a drain leaves the node drained.

shell, and exec in one pod, exit with the status of the command in the pod. Any of these statuses from them may come from that command.

Examples

See which targets exist and how they are doing:

acd-admin list
acd-admin status
acd-admin status postgresql

Check the cluster’s storage, then look into one Volume and its 5-minute IO average:

acd-admin volumes
acd-admin volumes data-kafka-0
acd-admin volumes data-kafka-0 --avg

Follow the manager’s logs from ten minutes ago:

acd-admin logs manager -f --since '10m ago'

See why a container crashed:

acd-admin logs manager -p

Restart Kafka, allowing up to 20 minutes:

acd-admin restart kafka --timeout 20m

Restart only the manager’s pods on one node:

acd-admin restart manager --node manager-3

Open psql on the PostgreSQL primary:

acd-admin shell postgresql

List the Kafka topics:

acd-admin exec kafka -- sh -c 'kafka-topics.sh --bootstrap-server "$BOOTSTRAP_SERVER" --list'

Run a command in every Redis pod:

acd-admin exec redis --all -- redis-cli ping

Open psql on a PostgreSQL replica rather than the primary:

acd-admin shell postgresql -l cnpg.io/instanceRole=replica

Take a node out of service and return it:

acd-admin drain manager-3
acd-admin uncordon manager-3

See Also

• Operations Guide: day-to-day procedures
• Troubleshooting Guide
• deploy: installing, upgrading and reconfiguring the cluster