Changing the Cluster

Overview

The machines in the cluster are whatever inventory.ini lists. To change them, edit the inventory and run deploy. This page covers the changes that need more than that.

Before you start, check that the cluster is healthy: every node yes under READY and every admin target ok under HEALTH in acd-admin status.

Adding an Agent

1. Install the new machine as for the others: see Before You Begin. If you use a backplane, give it an address in backplane_network.

2. From the first server, set up SSH to it:

   ssh-copy-id root@192.0.2.14
   ssh -o BatchMode=yes root@192.0.2.14 hostname
   

3. Add it under [agent] in /etc/acd/deploy/inventory.ini:

   [agent]
   manager-4 ansible_host=192.0.2.14
   

4. Run deploy:

   /mnt/esb3027/deploy
   

deploy configures the new machine’s network, joins it to the cluster, and updates every other machine’s firewall to trust it.

Adding Servers

The number of servers must stay odd, so add servers two at a time, for example to go from one to three. Prepare both machines as for an agent, list both under [server], and run deploy once. deploy joins them one at a time.

Changing a Machine’s Role

k3s cannot turn a server into an agent, or an agent into a server, in place: the machine leaves the cluster and joins again in its new role. Because the number of servers must stay odd, a role change always moves two machines: two agents promoted to go from one server to three, two servers demoted to go from three to one, or a server and an agent swapped.

1. Take the first machine out of the cluster, whether it is a server or an agent. From any server, drain and remove it:

   acd-admin drain <machine>
   kubectl delete node <machine>
   

   The drain moves the machine’s pods elsewhere. It can take several minutes while storage moves, and reports what it is waiting for. If it does not finish, see When a Drain Does Not Finish. Removing the node needs kubectl: acd-admin has no command for it. Read Safety first.

   Then, on the machine itself, uninstall k3s: run /usr/local/bin/k3s-uninstall.sh if it was a server, or /usr/local/bin/k3s-agent-uninstall.sh if it was an agent.

2. Only when the first machine is out, do the same for the second. Taking two out at once can leave too few machines for the workloads.

3. List both machines under their new roles in inventory.ini, and run deploy.

With a single server, promoting two agents leaves the server alone to run every workload while both agents are out. Expect pods to stay Pending until the machines rejoin. The deploy that brings them back allows for this, and waits up to 10 minutes for the pods to start.

If a machine is listed under a role other than the one it runs, deploy stops with [ACD-INVENTORY-ROLE-CHANGED] before changing anything.

When a Drain Does Not Finish

acd-admin drain waits for every pod it moves, and has no timeout of its own. Its session ends after 12 hours; running drain again carries on where it stopped. Storage does not hold it up for long: each volume has replicas on other machines, and for the Kafka data volumes, which have a single replica, the storage layer (Longhorn) copies the replica to another machine before it lets the drain go on. While that happens, the drain names the pod whose eviction a disruption budget refuses for now, and keeps retrying.

If it still waits, read what it names:

• A pod that is terminating but not gone, with the finalizer holding it. On a machine that is not Ready, such a pod stays until the machine returns. Once the drain waits only on such pods, press Ctrl-C: the machine stays drained.
• A pod whose eviction is refused for a long time. Check acd-admin status for an admin target that is not ok, and kubectl -n longhorn-system get volumes.longhorn.io for a volume that is not healthy. A disruption budget refuses to evict a pod while its other pods are not Ready. Fix that, and the drain carries on.

See acd-admin drain.

Things You Cannot Change

The cluster is built around a few settings that cannot change once it is installed: backplane_network, cluster_pod_cidr, cluster_svc_cidr, the firewall zone and policy names, each machine’s node_ip and node_external_ip, and the machines’ hostnames. See Locked at First Install.