node-maintenance.md

Node Maintenance

These steps only apply when using the TopoLVM StorageClass and PVCs. If only generic ephemeral volumes are used, these steps are not necessary.

Retiring Nodes

To remove a node and volumes/pods on the node from the cluster, follow these steps:

1. Run kubectl drain NODE --ignore-daemonsets=true.

   --ignore-daemonsets=true allows the command to succeed even if pods managed by Daemonset exist (e.g. topolvm-node). If drain stacks due to PodDisruptionBudgets or something, try --force option.

2. Run kubectl delete nodes NODE

3. TopoLVM will remove Pods and PersistentVolumeClaims on the node.

4. StatefulSet controller reschedules Pods and PVCs on other nodes.

Rebooting Nodes

To reboot a node without removing volumes, follow these steps:

1. Run kubectl drain NODE --ignore-daemonsets=true.

   --ignore-daemonsets=true allows the command to succeed even if pods managed by Daemonset exist (e.g. topolvm-node).

2. Reboot the node.

3. Run kubectl uncordon NODE after the node comes back online.

4. After reboot, Pods will be rescheduled to the same node because PVCs remain intact.