# Summary Use proxmox to create a Kubernetes cluster using Talos with one control plane and 3 worker nodes for k8s workloads. # Install talosctl on Localhost On your local desktop/laptop, install the `talosctl` CLI I am doing all this from my `mrbluefin` Computer, which happens to use Homebrew on Linux ```bash brew install siderolabs/tap/talosctl ``` # Create a Proxmox VM Template - Read the proxmox specific docs https://docs.siderolabs.com/talos/v1.12/platform-specific-installations/virtualized-platforms/proxmox - I will ENABLE QEMU guest agent support (so proxmox can manage the VM power). This means I build a customer ISO with qemu extension enabled. - The custom ISO is https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.12.4/metal-amd64.iso - For the initial installation of Talos Linux (not applicable for disk image boot), add the following installer image to the machine configuration: - factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4 - I will use this string later in my `talosctl gen config` command! - Setup a new VM using the above ISO - OVMF, Q35, 4-core, 4096M ram, **VirtIO SCSI** (NOT VirtIO SCSI Single), 32G root disk for now, QCOW2 on SSD storage tier, Default no cache, Discard, SSD emulation... - Do not start the VM, but convert it to a template # Create and Configure the Control Plane - Use the new talos VM template to create a new vm called `talos` - which is the control plane node - Boot the new `talos` VM - Watch the VNC console output and grab the DHCP IP address (mine was 10.23.47.219) - On localhost export these from your `~/.zshrc` or `~/.bashrc` or just in a current shell ``` export CLUSTER_NAME=talos-proxmox-cluster export CONTROL_PLANE_IP=10.23.47.219 ``` - Get the disk information from the new VM ``` # Get disk information from the VM talosctl get disks --insecure --nodes $CONTROL_PLANE_IP ``` - Mine was `sda`, we'll use that in the `gen config` command below - In a known directory, generate the cluster configs in the current working directory - NOTICE, that `factory.talos.dev/metal-installer/ce4...` string was from the custom ISO page above. Substutute your own installer string ``` cd ~/Downloads mkdir talos cd talos talosctl gen config \ $CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 \ --install-image factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4 \ --install-disk /dev/sda ``` - I want the control plane to have a static IP, so first get a list of interfaces on the VM ``` talosctl get links --insecure --nodes $CONTROL_PLANE_IP ``` - My VM uses `ens18` as the main interface. So now hand edit the generated `controlplane.yml` and modify the `network` section like so ``` machine: network: interfaces: - interface: ens18 dhcp: false addresses: - 10.23.41.50/21 routes: - network: 0.0.0.0/0 gateway: 10.23.47.1 nameservers: - 10.23.47.1 ``` - I also set the hostname, the very bottom of the file like this ``` apiVersion: v1alpha1 kind: HostnameConfig hostname: talos ``` - NOTICE: Also change the `cluster.controlPlane.endpoint` to the new control planes static IP ``` endpoint: https://10.23.41.50:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname. ``` - Finally APPLY the config. This essentially "INSTALLS" talos on the VM. At this point its simply booted in memory. Watch the VNC terminal output while doing this! ``` talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file ./controlplane.yaml ``` - The VM will reboot if the config was success, or error with config mistakes from VNC console you can manually fix and try again. - Remove the ISO form proxmox VM so it doesn't accidentally boot from the CD # Create and Configure the Nodes - When building the control plane node, we already have the 3 files in our ~/Downloads/talos directory - Copy the `worker.yml` to different files per worker because we need to tweak the IP address of each ``` mv worker.yml worker1.yml cp worker1.yml worker2.yml cp worker1.yml worker3.yml ``` - Edit each worker yml and adjust the IP address properly. See the notes above for the static IP address modifications - Now use the Proxmox talos VM template to build 3 more VMs, calling them `talos1`, `talos2` and `talos3` (or however many you want) - Workers like more RAM, so adjust CPU and RAM as needed - Boot each new VM and record the IP address of the workers ``` export WORKER_IP=("10.23.47.243" "10.23.47.238" "10.23.47.211") ``` - Apply the configs to each worker ``` talosctl apply-config --insecure --nodes 10.23.47.243 --file worker1.yaml talosctl apply-config --insecure --nodes 10.23.47.238 --file worker2.yaml talosctl apply-config --insecure --nodes 10.23.47.211 --file worker3.yaml ``` - Remove the ISO form each proxmox VM so it doesn't accidentally boot from the CD # Bootstrap Etcd and get K8s Config ``` # Still in ~/Downloads/talos talosctl --talosconfig=./talosconfig config endpoints $CONTROL_PLANE_IP # Bootstrap etcd talosctl bootstrap --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig # Get k8s access (saves ~/.kube/config) talosctl kubeconfig --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig # Check cluster stats talosctl --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig health # Verify k8s nodes kubectl get nodes ``` # Changes I forgot to change the `controlPlane` to new 10.23.41.50, so I edited the configs for control plane and wokers and ran this to fix them ``` talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.50 --file ./controlplane.yaml --mode staged talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.51 --file ./worker1.yaml --mode staged talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.52 --file ./worker2.yaml --mode staged talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.53 --file ./worker3.yaml --mode staged talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.50 talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.51 talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.52 talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.53 ``` # Daily Usage - I copied my temp `~/Downloads/talos/talosconfig` to `~/.talos/config` so I can run all commands without explicit `--talosconfig` - DNS name of my main talos control plane vm is `talos`, so easy to `-n talos` - Get health `talosctl health -n talos` - Get link info `talosctl get links -n talos` - `talosctl get links -n 10.23.41.51` - `talosctl get links -n 10.23.41.52` - `talosctl get links -n 10.23.41.53` - Get disk info `talosctl get disks -n talos` - Show images `talosctl image list -n talos` -