# Summary

Use proxmox to create a Kubernetes cluster using Talos with one control plane and 3 worker nodes for k8s workloads.


# Install talosctl on Localhost

On your local desktop/laptop, install the `talosctl` CLI

I am doing all this from my `mrbluefin` Computer, which happens to use Homebrew on Linux

```bash
brew install siderolabs/tap/talosctl
```


# Create a Proxmox VM Template

- Read the proxmox specific docs https://docs.siderolabs.com/talos/v1.12/platform-specific-installations/virtualized-platforms/proxmox
- I will ENABLE QEMU guest agent support (so proxmox can manage the VM power).  This means I build a customer ISO with qemu extension enabled.
- The custom ISO is https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.12.4/metal-amd64.iso
- For the initial installation of Talos Linux (not applicable for disk image boot), add the following installer image to the machine configuration:
	- factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4
	- I will use this string later in my `talosctl gen config` command!
- Setup a new VM using the above ISO
	- OVMF, Q35, 4-core, 4096M ram, **VirtIO SCSI** (NOT VirtIO SCSI Single), 32G root disk for now, QCOW2 on SSD storage tier, Default no cache, Discard, SSD emulation...
- Do not start the VM, but convert it to a template

# Create and Configure the Control Plane

- Use the new talos VM template to create a new vm called `talos` - which is the control plane node
- Boot the new `talos` VM
- Watch the VNC console output and grab the DHCP IP address (mine was 10.23.47.219)
- On localhost export these from your `~/.zshrc` or `~/.bashrc` or just in a current shell
```
export CLUSTER_NAME=talos-proxmox-cluster
export CONTROL_PLANE_IP=10.23.47.219
```
- Get the disk information from the new VM
```
# Get disk information from the VM
talosctl get disks --insecure --nodes $CONTROL_PLANE_IP
```
- Mine was `sda`, we'll use that in the `gen config` command below
- In a known directory, generate the cluster configs in the current working directory
- NOTICE, that `factory.talos.dev/metal-installer/ce4...` string was from the custom ISO page above.  Substutute your own installer string
```
cd ~/Downloads
mkdir talos
cd talos

talosctl gen config \
	$CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 \
	--install-image factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4 \
	--install-disk /dev/sda
```
- I want the control plane to have a static IP, so first get a list of interfaces on the VM
```
talosctl get links --insecure --nodes $CONTROL_PLANE_IP
```
- My VM uses `ens18` as the main interface.  So now hand edit the generated `controlplane.yml` and modify the `network` section like so
```
machine:
    network:
        interfaces:
            - interface: ens18
              dhcp: false
              addresses:
                - 10.23.41.50/21
              routes:
                - network: 0.0.0.0/0
                  gateway: 10.23.47.1
        nameservers:
            - 10.23.47.1
```
- I also set the hostname, the very bottom of the file like this
```
apiVersion: v1alpha1
kind: HostnameConfig
hostname: talos
```
 - NOTICE: Also change the `cluster.controlPlane.endpoint` to the new control planes static IP
```
endpoint: https://10.23.41.50:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
```
 - Finally APPLY the config.  This essentially "INSTALLS" talos on the VM.  At this point its simply booted in memory.  Watch the VNC terminal output while doing this!
```
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file ./controlplane.yaml
```
- The VM will reboot if the config was success, or error with config mistakes from VNC console you can manually fix and try again.
- Remove the ISO form proxmox VM so it doesn't accidentally boot from the CD

# Create and Configure the Nodes
- When building the control plane node, we already have the 3 files in our ~/Downloads/talos directory
- Copy the `worker.yml` to different files per worker because we need to tweak the IP address of each
```
mv worker.yml worker1.yml
cp worker1.yml worker2.yml
cp worker1.yml worker3.yml
```
- Edit each worker yml and adjust the IP address properly.  See the notes above for the static IP address modifications
- Now use the Proxmox talos VM template to build 3 more VMs, calling them `talos1`, `talos2` and `talos3` (or however many you want)
- Workers like more RAM, so adjust CPU and RAM as needed
- Boot each new VM and record the IP address of the workers
```
export WORKER_IP=("10.23.47.243" "10.23.47.238" "10.23.47.211")
```
- Apply the configs to each worker
```
talosctl apply-config --insecure --nodes 10.23.47.243 --file worker1.yaml
talosctl apply-config --insecure --nodes 10.23.47.238 --file worker2.yaml
talosctl apply-config --insecure --nodes 10.23.47.211 --file worker3.yaml
```
- Remove the ISO form each proxmox VM so it doesn't accidentally boot from the CD


# Bootstrap Etcd and get K8s Config

```
# Still in ~/Downloads/talos
talosctl --talosconfig=./talosconfig config endpoints $CONTROL_PLANE_IP

# Bootstrap etcd
talosctl bootstrap --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig

# Get k8s access (saves ~/.kube/config)
talosctl kubeconfig --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig

# Check cluster stats
talosctl --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig health

# Verify k8s nodes
kubectl get nodes


```


# Changes

I forgot to change the `controlPlane` to new 10.23.41.50, so I edited the configs for control plane and wokers and ran this to fix them

```
talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.50 --file ./controlplane.yaml --mode staged
talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.51 --file ./worker1.yaml --mode staged
talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.52 --file ./worker2.yaml --mode staged
talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.53 --file ./worker3.yaml --mode staged

talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.50
talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.51
talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.52
talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.53
```


# Daily Usage

- I copied my temp `~/Downloads/talos/talosconfig` to `~/.talos/config` so I can run all commands without explicit `--talosconfig`
- DNS name of my main talos control plane vm is `talos`, so easy to `-n talos`
- Get health `talosctl health -n talos`
- Get link info `talosctl get links -n talos`
	- `talosctl get links -n 10.23.41.51`
	- `talosctl get links -n 10.23.41.52`
	- `talosctl get links -n 10.23.41.53`
- Get disk info `talosctl get disks -n talos`
- Show images `talosctl image list -n talos`
-