Like 0
Fork 0

Last active 1 month ago

mreschke's Avatar mreschke revised this gist 1 month ago. Go to revision

1 file changed, 167 insertions

talos.md(file created)

@@ -0,0 +1,167 @@
1 + # Summary
2 +
3 + Use proxmox to create a Kubernetes cluster using Talos with one control plane and 3 worker nodes for k8s workloads.
4 +
5 +
6 + # Install talosctl on Localhost
7 +
8 + On your local desktop/laptop, install the `talosctl` CLI
9 +
10 + I am doing all this from my `mrbluefin` Computer, which happens to use Homebrew on Linux
11 +
12 + ```bash
13 + brew install siderolabs/tap/talosctl
14 + ```
15 +
16 +
17 + # Create a Proxmox VM Template
18 +
19 + - Read the proxmox specific docs https://docs.siderolabs.com/talos/v1.12/platform-specific-installations/virtualized-platforms/proxmox
20 + - I will ENABLE QEMU guest agent support (so proxmox can manage the VM power). This means I build a customer ISO with qemu extension enabled.
21 + - The custom ISO is https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.12.4/metal-amd64.iso
22 + - For the initial installation of Talos Linux (not applicable for disk image boot), add the following installer image to the machine configuration:
23 + - factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4
24 + - I will use this string later in my `talosctl gen config` command!
25 + - Setup a new VM using the above ISO
26 + - OVMF, Q35, 4-core, 4096M ram, **VirtIO SCSI** (NOT VirtIO SCSI Single), 32G root disk for now, QCOW2 on SSD storage tier, Default no cache, Discard, SSD emulation...
27 + - Do not start the VM, but convert it to a template
28 +
29 + # Create and Configure the Control Plane
30 +
31 + - Use the new talos VM template to create a new vm called `talos` - which is the control plane node
32 + - Boot the new `talos` VM
33 + - Watch the VNC console output and grab the DHCP IP address (mine was 10.23.47.219)
34 + - On localhost export these from your `~/.zshrc` or `~/.bashrc` or just in a current shell
35 + ```
36 + export CLUSTER_NAME=talos-proxmox-cluster
37 + export CONTROL_PLANE_IP=10.23.47.219
38 + ```
39 + - Get the disk information from the new VM
40 + ```
41 + # Get disk information from the VM
42 + talosctl get disks --insecure --nodes $CONTROL_PLANE_IP
43 + ```
44 + - Mine was `sda`, we'll use that in the `gen config` command below
45 + - In a known directory, generate the cluster configs in the current working directory
46 + - NOTICE, that `factory.talos.dev/metal-installer/ce4...` string was from the custom ISO page above. Substutute your own installer string
47 + ```
48 + cd ~/Downloads
49 + mkdir talos
50 + cd talos
51 +
52 + talosctl gen config \
53 + $CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 \
54 + --install-image factory.talos.dev/metal-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.4 \
55 + --install-disk /dev/sda
56 + ```
57 + - I want the control plane to have a static IP, so first get a list of interfaces on the VM
58 + ```
59 + talosctl get links --insecure --nodes $CONTROL_PLANE_IP
60 + ```
61 + - My VM uses `ens18` as the main interface. So now hand edit the generated `controlplane.yml` and modify the `network` section like so
62 + ```
63 + machine:
64 + network:
65 + interfaces:
66 + - interface: ens18
67 + dhcp: false
68 + addresses:
69 + - 10.23.41.50/21
70 + routes:
71 + - network: 0.0.0.0/0
72 + gateway: 10.23.47.1
73 + nameservers:
74 + - 10.23.47.1
75 + ```
76 + - I also set the hostname, the very bottom of the file like this
77 + ```
78 + apiVersion: v1alpha1
79 + kind: HostnameConfig
80 + hostname: talos
81 + ```
82 + - NOTICE: Also change the `cluster.controlPlane.endpoint` to the new control planes static IP
83 + ```
84 + endpoint: https://10.23.41.50:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
85 + ```
86 + - Finally APPLY the config. This essentially "INSTALLS" talos on the VM. At this point its simply booted in memory. Watch the VNC terminal output while doing this!
87 + ```
88 + talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file ./controlplane.yaml
89 + ```
90 + - The VM will reboot if the config was success, or error with config mistakes from VNC console you can manually fix and try again.
91 + - Remove the ISO form proxmox VM so it doesn't accidentally boot from the CD
92 +
93 + # Create and Configure the Nodes
94 + - When building the control plane node, we already have the 3 files in our ~/Downloads/talos directory
95 + - Copy the `worker.yml` to different files per worker because we need to tweak the IP address of each
96 + ```
97 + mv worker.yml worker1.yml
98 + cp worker1.yml worker2.yml
99 + cp worker1.yml worker3.yml
100 + ```
101 + - Edit each worker yml and adjust the IP address properly. See the notes above for the static IP address modifications
102 + - Now use the Proxmox talos VM template to build 3 more VMs, calling them `talos1`, `talos2` and `talos3` (or however many you want)
103 + - Workers like more RAM, so adjust CPU and RAM as needed
104 + - Boot each new VM and record the IP address of the workers
105 + ```
106 + export WORKER_IP=("10.23.47.243" "10.23.47.238" "10.23.47.211")
107 + ```
108 + - Apply the configs to each worker
109 + ```
110 + talosctl apply-config --insecure --nodes 10.23.47.243 --file worker1.yaml
111 + talosctl apply-config --insecure --nodes 10.23.47.238 --file worker2.yaml
112 + talosctl apply-config --insecure --nodes 10.23.47.211 --file worker3.yaml
113 + ```
114 + - Remove the ISO form each proxmox VM so it doesn't accidentally boot from the CD
115 +
116 +
117 + # Bootstrap Etcd and get K8s Config
118 +
119 + ```
120 + # Still in ~/Downloads/talos
121 + talosctl --talosconfig=./talosconfig config endpoints $CONTROL_PLANE_IP
122 +
123 + # Bootstrap etcd
124 + talosctl bootstrap --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig
125 +
126 + # Get k8s access (saves ~/.kube/config)
127 + talosctl kubeconfig --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig
128 +
129 + # Check cluster stats
130 + talosctl --nodes $CONTROL_PLANE_IP --talosconfig=./talosconfig health
131 +
132 + # Verify k8s nodes
133 + kubectl get nodes
134 +
135 +
136 + ```
137 +
138 +
139 + # Changes
140 +
141 + I forgot to change the `controlPlane` to new 10.23.41.50, so I edited the configs for control plane and wokers and ran this to fix them
142 +
143 + ```
144 + talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.50 --file ./controlplane.yaml --mode staged
145 + talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.51 --file ./worker1.yaml --mode staged
146 + talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.52 --file ./worker2.yaml --mode staged
147 + talosctl --talosconfig=./talosconfig apply-config --nodes 10.23.41.53 --file ./worker3.yaml --mode staged
148 +
149 + talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.50
150 + talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.51
151 + talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.52
152 + talosctl --talosconfig=./talosconfig reboot --nodes 10.23.41.53
153 + ```
154 +
155 +
156 + # Daily Usage
157 +
158 + - I copied my temp `~/Downloads/talos/talosconfig` to `~/.talos/config` so I can run all commands without explicit `--talosconfig`
159 + - DNS name of my main talos control plane vm is `talos`, so easy to `-n talos`
160 + - Get health `talosctl health -n talos`
161 + - Get link info `talosctl get links -n talos`
162 + - `talosctl get links -n 10.23.41.51`
163 + - `talosctl get links -n 10.23.41.52`
164 + - `talosctl get links -n 10.23.41.53`
165 + - Get disk info `talosctl get disks -n talos`
166 + - Show images `talosctl image list -n talos`
167 + -
Newer Older

Powered by Opengist ⋅ Load: 16ms⋅

English