[Kubernetes Data Platform][Part 2.3]: Creating highly available Kubernetes cluster with rke2

Viet_1846

·

Following

8 min read

·

Aug 2, 2024

This guide provides a step-by-step approach to deploying a highly available (HA) Kubernetes cluster using rke2. The cluster will utilize “VM-like” containers using docker-compose and sysbox.

For high availability, kube-vip will act as the cluster’s virtual IP and load balancer. This eliminates the need for external hardware or software, simplifying deployment and management.

Deeper Dive into Technologies Used:

Rke2 (Rancher Kubernetes Engine, 2nd Generation):

Rancher’s next-generation Kubernetes distribution built for security and compliance. Designed specifically for the U.S. Federal Government sector, rke2 is a fully conformant Kubernetes distribution.

Sysbox:

Sysbox is an open-source, free container runtime that enhances container security and functionality. Originally developed by Nestybox, Sysbox offers two key advantages:

• Improved Container Isolation: Sysbox utilizes Linux user namespaces for all containers. This ensures the container’s “root user” has no privileges on the host system, significantly improving isolation.
• Enhanced Workload Capabilities: Unlike traditional containers, Sysbox allows you to run system-level software directly within the container. This includes programs like systemd, Docker, Kubernetes, K3s, buildx, legacy applications, and more. This functionality eliminates the need for modifications or special versions of the software, simplifying deployment. Additionally, Sysbox virtualizes portions of the container’s file systems (procfs & sysfs) and hides host information, further enhancing security. The runtime also locks initial mounts within the container, providing an additional layer of protection.

Benefits of Sysbox:

• Enables containers to run workloads traditionally reserved for virtual machines.
• Simplifies deployment by allowing unmodified, standard software versions within containers.
• Eliminates the need for privileged containers, complex images, tricky entrypoints, and special volume mounts.

Kube-vip: A Kubernetes add-on that provides a virtual IP and load balancing functionality for both the control plane (enabling HA) and LoadBalancer type Kubernetes services, simplifying deployment and reducing external dependencies.

RKE2 HA With VIP Load Balance Architecture:

INSTALLATION STEPS

1. Create “VM-like” containers using docker-compose and sysbox. (Details in docker-compose.yaml)
2. This will create 6 “VM-like” containers: master-1(172.25.2.3), master-2(172.25.2.4), master-3(172.25.2.5), worker-1(172.25.2.6), worker-2(172.25.2.7), worker-3(172.25.2.8).
3. Prepare First Control Plane Node (Master 1)
4. Deploy kube-vip VIP(172.25.2.2)
5. Configure Remaining Control-Plane Nodes (Master 2, 3)
6. Deploy Worker Nodes (Worker 1, 2, 3)
7. Install basic Kubernetes components
8. Install testing tools
9. Destroy the cluster

HANDS-ON STEP

Reference Repository: https://github.com/viethqb/data-platform-notes/tree/main/kubernetes/rke2

IMPORTANT: Convention for Bash Scripts

Throughout this series, the following convention applies to Bash scripts:

Commands in the form “> command” should be executed on the local laptop.

Commands in the form “root@kworker1 command” should be executed on the kworker1 VM as the root user.

1. Create “VM-like” containers using docker-compose and sysbox

docker-compose.yaml

services:
  master-1:
    hostname: master-1
    container_name: master-1
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.3
  master-2:
    hostname: master-2
    container_name: master-2
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.4
  master-3:
    hostname: master-3
    container_name: master-3
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.5
  worker-1:
    hostname: worker-1
    container_name: worker-1
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.6
  worker-2:
    hostname: worker-2
    container_name: worker-2
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.7
  worker-3:
    hostname: worker-3
    container_name: worker-3
    build:
      context: .
      dockerfile: Dockerfile
    runtime: sysbox-runc
    networks:
      kubernetes_network:
        ipv4_address: 172.25.2.8
networks:
  kubernetes_network:
    driver: bridge
    ipam:
     config:
       - subnet: 172.25.2.0/24
         gateway: 172.25.2.1

Dockerfile

FROM ghcr.io/nestybox/ubuntu-jammy-systemd:latest

# Install Sshd
RUN apt-get update \
    && apt-get install --no-install-recommends -y openssh-server uuid-runtime\
    apt-transport-https \
    bash-completion vim less man jq bc \
    lsof tree psmisc htop lshw sysstat dstat \
    iproute2 iputils-ping iptables dnsutils traceroute \
    netcat curl wget nmap socat netcat-openbsd rsync net-tools telnet\
    p7zip-full \
    git tig \
    binutils acl pv \
    strace tcpdump \
    open-iscsi nfs-common \
    && rm -rf /var/lib/apt/lists/* \
    && mkdir /home/admin/.ssh \
    && chown admin:admin /home/admin/.ssh

# Enable sudo without password
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# Enable ssh password authentication
RUN sed -i 's/^PasswordAuthentication .*/PasswordAuthentication yes/' /etc/ssh/sshd_config
RUN echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config

# Set Root password
RUN echo "admin\nadmin" | passwd root

EXPOSE 22

# Set systemd as entrypoint.
ENTRYPOINT ["/sbin/init", "--log-level=err"]

Start “VM-like” containers

> cd  ~/Documents 
> git clone https://github.com/viethqb/data-platform-notes.git
> cd data-platform-notes/kubernetes/rke2
> docker-compose up -d
> docker ps -q | xargs -n 1 docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} {{ .Name }}' | sed 's/ \// /'
#ssh user: admin(root)/admin

> ssh-copy-id [email protected]
> ssh-copy-id [email protected]
> ssh-copy-id [email protected]
> ssh-copy-id [email protected]
> ssh-copy-id [email protected]
> ssh-copy-id [email protected] 

> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'
> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'
> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'
> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'
> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'
> ssh [email protected] 'echo $(uuidgen) > /etc/machine-id'

2. Prepare First Control Plane Node (Master 1)

Run on master-1

> ssh [email protected]
root@master-1:~# mkdir -p /etc/rancher/rke2/
root@master-1:~# mkdir -p  /var/lib/rancher/rke2/server/manifests/

root@master-1:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
tls-san:
  - 172.25.2.2
  - 172.25.2.3
  - 172.25.2.4
  - 172.25.2.5
write-kubeconfig-mode: "0600"
etcd-expose-metrics: true
cni:
  - canal
EOF

root@master-1:~# cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

root@master-1:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
root@master-1:~# systemctl start rke2-server
root@master-1:~# systemctl enable rke2-server
root@master-1:~# echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
root@master-1:~# echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc
root@master-1:~# cat /var/lib/rancher/rke2/server/node-token
# K10320034a05e280ef37f0753cf1263586ae04d471a8b10a8d87f330150e0325475::server:c32246c40f9eb288ec6992065c971669

Save this token for installation on other nodes:

K10320034a05e280ef37f0753cf1263586ae04d471a8b10a8d87f330150e0325475::server:c32246c40f9eb288ec6992065c971669

3. Deploy kube-vip

Run on master-1

root@master-1:~# export VIP=172.25.2.2
root@master-1:~# ip a
# Find the network interface corresponding to IP 172.25.2.3
root@master-1:~# export INTERFACE=eth0
root@master-1:~# curl https://kube-vip.io/manifests/rbac.yaml > /var/lib/rancher/rke2/server/manifests/kube-vip-rbac.yaml
root@master-1:~# /var/lib/rancher/rke2/bin/crictl -r "unix:///run/k3s/containerd/containerd.sock"  pull ghcr.io/kube-vip/kube-vip:latest

root@master-1:~# CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock  ctr -n k8s.io run \
--rm \
--net-host \
ghcr.io/kube-vip/kube-vip:latest vip /kube-vip manifest daemonset --arp --interface $INTERFACE --address $VIP --controlplane  --leaderElection --taint --services --inCluster | tee /var/lib/rancher/rke2/server/manifests/kube-vip.yaml

root@master-1:~# source ~/.bashrc 
root@master-1:~# kubectl rollout status daemonset kube-vip-ds -n kube-system --timeout=650s
root@master-1:~# kubectl get ds -n kube-system kube-vip-ds

4. Configure Remaining Control-Plane Nodes (Master 2, 3)

Run on master-2

root@master-2:~# mkdir -p /etc/rancher/rke2/
root@master-2:~# mkdir -p  /var/lib/rancher/rke2/server/manifests/

root@master-2:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://172.25.2.2:9345
token: <token obtained in step 2 >
write-kubeconfig-mode: "0644" 
tls-san:
  - 172.25.2.2
  - 172.25.2.3
  - 172.25.2.4
  - 172.25.2.5
write-kubeconfig-mode: "0644"
etcd-expose-metrics: true
cni:
  - canal

EOF

root@master-2:~# cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

root@master-2:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
root@master-2:~# systemctl start rke2-server
root@master-2:~# systemctl enable rke2-server

root@master-2:~# echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
root@master-2:~# echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc 
root@master-2:~# source ~/.bashrc

Run on master-3

root@master-3:~# mkdir -p /etc/rancher/rke2/
root@master-3:~# mkdir -p  /var/lib/rancher/rke2/server/manifests/

root@master-3:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://172.25.2.2:9345
token: <token obtained in step 2>
write-kubeconfig-mode: "0644" 
tls-san:
  - 172.25.2.2
  - 172.25.2.3
  - 172.25.2.4
  - 172.25.2.5
write-kubeconfig-mode: "0644"
etcd-expose-metrics: true
cni:
  - canal

EOF

root@master-3:~# cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

root@master-3:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
root@master-3:~# systemctl start rke2-server
root@master-3:~# systemctl enable rke2-server

root@master-3:~# echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
root@master-3:~# echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc 
root@master-3:~# source ~/.bashrc

5. Deploy Worker Nodes (Worker 1, 2, 3)

Run on worker-1

root@worker-1:~# mkdir -p /etc/rancher/rke2/

root@worker-1:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://172.25.2.2:9345
token: K10320034a05e280ef37f0753cf1263586ae04d471a8b10a8d87f330150e0325475::server:c32246c40f9eb288ec6992065c971669
write-kubeconfig-mode: \"0644\"
EOF

root@worker-1:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -
root@worker-1:~# systemctl start rke2-agent.service
root@worker-1:~# systemctl enable rke2-agent.service

Run on worker-2

root@worker-2:~# mkdir -p /etc/rancher/rke2/

root@worker-2:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://172.25.2.2:9345
token: <token obtained in step 2>
write-kubeconfig-mode: \"0644\"
EOF

root@worker-2:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -
root@worker-2:~# systemctl start rke2-agent.service
root@worker-2:~# systemctl enable  rke2-agent.service

Run on worker-3

root@worker-3:~# mkdir -p /etc/rancher/rke2/

root@worker-3:~# cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://172.25.2.2:9345
token: <token obtained in step 2>
write-kubeconfig-mode: \"0644\"
EOF

root@worker-3:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -
root@worker-3:~# systemctl start rke2-agent.service
root@worker-3:~# systemctl enable  rke2-agent.service

6. Install basic Kubernetes components

> scp [email protected]:/etc/rancher/rke2/rke2.yaml .
> sed -i -e 's/127.0.0.1/172.25.2.2/g' ./rke2.yaml
> k get no --kubeconfig ./rke2.yaml

# Install local-path-storage
> export KUBECONFIG=./rke2.yaml
> kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.26/deploy/local-path-storage.yaml
> kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
> kubectl get sc
> kubectl get ingressclass
> kubectl top no

7. Install testing tools

minio-values.yaml

auth:
  rootUser: "admin"
  rootPassword: "password"
ingress:
  enabled: true
  ingressClassName: "nginx"
  hostname: minio.lakehouse.local
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
defaultBuckets: "lakehouse, airflow, risingwave, kafka"
persistence:
  size: 50Gi

Install Minio on Kubernetes

# Install Minio
> helm repo add bitnami https://charts.bitnami.com/bitnami
> helm repo update
> helm upgrade --install minio -n minio -f minio-values.yaml bitnami/minio --create-namespace --debug

> kubectl -n minio get all -owide
> kubectl -n minio get pvc
> kubectl -n minio get ing

Update local hosts file

> sudo vim /etc/hosts
# Add the following lines to the end of the /etc/hosts
# 172.25.2.6   minio.lakehouse.local

Access Minio at http://minio.lakehouse.local in your web browser with user: admin & pass: password

8. Destroy the cluster

> docker-compose down