How to migrate a pod?

Before the advent of containers, VM migrations were a well established solution to move application state across machines. Luckily, the CRIU project has made it possible to checkpoint and restore containers. In the past months, I set up a Kubernetes cluster with pod migration support based on prior work, exploring the migration performance through high-speed file servers on Azure and investigated the feasibility and potential for a commercial HPC service run on Kubernetes. You can find a tutorial for the cluster setup here and a short pod migration demo here.

Case study results and the potential of migration

The research findings show that the migration time is mainly influenced by memory load and just takes a few seconds for containers with a few gigabytes of memory usage and still less than 2 minutes for 50 GB. A migration controller was used to observe node memory consumption and preemptively migrate workloads upon imminent resource shortage. Different intervention heuristics were investigated, and the best performing one used a memory threshold to trigger migration. By selecting the pods with the steepest memory slope for migration, the heuristic could prevent forceful pod evictions due to resource over-commitment in most cases. The analysis was performed with production cluster data on a Kubernetes scheduler simulator. For the investigated HPC service, the simulations showed that cluster utilization could be improved by a factor of up to 4 times compared to status quo production scenarios. You can see the migration controller in action here:

The status quo and conclusion

Research for stateful pod migration on Kubernetes is rare, but prior proof of concepts have shown the practical feasibility. However, they did not evaluate its applicability or benefits for real-world problems. But there is demand for such a feature3,4 and just recently, a checkpointing solution (without restore) was officially integrated in Kubernetes v1.24. Container migration is a great option to migrate stateful pods without application knowledge. It's a workload agnostic approach, whose downtime is fairly predictable through the memory load of the container. While some applications might have features to persist state to disk, this does not apply to all, and when available the effect of migration (e.g. downtime, checkpointable state) is often not predictable. The hurdle of the technical setup is still the biggest one to overcome, but there exists an easier-to-use project called KubeVirt that allows running a VM inside a pod, which can be live migrated through a Kubernetes API extension object.

If you have a use-case for pod migration or want to know more about my findings, please reach out! I would like to make stateful pod migration more accessible!

Recently, I tried to configure calico networking on a self-managed Kubernetes cluster on Azure. It did not work out of the box and many instructions on the internet did not work for me. In the following, I want to share my setup. To set up the network and VMs, I followed this tutorial. After installing the default configuration of calico, inter-node communication between pods did not work.

My working approach uses User-Defined-Routes (UDR) on Azure to route traffic from the different pod-subnets of each node. Be sure to only have one IP address assigned to each network interface! I installed the Azure CNI plugin before, which assigns pod IPs from the secondary IPs of the vnet that are assigned to the network interface. This caused problems in the IP detection in Calico, but it can be easily fixed by deleting the secondary IPs.

When bootstrapping Kubernetes, you should also set the pod subnet (CIDR) to avoid address overlap with the virtual network. The default for kubeadm is 192.168.0.0/16 which indeed overlapped with my Azure network.

This is my kubeadm config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


cat conf.yaml
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.19.16 #-beta.0.1017_d0acb1e3ae68d3-dirty
networking:
  podSubnet: "172.16.0.0/16"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  criSocket: "/run/containerd/containerd.sock"

To bootstrap the cluster, execute: ./kubeadm init --upload-certs --cri-socket "/run/containerd/containerd.sock" --config conf.yaml

Then, get the calico setup from here.

In the yaml set the pod CIDR:

1
2


- name: CALICO_IPV4POOL_CIDR
  value: "172.16.0.0/16" 

To configure the inter-node pod communication, I followed these steps here.

Before applying I recommend to set up VXLAN instead of ipip mode (see below). By default, calico uses ipip tunneling for pod communication. A tunnel is a virtual network interface to connect subnets. This did not work in my Azure network. Instead, I configured VXLAN.

1
2
3
4
5
6
7
8
9


cat vxlan.yaml
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: ippool-vxlan-1
spec:
  cidr: 172.16.0.0/16
  vxlanMode: Always
  natOutgoing: true

Traffic to the pod network (outside the node) should then not be routed to tunl0 anymore:

1
2
3


ip a
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0

Install calioctl and verify that there is only one ippool: calicoctl get ippools

If not, delete the other pool: calicoctl delete pool default-ipv4-ippool

You need to reboot, to force a reassignment of the pod IPs to the new network.

Then doublecheck that the the CIDR did not change:

1
2


kubectl get ipamblocks.crd.projectcalico.org \
-o jsonpath="{range .items[*]}{'podNetwork: '}{.spec.cidr}{'\t NodeIP: '}{.spec.affinity}{'\n'}"

Otherwise, you need to update the Azure routing table!

If you want to dig deeper into the different networking options in calico for Azure, I recommend this video:

• Requirements
• Build images
• Tag images and push to registry
• Kubeadm bootstrap

In this tutorial, I want to show you how to bootstrap a Kubernetes cluster with kubeadm using your customized Kubernetes fork. This might be useful if you want to use new features that are not yet merged in the upstream. For development, it’s of course much easier to set up a local cluster (./hack/local-up-cluster.sh), but to test functionality across different nodes, you might need a distributed cluster.

One option is to install it The hard way, but I think it’s more convenient to use kubeadm. The challenge is that you need container images for the kube control-plane components.

I found it difficult to find good documentation on this process, so I hope to help you along with this.

Requirements

By default Kubeadm pulls the Kubernetes version matching the version of kubeadm from the official registry. But it also allows to specify a custom registry. To install our custom Kubernetes, we will need to build and pull the required container images, tag them with the expected version of kubeadm, and the push them to our registry.

Build images

I built the images on a Ubuntu machine, since the build is resource intensive. Kubernetes is big, so be sure to have enough space on the VM! The build happens inside a docker container, so make sure to have docker installed. VMs usually only have a small disk image, so I created a symbolic link to save the docker files on another attached drive (here mounted on /mnt):

1
2
3
4
5
6


sudo -s
systemctl stop docker
rm -rf /var/lib/docker
mkdir -p /mnt/docker
ln -s /mnt/docker /var/lib/docker
systemctl restart docker

Then, inside the kubernetes root path, specify your docker registry and run: KUBE_SERVER_PLATTFORMS="linux/amd64" KUBE_DOCKER_REGISTRY="sadrian99" KUBE_RELEASE_RUN_TESTS=n ./build/release-images.sh I recommend to set a flag for the image version such that the version coincides with what kubeadm expects (version of kubeadm). In my case, this was v1.19.16.

Tag images and push to registry

The generated image names have a trailing amd64 which needs to be removed for kubeadm: IMGN: new name for kubadm IMG: old image name VER: old version tag VERN: new version tag REPO: your container registry url In my case, REPO=sadrian99 and VERN=v.19.16. Then we tag the image with the new name:

1
2


export IMG=kube-apiserver-amd64; export IMGN=kube-apiserver
docker tag $REPO/$IMG:$VER sadrian99/$IMGN:$VERN && docker push $REPO/$IMGN:$VERN```

You need to do this for these images:

• kube-apiserver:v1.19.16
• kube-controller-manager:v1.19.16
• kube-scheduler:v1.19.16
• kube-proxy:v1.19.16

Additionally you need to pull these images and copy them to your registry:

• pause:3.2
• etcd:3.4.7-0

• coredns:1.6.7

  1 2

export D=coredns:1.6.7
docker pull k8s.gcr.io/$D && docker tag k8s.gcr.io/$D $REPO/$D && docker push $REPO/$D

Kubeadm bootstrap

You can test if the images are all available: kubeadm config images pull --image-repository sadrian99

Then you can add --image-repository sadrian99 to the init command to bootstrap your cluster.

Congrats, you should now have a cluster with your own Kubernetes version! I used this to migrate pods across nodes. Let me know what your use case is!

• Why
• Status quo
• Goal
• Demo
• 1. Cluster setup
  • Network setup
  • VM provisioning
• 2. Kernel downgrade
  • Worker
• 3. Install Prequisites
  • Master node
    • Containerd + Kubelet
    • Kubeadm
    • Kubectl
  • Worker node
    • CRIU
    • Golang
    • Containerd
      • Install dependencies
      • Systemd service
      • Build
    • Update systemd svc to custom binary
    • Configure CNI plugins
    • Kubelet
    • Kubeadm
• 4. Kubernetes bootstrapping
  • Networking
  • Kubeadm init
  • Verify cluster state
  • (Debugging)
  • Join worker
  • Copy kubeconfig
• 5. Demo (test migration)
  • Deploy stateful pod
  • Clone pod
• 6. Set up file server
  • Important warning
  • Steps
• Development
• End

This is an extensive tutorial on how to set up a Kubernetes cluster that supports pod migration.

Why

Statelessness is the basic foundation for microservices run inside Kubernetes. Outside it’s main application domain, the platform also appeals to the High Performance Computing (HPC) community for that infrastructure management can be delegated to cloud providers and it’s on-demand scaling. The challenge is that HPC jobs are usually long running and stateful. Jobs such as simulations or optimization problems usually keep their state in memory and state checkpointing on disk is not always available. This is undesirable because failures are expected to occur. Matters becomes even worse for jobs with unpredictable resource requirements. Unexpected spikes in memory can lead to out-of-memory node situations, which results in pods being killed. The catastrophic consequence is the complete loss of job progress from many hours or even days of compute time. To avoid this, a migration of stateful pods to another node would be desirable.

Status quo

Currently, Kubernetes does not support pod migration. However, a PoC of a pod migration n prior work by Jakob Schrettenbrunner showed the feasibility. A proposal to support very basic checkpointing (forensic checkpoiting without restore) functionality has recently been accepted by the Kubernetes community as well and is expected to be available in future releases.

Goal

Building on the prior PoC of Jakob Schrettenbrunner, I want to show you step by step how to set up a Kubernetes cluster with pod migration functionality. Bootstrapping a Kubernetes cluster from scratch is not a trivial task, but kubeadm will help us. Jakob also provided some documentation on his setup and while very helpful it is far from complete and does not mention all potential gotchas. You might suspect already that this won’t be a quick and easy process, but I hope to make it a lot easier for you through this extensive tutorial.

Demo

To see what to expect, here is a quick demo of the steps to migrate a pod:

1. Cluster setup

The cluster consists of 1 master node and 2 worker nodes. The VMs are provisioned in Microsoft Azure. For migrating the pod across a worker node, Azure’s SMB file share server) is used. You might also use an NFS server (and it might even make things easier as mentioned later..), but this was not possible for company policy reasons in my case.

Kubernetes is bootstrapped using kubeadm. It’s tested with version v1.19.0-beta.0.1015+b521fb5114995f-dirty ( binaries are available here, but I recommend building from source).

Network setup

To set up the cluster network, I followed this tutorial. You can use the web shell on Azure for this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


az group create -n kubeadm -l westus2

az network vnet create \
    --resource-group kubeadm \
    --name kubeadm \
    --address-prefix 192.168.0.0/16 \
    --subnet-name kube \
    --subnet-prefix 192.168.0.0/16

az network nsg create \
    --resource-group kubeadm \
    --name kubeadm

az network nsg rule create \
    --resource-group kubeadm \
    --nsg-name kubeadm \
    --name kubeadmssh \
    --protocol tcp \
    --priority 1000 \
    --destination-port-range 22 \
    --access allow

az network nsg rule create \
    --resource-group kubeadm \
    --nsg-name kubeadm \
    --name kubeadmWeb \
    --protocol tcp \
    --priority 1001 \
    --destination-port-range 6443 \
    --access allow

az network vnet subnet update \
    -g kubeadm \
    -n kube \
    --vnet-name kubeadm \
    --network-security-group kubeadm
# load balancer:
az network public-ip create \
    --resource-group kubeadm \
    --name controlplaneip \
    --sku Standard \
    --dns-name nilfrankubeadm

 az network lb create \
    --resource-group kubeadm \
    --name kubemaster \
    --sku Standard \
    --public-ip-address controlplaneip \
    --frontend-ip-name controlplaneip \
    --backend-pool-name masternodes

az network lb probe create \
    --resource-group kubeadm \
    --lb-name kubemaster \
    --name kubemasterweb \
    --protocol tcp \
    --port 6443

az network lb rule create \
    --resource-group kubeadm \
    --lb-name kubemaster \
    --name kubemaster \
    --protocol tcp \
    --frontend-port 6443 \
    --backend-port 6443 \
    --frontend-ip-name controlplaneip \
    --backend-pool-name masternodes \
    --probe-name kubemasterweb \
    --disable-outbound-snat true \
    --idle-timeout 15 \
    --enable-tcp-reset true

az network nic ip-config address-pool add \
    --address-pool masternodes \
    --ip-config-name ipconfigkube-master-1 \
    --nic-name kube-master-1VMNic \
    --resource-group kubeadm \
    --lb-name kubemaster

az network nic ip-config address-pool add \
    --address-pool masternodes \
    --ip-config-name ipconfigkube-master-2 \
    --nic-name kube-master-2VMNic \
    --resource-group kubeadm \
    --lb-name kubemaster

VM provisioning

Master

The specs of my master node VM are as follows on Ubuntu 21.04 (20.04 LTS could also be used):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
  "name": "kube-master-3",

  "location": "westeurope",

  "name": "ubuntu-21-04-lts",

  "publisher": "tidalmediainc",

  "product": "ubuntu-21-04-lts",

  "vmSize": "Standard_D2ds_v4"
}

Worker

Both worker VMs share the same specs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{

"name": "zone2/zone3",

"location": "westeurope",

"vmSize": "Standard_E16-4ds_v4",


"imageReference": {

"publisher": "canonical",

"offer": "0001-com-ubuntu-server-focal",

"sku": "20_04-lts-gen2",

"version": "latest"

}

After connecting to the VM, go into root mode: sudo -s All following steps assume this!

2. Kernel downgrade

Worker

As mentioned here, recent Ubuntu kernels broke compatibility with CRIU. Hence we downgrade the kernel: apt install -y linux-image-unsigned-5.4.0-1068-azure

Follow these steps to change the boot kernel.

3. Install Prequisites

Master node

Containerd + Kubelet

On the master node, you can install the official releases:

1
2
3
4
5
6


curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

apt update

apt install -y containerd kubelet

But for safety, you should replace the kubelet with the binary path defined in the systemd service (/usr/bin/kubelet). You can get the customized kubelet like this:

1
2
3


wget https://github.com/elchead/kubernetes/releases/download/v8.1.0/kubelet
chmod +x ./kubelet
cp ./kubelet /usr/bin

Kubeadm

To get the kubeadm version compatible with our modified kubernetes. Inside your home directory, do:

1
2


wget https://github.com/elchead/kubernetes/releases/download/v8.1.0/kubeadm
chmod +x ./kubeadm

Kubectl

Install as described here

Worker node

I recommend to perform the following steps in parallel on both worker VMs. You might use tmux or iTerm on Mac ( Cmd+Shift+i).

CRIU

As described here:

1
2
3
4


echo 'deb http://download.opensuse.org/repositories/devel:/tools:/criu/xUbuntu_20.04/ /' | sudo tee /etc/apt/sources.list.d/devel:tools:criu.list
curl -fsSL https://download.opensuse.org/repositories/devel:tools:criu/xUbuntu_20.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/devel_tools_criu.gpg > /dev/null
sudo apt update
sudo apt install -y criu

Verify the version to be 3.16.1: criu --version

Golang

Follow here. I installed v1.17.8:

1
2
3


wget https://go.dev/dl/go1.17.8.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.17.8.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Containerd

Install dependencies

1


apt install btrfs-progs libbtrfs-dev runc

Systemd service

I took the dirty path and used apt to install an official release :

1
2


apt update
apt install containerd

Then, I later replaced the binary defined in the systemd config with my fork binary. The file looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


cat /lib/systemd/system/containerd.service

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

Build

I recommend to build from source, but you may also use the binaries inside bin.

Clone my fork and checkout the checkpoint branch. If you want to use the version that only uploads a zip to the file server (please read under [6. Set up file server](#6. Set up file server), use checkpoint-zip

1
2
3
4


mkdir -p /root/go/src/github.com/containerd && cd /root/go/src/github.com/containerd
git clone https://github.com/elchead/containerd.git
cd containerd
git checkout checkpoint

Build from source: make && make install

Update systemd svc to custom binary

Stop the service: `systemctl stop containerd

Inside root of the repository, do:

1
2


cp ./bin/containerd /usr/bin/containerd
systemctl start containerd

Configure CNI plugins

1
2


export GOPATH=/root/go
./script/setup/install-cni

In the output the CNI version is set to 1.0.0 which is wrong. So we change it to a supported version such as 0.3.0 : vim /etc/cni/net.d/10-containerd-net.conflist

To be safe, restart the containerd service after after configuring the CNI plugins: `systemctl restart containerd

Kubelet

1
2
3
4
5
6
7
8
9


curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update

sudo apt -y install kubelet

Then replace the binary:


1
2
3


wget https://github.com/elchead/kubernetes/releases/download/v8.1.0/kubelet
chmod +x ./kubelet
cp ./kubelet /usr/bin

Modify the systemd service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --container-runtime-endpoint=/run/containerd/containerd.sock --v=9 --read-only-port=0 --anonymous-auth=true --authorization-mode=AlwaysAllow --container-runtime=remote
~

The kubelet config should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 50m0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

It’s important to set the runtimeRequestTimeout to a higher value (default is 2 minutes), if you intend to migrate big containers (~50GB+)!

Kubeadm

Install as above for the master node:

1
2


wget https://github.com/elchead/kubernetes/releases/download/v8.1.0/kubeadm
chmod +x ./kubeadm

4. Kubernetes bootstrapping

Networking

First, we set up the networking and containerd. I followed this tutorial, but it might not be necessary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

sudo systemctl restart containerd

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Kubeadm init

On the master node, create a config file in your home directory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


vim conf.yaml
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
imageRepository: sadrian99
kind: ClusterConfiguration
kubernetesVersion: v1.19.16
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  criSocket: "/run/containerd/containerd.sock"

Through this, the registry with the customized Kubernetes components will be used.

Then, start the cluster with: ./kubeadm init --upload-certs --cri-socket "/run/containerd/containerd.sock" --config conf.yaml

Note: Ideally, the cluster is exposed with a DNS endpoint, but this did not work for me!

As the output indicates, perform:

1
2
3


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Then, copy the output to join the cluster on a worker node. It looks like this:

1
2


kubeadm join 192.168.0.7:6443 --token 90djfo.2386gkxicg6y2ywo \
    --discovery-token-ca-cert-hash sha256:0834d5d47c16799e2b1b4df3431923570549fe903f9339f875dd1f2f9d2dd2ef

Finally, set up CNI:

1


kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.19"

Verify cluster state

kubectl get node -w Should show a ready node after a few seconds.

kubectl get po -A Should show all pods running (including coredns!).

(Debugging)

If the node does not get ready, check the logs of the kubelet:

1


journalctl -u kubelet [-f]

Or if you don’t find any hints from there, look here:

1


journalctl -u  containerd [-f]

It happened to me, that I got the error cni plugin not initialized. If this is the case, be sure to repeat step the CNI plugin installation from above again.

If a pod is not running, use kubectl describe to debug.

Otherwise, a cluster reset might also help: ./kubeadm reset --cri-socket "/run/containerd/containerd.sock"

Join worker

Install crictl as prequisite:

1
2
3
4


VERSION="v1.23.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

On the worker node, run the previously copied join command. You might need to add --cri-socket "/run/containerd/containerd.sock":

1


./kubeadm join 192.168.0.7:6443 --token 90djfo.2386gkxicg6y2ywo     --discovery-token-ca-cert-hash sha256:0834d5d47c16799e2b1b4df3431923570549fe903f9339f875dd1f2f9d2dd2ef --cri-socket "/run/containerd/containerd.sock"

To verify, go back to the master node and check that the new node appears ready.

Copy kubeconfig

Now, copy the kubeconfig from the master node ($HOME/.kube/config) to the worker node inside config and export it: export KUBECONFIG=$HOME/config

The migration PoC needs the kubeconfig inside a special directory:

1
2


mkdir -p /var/lib/kubelet
cp $HOME/config /var/lib/kubelet/kubeconfig

5. Demo (test migration)

Deploy stateful pod

Now it’s time to test the migration, with a simple memory allocating pod (here 50 MB): kubectl run counter1 --restart=Never --image "ghcr.io/schrej/podmigration-testapp:latest" -- -m 50. It’s important to set restartPolicy:Never to prevent the original container from restarting during migration (relevant for large migrations)!

Through kubectl get po -owide, you can get pod IP and increment a stateful counter. Be sure to do this on the worker node: curl $POD_IP:8080 Repeat the counter increment a few times, to validate the successful migration later.

Clone pod

The pod spec is identical, except that it has an additional field spec.clonePod :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: counter1
  name: mcounter1
spec:
  clonePod: counter1
  containers:
    - args:
        - -m
        - "50"
      image: ghcr.io/schrej/podmigration-testapp:latest
      name: counter1
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

The migration should be very fast. Currently, the old pod gets broken during the migration. But the cloned pod should be running. Requesting it’s endpoint with curl should return a number bigger than 1. Voila - you have successfully cloned a stateful pod in Kubernetes!

6. Set up file server

Important warning

I had consistency problems for bigger file uploads with SMB. The container restore command is issued 1 second after the disk checkpoint has been saved completely. However, at this time not all files of the checkpoint directory were uploaded successfully.

I circumvented this problem by storing the checkpoint on local disk and only storing a zipped archive on the server. The temporary local-disk location is /var/lib/kubelet/check. Since, the OS disk is usually only 30GB, you will need to create a symbolic link to a bigger disk. In my case, a temporary disk with 500GB was mounted in /mnt. To solve this, do:

1
2


rm -r /var/lib/kubelet/check/
ln -s /mnt /var/lib/kubelet/check

Interestingly, the compression immensly reduced the checkpoint size for the simple example app. For 50GB of allocated memory, the compressed zip was only around 20MB! This modification was done inside containerd in the branch checkpoint-zip.

Steps

The procedure is specific to Azure and is well documented here. The server should be mounted inside /var/libe/kubelet/migration. I used the static mount and my /etc/fstab entry looks like this:

1


//SERVER_URL/checkpoints /var/lib/kubelet/migration cifs nofail,credentials=/etc/smbcredentials/STORAGECLASSNAME.cred,serverino,cache=none

Development

If you want to develop further or quickly test changes, it is much easier to work with a local cluster. Inside the kubernetes repo root, run:

1


CONTAINER_RUNTIME=remote CONTAINER_RUNTIME_ENDPOINT="unix:///run/containerd/containerd.sock" CGROUP_DRIVER="systemd" KUBELET_AUTHORIZATION_WEBHOOK="false" KUBELET_FLAGS="--read-only-port=0 --anonymous-auth=true --authorization-mode=AlwaysAllow" ./hack/local-up-cluster.sh

To read the logs for the kubelet, you can use: tail -f /tmp/kubelet.log.

End

I admit that this was a long tutorial and it’s likely not everything went smooth while following along. If you are stuck at some step, you can contact me and I can try to help :)

Contents

• • 1. Download helm chart repo and add to helm:
  • 2. Deploy Influxdb2
  • 3. Add Telegraf Configuration in Influxdb
  • 4. Deploy Telegraf
  • 5. View metrics data
  • 6. Connect local client with DB

InfluxDB is an Open-Source timeseries database which can be used for monitoring Kubernetes clusters. In this tutorial, I want to show you step by step how to get a dashboard with metrics on Kuberntes resource usage of podes and nodes. It’s not difficult, but the official documentation is outdated and confusing, so I hope to make it easier for you. We will use InfluxDB2 which includes a nice dashboard and supports the Flux QL query language.

1. Download helm chart repo and add to helm:

helm repo add influxdata https://helm.influxdata.com/

2. Deploy Influxdb2

No settings need to be adjusted:

helm upgrade --install influx2 influxdata/influxdb2

But you should note down the password for the admin user:

echo $(kubectl get secret influx2-influxdb2-auth -o "jsonpath={.data['admin-password']}" --namespace tick | base64 --decode)

To open the Influx UI, do:

kubectl port-forward svc/influx2-influxdb2 8081:80

Now, you can open localhost:8081 and log in with user: admin and the before copied password.

3. Add Telegraf Configuration in Influxdb

In the UI sidebar, click Data and add “Telegraf Configuration”. Choose Kubernetes and enter a name: (e.g telegraf). Then, copy the API token.

4. Deploy Telegraf

We choose the Daemonset telegraf-ds, because it is expected for the Kuberntes input plugin, which among many others observes the node and pod resources. Now open the values.yaml and add the following information under config.outputs:

    - influxdb_v2:
        urls:
          - "http://influx2-influxdb2.tick.svc"
        token: INSERT_TOKEN
        organization: "influxdata"
        bucket: "default"

The input plugins are configured by default.

Then deploy the chart:

helm upgrade --install ds-telegraf influxdata/telegraf-ds -f values.yaml

5. View metrics data

If all went well, you should now receive metrics data and see them in the UI under Explore:

6. Connect local client with DB

If you want to query the DB through a locally installed CLI client or their Go-client, you can use the port-forwarding from earlier and use this adress to connect: kubectl port-forward svc/influx2-influxdb2 8081:80.

This step was not so intuitive to me, because in InfluxDb 1 the service exposed the 8086 port for this. But now the UI and the database connection are both reachable through this port.

To test the connection with the CLI you can use:

influx ping --host http://localhost:8081

This is all there is to it. If you have any questions or find that my tutorial got outdated, please contact me :)

Integration vs Operation

When solving a problem the process is always to break it up into smaller pieces. The problem solution is then just the composition of the smaller units. So we define functions for each subproblem and one integration function to solve the big problem. There should be a clear distinction between operation and integration (also see Integration Operation Segregation Principle). Each small unit should be independent, i.e. unaware of the other parts in the composition.

Mocking is required when our decomposition strategy has failed, Eric Elliott

Function composition

What was new to me is that composing functions do not need to be unit tested when they are truly independent. Because in such case we can use a generic composition utility.

Let's look at an example[^1]. The imperative and obvious solution to integrate is this:

[^1]: adopted from Source

// Imperative composition
const composition = (x) => {
  const afterG = g(x);
  const afterF = f(afterG);
  return afterF;
};

In languages without first-class functions, there might be no way around this. But in most popular languages, such as JavaScript, you can do better. Function composition is declarative and avoids bugs such as passing or returning the wrong variable.

For the declarative composition, you can either define your own pipe (which could be error-prone) or use a library[^1]:

// import pipe from 'lodash/fp/flow';
const pipe = (...fns) => x => fns.reduce((y, f) => f(y), x);
// Functions to compose
const g = n => n + 1;
const f = n => n * 2;
// Declarative composition
const doStuffBetter = pipe(g, f);
doStuffBetter(20), // 42

Note that reduce applies the accumulator on each value from left to right! There is also a reverse variant called reduceRight. This reduction would have given 41 as result.

Commonly, we have asynchronous calls in our code, but we can also compose promises! I think this is also where it really pays off - when you compose calls with side effects. Unit testing the integration function becomes tedious, because we need to stub all participants[^1]:

// imperative
async function uploadFiles({ user, folder, files }) {
  const dbUser = await readUser({ user });
  const folderInfo = await getFolderInfo({ folder });
  if (await haveWriteAccess({ dbUser, folderInfo })) {
    return uploadToFolder({ dbUser, folderInfo, files });
  } else {
    throw new Error("No write access to that folder");
  }
}

// declarative
const asyncPipe =
  (...fns) =>
  (x) =>
    fns.reduce(async (y, f) => f(await y), x);
const uploadFiles = asyncPipe(
  readUser,
  getFolderInfo,
  haveWriteAccess,
  uploadToFolder
);

uploadFiles({ user, folder, files }).then(log);

As you see, the declarative uploadFiles function is just a function call - no logic to be tested! The correctness of the step order is not assured, but in most cases this is covered in integration tests. If it's complex logic, you might still write a unit test to test the correct step order of the composition.

I agree that the declarative implementation (asyncPipe) is more difficult to understand at first, but it is less error prone and more concise. asyncPipe is given an array of functions that it reduces. y is the previous result and we apply f on it's result. The second return value x is the initial value. This syntax confused me a bit, but the initial value obviously needs to be provided at some place. This is functional programming!

The merits of Functional Programming

The paradigm leads to code that is easier to test, because it is a stateless input / output machine. Moreover, it leaves less room for bugs, because you only declare what you want to perform instead of how to do it (imperative). However, it's not possible to only rely on functional programming, because applications are stateful and have side effects (network requests, file operations, logging...). A good strategy is to keep the business logic functional and move side effects to the outer boundary.

Let's consider the example of an online shop that decides to give some premium benefits to its loyal customers. Premium customers might benefit from special discounts and free shipping so we want to update their status in the database. On the other hand, we might want to reach out to the inactive customers. The status of our customer is clearly business logic and it should not be mixed with database logic. One approach would be to pass a database interface and mock it in the test.

function updateCustomer(today,entry,db) {
    if(isLoyalCustomer(today,entry.date)){
        newEntry = {premium: true, ...entry}
        db.UpdateEntry(newEntry)
    }
    if(isInactiveCustomer(today,entry.date){
        newEntry = {inactive: true, ...entry}
        return db.UpdateEntry(newEntry)
    })
}

But there is a better approach that is declarative and easier to test:

function updateCustomer(today,entry) {
    if(isLoyalCustomer(today,entry.date)){
        newEntry = {premium: true, ...entry}
        return new FileUpdate(newEntry)
    }
    if(isInactiveCustomer(today,entry.date){
        newEntry = {inactive: true, ...entry}
        return new FileUpdate(newEntry)
    })
    return new NoUpdate(entry)

}

It's clearer in the intent that the output is a return value instead of an input value with side effects. Also, there is no need to mock! Of course there is still a missing piece for this variant - the mutable shell that applies the side effects. In our case, there would be a Persister, which is a database wrapper that reads the update instructions. It's correctness would be covered in the integration test.

When to use mocks

Be aware that mocks are sometimes the only way to test logic. But think what exactly needs to be tested. For testing the request handler of our express app, it's not necessary to create a mockserver. We only want to test the handler logic. Express logic to create the server with port allocation etc. does not need to be tested by us!

I find the distinction between handlers and servers especially clear in Golang:

handler := newHelloHandler()

request := newRequest("Floyd")
response := httptest.NewRecorder()

handler.ServeHTTP(response, request)
assertStatus(t, response.Code, http.StatusOK)
assertResponseBody(t, response.Body.String(), "Hello Floyd")

The handler object includes all logic how to process requests, so we can use it to test the correct behavior. Note how only the response needs to be mocked (spied to be precise).

In Express.js, we do the same. We test the handler function and spy the response object.

const helloHandler  = (req, res) => res.send('Hello World!');

const expected = 'Hello World!';
const resSpy = {
    send: (actual) => assert.Equal(actual,expected)
}
hello({}, resSpy);

The request handler is at the outer layer of our application, i.e. it connects with external dependencies to cause side effects. In our case the external dependency is the response object which makes a network call. In that case it is justified and necessary to mock. The general guideline is to only mock external dependencies.

Conclusion

By pushing the side effects to the outer layer of the application, we can mostly avoid mocks. Functional code is easy to test and allows to use function composition. This reduces the surface for bugs and saves us to write unit tests for integration functions. Mocks have its place, but if it's not an external dependency it might be code smell.

If you want to learn more about testing, I can highly recommend Unit Testing Principles, Practices, and Patterns.