[Tutorial]: Bootsrapping your custom Kubernetes with kubeadm
3 min read
In this tutorial, I want to show you how to bootstrap a Kubernetes cluster with
kubeadm using your customized Kubernetes fork. This might be useful if you want to use new features that are not yet merged in the upstream. For development, it’s of course much easier to set up a local cluster (
./hack/local-up-cluster.sh), but to test functionality across different nodes, you might need a distributed cluster.
One option is to install it The hard way, but I think it’s more convenient to use
kubeadm. The challenge is that you need container images for the kube control-plane components.
I found it difficult to find good documentation on this process, so I hope to help you along with this.
By default Kubeadm pulls the Kubernetes version matching the version of kubeadm from the official registry. But it also allows to specify a custom registry. To install our custom Kubernetes, we will need to build and pull the required container images, tag them with the expected version of kubeadm, and the push them to our registry.
I built the images on a Ubuntu machine, since the build is resource intensive. Kubernetes is big, so be sure to have enough space on the VM! The build happens inside a docker container, so make sure to have docker installed. VMs usually only have a small disk image, so I created a symbolic link to save the docker files on another attached drive (here mounted on
1 2 3 4 5 6 sudo -s systemctl stop docker rm -rf /var/lib/docker mkdir -p /mnt/docker ln -s /mnt/docker /var/lib/docker systemctl restart docker
Then, inside the kubernetes root path, specify your docker registry and run:
KUBE_SERVER_PLATTFORMS="linux/amd64" KUBE_DOCKER_REGISTRY="sadrian99" KUBE_RELEASE_RUN_TESTS=n ./build/release-images.sh I recommend to set a flag for the image version such that the version coincides with what kubeadm expects (version of kubeadm). In my case, this was
Tag images and push to registry
The generated image names have a trailing
amd64 which needs to be removed for
IMGN: new name for kubadm
IMG: old image name
VER: old version tag
VERN: new version tag
REPO: your container registry url In my case,
VERN=v.19.16. Then we tag the image with the new name:
1 2 export IMG=kube-apiserver-amd64; export IMGN=kube-apiserver docker tag $REPO/$IMG:$VER sadrian99/$IMGN:$VERN && docker push $REPO/$IMGN:$VERN```
You need to do this for these images:
Additionally you need to pull these images and copy them to your registry:
export D=coredns:1.6.7 docker pull k8s.gcr.io/$D && docker tag k8s.gcr.io/$D $REPO/$D && docker push $REPO/$D
You can test if the images are all available:
kubeadm config images pull --image-repository sadrian99
Then you can add
--image-repository sadrian99 to the
init command to bootstrap your cluster.
Congrats, you should now have a cluster with your own Kubernetes version! I used this to migrate pods across nodes. Let me know what your use case is!