Introduction
I started learning to code at home during covid in fall 2020.
I went through the excellent CS50x https://cs50.harvard.edu/x/2025/ by David J. Malan.
Then continued with different free resources, mostly:
I then worked freelance for a while and a few months later in 2021 got hired as a junior backend engineer by a fintech startup.
Up until then I had been deploying things using docker and docker compose but discovered Kubernetes upon joining.
To be able to understand what it was about I thought I would build a home lab and experiment.
Not having prior knowledge to run a cluster I decided instead to build a single desktop machine and to virtualize the cluster within.
I got to experiment with different combinations of tools such as:
| category | website | feedback |
|---|---|---|
| VM management | https://canonical.com/multipass | CLI was nice but unstable, had episodes of stuck VMs, etc. Also was limited in passthrough capabilities. |
| VM management | https://canonical.com/lxd | Was lighter than multipass but still could have issues with non responsive VMs. |
| VM management | https://www.proxmox.com/en/ | What I use now. It just works, passthrough is easy. UI is nice and it is comfortable to be able to just open a shell to a regular Debian machine |
| Distributed storage | https://openebs.io/ | Did not use it very long, not a good memory |
| Distributed storage | https://rook.io/docs/rook/v1.9/ceph-storage.html | Only used Rook to connect to an external Ceph cluster |
| Distributed storage | https://ceph.io/en/ | Ceph cluster using 3 VMs as nodes. It was unstable, had corruption, had to debug manually in the object store, lost some data, gave up and reverted to Longhorn. |
| Distributed storage | https://min.io/ | Rock solid S3 solution. I use it as SNSD because multi disks was actually less resilient for me. |
| Distributed storage | https://juicefs.com/ | Used for a while to mount a fuse file system accessible to Kubernetes pods on top of MinIO. It was working well but also constantly writing. |
| Distributed storage | https://longhorn.io/ | Still what I use, it improved in stability over time. Volumes are backed up automatically to MinIO S3. |
| Distributed storage | https://github.com/democratic-csi/democratic-csi | Used along with TrueNAS to provide PV to Kubernetes |
| Distributed storage | https://www.truenas.com/truenas-scale/ | I still use it to provide iSCSI (block storage) as well as NFS to Kubernetes |
| Local storage | - | Used openEBS and k3s implementations with real volumes or using loop devices mounted on .img files. I still occasionally use local path for things like LLM models since I only have one node with a GPU anyway. |
| Kubernetes distribution | https://microk8s.io/ | Used for a few months, did not leave a great impression. |
| Kubernetes distribution | https://k3s.io/ | Used for the past 3 years, it's good |
| Kubernetes distribution | https://k3d.io/v5.6.3/ | It's nice to test quickly something in docker |
| Kubernetes distribution | https://minikube.sigs.k8s.io/docs/ | Alternative to the above |
Over time I invested and my current hardware is the following:
And from the above table, what I currently still use:
- Proxmox for the system of the 3 nodes
- Ubuntu server/desktop for the VMs
- k3s
- Longhorn for in-cluster storage on top of each node NVME disks (virtualized through proxmox)
- TrueNAS for out of cluster storage via democratic-csi on top of physical HDD for NFS as well as virtual NVME storage for iSCSI persistent volumes.
- MinIO for S3 with a dedicated HDD formatted in XFS
For the past 3 years I have been deploying and testing different self hostable open source projects. Some of them I have adopted and use everyday as permanent replacement of cloud solutions while others I have discarded temporarily or forever.