Ceph

This is the next post in the Current Homelab series, where I give an overview of what my lab is currently looking like. This time, I will be talking about my storage layer, which is mostly made up of Ceph. I chose Ceph around spring 2021, when I decided to go from a baremetal+docker-compose setup to a VM based setup with LXD. At the time, my main storage consisted of a pair of WD Red 4TB disks for my main storage requirements, and a 60GB crucial SATA SSD for my server’s root FS. While going through the LXD docs, I saw that it supported something called “Ceph RBD” for its VM volumes. ...

In the course of spreading my homelab over a couple more machines, I finally arrived at the Ceph cluster’s MON daemons. These were running on three Ceph VMs on my main x86 server up to now. In this post, I will describe how I moved them to three Raspberry Pis. While the cluster was up the entire time. First, a couple of considerations: MON daemons use on average about 1GB of memory in my cluster My cluster, and most of my services, went down during the migration. So please be cautious if you plan to do your own migration The MON daemons are something of a control plane for Ceph clusters. They hold the MON map of daemons and data locations. Every client which uses the Ceph cluster will use them to access a map of available OSDs to work with. ...

Today, I wanted to migrate my single Ceph NFS daemon from my LXD VM on an x86 server to my Hardkernel H3. This did not work out too well. Yesterday, I had finally finished the migration of my Ceph cluster’s MON daemons to my three Raspberry Pi controllers. This move had it’s own problems, which I will detail a bit in another blog entry. Part of this migration was to also move over the MGR daemons to my Pis. This actually worked without any problem - I thought. ...