What is Microceph and Ceph
Ceph is a highly scalable, open-source distributed storage platform. It supports object, block, and file storage, and is designed for large-scale, enterprise-level environments. Ceph provides robust features for redundancy, fault tolerance, and scalability, making it suitable for complex and demanding storage needs.
MicroCeph is a lightweight deployment and management tool for Ceph clusters. It simplifies the process of setting up and operating a Ceph-based distributed storage system, making it accessible even for smaller-scale environments like home clusters based, for example, on Raspberry Pi.
How to add a host to a cluster?
Once you setup your first Microceph node and initialized (bootstrapped) the cluster, it’s time to start adding more hosts. After all, a cluster with only one node does not make sense!
To do it, on every node you want to add, run following commands:
Install Microceph via snap
sudo snap install microceph
On the node you initialized the cluster run
sudo microceph cluster add {node-name}
As a response you will get a token like
eyJzZWNyZXQiOiI1NDgyYjAzMjExMzMyNjkwNWZiM2FiMzMwZGZkY2QwNTdjNDgzZmNmN2MyMjQwOTUzYWQxMWQ3ODQxMDY4OTkyIiwiZmluZ2VycHJpbnQiOiIzMTA3ZDkxYWNhZjZhZGRmNzk1MmJhYzBkZTM0NDZhM2JhNDlmZTU1ZjkxZTE1MDViNzA3NTZmMzdjNWVjOTY2Iiwiam9pbl9hZGRyZXNzZXMiOlsiMTAuMC4yLjI6NzQ0MyIsIjEwLjAuMi4zOjc0NDMiLCIxMC4wLjIuNDo3NDQzIiwiMTAuMC4yLjU6NzQ0MyJdfQ==
Copy it to the clipboard, in will be needed in the next step
On the new node run
sudo microceph cluster join {token}
At this point, your node is a member of a cluster! However, it’s not the end, as it still does not contribute the storage. We still have to add a disk.
As most probably your Raspberry Pi does not have separate drive attached you can spare as a Ceph OSD, we can just make a file that will act as a storage spece. Following line adds a 80 gigabytes OSD to the cluster
sudo microceph disk add loop,80G,1
Finally, we can check if everything works like expected by running:
sudo microceph status
and
sudo microceph.ceph status

How to remove a host
Host removal from Microceph cluster is basically an oposite of the adding process.
First let’s check if our cluster is in a good health.
sudo microceph.ceph status

Remember that your cluster needs at least as many OSD (and usually nodes) as it’s replication factor. By default, replication factor is 3, so at least 3 nodes should always exist. I my case I have 4 healthy nodes: rpi502, rpi503, rpi504 and rpi505. Plus, one not healthy rpi501 but we will not worry about it at the moment. I want to remove the node rpi502 and cluster has enough spares to handle that.
First let’s see which disks (OSD) are attached to which nodes by showing OSD tree
sudo microceph.ceph osd tree

Or just by listing all disks
sudo microceph disk list

In both cases result is the same. Host rpi502 has one OSD attached with ID 11.
So, first let’s remove the OSD
sudo microceph disk remove osd.11
And then remove host rpi502 from the cluster
sudo microceph cluster remove rpi502
Finally, run
sudo microceph status
and
sudo microceph.ceph status
to confirm Ceph cluster is still in a good shape.
Leave a Reply