Ceph installation inside Kubernetes can be provisionned using Rook. Currently doing an internship at Adaltas, I was in charge of participating in the setup of a Kubernetes (k8s) cluster. To avoid breaking anything on our production cluster, we decided to experiment the installation of a k8s cluster on 3 virtual machines (one master node n1, two worker nodes n2 and n3) using Vagrant with VirtualBox on the backend and Ubuntu 18.10 as the OS.

During the installation of the test cluster, we encountered a problem with Rook using Ceph that prevented it from provisioning any Persistent Volume Claims (PVC). This article will detail how to make a basic installation of Rook with Ceph on virtual machines, the problem we experienced and how to solve it. But first…

…a quick reminder about the role of PVCs!

When a pod needs to store various data (logs or metrics for example) in a persistent fashion, it has to describe what kind of storage it needs (size, performance, …) in a PVC. The cluster will then provision a Persistent Volume (PV) if one matches the requirements of the PVC. The PV can either be provisionned statically if an administrator manually created a matching PV or dynamically. Manually creating PVs can be time consuming if a lot of them are required by pods, which is why it is interesting for the cluster to be able to provision them dynamically. To make the cluster able to dynamically provision a PV, the PVC must indicate a Storage Class it wants to use. If such a Storage Class is available on the cluster, a PV will be dynamically provisionned to the pod.

Here are some links you can follow if you want to learn more about PVCs, PVs and Storage Classes:

Installing Rook on your k8s cluster

The installation of a k8s cluster being out of the scope of this article, I will assume you already have a working k8s cluster up and running. If it is not the case you can easily find some documentation on the internet on how to quickly bootstrap a k8s cluster.

The process of installing Rook isn’t hard, it’s just a matter of applying some manifests. First step, clone the Rook git repo:

Then switch to the latest release tag (which is v1.0.1 at the time of this writing) using:

The files of interests (which are listed below) are located inside the folder cluster/examples/kubernetes/ceph.

  • common.yaml
  • operator.yaml
  • cluster.yaml
  • storageclass.yaml

Apply each of them in the order listed above using:

One last step is to set the storageclass resource, defined inside of the storageclass.yaml file we just applied, to be as the default storageclass in our cluster. This is achieved with the command:

The problem

The Rook cluster will take some time to deploy, while it pulls the Rook images and deploy the pods. After a few minutes the output of kubectl get pods -n rook-ceph should look like this:

As we can see here, there are two pods called rook-ceph-osd-prepare... for which status is “Completed”. We expected some Object Storage Device (OSD) pods to appear once the status of the rook-ceph-osd-prepare... pods is completed but it is not the case here. Since the OSD pods are not appearing, anytime we will have a PVC, it won’t be provisioned by Rook and stays pending. We can see an exemple of this happening when trying to deploy a Gitlab instance with Helm. Here is the result of kubectl get pvc -n gitlab:

We can see no PVC is being provisioned even though they are assigned the correct Storage Class.

The solution

After some research we found that in fact, in order for Rook to work, we need to have a dedicated Storage Device that it can use to store the PVs. To fix this, we needed in our case to add a new virtual disk to our VMs through the VagrantFile file.

To create and attach a new virtual disk to a VirtualBox VM, we need to use the vboxmanage command or we can more conveniently define it directly in the VagrantFile like in this extract:

If we run again the whole installation process, we can see that the OSD pods are appearing:

And if we check the PVCs created by the Helm installation of Gitlab:

The PVCs are finaly provisioned!

A step further, customizing cluster.yaml

You may have noticed that we didn’t give any information to Rook on how to find an appropriate device to use for storage; it just autonomously detected the one we attached to the VM and used it. For a lot of obvious reasons, this certainly isn’t a desired behavior as in a real-life context we could have numerous devices attached having different purpose than simply providing storage to Rook. It is of course possible to customize the way it finds and use the storage. It is defined inside the cluster.yaml manifest. The related category of the manifest is storage. Below is the default configuration:

The useAllDevices field is set to true. From the official documentation of Rook: it indicates “whether all devices found on nodes in the cluster should be automatically consumed by OSDs“. The solution is to indicate to Rook where to look instead of automatically select any available device. If we set useAllDevices to false, we can use the following fields:

  1. deviceFilter to set a regex filter; for example ^sd[a-d] to find a device that starts with “sd” followed by a, b, c or d,

  2. devices to define a list of individual devices that will be used,

  3. directories to set a list of directories which will be used as the cluster storage.

It is also possible to define per-node configurations by setting useAllNodes to false, but this is out of the scope of this article. If you want to learn more about storage configuration for Rook, please take a look a the documention.

The end

Thank you for having read this article, I hope it has brought you some light if you were having the same problem!