Last updated on June 6th, 2022 at 08:28 pm
In this guide, we are going to learn on how to install and configure Ceph Block Device on Ubuntu 18.04. Ceph Block Devices is one of the deployments options of Ceph Storage Cluster. Other deployments include Ceph Object Storage and Ceph File System.
Ceph block device is also known as Reliable Autonomic Distributed Object Store (RADOS) Block Device (RBD). They interact with Ceph OSDs using the librbd library.
Ceph can be mounted as block device just like how the normal hard drives are mounted. When data is written to Ceph block device, it is stripped across multiple Object Storage Devices (OSD) in a Ceph storage cluster.
RBD integrates with KVM and hence, can be used to provide block storage devices for cloud computing systems such as OpenStack
Install and Configure Ceph Block Device on Ubuntu 18.04
Before you can proceed, ensure that you have a running Ceph storage cluster.
Follow the guide below to learn how to install and setup Ceph Storage Cluster on Ubuntu 18.04.
Setup Three Node Ceph Storage Cluster on Ubuntu 18.04
Deployment Architecture
Prepare Ceph Client for Ceph Deployment
Create Ceph Deployment User
On a Ceph client, create the ceph user with passwordless sudo rights for installing ceph packages and configurations just like it was done on the Ceph OSD nodes. Do not use the username ceph as it is reserved.
Replace cephadmin username accordingly.
useradd -m -s /bin/bash cephadmin
passwd cephadmin
echo -e "Defaults:cephadmin !requiretty\ncephadmin ALL=(ALL:ALL) NOPASSWD:ALL" >> /etc/sudoers.d/cephadmin
chmod 0440 /etc/sudoers.d/cephadmin
Configure Time Synchronzation
To ensure that the time is synchronized between the Ceph cluster and the Ceph client, configure your Ceph client to use the same NTP server as your Ceph cluster.
apt install chrony
vim /etc/chrony/chrony.conf
... # pool ntp.ubuntu.com iburst maxsources 4 # pool 0.ubuntu.pool.ntp.org iburst maxsources 1 # pool 1.ubuntu.pool.ntp.org iburst maxsources 1 # pool 2.ubuntu.pool.ntp.org iburst maxsources 2 pool ntp.kifarunix-demo.com iburst ...
Restart and enable chronyd to run on system boot.
systemctl enable chronyd
systemctl restart chronyd
Install Python 2
Python 2 is required to deploy Ceph on Ubuntu 18.04. You can install Python 2 by executing the command below;
apt install python-minimal
Setup Password-less SSH login to Ceph Client
Login as Ceph user you have created on your Ceph Admin node, generate password-less SSH keys and copy them to the client.
We have already done this in our guide on setting up Ceph cluster on Ubuntu 18.04. If you followed the guide, simply update the user SSH configuration file, ~/.ssh/config, with the connection details of the Ceph client.
su - cephadmin
vim ~/.ssh/config
Host ceph-osd01 Hostname ceph-osd01 User cephadmin Host ceph-osd02 Hostname ceph-osd02 User cephadmin Host ceph-osd03 Hostname ceph-osd03 User cephadmin Host ceph-client Hostname ceph-client User cephadmin
Update the hosts file, if you do not have a DNS server.
echo "192.168.2.118 ceph-client.kifarunix-demo.com ceph-client" >> /etc/hosts
Next, copy the SSH keys to Ceph client.
ssh-copy-id ceph-client
Install Ceph Utilities on Ceph Admin Node
Run the command below on Ceph Admin node to install common utilities used to mount and interact with ceph storage cluster. These are provided by the package, ceph-common.
sudo apt install ceph-common
Install Ceph on Ceph Client
Install Ceph packages on the Ceph client by running the command below. Run Ceph deployment from the ceph user configuration directory.
su - cephadmin
cd kifarunix-cluster/
ceph-deploy install ceph-client
Copy Ceph configuration and keys to Ceph client.
ceph-deploy admin ceph-client
Create Block Device Pools
In order to use the Ceph block device client, you need to create a pool for the RADOS Block Device RBD) and initialize it.
- A pool is a logical group for storing objects. They manage placement groups, replicas and the CRUSH rule for the pool.
- A placement group is a fragment of logical object pool that places objects as a group into OSDs. Ceph client calculates which placement group an object should be stored.
To create Ceph pool use the command below;
ceph osd pool create {pool-name} pg_num pgp_numWhere:
{pool-name}is the name of the Ceph pool you are creating.pg_numis the total number of placement groups for the pool. It is recommended thatpg_numis set to 128 when using less than 5 OSDs in your Ceph cluster.pgp_numspecifies total number of placement groups for placement purposes. Should be equal to the total number of placement groups.
Hence, on the Ceph Client, run the command below to create a pool called kifarunixrbd.
[email protected]:~$ ssh ceph-ceph-client
Next, create an OSD pool with placement group number 128
[email protected]:~$ sudo ceph osd pool create kifarunixrdb 128 128
Associate the Pool created with the respective application to prevent unauthorized types of clients from writing data to the pool. An application can be;
cephfs(Ceph Filesystem).rbd(Ceph Block Device).rgw(Ceph Object Gateway).
To associate the pool created above with RBD, simply execute the command below. Replace the name of the pool accordingly.
[email protected]:~$ sudo ceph osd pool application enable kifarunixrdb rbd
Once you have created and enabled an application for a pool, you can initialize it using the command below;
[email protected]:~$ sudo rbd pool init -p kifarunixrdb
To list available pools;
sudo ceph osd lspools
Creating Block Device Images
Create an image for a block device in the Ceph storage cluster before adding it to a node using the command below in a Ceph Client.
rbd create <image-name> --size <megabytes> --pool <pool-name>
For example, to create an block device image of 1GB in our pool created above, kifarunixrbd, simply run the command;
[email protected]:~$ sudo rbd create disk01 --size 1G --pool kifarunixrbd
To list the images in your pool, eg kifarunixrbd;
sudo rbd ls -l kifarunixrbd
NAME SIZE PARENT FMT PROT LOCK disk01 1 GiB 2
To retrieve information about the image created, run the command;
sudo rbd --image disk01 -p kifarunixrbd info
rbd image 'disk01':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
id: 37886b8b4567
block_name_prefix: rbd_data.37886b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Sat Mar 14 21:39:19 2020To remove an image from the pool;
sudo rbd rm disk01 -p kifarunixrbd
To move it to trash for later deletion;
sudo rbd trash move kifarunixrbd/disk01
To restore image from trash to the pool, obtain the image ID as assigned on the trash store then restore the image using the ID;
sudo rbd trash list kifarunixrbd
37986b8b4567 image01
Where kifarunixrbd is the name of the pool.
sudo rbd trash restore kifarunixrbd/37986b8b4567
To empty image from trash for permanent deletion;
rbd trash remove kifarunixrbd/37986b8b4567
Mapping Images to Block Devices
After creating an Image, you can map it to block devices using. However, before you can map the image, disable any features that are unsupported by the kernel by running the command below. Replace the pool and image names.
[email protected]:~$ sudo rbd feature disable kifarunixrbd/disk01 object-map fast-diff deep-flatten
The RBD kernel module will be loaded by rdb if it is not already loaded.
Next, map the image to block device.
sudo rbd map disk01 --pool kifarunixrbd
/dev/rbd0
To show block device images mapped to kernel modules with the rbd command;
[email protected]:~$ sudo rbd showmapped
id pool image snap device 0 kifarunixrbd disk01 - /dev/rbd0
To unmap a block device image, use the command, rbd unmap /dev/rbd/{poolname}/{imagename} for example;
sudo rbd unmap /dev/rbd/kifarunixrbd/disk01
Create FileSystem on Ceph Block Device
The Ceph mapped block device is now ready. All is left is to create a file system on it and mount it to make it useable.
For example, to create an XFS file system on it;
sudo mkfs.xfs /dev/rbd0 -L cephbd
meta-data=/dev/rbd0 isize=512 agcount=9, agsize=31744 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Mounting Ceph Block Device on Ubuntu 18.04
You can now mount the block device. For example, to mount it under /media/ceph directory;
sudo mkdir /media/ceph
sudo mount /dev/rbd0 /media/ceph
You can as well mount it as follows;
sudo mount /dev/rbd/kifarunixrbd/disk01 /media/ceph/
Check mounted Filesystems;
df -hT -P /dev/rbd0
Filesystem Type Size Used Avail Use% Mounted on /dev/rbd0 xfs 1014M 34M 981M 4% /media/ceph
There you go.
If you check the Ceph cluster health;
[email protected]:~# ceph --status
cluster:
id: ecc4e749-830a-4ec5-8af9-22fcb5cadbca
health: HEALTH_OK
services:
mon: 2 daemons, quorum ceph-osd01,ceph-osd02
mgr: ceph-osd01(active), standbys: ceph-osd02
osd: 3 osds: 3 up, 3 in
data:
pools: 1 pools, 128 pgs
objects: 18 objects, 14 MiB
usage: 3.1 GiB used, 8.9 GiB / 12 GiB avail
pgs: 128 active+clean
That marks the end of our guide on how to install and Configure Ceph Block Device on Ubuntu 18.04.
Reference
Other Tutorials
Install and Setup GlusterFS on Ubuntu 18.04
How to Install and Configure NFS Server on RHEL/CentOS 7
