Today, we create a ceph network storage on our Centos 8 Stream with cephadm command. We will installing with manual page: https://docs.ceph.com/en/pacific/install/
In this example, we will have three systems (nodes), with identical HW resources (4 GB ram, 4 vCPU, two NICs – one internal for ceph and one for world, and dedicated 4 TB SSD disk for ceph storage). In this article, every command must be run on all nodes. Public network is 192.168.1.0/24 and Ceph separate network is 192.168.2.0/24
So, as a first step, we need to set up our Centos for synchronized time.
Setting up time
As the first step, we must set up a time, I use chrony:
dnf install chrony -y systemctl enable chronyd timedatectl set-timezone Europe/Bratislava timedatectl
Now, edit some variables in configurations file for chronyd. Add some servers from pool, and edit local subnets, where we delived time:
vim /etc/chrony.conf pool 2.centos.pool.ntp.org iburst pool 1.centos.pool.ntp.org iburst pool 3.centos.pool.ntp.org iburst
Now start/restart our service, and check, if it is working:
systemctl restart chronyd systemctl status chronyd.service chronyc sources
Create hostnames, ssh rsa-keys and update
Now, we must edit on all nodes our hostnames, set it permanent:
hostnamectl set-hostname ceph1
Now, add all hostnames, and IPs to file /etc/hosts:
tee -a /etc/hosts<<EOF 192.168.1.1 ceph1 192.168.1.2 ceph2 192.168.1.3 ceph3 EOF
Now, create rsa-key pair, for password-less connect to and from each node for root user for installing and updating:
ssh-keygen -t rsa -b 4096 -C "ceph1" means: -b bits. Number of bits in the key to create -t type. Specify type of key to create -C comment
And copy it to other nodes:
for host in ceph1 ceph2 ceph3; do ssh-copy-id root@$host done
Now update:
dnf update -y dnf install podman gdisk jq -y -reboot-
Preparing for ceph
Now, setup a yum/dnf based repository for ceph packages and updates and install package cephadm:
dnf install -y centos-release-ceph-pacific.noarch dnf install -y cephadm
Now, we can Bootstrap a new cluster. The first step in creating a new Ceph cluster is running the cephadm bootstrap command on the Ceph cluster’s first host. The act of running the cephadm bootstrap command on the Ceph cluster’s first host creates the Ceph cluster’s first “monitor daemon”, and that monitor daemon needs an IP address. You must pass the IP address of the Ceph cluster’s first host to the ceph bootstrap command, so you’ll need to know the IP address of that host.
Now, we can bootstrap our first monitor by command (only on one node!!!):
cephadm bootstrap --mon-ip 192.168.1.1 --cluster-network 192.168.2.0/24
And after some time, we have a working cluster. And we can connect to dashboard.
Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... firewalld ready Enabling firewalld port 8443/tcp in current zone... Ceph Dashboard is now available at: URL: https://ceph1.example.com:8443/ User: admin Password: tralala Enabling client.admin keyring and conf on hosts with "admin" label You can access the Ceph CLI with: sudo /usr/sbin/cephadm shell --fsid xxx -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/pacific/mgr/telemetry/
Now, we can log in with web interface (dashboard). At first login, we use mentioned username and password and we have to change our password.
But our cluster is not finished, yet 🙂
So, we continue. On first node (ceph1), where we bootstrap ceph, we can view status ceph by command:
cephadm shell -- ceph -s
cluster:
id: 77e12ffa-c017-11ec-9124-c67be67db31c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph1 (age 26m)
mgr: ceph1.rgzjga(active, since 23m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
But checking status of ceph by this way is difficulty, so we install ceph-common package by cephadm on every node:
cephadm add-repo --release pacific
cephadm install ceph-common
ceph status
cluster:
id: 77e12ffa-c017-11ec-9124-c67be67db31c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph1 (age 27m)
mgr: ceph1.rgzjga(active, since 24m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
Now, we copy ceph ssh pubkeys to other hosts for working each-other with ceph and passwordless:
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2 ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3
And now, we can add this nodes to ceph, runnig from first node (where we bootstrap ceph).After this commands, wait some time, for podman to deploy containers (monitor, manager). And label them as admin.
ceph orch host add ceph2 192.168.1.2 ceph orch host add ceph3 192.168.1.3 ceph orch host label add ceph2 _admin ceph orch host label add ceph3 _admin
Now, we can look, which disks are available for us:
ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS ceph1 /dev/sda ssd QEMU_HARDDISK_drive-scsi2 4000G Yes ceph2 /dev/sda ssd QEMU_HARDDISK_drive-scsi2 4000G Yes ceph3 /dev/sda ssd QEMU_HARDDISK_drive-scsi2 4000G Yes
So, now we can create an OSD disks from these devices.
ceph orch daemon add osd ceph1:/dev/sda
Created osd(s) 0 on host 'ceph1'
ceph orch daemon add osd ceph2:/dev/sda
Created osd(s) 1 on host 'ceph2'
ceph orch daemon add osd ceph3:/dev/sda
Created osd(s) 2 on host 'ceph3'
ceph -s
cluster:
id: 77e12ffa-c017-11ec-9124-c67be67db31c
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph1,ceph3,ceph2 (age 8m)
mgr: ceph1.vsshgj(active, since 8m), standbys: ceph3.ctsxnh
osd: 3 osds: 3 up (since 21s), 3 in (since 46s)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 15 MiB used, 11 TiB / 11 TiB avail
pgs: 1 active+clean
Now, if we want create a ceph filesystem cephfs, we must create two pool. One for data and one for metadata. so, execute commands below on one ceph node:
ceph osd pool create cephfs_data ceph osd pool create cephfs_metadata ceph fs new cephfs cephfs_metadata cephfs_data
Now, log into Ceph dashboard a we can see, that health is RED and there is error.We must create a mds services:
This deploys mds services to our nodes (one become active and one become standby).
Now, we can continue by command line and create an user, which can mount and write to these cephfs:
ceph auth add client.cephfs mon 'allow r' osd 'allow rwx pool=cephfs_data' ceph auth caps client.cephfs mds 'allow r,allow rw path=/' mon 'allow r' osd 'allow rw pool=cephfs_data' osd 'allow rw pool=cephfs_metadata' #and see our caps: ceph auth get client.cephfs [client.cephfs] key = agvererbrtbrttnrsasda/a5/dd== caps mds = "allow r,allow rw path=/" caps mon = "allow r" caps osd = "allow rw pool=cephfs_data"
Now, we can export or copy out our key and save it to a file. And now, we can mount those cephfs on another linux:
mount -t ceph ceph1.example.com:/ /mnt/cephfs -o name=cephfs,secretfile=/root/cephfs.key -v df -h Filesystem Size Used Avail Use% Mounted on 192.168.1.1:/ 3.5T 0 3.5T 0% /mnt/cephfs
If we want check, if we have enabled compression, so, execute:
ceph osd pool get cephfs_data compression_algorithm
Error ENOENT: option 'compression_algorithm' is not set on pool 'cephfs_data'
ceph osd pool get cephfs_data compression_mode
Error ENOENT: option 'compression_mode' is not set on pool 'cephfs_data'
If we want compression, enable it and set algorithm. Mode of compression, you can learn: read about – https://docs.ceph.com/en/latest/rados/operations/pools/
We can see, that there are data, but no compression:
ceph df detail
So enable it on both cephfs pools:
ceph osd pool set cephfs_data compression_mode aggressive
ceph osd pool set cephfs_data compression_algorithm lz4
ceph osd pool set cephfs_metadata compression_mode aggressive
ceph osd pool set cephfs_metadata compression_algorithm lz4
# and see:
ceph osd pool get cephfs_data compression_algorithm
compression_algorithm: lz4
ceph osd pool get cephfs_data compression_mode
compression_mode: aggressive
And after som copy data, we can see:
Have a nice day