Shared Storage (Ceph) - Funky Penguin's Geek Cookbook

after ceph luminous “ceph-disk” command has been replaced by the “ceph-volume” command. I successfully passing this step by using ceph/daemon:latest-luminous images. But, now I’m stuck at OSD step

2019-06-19 07:32:03.898799 7f9faef02d80 -1 osd.1 0 log_to_monitors {default=true}
2019-06-19 07:32:05.015088 7f9f968c8700 -1 osd.1 0 waiting for initial osdmap

Has anyone been able to make it work in ubuntu 18.04?

Hi,

my OSD container is not running with the same error. I don’t know why it’s getting different id.

Can you help me?

My OSD container log:

docker run --net=host
–privileged=true
–pid=host
-v /etc/ceph/:/etc/ceph/
-v /var/lib/ceph/:/var/lib/ceph/
-v /dev/:/dev/
-e OSD_FORCE_ZAP=1
-e OSD_DEVICE=/dev/sdb
-e OSD_TYPE=disk
–name=“ceph-osd”
–restart=always
ceph/daemon:latest-luminous osd_ceph_disk
2020-04-01 21:10:57 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config
HEALTH_WARN 1 MDSs report slow metadata IOs; noscrub,nodeep-scrub flag(s) set; Reduced data availability: 16 pgs inactive; OSD count 0 < osd_pool_default_si ze 3
2020-04-01 21:10:57 /opt/ceph-container/bin/entrypoint.sh: INFO: It looks like /dev/sdb is an OSD
2020-04-01 21:10:57 /opt/ceph-container/bin/entrypoint.sh: You can use the zap_device scenario on the appropriate device to zap it
2020-04-01 21:10:57 /opt/ceph-container/bin/entrypoint.sh: Moving on, trying to activate the OSD now.
main_activate: path = /dev/sdb1
get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
command: Running command: /usr/sbin/blkid -o udev -p /dev/sdb1
command: Running command: /sbin/blkid -p -s TYPE -o value – /dev/sdb1
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.KtFUpi with options noatime,inode64
command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 – /dev/sdb1 /var/lib/ceph/tmp/mnt.KtFUpi
command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.KtFUpi
activate: Cluster uuid is d50c7ccf-5a64-4726-bdd0-5e9a6084298d
command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
mount_activate: Failed to activate
unmount: Unmounting /var/lib/ceph/tmp/mnt.KtFUpi
command_check_call: Running command: /bin/umount – /var/lib/ceph/tmp/mnt.KtFUpi
Traceback (most recent call last):
File “/usr/sbin/ceph-disk”, line 9, in
load_entry_point(‘ceph-disk==1.0.0’, ‘console_scripts’, ‘ceph-disk’)()
File “/usr/lib/python2.7/site-packages/ceph_disk/main.py”, line 5736, in run
main(sys.argv[1:])
File “/usr/lib/python2.7/site-packages/ceph_disk/main.py”, line 5687, in main
args.func(args)
File “/usr/lib/python2.7/site-packages/ceph_disk/main.py”, line 3777, in main_activate
reactivate=args.reactivate,
File “/usr/lib/python2.7/site-packages/ceph_disk/main.py”, line 3540, in mount_activate
(osd_id, cluster) = activate(path, activate_key_template, init)
File “/usr/lib/python2.7/site-packages/ceph_disk/main.py”, line 3687, in activate
’ with fsid %s’ % ceph_fsid)
ceph_disk.main.Error: Error: No cluster conf found in /etc/ceph with fsid d50c7ccf-5a64-4726-bdd0-5e9a6084298d

Hi There!

When I run the following commands I get the following errors (as root on Ubuntu 18.04.4 LTS)

$ chcon -Rt svirt_sandbox_file_t /etc/ceph/; chcon -Rt svirt_sandbox_file_t /var/lib/ceph/
chcon: can’t apply partial context to unlabeled file ‘/etc/ceph’
chcon: can’t apply partial context to unlabeled file ‘/var/lib/ceph’

Any idea whether I need to be concerned with the errors (and if so how to fix) before I continue with your guide?

Thanks!

I am trying also to do the same on Ubuntu 18.04 (also tried 20.04 beta). I also struggle with following errors in early config steps:

cd@node1:~$ sudo apt install ceph-base
Reading package lists... Done
Building dependency tree
Reading state information... Done
ceph-base is already the newest version (15.2.1-0ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 

cd@node1:~$ sudo chcon -Rt svirt_sandbox_file_t /var/lib/ceph
[sudo] password for cd:
chcon: can't apply partial context to unlabeled file 'bootstrap-rbd-mirror'
chcon: can't apply partial context to unlabeled file 'bootstrap-mds'
chcon: can't apply partial context to unlabeled file 'posted'
chcon: can't apply partial context to unlabeled file 'crash'
chcon: can't apply partial context to unlabeled file 'bootstrap-osd'
chcon: can't apply partial context to unlabeled file 'mds'
chcon: can't apply partial context to unlabeled file 'bootstrap-rbd'
chcon: can't apply partial context to unlabeled file 'bootstrap-mgr'
chcon: can't apply partial context to unlabeled file 'bootstrap-rgw'
chcon: can't apply partial context to unlabeled file 'tmp'
chcon: can't apply partial context to unlabeled file '/var/lib/ceph'
cd@node1:~$

cd@node1:~$ sudo chcon -Rt svirt_sandbox_file_t /etc/ceph/
chcon: can't apply partial context to unlabeled file 'rbdmap'
chcon: can't apply partial context to unlabeled file '/etc/ceph/'
cd@node1:~$

cd@node1:~$ sudo docker run -d --net=host --restart always -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -e MON_IP=192.168.254.10 -e CEPH_PUBLIC_NETWORK=192.168.254.0/24 --name="ceph-mon" ceph/daemon mon
bd1244fc9acf92c5bdff9050a3297aeddefb56165817968fba364906d2687675
docker: Error response from daemon: error while creating mount source path '/var/lib/ceph': mkdir /var/lib/ceph: read-only file system.
cd@node1:~$

cd@node1:~$ sudo ls -lisa /var/lib/ceph/
total 44
65541 4 drwxr-x--- 11 ceph ceph 4096 Apr 22 17:42 .
14226 4 drwxr-xr-x 41 root root 4096 Apr 22 17:42 ..
65549 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-mds
65550 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-mgr
65551 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-osd
65552 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-rbd
65553 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-rbd-mirror
65554 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 bootstrap-rgw
65555 4 drwxr-xr-x  3 ceph ceph 4096 Apr 22 17:42 crash
65559 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 mds
65557 4 drwxr-xr-x  2 ceph ceph 4096 Apr 17 21:08 tmp
cd@node1:~$

Any idea?

Hey, sorry that this recipe is giving you so much grief! I know it’s based on an old version of ceph, and certain elements don’t work properly anymore. There’s an effort underway to update the recipe to the latest ceph versions - you’re welcome to join in the conversation at http://chat.funkypenguin.co.nz, in the #dev channel. Else watch this space for an update in the next few days :wink:

D

@funkypenguin Any update on when the updated recipe will be ready? Thank you for all your hard work :slight_smile:

@waynehaffenden I’ve been working through the instructions Ceph has posted to use cephadm to install and configure a new cluster using Docker containers but have run into an issue. I filed this bug (https://tracker.ceph.com/issues/45672) a bit ago (had to wait until they activated my account on their bug tracker).

1 Like

Thanks to @TNTechnoHermit’s persistence, I spent the morning refreshing this recipe :slight_smile: Here’s the freshly-baked version : https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/

D

1 Like

Thank you @funkypenguin and @TNTechnoHermit for all your incredible hard work, looking forward to testing this out later.

Quick question, assuming there is no way to run the ceph dashboard as a docker container so it can run on the swarm and be load balanced/always available regardless of which node may become unavailable?

Hey @waynehaffenden - I assume that it’s possible, since:

  1. the MGR is what provides the dashboard,
  2. it doesn’t depend on any sort of persistence, and
  3. you can have more than one MGR.

I’m just unsure whether you can have more than one MGR concurrently. I’m happy to accept PRs to the recipe if someone wants to test it out :wink:

D

I’m having a bit of trouble at the point of trying to mount the file system. Your commands will set the node IPs, as well as the host IP, but then your fstab file entry references none of that, but instead your node names. I tried replacing them with my own, but that doesn’t seem to allow it to mount any better than before. The error I get is “mount error 22 = Invalid argument”. My fstab entry:
swarmtest1,swarmtest2,swarmtest3:/ /var/data ceph name=admin,noatime,_netdev 0 0

The ingredients section says:

Each node should have the IP of every other participating node hard-coded in /etc/hosts ( including its own IP )

Maybe that’s too obscure. The intention is that you manually populate /etc/hosts on each node, so that you’re resilient to DNS failures, but you still get the improved readability of using actual node names instead of IP addresses.

Does it work if you populate /etc/hosts?

Sadly not, either with the IPs in the /etc/hosts file or even using the IPs themselves in the fstab file

What OS are you using?

I’m using Ubuntu 18.04

You’re missing the port number. Also, as I understand it, you just need to specify one node, not all three - If all 3 nodes are up when the initial mount to say Node 1 is made, the mount will be aware of the other 2 nodes?

Perhaps just test the mount (using mount command) direct to Node 1 (and only node 1) IP first?

FYI, it’s working for me without the port number (provided you’ve not deviated from the default port). And yes, you only need to specify one node, but it can’t be the node you’re on, since the mon daemon may not have started yet (@TNTechnoHermit had this problem initially). Since you can’t predict which of your nodes will be up/down when this particular node boots, it’s more resilient to just add them all :slight_smile:

@funkypenguin I’ve finally got round to setting this up and it all appears to be working great, thank you. My only question is that the output from ceph status shows a total of 30GiB (3 x 10GB disks) with an availability of 27GiB but when doing a df -h the mounted ceph path shows only 8.5G available? Here’s the output from ceph status:

cluster:
id: 630cf0f4-a389-11ea-978c-fa163ec47a7e
health: HEALTH_OK

services:
mon: 3 daemons, quorum dn0,dn1,dn2 (age 11h)
mgr: dn0.qomczu(active, since 11h), standbys: dn1.cpxaem
mds: data:1 {0=data.dn2.moduyx=up:active} 1 up:standby
osd: 3 osds: 3 up (since 11h), 3 in (since 12h)

task status:
scrub status:
mds.data.dn2.moduyx: idle

data:
pools: 3 pools, 65 pgs
objects: 27 objects, 74 KiB
usage: 3.0 GiB used, 27 GiB / 30 GiB avail
pgs: 65 active+clean

Thank you

Edit: Also, notice only one of my nodes is in standby as a manager, how do I get the 3rd node in standby too? If I reboot any of the VPS (for a kernel update for example), rebooting dn1 (in standby) or dn2 (the one not in standby) ceph continues to operate with a warning. However, rebooting dn0 (the one with cephadm on) ceph hangs and doesn’t operate (this concerns me to what would happen once I have docker volumes mounted to here with DBs running etc if ceph stops!) until the VPS is back online. Any ideas?

Edit 2: (Sorry) I’m also receiving a health warning with 1 hosts fail cephadm check (How do I go about finding out which host has the issue) and any ideas on what might cause this. This is only a mess around so happy to wipe and try again but thought I’d best ask first :smiley:

P.S I’m a complete noob to all of this stuff so forgive my ignorance :slight_smile:

This’ll mean that you have 30GiB of “raw” storage, but due to the amount of replicas you’ve chosen (3, by default) in the pool used to back cephfs, you’re only presented with an availability of 8.5G for cephfs.

If (for example), you created another pool (say, for rbd storage) with 2 replicas instead of 3, you’d have 17G available in this pool, and as you consumed each pool, the available storage would be reduced accordingly.

1 Like