To answer my own question, I have this all fixed. The problem was a couple of things, first I needed cephfs installed on the machine (this was a Debian 9 machine I was using to test on, I haven’t tested the coreos ones yet) but you needed the following packages:
I also miss copied your cmd which created the key for
dockerswarm which was causing another error. I use the following in the end:
ceph auth get-or-create client.docker osd 'allow rw' mon 'allow rw' mds 'allow rw' > /etc/ceph/keyring.docker
Then I tested a mount with:
mount -t ceph 172.30.1.200:6789:/ /volumes/ -o name=docker,secretfile=/tmp/secret
You also don’t need the ceph-authtool. You can just use:
ceph auth list
This will display the keys, look for the dockerswarm client or whatever you called it. Put the key and only the key into a text file and use that.
The only extra thing I had to do because my cluster wasnt going healthy was that the tweaks you suggested, I had to remove these with osd unset because they were causing a healthy warning and I hate warnings
Again, thank you for such a great guide as without it, I doubt I would of got it all working.
The only last thing I would really like to know more about is how to set the PG’s correctly and what really are they?
As I said earlier mines set to 70 but this is just a guess really as calc says 256 which doesnt work
I currently have 3x OSD each one has a single disk/partican which is 1.5TB in size (each is the same). What should the PG be? I do plan to push this up to 4 nodes shortly.
One last thing which readers might like to know is, you can use a mount point instead if your on a VPS and you only have one disk. You need to use the following…
docker run -d --net=host \
--name ceph-mds \
--hostname ceph-mds02 \
--restart always \
-v /var/lib/ceph/:/var/lib/ceph/ \
-v /etc/ceph:/etc/ceph \
-e CEPHFS_CREATE=1 \
-e CEPHFS_DATA_POOL_PG=70 \
-e CEPHFS_METADATA_POOL_PG=70 \
Replace with your PG etc…
One final note before I sign off, I found that it wont work with
ext4 file systems. I was getting some strange error which took a while to figure a solution/help. If you format it as xfs, you wont have any problems and it will just format/work. If you dont do this then you need to manually create everything which was a nightmare. So to make it easy use xfs or btrfs.