13:54 <devhell> heya, can anyone enlighten me please as to how to get a Ceph RGW working properly. I've deployed my cluster, all systems are fine, MON, MGR, MDS, all good. I've tried deploying RGW to one of the nodes, and used https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/installation_guide_for_red_hat_enterprise_linux/manually-installing-ceph-object-gateway to bootstrap the RGW

13:54 <devhell> service. Specifically, the 'ceph auth ...' part, which worked fine. When I restart the RGW systemd service it errors out saying: "... -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]" and "failed to fetch mon config (--no-mon-config to skip)

13:54 <devhell> they keyring is in the right place

13:56 <devhell> both, ceph.client.admin.keyring and the one I had it fetch and place in /var/lib/ceph/radosgw/ceph-foobar-node1/keyring

13:56 <devhell> checked the perms which seem to be fine

13:56 <devhell> is there something I should be adding to the RGW nixos configuration?

13:57 <devhell> right now it just enables RGW on my nodes

15:27 <devhell> alright, there is also a different issue, when rebooting the cluster, /var/lib/ceph/osd/ceph-$i/ is empty

15:28 <devhell> and the systemd service for each ceph osd fails because keyring for that osd is missing in that directory

15:28 <devhell> so /var/lib/ceph/osd/ceph-0/ is now empty after reboot, and ceph-osd-0.service is failing because keyring is missing

15:29 <devhell> I'm sure I've forgotten something, but I can't put my finger on what.

15:37 <devhell> ah, looks like the OSDs need to be manually activated or via a custom systemd unit...

15:53 <devhell> okay, the OSD stuff is fine now

15:53 <devhell> however RGW still won't work :/