Talos + Ceph CSI?

[duncanam]

Has anyone successfully set this up on Talos? I have an already existing Ceph cluster set up with both CephFS and RBD, but when applying Ceph's suggested helm chart, I get numerous errors. The node plugins seem to be failing due to the immutability of Talos, e.g. logs giving me

"failed to generate container X spec: failed to mkdir "/etc/selinux"

And my PVC is showing "Waiting for a volume go be created either by the external provisioner...".

The only guide to Ceph on the Talos documentation is using Rook, which I cannot do as my cluster is external to k8s.

Relevant links:

• https://devopstales.github.io/home/k8s-cephfs-storage-with-csi-driver/
• https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-cephfs/README.md

Edit: I figured out part of the problem, but not all. My node plugins now seem happy and running when I noticed there was an selinux mount bool in the config and turned that off, but still running into a failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = InvalidArgument desc = volume not found

Edit 2: I gave up on CephFS and tried RBD, which gave me multi node access modes are only supported on rbd block type volumes, so I tried setting my PVC to ReadWriteOnce instead of ReadWriteManyand it seems like it bound?

Edit 3: It would be cool if CephFS would work as it means it's easier to mount and access in other spots

[BonzTM]

I am running both RBD and CephFS on Talos via CSI and an external Ceph cluster (Not Rook)
RBD == dynamic PVs/PVCs
CephFS == mounting existing file system structure using manual PV creation (not block device creation on top of CephFS)

Keeping it somewhat brief, here's how I am configured.

1. CephRBD/FS namespaces are privileged using label. Not the most ideal, but my clusters are pretty locked down otherwise.
2. Manually installed snapshot and groupsnapshot CRDs (for snapshot support)
3. Manually install snapshot controller w/ appropriate RBAC
4. Manually create snapshotclasses
5. Manually create storageclasses for each CephRBD and CephFS (once had helm creating these, but had some unexpected results during upgrades)
6. Install both RBD/CephFS via helm

Here's an example of creating/mounting a CephFS directory/fs

Sorry for the brevity. I really wanted to respond but couldn't spend a ton of time writing up a big beautiful response at the moment. If you'd like, I can meet in discord to help out.

[duncanam]

Ack I forgot to reply. I did figure this out, getting Ceph-CSI to work with CephFS. It was a dumb issue, my cephfs name made in proxmox didn't match my configuration in the CSI plugin. I now have CephCSI working. I'll read through this though and see if I can make mine sleeker.