Blob protocol=nfs and uncommon permissions

[pniederlag]

What happened:

• create volume in kubernetes with blob.csi driver and protocol=nfs
• mount volume to pod
• volume is owned by root:root and has no permissions for world (neither x for seek nor r for read)

What you expected to happen:

• get sensible defaults (read)
• have an option to modify behaviour
• be able to use the volume mount in practical way

How to reproduce it:

create and use a phyiscal-volume in azure kubernetes with

storageClassName: azureblob-nfs-premium
protocol nfs

Anything else we need to know?:

• container "blob" in daemonset "csi-blob-node" has --enable-aznfs-mount=true'
• CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType
• any attempt for a safe journey to proper permissions with fsGroup did not succeed yet

Environment:

aks/kubernetes: v1.29.9
mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.14.0
mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.12.0
mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.23.9

[andyzhangx]

the gid depends on securityContext setting, have you tried below example pod?

blob-csi-driver/deploy/example/statefulset-nonroot-nfs.yaml

Lines 16 to 19 in 54f7967

	securityContext: # optional just to make sure user is non-root
	runAsUser: 10001
	runAsGroup: 10001
	fsGroup: 10001

[pniederlag]

@andyzhangx thx for your feedback. We'd be very delighted if it would work just way. However, it does not!. According to docs there must be a special flag on the CSI driver, which seems not enabled on our cluster.

Requirement:

https://github.com/kubernetes-sigs/blob-csi-driver/blob/54f796709c133e8c28b230096e61cc927da40ef7/deploy/example/fsgroup/README.md

Implementation:
however our installation does not set this as we basicly just followed https://learn.microsoft.com/en-us/azure/aks/azure-blob-csi?tabs=NFS by '--enable-blob-driver' for aks

[andyzhangx]

I also used the managed blob csi driver with fsGroupPolicy: ReadWriteOnceWithFSType, check below contents:

# k get csidriver blob.csi.azure.com -o yaml
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    csiDriver: v1.23.9
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"CSIDriver","metadata":{"annotations":{"csiDriver":"v1.23.9"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile"},"name":"blob.csi.azure.com"},"spec":{"attachRequired":false,"podInfoOnMount":true,"tokenRequests":[{"audience":"api://AzureADTokenExchange"}],"volumeLifecycleModes":["Persistent","Ephemeral"]}}
  creationTimestamp: "2024-04-15T03:53:53Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: blob.csi.azure.com
  resourceVersion: "76152616"
  uid: bd4937db-b878-40fb-b2a4-96197419e954
spec:
  attachRequired: false
  fsGroupPolicy: ReadWriteOnceWithFSType
  podInfoOnMount: true
  requiresRepublish: false
  seLinuxMount: false
  storageCapacity: false
  tokenRequests:
  - audience: api://AzureADTokenExchange
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

# k exec -it statefulset-blob-nonroot-nfs2-0 -- sh
$ df -h
Filesystem                                                                        Size  Used Avail Use% Mounted on
overlay                                                                           126G   36G   85G  30% /
tmpfs                                                                              64M     0   64M   0% /dev
10.161.100.100:/nfs40fc7f659b604080af74/pvc-a1d9d241-068b-471c-b430-b8e157729ba1  5.0P     0  5.0P   0% /mnt/blob
/dev/sda3                                                                         126G   36G   85G  30% /etc/hosts
shm                                                                                64M     0   64M   0% /dev/shm
tmpfs                                                                             6.8G   12K  6.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                             3.8G     0  3.8G   0% /proc/acpi
tmpfs                                                                             3.8G     0  3.8G   0% /proc/scsi
tmpfs                                                                             3.8G     0  3.8G   0% /sys/firmware
$ cd /mnt/blob
$ ls -lt
total 1
-rw-r--r-- 1 nonroot root 551 Dec 20 03:04 outfile
$ touch a
$ ls -lt
total 1
-rw-r--r-- 1 nonroot root 754 Dec 20  2024 outfile
-rw-r--r-- 1 nonroot root   0 Dec 20 03:04 a
$ chmod 0777 a
$ ls -lt
total 2
-rw-r--r-- 1 nonroot root 1044 Dec 20 03:04 outfile
-rwxrwxrwx 1 nonroot root    0 Dec 20 03:04 a

[pniederlag]

Thx for your valuable feedback, highly appreciated. Questions that come to my mind:

• does it require runAsUser and runAsGroup to be set as well?
• Could there be a difference between a deployment(our setup) and a statefulset(your test)?
• does your csi-blob-node blob container have enable-aznfs-mount=true enabled as well?

chmod/chown in general is working but does not scale on a container with > 2.000.000 blobs in 750 directories. (it took az storage fs access set-recursive --acl 45 minutes to apply changes).

I'll double check things on our side once more to see where it is failing

[andyzhangx]

does it require runAsUser and runAsGroup to be set as well?

The securityContext field in Kubernetes is used to specify security-related attributes for a container. In this case, the securityContext is being used to ensure that the container runs as a non-root user with a specific user ID and group ID.

The runAsUser field specifies the user ID that the container should run as. In this case, the value is set to 10001, which means that the container will run as a user with the ID 10001.

The runAsGroup field specifies the group ID that the container should run as. In this case, the value is set to 10001, which means that the container will run as a group with the ID 10001.

The fsGroup field specifies the group ID that should be used for file system access checks. In this case, the value is set to 10001, which means that the container will have access to files and directories that are owned by the group with the ID 10001.

By setting these fields in the securityContext, the container is being run with reduced privileges, which can help to improve the security of the overall system.

Could there be a difference between a deployment(our setup) and a statefulset(your test)?

there is no difference

does your csi-blob-node blob container have enable-aznfs-mount=true enabled as well?

yes