configurer and vault in different kubernetes clusters

[johnny990]

Hi,
is it possible to deploy vault configurer and vault itself in different clusters? Or is it possible to disable configurer somehow?

I have the following security critical use case:

• there is unsealing vault which deployed on separate machine highly secured (access restricted only to several trusted persons)
• there are multiple tenant vaults where we have configured autounseal via unsealing vault. These tenants less secured, but we'd like to secure vault itself as much as possible (no root tokens in secrets on these tenants, central vault token should allow only autounseal). The ratio is that if somebody get access to tenant vault namespace it will be difficult to get access to tenant vault's secrets without its root token.

But this scenario is almost not possible because we have configurer which require root token to configure roles, policies, etc on tenant vaults. If we remove root token, it complains and can't access vault.

So, the idea is to configure tenant vaults externally either by configurer deployed remotely (if it's possible) or via custom external scripts.

Does it make sense? Or it's overengineering without real security benefits?

[akijakya]

Hi @johnny990, thank you for using Bank-Vaults!

Do I understand correctly that the problem here is that the configurer stores the root and unseal tokens in a Kubernetes secret? As per the docs this option is only for development purposes, these tokens should be stored in a more secure environment indeed. In vault-operator, the location of the tokens can be configured in the crd under spec.unsealConfig, you can find some examples here!

[johnny990]

Hi @akijakya , thanks for a prompt reply!
actually I store root and other tokens in unsealing vault, but that doesn't help much, since if the intruder gets access to the tenant cluster he will be able to access secret where I store VAULT_TOKEN to access anything in spec.unsealConfig. That's why I'd like to leave on tenant cluster only autounseal token and move configurer part out of it.

So, here's my configuration if it will help.
tenant vault:

unsealConfig:
    options:
      preFlightChecks: true
      storeRootToken: true
    vault:
      address: "{{ .Values.unsealingVaultHost }}"
      unsealKeysPath: secret/data/vault/{{ .Release.Namespace }}/vault-keys

vaultEnvsConfig:
  - name: VAULT_TOKEN
    valueFrom:
      secretKeyRef:
        name: vault-central-token
        key: VAULT_TOKEN
  - name: VAULT_SKIP_VERIFY
    value: "true"

envsConfig:
  - name: VAULT_SKIP_VERIFY
    value: "true"
 - name: VAULT_TOKEN
   valueFrom:
     secretKeyRef:
      name: vault-central-token
       key: VAULT_TOKEN

policy in unsealing vault:

      {{ range .Values.tenants }}
      - name: allow_{{ . }}_transit
        rules: path "{{ . }}/transit/encrypt/autounseal" {
                 capabilities = ["update"]
                }
               path "{{ . }}/transit/decrypt/autounseal" {
                 capabilities = ["update"]
               }
               path "secret/data/vault/{{ . }}/*" {
                 capabilities = ["create", "read", "update", "delete", "list"]
               }
      {{ end }}

Let me know if you need additional information about my setup.

So, reducing to a short statement: to configure tenant vault I need root token, but if I put this root token somewhere where configurer is deployed (or if I put it in another vault but provide configurer with permissions to get it) means that somebody else could get it.
The only way to avoid this I see the option to separate tenant vault and its configurer to different clusters. Does it make sense?

[csatib02]

Hey @johnny990,

Technically it should be possible to deploy the Vault Configurer in a different Kubernetes cluster than the tenant Vaults. Since the Configurer communicates with the Vault API to configure policies and roles, it does not have to reside in the same Kubernetes cluster as the Vault it is configuring. You can configure it to connect to the tenant Vaults remotely via their API endpoint, but this is something that will most probably require additional resources to be created.

The idea makes sense from a security perspective, as it would reduce the exposure of sensitive tokens (such as the root token) in the tenant clusters. Some key points to keep in mind while implementing this:

• Ensure secure communication between the Configurer cluster and the tenant Vault clusters, likely using TLS or VPNs for safe API access.
• The Configurer’s permissions should be limited to only what is necessary to manage roles and policies in the tenant Vaults.

[johnny990]

Hey @johnny990,

Technically it should be possible to deploy the Vault Configurer in a different Kubernetes cluster than the tenant Vaults. Since the Configurer communicates with the Vault API to configure policies and roles, it does not have to reside in the same Kubernetes cluster as the Vault it is configuring. You can configure it to connect to the tenant Vaults remotely via their API endpoint, but this is something that will most probably require additional resources to be created.

Hi, @csatib02
thanks for the reply, I agree that it is technically possible, but I raised this question more about to propose this 'vault <-> configurer' separation out of the box. The point is that I can't deploy vault without configurer and can't deploy configurer wthout vault without too many extra efforts (like creating my own helm chart, maybe even repackaging images to support required configurations and parameters, and some lack of documentation).
So, it would be great to have either two separate helm charts (vault, vault-configurer) or to manage all these configurations via values.yaml (like disable configurer on tenant vault cluster and disable vault on manager cluster (to have just a configurer)). Or even more - to allow the configurer manage multiple vault instances, 'cause like you said, everything it requires - connectivity, permissions and configuration to apply.

[csatib02]

Well the configurer is actually a separate project, it's the Bank-Vaults CLI tool: https://github.com/bank-vaults/bank-vaults
Since the tool already handles a lot of tedious tasks, the operator was developed on top of its features. You can read about those here: https://bank-vaults.dev/docs/operator/

The use-case you are talking about makes a lot of sense and the Vault-Operator has a lot of built-in components to support that, but supporting a multi-cluster setup is not supported out of the box, and It would definitely be a huge load of work, to have that.

I believe that no CRD in the K8s landscape come with a suitable configuration out of the box. When someone starts to adopt a solution entirely, there will be a lot of things that need to be tailored to the specific needs of the organisation. Whether it's custom networking, security policies, or multi-cluster setups, every environment comes with its own unique set of challenges.

While it would be incredibly useful to support multi-cluster deployments natively within the Vault-Operator, achieving this would require significant development work—beyond just separating the configurer and making it customisable via a Helm chart. It would involve ensuring that the operator and configurer can securely and efficiently communicate across clusters, handle RBAC in multiple environments, and manage the lifecycle of Vault instances at scale.