CMAK operator

Table of Contents

• Prerequisites
• Installation
• Configuration
• Internals
• Design choices
• Troubleshooting

CMAK operator is a set of tools packaged as Helm chart, that allows to install and configure CMAK (previously Kafka Manager) into Kubernetes cluster.

CMAK (previously Kafka Manager) is well-known and mature tool for monitoring and managing Apache Kafka clusters.

Prerequisites

CMAK operator creates Kubernetes Jobs for configuring CMAK. Kubernetes Jobs are not automatically garbage collected. To enable cleanup it is required to enable TTLAfterFinished feature gate.

Installation

CMAK operator could be installed only with Helm 3. Helm chart is published to public Helm repository, hosted on GitHub itself.

It’s recommended to install CMAK operator into a dedicated namespace.

Add Helm repository

$ helm repo add cmak https://eshepelyuk.github.io/cmak-operator
$ helm repo update

Install latest version

$ helm install --create-namespace -n cmak-ns mycmak cmak/cmak-operator

Seach for available versions

$ helm search repo cmak-operator --versions
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
cmak/cmak-operator      0.2.1           3.0.0.5         CMAK operator for K8S.
cmak/cmak-operator      0.2.0           3.0.0.5         CMAK operator for K8S.

Install specific version

$ helm install --create-namespace -n cmak-ns --version 0.2.1 mycmak cmak/cmak-operator

Verify installation

❗	Exposing CMAK UI Currently, CMAK operator doesn’t create neither Ingress nor any other K8S resources to expose UI via HTTP. Users of CMAK operator should take care of this by themselves.

The simpliest test is to port forward CMAK UI HTTP port and access it from browser .

$ kubectl port-forward -n cmak-ns service/cmak 9000:9000

Then, open http://localhost:9000 in your browser.

Configuration

It’s extremely easy to configure multiple clusters in CMAK, starting from cluster setup, connection settings and ending with authorization using Helm values files.

Check CMAK operator values for all available options and their description.

1. Minimal values.yaml configuration for adding a several Kafka clusters to CMAK.

cmak:
  clusters:
    - name: "cluster-stage"
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

Connection settings could be configured for all clusters at once or per dedicated cluster.

1. Configuring connection settings

cmak:
  commonClusters:
    curatorConfig:
      zkMaxRetry: 100 # (1)
  clusters:
    - name: "cluster-stage"
      kafkaVersion: "2.5.0" # (2)
      curatorConfig:
        zkConnect: "kafka01.stage:2181,kafka02.stage:2181"
    - name: "cluster-prod"
      kafkaVersion: "2.1.0" # (3)
      curatorConfig:
        zkConnect: "kafka01.prod:2181,kafka02.prod:2181,kafka03.prod:2181"

1. this setting is applied to both clusters.

2. applied only to cluster-stage.

3. applied only to cluster-prod.

Configuration should be passed to helm via command line during installation or upgrade.

$ helm install --create-namespace -n cmak-ns -f cmak-values.yaml mycmak cmak/cmak-operator

Internals

CMAK operator comprises following components:

• CMAK, powered by Docker image from @hleb-albau.

• Apache ZooKeeper, powered by official Docker image.

• K8S Job used to configure Kafka clusters, powered by custom Docker image.

Design choices

Using own version of Zookeeper.

TO BE DEFINED.

Do not using REST for configuring clusters.

TO BE DEFINED

Troubleshooting

Cluster settings are not applied after changing values file.

Try to remove a K8S job and re-apply changes via helm upgrade.

$ kubectl delete job cmak-cluster -n cmak-ns
$ helm upgrade ...