DDD-6 Debezium Operator

[jcechace]

First draft of DDD for Debezium Operator. This should be iteratively improved and serve as a preliminary documentation.

[jpechane]

I don't think we need service right now. Debezium does not expose any data endpoint.

[jcechace]

+1 I will remove the service then

[jpechane]

I suppose type is just plain string, not enum of allow sinks, is it correct?

[jcechace]

yes, but changing it to enum would be quite simple. However an "outdated" operator would be unusable in case of a new sink release.

[jpechane]

I personally prefer the plain string - for sake of flexibility.

[jpechane]

Probably not now byt we might consider to use a logical name instead of a class name.

[jpechane]

Probably not now byt we might consider to use a logical name instead of a class name.

[jcechace]

Yes, for the "typed" configuration this would make sense.

[jpechane]

Let's start with the dynamic approach. It is the easiest to implement.
To de-couple sink/operator development we might allow combination of both. Does it make sense to introduce a separate namespace already that will represent the dynamic configuration and will be the only option right now while later on we could introduce statics too?

[jcechace]

I'm not really sure what you mean by namespace in this context. The idea here is that the properties under sink are mostly just gathered into a map.

In theory in future we could do something such as

sink:
  kafka:
    bootstraps: kafka:9092

so esentially we would introduce a typed property for each sink

[jpechane]

Yes, I understand, my plan was to hve something like

sink:
  kafka:
    bootstraps: kafka:9092
    additional:
       prop1:
       prop2:

So you could use arbitrary props even with typed properties in place - for cases when the is a delay between operator/server, or if something is omitted.
Namespace in this case is the separate field - here named additional that contains this dynamic configuration.
What I was talking about was placing the conf options under such distinct field so it will stay bacward compatible when typed configuration will be available.

[jcechace]

Yes, this can be there for sure. The question is what should be the initial state before we introduce these typed sink configurations. The way I see it we have three option

1. Current version

sink:
  type: kafka
  prop1:
  prop2: 

then later we could add the typed versions

sink:
  kafka:
    bootstraps: kafka:9092
    additional: 
      prop1:
      prop2:

in this case the type and additional props would still be directly part of Sink but validation would be put in place to ensure these configs are mutually exclusive

2. Slightly modified current verison

sink:
  type: kafka
  additional: 
    prop1:
    prop2:

Then again, after introducing the typed kafka (and so on) config we would ensure that type is mutually exclusive (sink.additonal could be then merged into sink.kafka.additional)

3. We initially introduce an arbitrary sink type

sink:
  config:
    type: kafka
    bootstraps: kafka:9092
    prop1:
    prop2:

In this scenario sink.config is essentially an extra nesting of current sink.

[jcechace]

Or... thinking about it.. which is probably the best from implementation point of view.

The initial version is going to be

sink:
  type: kafka
  config:
     prop1:
     prop2:

Once again, config is essentially an arbitrary dynamic sink type (and also technical equivalent of the additional filed)

And then later once typed configs are introduced

sink:
  type: kafka # must be kafka for kafkaConfig
  kafkaConfig:
    bootstraps: kafka:9092
  config: 
    prop1:
    prop2:

This gives as the best backwards compatibility and also allows us to use the type field as a decision point for which typed config to read (if none of known values matched, then the dynamic config would be expected)

WDYT?

[jpechane]

I'd start with per-namespace as it is least compplicated from k8s - speaking of security and privileges.

[jpechane]

I'm sorry but I don't follow this question. Could you please share an example?

[jcechace]

Essentially an equivalent of https://strimzi.io/docs/operators/latest/configuring.html#type-KafkaClusterTemplate-reference