(Why not) use message batching?

[SimonHeybrock]

One of the things Beamlime does is to consume, say, 14 ev44 messages per second, push them through the internal message router, collect them for a time interval, concat and hand them to the data reduction handler via the internal message router.

Can we just use message batching? That might significantly simplify the logic.

[YooSunYoung]

How exactly batch them...?

[SimonHeybrock]

Not sure exactly how it would work in practice, but something like

messages = detector_consumer.consume(num_messages=20, timeout=1.0)

[YooSunYoung]

If it's about consuming multiple messages at once, yes, we should probably do that.

But I didn't do that since that timeout blocks the whole process.

AOI kafka seems to have a better interface for that and it probably doesn't block the event loop:

https://aiokafka.readthedocs.io/en/stable/api.html#aiokafka.AIOKafkaConsumer

It also supports batch processing of messages but it seems like it only works if producer publishes messages in batch according to this documentation: https://docs.confluent.io/kafka/design/efficient-design.html ...?

[SimonHeybrock]

That is not how understood the mechanism. The consumer can decide how to fetch. Obviously, if we fetch too frequently we may only get a single message. But if we are willing to accept some more latency (which we also have with the "manual" beamlime-side batching) then we can fetch batches.

[YooSunYoung]

The consumer can decide how to fetch.

Yes. What I said is, the batch messages of kafka and consuming multiple messages are two different concepts.

But if we are willing to accept some more latency (which we also have with the "manual" beamlime-side batching) then we can fetch batches.

Yeah, but beamlime-side batching is to concatenate each fields of the dataset. How can it be handled by consumer itself...?