Kafka Support as a Broker with Enhanced Commit Features

[anupsingh3292]

We propose adding Kafka support as a broker for Celery, including the following features:

1. Kafka as a Broker: Enable Celery to use Kafka for message queuing, offering improved scalability and fault tolerance for high-throughput systems. Kombu already has confluentkafka implementation but needs support for interval based commit.

2. Partition-Based and Topic based Task Listening: Allow Celery workers to listen to specific Kafka partitions, enhancing parallel task processing and load distribution.

3. Interval-Based Commit: Introduce an interval-based offset commit mechanism, enabling periodic commits for better control and fault tolerance in task processing.

4. Out-of-Order Commit (Optional): Optionally support out-of-order offset commits, allowing flexibility in task processing order while ensuring proper acknowledgment.