Why JetStream doesn't achieve linear scalability through partitioning?

[penghuazhou]

If a topic has a data volume of tens of millions, does it need to be split into multiple streams to support it. Because streams cannot be split and stored on multiple machines, they can only be split into multiple streams from the business side, as shown in the following example.

nats stream add ORDERS --subjects "orders.partition0,orders.partition1"

package main

import (
    "fmt"
    "github.com/nats-io/nats.go"
)

func main() {
    nc, _ := nats.Connect(nats.DefaultURL)
    js, _ := nc.JetStream()

    // 模拟消息
    messages := []string{"msg1", "msg2", "msg3", "msg4"}

    for _, msg := range messages {
        // 分区规则：轮询分区
        partition := len(msg) % 2
        subject := fmt.Sprintf("orders.partition%d", partition)

        // 发布消息
        _, err := js.Publish(subject, []byte(msg))
        if err != nil {
            fmt.Println("Error publishing message:", err)
        } else {
            fmt.Printf("Published '%s' to %s\n", msg, subject)
        }
    }

    nc.Close()
}

[derekcollison]

A partition in Kafka is equivalent to a stream in NATS.

[penghuazhou]

thanks

[jnmoyne]

You can also leverage the built-in core NATS subject mapping functionality to automatically have NATS insert a partition number as part of the subject when the message is published (meaning you can control and adjust the number of partitions (or which part(s) of the subject the partition number is calculated from) administratively rather than doing the partitioning at the client code level (if you would rather avoid that)).
https://docs.nats.io/nats-concepts/subject_mapping