It should be added that the length of a chunk is sensitive to both the embedding model and the LLM (API), so it’s hard to say there’s a necessity to stick closely to the implementation of a specific tokenizer. However, providing an interface to measure chunk length would indeed be helpful for offering a degree of user customization. You could try implementing that.

It should be added that the length of a chunk is sensitive to both the embedding model and the LLM (API), so it’s hard to say there’s a necessity to stick closely to the implementation of a specific tokenizer. However, providing an interface to measure chunk length would indeed be helpful for offering a degree of user customization. You could try implementing that.

i did a quick pr, if u'r happy with the change, i will fix the examples & docs as well

further, I'm thinking that some dataclass or pydantic model shall be added to have a better input/output hint and validation