Add the ability for index to include a DocumentTransformer

[peterbraden]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Add an optional parameter to index, to allow a DocumentTransformer to be passed, which transforms the documents before they are inserted into the vector store.

ie.

def index(
    docs_source: Union[BaseLoader, Iterable[Document]],
    record_manager: RecordManager,
    vector_store: Union[VectorStore, DocumentIndex],
    *,
    document_transformer: DocumentTransformer | None = None, # <--- This Line
    batch_size: int = 100,
    cleanup: Literal["incremental", "full", None] = None,
    source_id_key: Union[str, Callable[[Document], str], None] = None,
    cleanup_batch_size: int = 1_000,
    force_update: bool = False,
) -> IndexingResult:

Motivation

The current index function allows us to pass a DocumentLoader, and will add the documents that it yields to a vector store. But sometimes we may want to apply transformations to the documents before insertion.

At the moment, you need to transform the documents before passing them to index, however if the transformation is slow (an example would be using an LLM to summarize a chunk) then we don't benefit from the indexing - the slow transformation must happen each time.

If we instead allowed the index function to take the transformer, we could only apply the expensive transformation to documents that had changed, thus preventing duplicate effort.

Proposal (If applicable)

No response