Add Multimodal Information Retrieval Supports

[muazhari]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

I predict there will be performance-significant multimodal (I suggest prioritizing text and image modality first) local embedding models starting with this model: https://huggingface.co/blog/paligemma. I have not yet found any open discussions or issues to support the proposed features like multimodal base document model and multi-embedding vector stores.

Motivation

Until reliable huge/infinite context LLM and cheap context usage costs are released, the RAG method will remain valuable. The increase in information (modalities in this case) likely leads to more effective and less risky use cases. LangChain here will make developers more efficient when developing the RAG system.

Proposal (If applicable)

Keeping the ideas afloat to related libraries likely incentivizes the initial effort.

[muazhari]

Sure, this performance significant recent Visual RAG came up.

• https://huggingface.co/spaces/vidore/vidore-leaderboard
• https://huggingface.co/vidore/colpali-3b-pt-448

[ahsannawazch]

Langchain should add support for colpali, colqwen retrievers for multimodal rag along with vector store used to index these matrix embeddings.

[muazhari]

Yep, also the current and next trend is free-parsing RAG. I guess the pipelining used by LangChain will need to adapt, as the preprocessing components needed to build RAG will be more reduced.