Ability to use IndexIDMap add_with_ids function with GPU based Cagra index

[navneet1v]

Description

I am trying to create a GPU Cagra index in a IndexIDMap index. But facing error like this:

  Traceback (most recent call last):
  File "/tmp/faiss-test.py", line 54, in <module>
    indexData(d, xb, ids, {}, "l2", "testgpuIndex.cagra.graph")
  File "/tmp/faiss-test.py", line 38, in indexData
    indexDataInIndex(idMapIVFPQIndex, ids, xb)
  File "/tmp/faiss-test.py", line 47, in indexDataInIndex
    index.add_with_ids(xb, ids)
  File "/opt/conda/lib/python3.11/site-packages/faiss/class_wrappers.py", line 251, in replacement_add_with_ids
    self.add_with_ids_c(n, swig_ptr(x), swig_ptr(ids))
  File "/opt/conda/lib/python3.11/site-packages/faiss/swigfaiss_avx2.py", line 11249, in add_with_ids
    return _swigfaiss_avx2.IndexIDMap_add_with_ids(self, n, x, xids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error in virtual void faiss::gpu::GpuIndex::add_with_ids(faiss::idx_t, const float*, const idx_t*) at /home/runner/miniconda3/conda-bld/faiss-pkg_1728491363260/work/faiss/gpu/GpuIndex.cu:120: Error: '!(this->is_trained)' failed: Index not trained

Based on my code reading what I can see is add_with_ids functionality is not supported with CagraIndex, but I think for the first time we should able to create the index, but subsequent adds should be blocked.

GpuIndexCagra is inherited from GpuIndex here and the exception is coming from here: and IndexIDMap uses add_with_ids

Is there any specific reason why this is blocked? If there is no specific reason I would like to contribute related to this.

Version

Faiss: 1.9.0

[navneet1v]

Sample code which caused the exception

import faiss
import logging
from timeit import default_timer as timer
import math
import numpy as np


def indexData(d:int, xb:np.ndarray, ids:np.ndarray, indexingParams:dict, space_type:str, file_to_write:str="gpuIndex.cagra.graph"):
    num_of_parallel_threads = 4
    logging.info(f"Setting number of parallel threads for graph build: {num_of_parallel_threads}")
    faiss.omp_set_num_threads(num_of_parallel_threads)
    res = faiss.StandardGpuResources()
    metric = faiss.METRIC_L2
    if space_type == "innerproduct":
        metric = faiss.METRIC_INNER_PRODUCT
    cagraIndexConfig = faiss.GpuIndexCagraConfig()
    cagraIndexConfig.intermediate_graph_degree = 64 if indexingParams.get('intermediate_graph_degree') is None else indexingParams['intermediate_graph_degree']
    cagraIndexConfig.graph_degree = 32 if indexingParams.get('graph_degree') == None else indexingParams['graph_degree']
    cagraIndexConfig.device = faiss.get_num_gpus() - 1
    cagraIndexConfig.store_dataset = False

    cagraIndexConfig.build_algo = faiss.graph_build_algo_IVF_PQ
    cagraIndexIVFPQConfig = faiss.IVFPQBuildCagraConfig()
    cagraIndexIVFPQConfig.kmeans_n_iters = 10 if indexingParams.get('kmeans_n_iters') == None else indexingParams['kmeans_n_iters']
    cagraIndexIVFPQConfig.pq_bits = 8 if indexingParams.get('pq_bits') == None else indexingParams['pq_bits']
    cagraIndexIVFPQConfig.pq_dim = 32 if indexingParams.get('pq_dim') == None else indexingParams['pq_dim']
    cagraIndexIVFPQConfig.n_lists = int(math.sqrt(len(xb))) if indexingParams.get('n_lists') == None else indexingParams['n_lists']
    cagraIndexIVFPQConfig.kmeans_trainset_fraction = 10 if indexingParams.get('kmeans_trainset_fraction') == None else indexingParams['kmeans_trainset_fraction']
    cagraIndexConfig.ivf_pq_params = cagraIndexIVFPQConfig

    cagraIndexSearchIVFPQConfig = faiss.IVFPQSearchCagraConfig()
    cagraIndexSearchIVFPQConfig.n_probes = 30 if indexingParams.get('n_probes') == None else indexingParams['n_probes']
    cagraIndexConfig.ivf_pq_search_params = cagraIndexSearchIVFPQConfig

    print("Creating GPU Index.. with IVF_PQ")
    cagraIVFPQIndex = faiss.GpuIndexCagra(res, d, metric, cagraIndexConfig)
    idMapIVFPQIndex = faiss.IndexIDMap(cagraIVFPQIndex)
    indexDataInIndex(idMapIVFPQIndex, ids, xb)
   

def indexDataInIndex(index: faiss.Index, ids, xb):
    if ids is None:
        index.train(xb)
    else:
        index.add_with_ids(xb, ids)

if __name__ == "__main__":    
    d = 128
    np.random.seed(1234)
    xb = np.random.random((100000, d)).astype('float32')
    ids = [i for i in range(len(xb))]
    indexData(d, xb, ids, {}, "l2", "testgpuIndex.cagra.graph")

[pankajsingh88]

Hi navneet1v! The error message states that the index is not trained (indexDataInIndex). IVF indexes do need training. I see that training is conditional in your indexDataInIndex method. Can you try with an explicit training (invoke index.train before invoking add_with_ids i.e something like

index = createIndex() # create the index
index.train(xt)
index.add_with_ids(xb, ids)

[navneet1v]

@pankajsingh88 actually I thought the same and tried the same thing but it didn’t work. I will put the exact trace for the same in few hours.

But to ans your question I have tried it. 😄

[navneet1v]

@pankajsingh88 here is the updated code with the exception.

import faiss
import logging
from timeit import default_timer as timer
import math
import numpy as np


def indexData(d:int, xb:np.ndarray, ids:np.ndarray, indexingParams:dict, space_type:str, file_to_write:str="gpuIndex.cagra.graph"):
    num_of_parallel_threads = 4
    logging.info(f"Setting number of parallel threads for graph build: {num_of_parallel_threads}")
    faiss.omp_set_num_threads(num_of_parallel_threads)
    res = faiss.StandardGpuResources()
    metric = faiss.METRIC_L2
    if space_type == "innerproduct":
        metric = faiss.METRIC_INNER_PRODUCT
    cagraIndexConfig = faiss.GpuIndexCagraConfig()
    cagraIndexConfig.intermediate_graph_degree = 64 if indexingParams.get('intermediate_graph_degree') is None else indexingParams['intermediate_graph_degree']
    cagraIndexConfig.graph_degree = 32 if indexingParams.get('graph_degree') == None else indexingParams['graph_degree']
    cagraIndexConfig.device = faiss.get_num_gpus() - 1
    cagraIndexConfig.store_dataset = False

    cagraIndexConfig.build_algo = faiss.graph_build_algo_IVF_PQ
    cagraIndexIVFPQConfig = faiss.IVFPQBuildCagraConfig()
    cagraIndexIVFPQConfig.kmeans_n_iters = 10 if indexingParams.get('kmeans_n_iters') == None else indexingParams['kmeans_n_iters']
    cagraIndexIVFPQConfig.pq_bits = 8 if indexingParams.get('pq_bits') == None else indexingParams['pq_bits']
    cagraIndexIVFPQConfig.pq_dim = 32 if indexingParams.get('pq_dim') == None else indexingParams['pq_dim']
    cagraIndexIVFPQConfig.n_lists = int(math.sqrt(len(xb))) if indexingParams.get('n_lists') == None else indexingParams['n_lists']
    cagraIndexIVFPQConfig.kmeans_trainset_fraction = 10 if indexingParams.get('kmeans_trainset_fraction') == None else indexingParams['kmeans_trainset_fraction']
    cagraIndexConfig.ivf_pq_params = cagraIndexIVFPQConfig

    cagraIndexSearchIVFPQConfig = faiss.IVFPQSearchCagraConfig()
    cagraIndexSearchIVFPQConfig.n_probes = 30 if indexingParams.get('n_probes') == None else indexingParams['n_probes']
    cagraIndexConfig.ivf_pq_search_params = cagraIndexSearchIVFPQConfig

    print("Creating GPU Index.. with IVF_PQ")
    cagraIVFPQIndex = faiss.GpuIndexCagra(res, d, metric, cagraIndexConfig)
    idMapIVFPQIndex = faiss.IndexIDMap(cagraIVFPQIndex)
    indexDataInIndex(idMapIVFPQIndex, ids, xb)


def indexDataInIndex(index: faiss.Index, ids, xb):
    if ids is None:
        index.train(xb)
    else:
        index.train(xb)
        index.add_with_ids(xb, ids)

if __name__ == "__main__":    
    d = 128
    np.random.seed(1234)
    xb = np.random.random((100000, d)).astype('float32')
    ids = [i for i in range(len(xb))]
    indexData(d, xb, ids, {}, "l2", "testgpuIndex.cagra.graph")

Creating GPU Index.. with IVF_PQ
Traceback (most recent call last):
  File "/tmp/faiss-test.py", line 55, in <module>
    indexData(d, xb, ids, {}, "l2", "testgpuIndex.cagra.graph")
  File "/tmp/faiss-test.py", line 38, in indexData
    indexDataInIndex(idMapIVFPQIndex, ids, xb)
  File "/tmp/faiss-test.py", line 48, in indexDataInIndex
    index.add_with_ids(xb, ids)
  File "/opt/conda/lib/python3.11/site-packages/faiss/class_wrappers.py", line 251, in replacement_add_with_ids
    self.add_with_ids_c(n, swig_ptr(x), swig_ptr(ids))
  File "/opt/conda/lib/python3.11/site-packages/faiss/swigfaiss_avx2.py", line 11249, in add_with_ids
    return _swigfaiss_avx2.IndexIDMap_add_with_ids(self, n, x, xids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error in virtual void faiss::gpu::GpuIndexCagra::addImpl_(faiss::idx_t, const float*, const idx_t*) at /home/runner/miniconda3/conda-bld/faiss-pkg_1728491363260/work/faiss/gpu/GpuIndexCagra.cu:106: adding vectors is not supported by GpuIndexCagra.

[pankajsingh88]

Looks like you don't need to add vectors explicitly in Cagra indexes. The training does that for you and you should be able to search directly. But this also implies that one doesn't have the ability to add vectors with ids.
check out faiss/gpu/test/TestGpuIndexCagra.cu as an example of how it's being used.

[navneet1v]

But this also implies that one doesn't have the ability to add vectors with ids.
check out faiss/gpu/test/TestGpuIndexCagra.cu as an example of how it's being used.

Absolutely correct. This is the reason why I created this issue but didn't call it as a bug :) . I am thinking of a way to go around this limitation. So what my idea here is we can support add/add_with_ids capability on the cagra index which internally calls train_index function only once. But once we have called the add api next calls to add/add_with_ids will throw the exception. This will at-least ensure that we can create an index with IndexIdMap as a wrapper.

Would like to know whats your thoughts on this? @pankajsingh88 I am happy to contribute this change.