knowledge base creation issue

[j-sieger]

code given in this notebook is failing with the below error. while creating the knowledge base.

err=ValidationException('An error occurred (ValidationException) when calling the CreateKnowledgeBase operation: The knowledge base storage configuration provided is invalid... The OpenSearch Serverless engine type associated with your vector index is invalid. Recreate your vector index with one of the following valid engine types: FAISS. Then retry your request.'), type(err)=<class 'botocore.errorfactory.ValidationException'>

i have changed the engine type to 'faiss' then the vector index creation itself failing. May be we need to change the openserach version used in this notebook.

`TransportError Traceback (most recent call last)
Cell In[34], line 2
1 # Create index
----> 2 response = oss_client.indices.create(index=index_name, body=json.dumps(body_json))
3 print('\nCreating index:')
4 print(response)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/client/utils.py:179, in query_params.._wrapper.._wrapped(*args, **kwargs)
176 if v is not None:
177 params[p] = _escape(v)
--> 179 return func(*args, params=params, headers=headers, **kwargs)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/client/indices.py:128, in IndicesClient.create(self, index, body, params, headers)
125 if index in SKIP_IN_PATH:
126 raise ValueError("Empty value passed for a required argument 'index'.")
--> 128 return self.transport.perform_request(
129 "PUT", _make_path(index), params=params, headers=headers, body=body
130 )

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/transport.py:409, in Transport.perform_request(self, method, url, headers, params, body)
407 raise e
408 else:
--> 409 raise e
411 else:
412 # connection didn't fail, confirm it's live status
413 self.connection_pool.mark_live(connection)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/transport.py:370, in Transport.perform_request(self, method, url, headers, params, body)
367 connection = self.get_connection()
369 try:
--> 370 status, headers_response, data = connection.perform_request(
371 method,
372 url,
373 params,
374 body,
375 headers=headers,
376 ignore=ignore,
377 timeout=timeout,
378 )
380 # Lowercase all the header names for consistency in accessing them.
381 headers_response = {
382 header.lower(): value for header, value in headers_response.items()
383 }

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/connection/http_requests.py:230, in RequestsHttpConnection.perform_request(self, method, url, params, body, timeout, allow_redirects, ignore, headers)
217 if (
218 not (200 <= response.status_code < 300)
219 and response.status_code not in ignore
220 ):
221 self.log_request_fail(
222 method,
223 url,
(...)
228 raw_data,
229 )
--> 230 self._raise_error(
231 response.status_code,
232 raw_data,
233 response.headers.get("Content-Type"),
234 )
236 self.log_request_success(
237 method,
238 url,
(...)
243 duration,
244 )
246 return response.status_code, response.headers, raw_data

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/opensearchpy/connection/base.py:301, in Connection._raise_error(self, status_code, raw_data, content_type)
298 except (ValueError, TypeError) as err:
299 logger.warning("Undecodable raw error response from server: %s", err)
--> 301 raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
302 status_code, error_message, additional_info
303 )

TransportError: TransportError(500, 'exception', 'Internal error occurred while processing request')`

[Hexecu]

Hi @j-sieger

I encountered a similar TransportError(500, 'exception', 'Internal error occurred while processing request') in my own project, although it happened in a different context. In my case, the error occurred randomly during certain query executions on OpenSearch Serverless.

Differences in Context:

• My Scenario: The error appeared while running specific queries on OpenSearch Serverless (engine: faiss), not during the creation of a knowledge base or vector index. The issue was not consistent but occurred very often across different queries.
• Resolution: AWS Support identified that the error was due to an internal issue where requests were not reaching the shard manager. They resolved it by stopping an unhealthy coordinator task that was experiencing shard priming errors.

I hope this information can help others who might be facing the same issue with OpenSearch Serverless. It can save a lot of time and frustration by pointing out that the problem might be on AWS’s side, and the best course of action is to reach out to their technical support team instead of spending too much time debugging it on your own.