Log: MR Python client REST rebase

[tarilabs]

This is just a convenience GitHub issue to log data from multiple sources, and provide a summary of the Model Registry Python client rebase on REST work, so far.

The challenge

Historically, the Model Registry Python client connected directly to the gRPC MLMD container/service.
Besides this requiring to replicate logic on GoLang REST server and the MR Python client itself (i.e.: implement the Logical Model mapping to/from MLMD both in GoLang and Python) so in pragmatic terms implementing the same logic in 2 places, it posed some additional challenges due to the only distributions available of the MLMD python library/wheel.

For example, but not limiting to:

• the MLMD python wheel only distributed for x86 (example at this link, only amd64/x86 available)
• the MLMD python wheel only distributed for CPython 3.9 and 3.10, not available for 3.11 (see, for example, https://github.com/google/ml-metadata/issues/198)

Introducing a workaround

Some of these concerns where partially addressed with:

• providing a remote-only, grpc-only version of the MLMD whl (see related document)

However, this posed some challenges to end-users, for example but not limiting to:

• can verify the MR manifest with the KF 1.9.rc0 #90 (comment)
• https://cloud-native.slack.com/archives/C073N7B6K3R/p1719234732313409
• https://groups.google.com/g/kubeflow-discuss/c/yMZ7q1gTSak/m/4MSuTkFJAQAJ

as can be seen, it was difficult to explain to end-user the constraint mainly came as the requirement from the MLMD whl, not necessarily a choice originating in the MR client itself.

The solution

This was overcome with:

• Switch Python client backend to use REST #152
  work by @isinyaaa which rebased the MR Python client to interact with the Model Registry server directly instead (and no longer the wrapped MLMD) via its exposed REST API.

Please notice the code-generated part is not exposed (and is not intended to be exposed in the future) to the end-user of the MR Python client; this is so that, if in the future we will need to change the underlying implementation, that would require a little less effort eventually.

What else was considered

Different REST code generators were evaluated, including Kiota, openapi-codegen, etc.

Each code generators had some limitations; for posterity in the case of Kiota a bunch were actually addressed upstream, for example:

• https://github.com/microsoft/kiota/issues/4428
• https://github.com/microsoft/kiota/issues/4325

While other remain (at the time of writing):

• https://github.com/microsoft/kiota/issues/4178
• https://github.com/microsoft/kiota/issues/4433
• https://github.com/microsoft/kiota/issues/4750#issuecomment-2160551178

In the case of openapi-codegen, limitations affected discriminator fields too, for example requiring to "manually" change the generated code anyway:

model-registry/clients/python/patches/fix-validators.patch

Lines 1 to 24 in aeb4bda

	diff --git a/clients/python/src/mr_openapi/models/doc_artifact.py b/clients/python/src/mr_openapi/models/doc_artifact.py
	index 5e6a34a5..4f7c29bd 100644
	--- a/clients/python/src/mr_openapi/models/doc_artifact.py
	+++ b/clients/python/src/mr_openapi/models/doc_artifact.py
	@@ -104,6 +104,7 @@ class DocArtifact(BaseModel):
	return cls.model_validate(obj)
	_obj = cls.model_validate({
	+ "artifactType": "doc-artifact",
	"customProperties": dict(
	(_k, MetadataValue.from_dict(_v))
	for _k, _v in obj["customProperties"].items()
	diff --git a/clients/python/src/mr_openapi/models/model_artifact.py b/clients/python/src/mr_openapi/models/model_artifact.py
	index 380d1480..306d74d6 100644
	--- a/clients/python/src/mr_openapi/models/model_artifact.py
	+++ b/clients/python/src/mr_openapi/models/model_artifact.py
	@@ -109,6 +109,7 @@ class ModelArtifact(BaseModel):
	return cls.model_validate(obj)
	_obj = cls.model_validate({
	+ "artifactType": "model-artifact",
	"customProperties": dict(
	(_k, MetadataValue.from_dict(_v))
	for _k, _v in obj["customProperties"].items()

Since the openapi-codegen is already used on the Go side to generate the server models, this was picked as the selected code-generator to go forward with at this time.

Additional considerations

This workaround is no longer needed:

https://github.com/kubeflow/model-registry/pull/116/files#diff-6b074bce6a463d7cd6b69e5b1901d4d48c6ff2cd150a40ce849f7a99cb68bce4R123-R124

thanks to the REST rebase.

With KF 1.9, this workaround is no longer needed:

#165 (comment)

thanks to the REST rebase.

---

I would like to express my gratitude to everyone who contributed directly or indirectly to this work, it benefits a lot the MR python client itself and its end-users 🙏

[tarilabs]

Considered Done.