Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): py bump docker ^7.1.0, requests ^2.32.2 #115

Merged
merged 4 commits into from
Jun 4, 2024
Merged

chore(deps): py bump docker ^7.1.0, requests ^2.32.2 #115

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1d4babf
chore(deps): py bump docker ^7.1.0, requests ^2.32.2
tarilabs May 30, 2024
2a0d132
add Mac OSX specific instructions
tarilabs May 30, 2024
285f488
Update CONTRIBUTING.md
tarilabs Jun 4, 2024
faa94f5
Update CONTRIBUTING.md
tarilabs Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,20 @@ and now you can substitute `gmake` every time the make command is mentioned in g
Several options of docker engines are available for Mac.
Having Docker installed is also helpful for Testcontainers.

### Podman

Podman also supports Rosetta (Apple specific) emulation which is handy since the Google MLMD project dependency is x86 specific.
You can install Podman from the Podman [website](https://podman.io/).
tarilabs marked this conversation as resolved.
Show resolved Hide resolved

We recommend setting up the Podman machine with root priviledges,
tarilabs marked this conversation as resolved.
Show resolved Hide resolved
and setting the environment variable

```sh
export TESTCONTAINERS_RYUK_PRIVILEGED=true
```

when running TestContainer-based Model Registry Python tests (for more information, see [here](https://pypi.org/project/testcontainers/#:~:text=TESTCONTAINERS_RYUK_PRIVILEGED)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC by exporting the env variable we're already following the recommended usage, so I'd rewrite the recommendation, mainly taking out the "and" (as it sounds like there is something additional to be done when using the Podman machine in that sense). Also, nit (typo).

Suggested change
We recommend setting up the Podman machine with root priviledges,
and setting the environment variable
```sh
export TESTCONTAINERS_RYUK_PRIVILEGED=true
```
when running TestContainer-based Model Registry Python tests (for more information, see [here](https://pypi.org/project/testcontainers/#:~:text=TESTCONTAINERS_RYUK_PRIVILEGED)).
We recommend using a [privileged Podman machine](https://pypi.org/project/testcontainers/#:~:text=TESTCONTAINERS_RYUK_PRIVILEGED)) when running TestContainer-based Model Registry Python tests.
You can do that by setting the following environment variable
```sh
export TESTCONTAINERS_RYUK_PRIVILEGED=true

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my understanding they're not the same thing, as this proposed correction would make it look like (since the "podman machine" is now linking to Ryuk container privileges).

In my understanding, one thing is the emulation used to run containers in general on macOsx, for podman that is the "podman machine"; another thing is the Ryuk container, used by TestContainers as a ~"garbage collector" which need to opt-in to this privileges.

I have podman machine as root by default,
but if I miss to specify the environment variable for ryuk specifically it won't work for Testcontainers.

In conclusion, to me it's indeed an "and".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the heads up!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

happy to revise this text if we later find anything I've summarized above is not accurate/correct 👍


### Colima

Colima offers Rosetta (Apple specific) emulation which is handy since the Google MLMD project dependency is x86 specific.
Expand Down Expand Up @@ -202,3 +216,31 @@ Then with the given setup MLMD is already installed inside the DevContainer:
At this point Poetry is already installed as well and can be used to build and run test of the Model Registry Python client.

<!-- to be continued with explanation of this "hack": https://github.com/tarilabs/ml-metadata-remote#readme -->

# FAQ

## Error `docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers ...`

This happens on Mac OSX when running Testcontainers-based Python tests, and [Ryuk](https://github.com/testcontainers/moby-ryuk) does not have the correct access priviledges.

```
-------------------------------------------- Captured stderr setup ---------------------------------------------
Pulling image testcontainers/ryuk:0.7.0
Container started: 4af385c2d670
---------------------------------------------- Captured log setup ----------------------------------------------
WARNING root:config.py:59 DOCKER_AUTH_CONFIG is experimental, see testcontainers/testcontainers-python#566
INFO testcontainers.core.container:container.py:88 Pulling image testcontainers/ryuk:0.7.0
INFO testcontainers.core.container:container.py:101 Container started: 4af385c2d670
=========================================== short test summary info ============================================
ERROR tests/store/test_wrapper.py::test_get_undefined_artifact_type_id - docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/4af385c2d67067875f6f0...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 1 error in 0.50s ===============================================
```

Solution:
- Use `export TESTCONTAINERS_RYUK_PRIVILEGED=true` (we recommend this option)
- Use `export TESTCONTAINERS_RYUK_DISABLED=true` to disable Ryuk entirely; the tests are configured to close all container resources, but we do not recommend this option

For more information:
- section [Docker engine](https://github.com/kubeflow/model-registry/blob/main/CONTRIBUTING.md#docker-engine) in this document
- https://pypi.org/project/testcontainers/#:~:text=the%20database%20version.-,Configuration,-Env%20Variable
3 changes: 2 additions & 1 deletion clients/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,13 +110,14 @@ Use `nox -l` to list sessions and execute them using `nox -s [session]`.

### Running Locally on Mac M1 or M2 (arm64 architecture)

If you want run tests locally you will need to set up a colima develeopment environment using the instructions [here](https://github.com/kubeflow/model-registry/blob/main/CONTRIBUTING.md#colima)
If you want run tests locally you will need to set up a development environment, including docker engine; we recommend following the instructions [here](https://github.com/kubeflow/model-registry/blob/main/CONTRIBUTING.md#docker-engine).

You will also have to change the package source to one compatible with ARM64 architecture. This can be actioned by uncommenting lines 14 or 15 in the pyproject.toml file. Run the following command after you have uncommented the line.

```sh
poetry lock
```

Use the following commands to directly run the tests with individual test output. Alternatively you can use the nox session commands above.

```sh
Expand Down
97 changes: 39 additions & 58 deletions clients/python/poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 6 additions & 2 deletions clients/python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@ homepage = "https://github.com/kubeflow/model-registry"
[tool.poetry.dependencies]
python = ">= 3.9, < 3.11"
attrs = "^21.0"
ml-metadata = "^1.14.0"
ml-metadata = "==1.14.0"
# you might consider using locally the following alternative, when developing on Apple-silicon/ARM-based computers:
# ml-metadata = { url = "https://github.com/opendatahub-io/ml-metadata/releases/download/v1.14.0%2Bremote.1/ml_metadata-1.14.0+remote.1-py3-none-any.whl" }
typing-extensions = "^4.8"

huggingface-hub = { version = ">=0.20.1,<0.24.0", optional = true }
# pin docker+requests dependencies of testcontainers to docker ^7.1.0 for compatibility with requests ^2.32.2; ref: https://github.com/testcontainers/testcontainers-python/issues/577#issuecomment-2123324367
requests = { version = "^2.32.2", optional = true }

[tool.poetry.extras]
hf = ["huggingface-hub"]
Expand All @@ -33,7 +35,9 @@ pytest-cov = "^4.1.0"
sphinx-autobuild = ">=2021.3.14,<2025.0.0"
ruff = "^0.4.4"
mypy = "^1.7.0"
testcontainers = "^3.7.1"
testcontainers = "^4.5.0"
# pin docker+requests dependencies of testcontainers to docker ^7.1.0 for compatibility with requests ^2.32.2; ref: https://github.com/testcontainers/testcontainers-python/issues/577#issuecomment-2123324367
docker = "^7.1.0"

[tool.coverage.run]
branch = true
Expand Down
Loading