Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop excluding unisolated packages from build dependencies #21

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

ryanking13
Copy link
Member

@ryanking13 ryanking13 commented Aug 21, 2024

This changes how we handle unisolated packages + hostsitepackages directory.

The current mechanism of unisolated packages is as follows:

  1. We have a few number of isolated packages. Numpy and Scipy are the most important ones.
  2. These isolated packages are often used as a build dependency for other packages.
  3. When some packages set these dependencies as build dependencies, we intercept the call of installing the build dependencies and exclude those packages from being installed.
  4. Instead, we pass PYTHONPATH to point to the WASM build of unisolated packages so that these unisolated packages can be used instead of the native packages when building the package.

However, this mechanism introduced a few issues, such as pyodide/pyodide#5012 (comment).

This PR changes it in the following way:

  1. Do not exclude the unisolated packages when installing build dependencies.
  2. Instead, install the packages as is, and only replace the cross build files of the installed package.

@agriyakhetarpal agriyakhetarpal mentioned this pull request Sep 17, 2024
@ryanking13 ryanking13 marked this pull request as ready for review September 24, 2024 12:50
@ryanking13 ryanking13 changed the title [DRAFT] Stop excluding unisolated packages from build dependencies Stop excluding unisolated packages from build dependencies Sep 24, 2024
@hoodmane
Copy link
Member

Sounds good to me. Relying on PYTHONPATH being around does indeed seem like it could cause trouble.

Comment on lines +226 to +228
return libdir, [
str(f.relative_to(libdir)) for f in package_dir.rglob("*") if f.is_file()
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is trying to get cross-build-files? How does it work in tree? Shouldn't we look at cross-build-files in meta.yaml in that case?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, site-packages-extras maybe should have been called site-packages-cross-files or something a bit more descriptive...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is trying to get cross-build-files? How does it work in tree? Shouldn't we look at cross-build-files in meta.yaml in that case?

As you can see in the buildpkg.py change, only cross-build-files are installed under the hostsitepackages on in-tree build (I think there are some room to optimize this logic though).

Also, site-packages-extras maybe should have been called site-packages-cross-files or something a bit more descriptive...

Totally agree, but the folder site-packages-extras is generated when creating the xbuildenv, so we'll need to fix there too.

@@ -205,33 +205,6 @@ def test_install_force(
assert (tmp_path / version / ".installed").exists()
assert manager.current_version == version

def test_install_cross_build_packages(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels like we could use an updated version of this test?

Copy link
Member

@hoodmane hoodmane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ryanking13! I made some comments but it generally looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants