Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ModuleNotFoundError: No module named 'cmake_build_extension' when building package #5187

Open
2 tasks done
stephanmg opened this issue Nov 11, 2024 · 6 comments · May be fixed by #5188 or pyodide/pyodide-build#60
Open
2 tasks done

ModuleNotFoundError: No module named 'cmake_build_extension' when building package #5187

stephanmg opened this issue Nov 11, 2024 · 6 comments · May be fixed by #5188 or pyodide/pyodide-build#60
Labels
bug Something isn't working

Comments

@stephanmg
Copy link

stephanmg commented Nov 11, 2024 •
edited
Loading

I tried to use pyodide skeleton pypi amici to create a package of the useful AMICI package.

However I fail when using pyodide build-recipes amici --install after.

AMICI seems to require cmake-build-extension. I cloned the pyodide repository, started the docker container ./run_docker and build pyodide-build according to the documentation.

Since cmake-build-extension seems to be required, I added it to the meta.yaml file under the subsection host in section requirements. While I can import the module from within the running Docker container, i.e. python -c 'import cmake_build_extension works, pyodide build-recipes amici --install fails, resulting in a ModuleNotFoundError for the former import statement.

My branch (forked from main branch of pyodide this morning) containing the meta.yaml is here:
https://github.com/stephanmg/pyodide/tree/package-amici/packages/amici

Any help / pointing to resources to help me create this package would be appreciated.

P.S.: Thank you @agriyakhetarpal for your help on Discord already.

🐍 Package Issue / Request

  • AMICI, version latest
  • Package URL
  • Package Dependencies that needs to be resolved first: A setup.py is provided and a Project.toml file.

Checklists

  • I have tried to install the package using micropip.install(...) and encountered the issue that the package is not pure Python and uses extensively C extensions
  • I have read the documentation and tried building the package myself.
@hoodmane
Copy link
Member

Thanks for the report @stephanmg. The build occurs in an isolated virtual environment, so installing cmake-build-extension into the top level environment doesn't help. The problem is that it gets dropped here:
https://github.com/pyodide/pyodide-build/blob/main/pyodide_build/pypabuild.py#L122
because cmake is in the list of "avoided requirements":
https://github.com/pyodide/pyodide-build/blob/main/pyodide_build/pypabuild.py#L36-L39
and the matching is sloppy: instead of checking avoid_name == req.name.lower() we check if avoid_name in req.name.lower(). Since "cmake" in "cmake-build-extension" is True, we drop it. This sloppy behavior was introduced in #2272, but unfortunately it looks like there was no discussion about it in PR review or anything. We can try changing that and see if it works in CI. It's a bit surprising this hasn't come up before tbh.

Once I fix this problem, the build gets a bit further but still fails at:

CMake Error at /tmp/build-env-t83_2naz/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find KLU (missing: KLU_LIBRARY KLU_LIBRARIES KLU_INCLUDE_DIR)

You should be able to get to this next error by applying this patch to pyodide-build.

--- a/pyodide_build/pypabuild.py
+++ b/pyodide_build/pypabuild.py
@@ -118,7 +118,7 @@ def remove_avoided_requirements(
     for reqstr in list(requires):
         req = Requirement(reqstr)
         for avoid_name in set(avoided_requirements):
-            if avoid_name in req.name.lower():
+            if avoid_name == req.name.lower():
                 requires.remove(reqstr)
     return requires

hoodmane added a commit to hoodmane/pyodide-build that referenced this issue Nov 11, 2024
@hoodmane
Use a less sloppy check in remove_avoided_requirements
a612e38 
See pyodide/pyodide#5187. In that issue, the build
requires a package called `cmake-build-extension` but because we have `cmake` in
the list of requirements to avoid we drop it and it breaks the build.

This logic was introduced in pyodide/pyodide#2272 and
did not come up in the review. Surprisingly this is the first problem it has
caused. Let's see if this change breaks anything in Pyodide CI.
@hoodmane hoodmane linked a pull request Nov 11, 2024 that will close this issue
Open
2 tasks
hoodmane added a commit to hoodmane/pyodide-build that referenced this issue Nov 11, 2024
@hoodmane
Use a less sloppy check in remove_avoided_requirements
953269d 
Resolves pyodide/pyodide#5187. In that issue, the
build requires a package called `cmake-build-extension` but because we have
`cmake` in the list of requirements to avoid we drop it and it breaks the build.

This logic was introduced in pyodide/pyodide#2272 and
did not come up in the review. Surprisingly this is the first problem it has
caused. Let's see if this change breaks anything in Pyodide CI.
hoodmane added a commit to hoodmane/pyodide that referenced this issue Nov 11, 2024
@hoodmane
Testing for pyodide/pyodide-build#60
721ba9f 
See upstream PR:
pyodide/pyodide-build#60

Resolves pyodide#5187. In that issue, the build requires a package
called `cmake-build-extension` but because we have `cmake` in the list of
requirements to avoid we drop it and it breaks the build.

This logic was introduced in pyodide#2272 and
did not come up in the review. Surprisingly this is the first problem it has
caused. Let's see if this change breaks anything in Pyodide CI.
@hoodmane hoodmane linked a pull request Nov 11, 2024 that will close this issue
Open
@agriyakhetarpal
Copy link
Member 

CMake Error at /tmp/build-env-t83_2naz/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find KLU (missing: KLU_LIBRARY KLU_LIBRARIES KLU_INCLUDE_DIR)

This comes from the SUNDIALS library, which needs to link with KLU from SuiteSparse. We do have a recipe for libsuitesparse already, though we might or might not need to bump its version, I haven't checked. We don't have a recipe for SUNDIALS yet (I've wanted to add it before but never found the time).

@hoodmane hoodmane changed the title Issue building AMICI Python package (using C extensions) ModuleNotFoundError: No module named 'cmake_build_extension' when building package Nov 11, 2024
@hoodmane
Copy link
Member

I'm going to take over this issue for the bug I'm currently working on fixing. Though maybe I should have opened a separate issue, since it seems that there will be more work to be done with amici after this is resolved.

@hoodmane
Copy link
Member

Maybe @stephanmg can open another issue about amici since I stole this one =)

@stephanmg
Copy link
Author

stephanmg commented Nov 11, 2024 •
edited
Loading

Maybe @stephanmg can open another issue about amici since I stole this one =)

Yes, will open later a new issue.

I can reproduce the error up to the linking error now (after applying the patch)

@stephanmg stephanmg mentioned this issue Nov 11, 2024
Open
2 tasks
@stephanmg
Copy link
Author

I think the label new package request can be remove in this issue.

@agriyakhetarpal agriyakhetarpal added bug Something isn't working and removed new package request labels Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@stephanmg @hoodmane @agriyakhetarpal