Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building and testing Python packages out of tree for Python package with C++ code produces error saying a required module (numpy) is not found. #44

Closed
aya49 opened this issue Oct 22, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@aya49
Copy link

aya49 commented Oct 22, 2024

🐛 Bug

I am in WSL2, trying to build a Python package (FlowUtils) out of tree for Pyodide because it has C++ scripts in the package: https://pyodide.org/en/stable/development/building-and-testing-packages.html#building-and-testing-packages-out-of-tree

After following the instructions in a virtual environment, I get into the Package directory and run pyodide build, it gives me an error saying that I don't have numpy, which I do.

I also tried building the package normally, and it works. The website said if that's the case, then to open an issue, so here it is! Am I doing something wrong?

Note: I also tried building the package in the tree but it also fails when I try to build pyodide build-recipes flowutils --install, failing at the first package... that's out of scope here, but this is why I tried building out of tree: https://pyodide.org/en/stable/development/new-packages.html

To Reproduce

Pre-requisite: I've already installed Python packages numpy and pyodide-dev. I also have python3-dev installed.

In Windows 11 WSL2 Ubuntu 22.02 terminal:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk  
PYODIDE_EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version)
./emsdk install ${PYODIDE_EMSCRIPTEN_VERSION}
./emsdk activate ${PYODIDE_EMSCRIPTEN_VERSION}
source emsdk_env.sh
cd ..
git clone https://github.com/whitews/FlowUtils.git
cd FlowUtils
pyodide build

Expected behavior

From the website:

"pyodide build invokes a slightly modified version of the pypa/build build frontend so the behavior should be similar to what happens if you do:"

pip install build
python -m build

"If you run into problems, make sure that building a native wheel with pypa/build works."

The packages builds normally when using pypa/build. But it shows and error when I use Pyodide's build (output below). I think the latter is suppose to build normally too but it doesn't.

Environment

  • Pyodide Version: 0.26.3
  • Any other relevant information: Python 3.10.12

Additional context

Error:

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pyodide build
xbuild environment already exists, skipping download
Installing xbuild environment
Traceback (most recent call last):
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 5, in <module>
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 5, in <module>
ModuleNotFoundError: No module named 'numpy'

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ python --version
Python 3.10.12
(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pip install numpy
Requirement already satisfied: numpy in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (2.1.2)

When building normally:

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pip install build
Requirement already satisfied: build in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (1.2.2.post1)
Requirement already satisfied: packaging>=19.1 in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (24.1)
Requirement already satisfied: pyproject_hooks in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (1.2.0)
Requirement already satisfied: tomli>=1.1.0 in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (2.0.2)
(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - oldest-supported-numpy
  - setuptools>=61.0
* Getting build dependencies for sdist...
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
running check
creating flowutils-1.1.0
creating flowutils-1.1.0/src/FlowUtils.egg-info
creating flowutils-1.1.0/src/flowutils
creating flowutils-1.1.0/src/flowutils/gating_c_ext
creating flowutils-1.1.0/src/flowutils/logicle_c_ext
copying files to flowutils-1.1.0...
copying LICENSE -> flowutils-1.1.0
copying MANIFEST.in -> flowutils-1.1.0
copying README.md -> flowutils-1.1.0
copying pyproject.toml -> flowutils-1.1.0
copying setup.py -> flowutils-1.1.0
copying src/FlowUtils.egg-info/PKG-INFO -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/SOURCES.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/dependency_links.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/requires.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/top_level.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/flowutils/__init__.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/_version.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/compensate.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/gating.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/transforms.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/gating_c_ext/_gate_helpers.c -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/gating_c_ext/gate_helpers.c -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/gating_c_ext/gate_helpers.h -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/logicle_c_ext/_logicle.c -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/flowutils/logicle_c_ext/logicle.c -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/flowutils/logicle_c_ext/logicle.h -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/FlowUtils.egg-info/SOURCES.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
Writing flowutils-1.1.0/setup.cfg
Creating tar archive
removing 'flowutils-1.1.0' (and everything under it)
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - oldest-supported-numpy
  - setuptools>=61.0
* Getting build dependencies for wheel...
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
* Building wheel...
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/_version.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/compensate.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/gating.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/__init__.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/transforms.py -> build/lib.linux-x86_64-cpython-310/flowutils
running build_ext
building 'flowutils.logicle_c' extension
creating build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/logicle_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/logicle_c_ext/_logicle.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/_logicle.o -std=c99
In file included from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/flowutils/logicle_c_ext/_logicle.c:2:
/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/logicle_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/logicle_c_ext/logicle.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/logicle.o -std=c99
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/_logicle.o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/logicle.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so
building 'flowutils.gating_c' extension
creating build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/gating_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/gating_c_ext/_gate_helpers.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/_gate_helpers.o -std=c99
In file included from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/flowutils/gating_c_ext/_gate_helpers.c:2:
/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/gating_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/gating_c_ext/gate_helpers.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/gate_helpers.o -std=c99
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/_gate_helpers.o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/gate_helpers.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/flowutils/gating_c.cpython-310-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/_version.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/compensate.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/gating.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/__init__.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/transforms.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/gating_c.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/./flowutils
running install_egg_info
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
Copying src/FlowUtils.egg-info to build/bdist.linux-x86_64/wheel/./FlowUtils-1.1.0-py3.10.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/FlowUtils-1.1.0.dist-info/WHEEL
creating '/mnt/d/schoolwork/projects/FlowUtils/dist/.tmp-6q9ebrg6/FlowUtils-1.1.0-cp310-cp310-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'flowutils/__init__.py'
adding 'flowutils/_version.py'
adding 'flowutils/compensate.py'
adding 'flowutils/gating.py'
adding 'flowutils/gating_c.cpython-310-x86_64-linux-gnu.so'
adding 'flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so'
adding 'flowutils/transforms.py'
adding 'FlowUtils-1.1.0.dist-info/LICENSE'
adding 'FlowUtils-1.1.0.dist-info/METADATA'
adding 'FlowUtils-1.1.0.dist-info/WHEEL'
adding 'FlowUtils-1.1.0.dist-info/top_level.txt'
adding 'FlowUtils-1.1.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built flowutils-1.1.0.tar.gz and FlowUtils-1.1.0-cp310-cp310-linux_x86_64.whl
@aya49 aya49 added the bug Something isn't working label Oct 22, 2024
@hoodmane hoodmane transferred this issue from pyodide/pyodide Oct 22, 2024
@hoodmane
Copy link
Member

Thanks for the report @aya49! @ryanking13 is this potentially related to #21?

@hoodmane
Copy link
Member

From the paths it looks like you are using python3.10. Can you try again with Python 3.12? If that's the problem, we should add a version check with a sensible error message.

@ryanking13
Copy link
Member

I am guessing that it is more related to the Python version mismatch.

@hoodmane
Copy link
Member

We definitely should add a test that we return a good error when there is a version mismatch.

@ryanking13
Copy link
Member

We do check the version when installing the xbuildenv, but looking at the build log, for some reason it is already installed so there was no additional verification.

@hoodmane
Copy link
Member

Probably pyodide build should always do a version check. It's much cheaper than installing the xbuildenv and we do that all the time even if it isn't needed.

@ryanking13
Copy link
Member

Yeah, I think we should

  1. Store the metadata including local Python and (optionally?) pyodide-build version when installing xbuildenv
  2. When running pyodide build check this metadata and raise if the local Python or pyodide-build version has changed.

@aya49
Copy link
Author

aya49 commented Oct 23, 2024

Thank you for the fast reply @hoodmane! I will try it out and post the results! Edit: It worked!!

@aya49 aya49 closed this as completed Oct 24, 2024
ryanking13 added a commit that referenced this issue Nov 25, 2024
This adds a check when initializing xbuildenv (== `init_environment`) to
make sure the local Python version used to install the xbuildenv was not
changed.

This prevents the following scenario:

1. One has a local Python version (3.11.3) and installs Pyodide
xbuildenv (0.25.1).
2. Then, one changes the local Python version to (3.12.1) using pyenv.
3. Pyodide xbuildenv 0.25.1 is no longer compatible to the local Python
version, so it should fail.

Related issue: #44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants