Nice diagrams. Sounds like a reasonable idea.

To implement this issue, I started looking at the behavior of pypabuild and pip. The tricky part is that pip treats multiple index urls equally (pypa/pip#8606)

So, say we have a package that have build dependencies: ["numpy", "setuptools", "setuptools-scm"]. The behavior we want is:

1. Install numpy from Pyodide-hosted package index (wasm32 wheel)
2. Install setuptools and setuptools-scm from PyPI (manylinux wheel or py3-none-any wheel)

but this is not achievable since numpy exists both in the Pyodide-hosted package index and PyPI, and pip will choose the latest version, which might not be the Pyodide-hosted one.

So, I think we have two options:

1. Host all build dependencies, such as setuptools, poetry, hatch, Cython, ... in the alternative package index as well.
2. Maintain a list of packages that need to be installed from Pyodide package index, and only these packages are installed from the Pyodide package index.

I think option 1 is the better long-term solution, but I'd be interested in implementing option 2 in the short term. It's logically equivalent to how we currently specify cross-build-env in meta.yaml.

Pseudo-code for Option 2 would be:

build_deps = ["numpy", "setuptools", "setuptools-scm"]

cross_build_deps, host_build_deps = split_build_deps(build_deps)
# cross_build_deps = ["numpy"]
# host_build_deps = ["setuptools", "setuptools-scm"]

build_env.install_reqs(cross_build_deps, env={"PIP_INDEX_URL": PYODIDE_PACKAGE_INDEX})
build_env.install_reqs(host_build_deps, env={"PIP_INDEX_URL": PYPI})

These are the list of build dependencies for packages in the Pyodide distribution as of 2024/11/26. It is not as much than I thought, and I believe most of them are pure Python packages except for a few packages. So maybe we can just host them (not included in the Pyodide distribution, but only for build time dependency)

build
cffi
cmake
colorama
cppy
cysignals
cython
ewah-bool-utils
expandvars
extension-helpers
flit_core
flit_scm
gmpy2
hatch-fancy-pypi-readme
hatch-vcs
hatchling
jinja2
lazy_loader
maturin
meson-python
meson
mypy_extensions
nanobind
ninja
numpy
oldest-supported-numpy
packaging
pip
pkgconfig
poetry-core
py-cpuinfo
pybind11
pythran
rebound
scikit-build-core
scikit-build
scipy
setuptools
setuptools-git-versioning
setuptools-rust
setuptools-scm
sphinx
tomli;
types-psutil
types-setuptools
typing-extensions
versioneer[toml]
wheel