Skip to content

Building portable wheels for ANTsPy

Jump to bottom
Philip Cook edited this page Oct 22, 2024 · 44 revisions

NOTE: remember to create the appropriate git tag when releasing.

Current system (updated 2024-08-12)

Release checklist

  • Tag vA.B.C on master
  • Update version in pyproject.toml and push
  • Create release from tag on Github
  • Add extra wheels on pypi as needed

Github actions

GHA builds wheels on release creation for Mac, Linux, and Windows. These are automatically pushed to PyPI.

For Mac, the PyPI builds are targeted for the runner host OS. This is required because the build needs libpng, installed via Homebrew (PRs welcome to work around this limitation). So wheels will have tags like macosx_12_0_x86_64 or macosx_14_0_arm64. However, some Mac python installations do not include the current OS in the compatible tags list, so pip will not install the macosx_12_0_x86_64 wheel even for a user running Mac OS 12+. More info and workarounds here.

For user convenience, I (PAC) have been able to build compatible wheels using a miniconda python installation on my Mac:

# activate conda environment for antspy. This example using python 3.11
export MACOSX_DEPLOYMENT_TARGET=10.14
python -m build --wheel
twine upload dist/antspyx-0.5.3-cp311-cp311-macosx_10_14_x86_64.whl -u __token__ -p ${ANTSPYX_PYPI_API_TOKEN}

MACOSX_DEPLOYMENT_TARGET is set to 10.14 because this is the earliest version that includes full C++17 support. For ARM64, I assume one would need MACOSX_DEPLOYMENT_TARGET=11.0 because 11 is the earliest version shipped with M1 Macs.

I would much appreciate a PR that let us set MACOSX_DEPLOYMENT_TARGET=10.14 for Intel and MACOSX_DEPLOYMENT_TARGET=11.0 for arm64 Macs in the Github Actions wheel build.

Older build instructions

OSX

use pyenv

https://www.chrisjmendez.com/2017/08/03/installing-multiple-versions-of-python-on-your-mac-using-homebrew/

install multiply python versions

for x in 3.6.12 3.7.12 3.8.12 3.9.8; do
  pyenv install -v  $x 
done

then it's straightforward to change python versions

eval "$(pyenv init -)"
pyenv global 3.5.2
pyenv local 3.5.2

then build the wheel as usual and upload.

for example run this script:

#!/bin/bash
py=$1
pyenv virtualenv $py pyenv${py}
source ~/.pyenv/versions/pyenv${py}/bin/activate
python3 --version
mkdir ANTsPy${py}
cd ANTsPy${py}
git clone https://github.com/ANTsX/ANTsPy.git
cd ANTsPy
python3 -m pip install --upgrade pip
python3 -m pip install twine 
python3 -m pip install wheel cmake twine
python3 setup.py bdist_wheel
python3 -m twine upload -u USERNAME -p PASSWORD dist/*

LINUX

Resources

https://github.com/joerick/cibuildwheel

https://github.com/astrofrog/autowheel

the docker images

https://github.com/pypa/manylinux

e.g.

quay.io/pypa/manylinux1_x86_64

run the docker image

docker run -it --rm quay.io/pypa/manylinux_2_24_x86_64 /bin/bash

or

docker run -it --rm quay.io/pypa/manylinux1_x86_64 /bin/bash

use docker ps to find name of container

use docker cp to copy cmake to container

ANTsPy

clone it

git clone https://github.com/ANTsX/ANTsPy.git

build wheels for various platforms

see cat ~/.pypirc 

pyguys="/opt/python/cp310-cp310/bin/python3  /opt/python/cp37-cp37m/bin/python3  /opt/python/cp39-cp39/bin/python3
/opt/python/cp36-cp36m/bin/python3   /opt/python/cp38-cp38/bin/python3"

git clone https://github.com/glennrp/libpng.git libpng
cd libpng
./configure --enable-maintainer-mode 
make 
make install
cd ..

PLAT=manylinux_2_24_x86_64

export TWINE_USERNAME=__token__
# also export your unique API token
export TWINE_PASSWORD=pypi-FIXMEOTHERSTUFFHERE
ct=100
for PYBIN in /opt/python/cp*/bin/python3 ; do
  ${PYBIN} -m pip install --upgrade pip
  ${PYBIN} -m pip install twine cryptography==2.9.1 cmake
  mkdir build${ct}
  cd build${ct}
  git clone https://github.com/ANTsX/ANTsPy.git
  cd ANTsPy
  ${PYBIN} setup.py install
  find build/ -name "*.so"|xargs strip
  ${PYBIN} setup.py sdist bdist_wheel doctor-wheel
  auditwheel repair dist/*whl --plat $PLAT
  /opt/_internal/cpython-3.${XX}*/bin/doctor-wheel wheelhouse/*whl
  ${PYBIN} -m twine upload wheelhouse/*
  ${PYBIN} -m twine upload dist/*tar.gz
  cd /
  let ct=$ct+1
done

advice from here https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh

and https://github.com/pypa/python-manylinux-demo/blob/master/.travis.yml

use clang

export TWINE_USERNAME=FIXME
export TWINE_PASSWORD=FIXME
PLAT=manylinux_2_24_x86_64
apt-get update
apt-get upgrade -y
apt-get install wget

apt-get update
apt-get install -y software-properties-common
apt-get install -y apt-transport-https

wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
apt-add-repository "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch main"

apt-get update
apt-get upgrade -y
apt-get install --allow-unauthenticated -y clang clang-format clang-tidy
update-alternatives --install /usr/bin/cc cc /usr/bin/clang 90
update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 90
update-alternatives --install /usr/bin/cpp cpp /usr/bin/clang++ 90

git clone https://github.com/glennrp/libpng.git libpng
cd libpng
./configure --enable-maintainer-mode 
make 
make install
cd ..

ct=100
for XX in 9 8 7 ; do
  cd /
  PYBIN=`ls /opt/python/cp3${XX}-cp3${XX}*/bin/python3`
  ${PYBIN} -m pip install --upgrade pip
  ${PYBIN} -m pip install twine cryptography==2.9.1
  ${PYBIN} -m pip install cmake  doctor-wheel
  mkdir build${ct}
  cd build${ct}
  git clone https://github.com/ANTsX/ANTsPy.git
  cd ANTsPy
  ${PYBIN} setup.py install
  find build/ -name "*.so"|xargs strip
  ${PYBIN} setup.py sdist bdist_wheel
  auditwheel repair dist/*whl --plat $PLAT
  ${PYBIN} -m twine upload wheelhouse/*
  ${PYBIN} -m twine upload dist/*tar.gz
  cd /
  let ct=$ct+1
done

CENTOS builds

export TWINE_USERNAME=FIXME
export TWINE_PASSWORD=FIXME

PLAT=manylinux2014_x86_64

git clone https://github.com/glennrp/libpng.git libpng
cd libpng
./configure --enable-maintainer-mode
make
make install
cd ..

ct=101
for XX in 6 9 8 7 ; do
  cd /
  PYBIN=`ls /opt/python/cp3${XX}-cp3${XX}*/bin/python3`
  ${PYBIN} -m pip install --upgrade pip
  ${PYBIN} -m pip install twine cryptography==2.9.1
  ${PYBIN} -m pip install cmake  doctor-wheel
  mkdir build${ct}
  cd build${ct}
  git clone https://github.com/ANTsX/ANTsPy.git
  cd ANTsPy
  ${PYBIN} -m pip install .
  find build/ -name "*.so"|xargs strip
  ${PYBIN} setup.py sdist bdist_wheel
  auditwheel repair dist/*whl --plat $PLAT
  ${PYBIN} -m twine upload wheelhouse/*
  if [[ $ct -eq 100 ]] ; then
    ${PYBIN} -m twine upload dist/*tar.gz
  fi
  cd /
  let ct=$ct+1
done

CENTOS builds: manylinux1_x86_64

export TWINE_USERNAME=FIXME
export TWINE_PASSWORD=FIXME

PLAT=manylinux1_x86_64

git clone https://github.com/glennrp/libpng.git libpng
cd libpng
./configure --enable-maintainer-mode
make
make install
cd ..

yum -y install libffi-devel

ct=100
for XX in 9 8 7 6 ; do
  cd /
  PYBIN=`ls /opt/python/cp3${XX}-cp3${XX}*/bin/python3`
  ${PYBIN} -m pip install --upgrade pip
  ${PYBIN} -m pip install twine cryptography==2.9.1
  ${PYBIN} -m pip install cmake  doctor-wheel
  mkdir build${ct}
  cd build${ct}
  git clone https://github.com/ANTsX/ANTsPy.git
  cd ANTsPy
  ${PYBIN} -m pip install .
  ${PYBIN} setup.py sdist bdist_wheel
  auditwheel repair dist/*whl --plat $PLAT
  ${PYBIN} -m twine upload wheelhouse/*
  if [[ $ct -eq 100 ]] ; then
    ${PYBIN} -m twine upload dist/*tar.gz
  fi
  cd /
  let ct=$ct+1
done

ubuntu manylinux (supposed to be futureproof)

#
# manylinux_2_24_x86_64
# docker run -it --rm quay.io/pypa/manylinux_2_24_x86_64 /bin/bash
#
git clone https://github.com/glennrp/libpng.git libpng
cd libpng
./configure --enable-maintainer-mode
make
make install
cd ..

ct=100
for XX in  9 8 7 6 ; do
  cd /
  PYBIN=`ls /opt/python/cp3${XX}-cp3${XX}*/bin/python3`
  ${PYBIN} -m pip install --upgrade pip
  ${PYBIN} -m pip install twine cryptography==2.9.1
  ${PYBIN} -m pip install cmake  doctor-wheel
  ${PYBIN} -m pip install scipy numpy
  mkdir build${ct}
  cd build${ct}
  git clone https://github.com/ANTsX/ANTsPy.git
  cd ANTsPy
  ${PYBIN} -m pip install .
  find build/ -name "*.so"|xargs strip
  ${PYBIN} setup.py sdist bdist_wheel
  auditwheel repair dist/*whl
  ${PYBIN} -m twine upload wheelhouse/*
  if [[ $ct -eq 100 ]] ; then
    ${PYBIN} -m twine upload dist/*tar.gz
  fi
  cd /
  let ct=$ct+1
done
Clone this wiki locally