Skip to content

Latest commit

 

History

History
143 lines (118 loc) · 7.64 KB

20230803.org

File metadata and controls

143 lines (118 loc) · 7.64 KB
Raw

Build another Python version than Guix provides

My Guix installation[fn:1] provides Python 3 version 3.10.7. This note describes how to use Guix to install another Python 3 version. This turned out to be much more elaborate than expected…

Management summary

To build a Python version that Guix does not provide out-of-the-box, you

  • find the package specification of the most recent Python version that is supported by Guix but still precedes the version you need,
  • create a package variant that derives from that specification and
  • build the variant.

To give an example, Guix repo file gnu/packages/python.scm contains the package specification of Python 3.10.7 that my Guix installation uses. I have created a package variant that inherits this specification and that overrides the parts required to build 3.10.12, see 20230803/python-3.10.12.scm. The following command builds that Python version:

guix build --file=python-3.10.12.scm

This only builds the derivation, it does not install it.

Build Python version 3.10.12

This whole exercise started when I wanted to use Python 3.9.17, at the time of writing the most recent 3.9 version. The command guix build creates a build of a known package using the source code you provide and the following command was my first attempt to build 3.9.17:

guix build python --with-source=https://www.python.org/ftp/python/3.9.17/python-3.9.17.tgz

This indeed compiled and linked 3.9.17, but because some of its unit tests failed, Guix didn’t create a package for it.

The previous command uses the package specification named python-3.10 in gnu/packages/python.scm to build 3.9.17. However, that specification is intended for 3.10.7, which is provided by my Guix installation, and I could imagine it doesn’t cover 3.9.17. So I tried to build the latest 3.10 version, 3.10.12, which should be more similar to 3.10.7:

guix build python --with-source=https://www.python.org/ftp/python/3.10.12/python-3.10.12.tgz

Unfortunately, the result was the same: the command compiled and linked 3.10.12 but some of its unit tests failed.

This really made me wonder: did 3.10.17 change so much from 3.10.7 to cause a failing build? Time for another experiment, namely rebuild the existing 3.10.7 under another name:

guix build python [email protected]=https://www.python.org/ftp/python/3.10.7/python-3.10.7.tgz

Again we ended up with a successful build of 3.10.7 and several failed unit tests. This really made me wonder how the official 3.10.7 was build.

After some extensive googling, I found this email on the guix-devel mailing list that mentioned the unit tests for Python3. It 1) reminded me that package specification python-3.10 disables some unit tests and 2) showed that some of the disables tests were tests that were failing for me. Then it dawned to me: the guix build --with-source option doesn’t just use a different tarbal to build Python, it replaces the complete source expression of python-3.10. The original source expression skips several unit tests but if you use --with-source, that complete expression is replaced.

So I created new expression and used that to build Python 3.10.7[fn:2]:

guix build --file=python-3.10.12.scm

And voilà, the build completed, the unit tests passed and a Guix package was created[fn:3].

Caveat emptor

The command retrieves the version of the software to build from the URL, but the URL has to be an exact match. For example, the original URL specifies Python-3.9.12.tgz so with a capital P. If you try that command, the command cannot determine the version that is build and still builds 3.10.7. If you specify a lowercase p, so python-3.9.12.tgz, it does build the right version.

Key takeways

To build 3.10.12 was much more work than expected. You have to copy the original package specification and hope that that specification can be reused with no or only trivial changes. If not, you have to delve into the Guix and Python3 build processes. This explains why Guix cannot just support a new Python release.

Going back to my original choice of Python version, 3.9.17, initially I used an even more elaborate approach. Its package specification is one for previous versions of Guix so I thought I have to “travel back in time” to one of those versions using the Guix time-machine command:

guix time-machine --commit=138115b012cd5b223d8f12fc62e3bd13202be737 -- build python --with-source=https://www.python.org/ftp/python/3.9.17/python-3.9.17.tgz

Unfortunately, most of the binaries this version uses aren’t available anymore so you have to build them yourself. This takes a lot of time on my machine and I cancelled the command after more than an hour with nothing to show for[fn:4].

The use of Guix time-machine builds a version using an older Guix, with an older toolchain. This lets you create an exact copy of a package that is not available anymore. However, that’s not really what I need: I’m fine with a new Python build with the current toolchain. So I could have created a derivation for 3.9.17 from the most recent 3.10.7 package specification. Whether that works and gives you a usable build is another question. For example, it could be that some of the patches have changed since 3.9 so using the patches from a 3.10.7 build might not work.

Conclusion

All in all, I found it to be quite an elaborate process to build a new Python 3 version with Guix. Fortunately I could reuse a package specification for the current supported Python 3 build. If that’s not the case, I can imagine you’ll have quite the work cut for you.

In the past I’ve used pyenv to install different Python versions. Just like Guix, it downloads the source code of the Python version you specify and builds it locally. Although pyenv doesn’t guarantee reproducibility, it’s more convenient to use and supports a large set of Python versions out-of-the box. This means I’ll keep using pyenv.

One other thing, it seems that Guix doesn’t support the installation of different Python 3 versions side-by-side, that is, not in the same Guix profile. The python testing tool tox needs this functionality to test code for multiple Python versions. Work has been done on a tox variant that uses Guix as its backend, viz. Guix-tox, but it’s very uncertain that variant is still functional: the latest change to its repo was 7 years ago.

Footnotes

[fn:1] commit hash 676508ac858928a2ec66f18ccfae17c9cec3dda2

[fn:2] I copied the original python-3.10 package specification and made the changes I deemed necessary. It took some trial and error to get it right, but the Guix error messages were really clear and helpful. You can find the specification to build Python 3.10.12 in 20230803/python-3.10.12.scm.

[fn:3] Although Guix builds the package for me, I haven’t found a way to install it, yet :).

[fn:4] My main development machine is an older ThinkPad T470 which has an [email protected], albeit with 32GB of internal memory. Maybe the machine isn’t powerful enough for a “quick” build.