Parallel builds support #392
Conversation
|
Cool thanks for looking into this! |
indygreg
force-pushed
the
gps/parallel-builds
branch
from
74a9dc0 to
76cdbd9
Compare
|
The 3.13 CI jobs aren't yet fully optimized and already this PR drops Linux builder total time from ~64h to ~43h, or ~67% of baseline. Not bad. |
I'm about to enable support for parallel CPython builds. This function was currently not safe if called in parallel. This was causing setuptools / pip downloads to race writing to the same file and failing builds.
This is less ergonomic (oh Make). But it will make it easier to build multiple Python versions at the same time in a future commit.
Previously we generated the same Makefile for every Python version. We can simplify things by only writing 1 Makefile per target configuration.
This refactor allows targets for each major Python version to be defined simultaneously. This opens the door to a single `make` invocation building multiple versions in parallel.
`build-main.py` now passes in space delimited Python versions to build, which the Makefile can happily translate to appropriate make targets for us. We added support for `--python all` to allow building all supported Python versions.
Our build system supports this now. So update CI to make use of the feature. On Linux, 3.13 freethreaded builds are still their own CI job. We may need to tweak `build-main.py` to allow intelligently selecting freethreaded compatible versions. We could potentially also allow building freethreaded in parallel with non-freethreaded. This can be done as a followup.
indygreg
force-pushed
the
gps/parallel-builds
branch
from
76cdbd9 to
d17e9dc
Compare
|
The macOS builds should be fixed. Issue turned out to be a compatibility issue changed in GNU Make 3.82 (macOS ships 3.81). Specifically commit 81f3e4babd128f6740d05b371122762924522fb6 (because I bisected it).
|
|
Ok. CI came back clean. macOS CI total time drops from 10:27 to 8:49. 529/627=84% of baseline. Not the percentage savings as Linux. And 3.13 is still on the table for optimization. But still a nice win. |
|
My changes in #428 will make this a little more complicated, but it's probably still worth pursuing. I'm happy to investigate it. |
This is an attempt to support building multiple Python versions at the same time. The impetus for this is reducing the amount of work that CI does by amortizing the costs for building common dependencies.
The first 3 commits should be relatively safe to merge. 4 and 5 need a bit of testing.
Open issues: