Repository navigation
Conditionally build Python-dependent Lyrical libraries across Pythons 3.12-3.14 - #193
EzraBrooks wants to merge 2 commits into
Conversation
EzraBrooks
changed the title
This comment has been minimized.
This comment has been minimized.
02e5102 to
0d337b2
Compare
0d337b2 to
0fb00ec
Compare
19b8b2d to
91a335f
Compare
e57c4fa to
561e02a
Compare
|
Ok I think we have all the building blocks for this now @EzraBrooks :) |
561e02a to
198e682
Compare
198e682 to
067dbeb
Compare
With several Python versions pinned (lyrical, #193), a recipe with Python content makes one build per version, and the estimate counted recipes, not builds: #193's linux-64 run had 3474 builds where the estimate said 2487 packages. build_timings.variant_builds counts each recipe's builds from the multi-valued variant keys its build/host requirements name without a version (zipped keys as one axis, conditions evaluated for a native build), from the work dir's conda_build_config.yaml. ci_estimate counts builds, and a cached build per variant (the cache holds each variant's package). build-ci records a recipe's time per build (each variant's span, averaged), and vinca_timings multiplies it by the recipe's builds for packing the jobs. check_dependency_compat: variant_values and zip_groups; variant_pins now uses variant_values (same output, checked on all distros/platforms). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
is it time to re-run CI here? |
|
I'm investigating the failure |
|
Claude suspects a race condition in build parallelism in rattler-build. Seems suspicious, but I'll chase that down. If that's the case then we might be able to just get away with a cached re-run here. |
|
Ah it is indeed prefix-dev/rattler-build#2931 which @Tobias-Fischer already found |
…d#2931 A build or test waited for any variant of a dependency, not the one it can install: e.g. a Python 3.13 test started once the 3.12 package it needs was built, and failed to solve. The pre-releases (built from main, for linux-64, osx-arm64 and win-64; built for glibc 2.28) from prefix.dev's rattler-build-pre-release channel, and the same commit built from git on osx-64 and linux-aarch64. patchelf, which only conda-forge's package depends on, is added on Linux. Locked with pixi 0.81.0, the workflows' version: the source packages' identity differs between pixi versions.
|
so @Tobias-Fischer @traversaro how do we move forward here? the build seems to be moving along happily, but my understanding is that this will take way too long for CI and just always time out? |
|
Yes .. I think we're really at a stage now where we would massively benefit from a faster CI without time limit. We've been discussing this a few times before, but never found a sponsor. If we can do something self-hosted, I could probably get machines together to do this, but wouldn't be able to guarantee 99.999999% uptime ;). For this particular PR, happy to merge, but we'll face this again soon .. workaround is to restart CI every 6 hours and rely on caching. |
|
Implementing noarch for pure-Python packages will contribute some speed-up but not enough to matter at this scale. I will start noodling on overall build speed improvements, though you guys probably have already hit all the low hanging fruit and more |
…valuated (vinca#189) Only packages with Python content get Python in host/run and with it the Python variants; the others get a python_min-pinned build-time interpreter and a test that they install no Python modules. Same recipes as vinca 74d1e85 for lyrical (linux-64, win-64), apart from vinca-rs's intended differences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the improved "conditional Python dependency" support from
to build Lyrical Python packages for Pythons 3.12, 3.13, and 3.14 while not unnecessarily rebuilding non-Python dependencies.