[Tracking issue] Ensure that non-variant tools continue working

[konstin]

We need to ensure that non-variant tools keep working with the variant additions. This concern specifically:

• PEP 517 frontends, through variant wheels
• The new variant markers, through requirements in wheels
• The variants.json file, in registries

This issue is a tracking issue to collect all backwards compatibility hazards and a point to check before submitting the final PEP.

[konstin]

For the variant markers, we have the same problem (and hopefully solution) as https://discuss.python.org/t/pep-780-abi-features-as-environment-markers/86013/31.

[mgorny]

For installers, we went for changing the filename so that it would be rejected/ignored by tools, per #5. We've tested a few tools (admittedly not many), and they did the right thing — i.e. rejected variant wheels, or fell back to non-variant when appropriate. However, there is still some risk of tools a. not performing sufficient validation and using a variant wheel in place of a regular wheel, and b. tools failing over invalid filename immediately without trying other wheels. This part has also been challenged during the earlier discussion.

One option that's still on the table is moving the variant label onto platform tag (i.e. manylinux_2_27_x86_64+label), while either a. requiring it to be repeated for every platform tag, or b. banning multiple platform tags. This shifts the rejection reason from "invalid filename" to potentially more commonly used "unsupported platform". This may make the discussion easier, reduces the risk of tools failing unexpectedly and improves compatibility with tools that don't care about variants (i.e. non-installers). I'm not pushing for it now, but I think we should keep it in mind in case of future problems.

[konstin]

Here's some quick pip statistics for the last 7 days from the PyPI BigQuery dataset. I wouldn't vouch for that data but it matches other data I've seen.

pip 23.0 was released 2023-01-30, so we're still seeing a lot of usage for a >2.5 year old pip, and even non-negligible numbers for pip 21.

[rgommers]

Thanks for the statistics @konstin. That's a little sad, such a long tail. I guess 90% of this is downloads from CI so this will largely reflect what is bundled with CPython? Or did you manage to exclude CI usage?

Not sure the answer to that question matters too much, either way it's high-impact. Even if it's solvable by every projects plastering pip install -U pip everywhere, that's still a ton of churn.

[mgorny]

Not really sure how to proceed here. PEP 780 is related but the discussion there sends to have stalled. We may want to do a separate DPO thread or even a short PEP, but I don't want to accidentally end up delaying 780 if it would otherwise just pass as-is.

The two main points are:

1. Do we actually need to remain compatible with old installers? Perhaps the "DPO consensus" will be "we don't care".
2. How complex solution is going to be acceptable? Donald Stufft pointed out that we don't want to carry an extra burden for years to come because of a transitional problem.

We can split the problem in two:

1. Do we need to avoid breaking this once? If yes, then Konstantin's solution of adding new metadata may be desirable (but will it justify the extra burden?)
2. Do we want a comprehensive solution for future additions? If yes, then my idea of supported_markers may be desirable.

Both of these are PEP material. There's always the option of emitting different metadata but that's a dangerous precedent, and doesn't solve 780.

[konstin]

I'd definitely try separating the discussion out from the main variants thread - both easily have enough discussion for their own thread. It makes it easier to iterate over solutions as we build consensus.

Do we want a comprehensive solution for future additions? If yes, then my idea of supported_markers may be desirable.

It would be great to have something like a supported_features or a spec_version marker that we can add all sorts of features into. It's something that can be reused e.g. exposing it in simple API pages to have installers skip files, so it becomes valuable to more initiatives in the futures, plus we get good error reporting.

[rgommers]

@konstin would you be able to refresh the pip statistics data? It's time to poke at this more seriously, and fresh data would help. Hopefully the tail has compressed a little (not holding my breath though).

[rgommers]

The uv preview feature for PEP 780 (see here on DPO) reminded me this needs work very soon.

I believe the key principle for Python packaging should be that the artifact producer/publisher side should be strict on (unknown things get rejected) while the consumer/install side should be forward-compatible, and either ignore new metadata or consider the presence of it as marking an artifact or release as invalid and skip over it.

That principle is largely followed already:

• Core Metadata: new minor-version fields are gated on build/publishing/index support, while older consumers should tolerate newer minor versions.
• pyproject.toml: new [project] fields such as PEP 639 license-files and PEP 794 import-names require producer-side support, but do not break older installers once represented in published metadata.
• Simple API: servers may add new fields, while older clients ignore fields they do not understand.
• Wheel format: newer minor Wheel-Versions should remain consumable; unsupported major versions fail.
  • And of course PEP 825 wheel filenames as well, they ensure variant wheels get ignored by older tools.
• Wheel compatibility tags: an unknown or unsupported tag makes that wheel ineligible rather than making the whole release or even the whole package invalid.

Environment markers are the main (or only?) exception to this pattern. The grammar for markers contains a closed list of marker names (see env_var in https://packaging.python.org/en/latest/specifications/dependency-specifiers/#grammar). Adding a new marker therefore changes the grammar itself. Older parsers raise a SyntaxError instead of parsing the dependency and treating that package version as unsupported, then falling back to an older compatible version. This requires a one-time grammar change so that unknown marker names are parse-able but semantically unsupported.

A separate supported_markers or supported_features mechanism may still be useful for capability discovery, diagnostics, or index-side filtering. But it seems optional to me - it could have value though on top of making the marker-name grammar forward-compatible (I can't fully see the implications of such a mechanism right now).

Does that sound right? If so, I think we need a proposal for a grammar change first. I'd be inclined to keep the proposal focused on that, unless we can think of a way to explain supported_markers in a way that doesn't increase the degrees of freedom and the expected discussion volume too much.

[mgorny]

Grammar change = have a default of "evaluates to false" or "evaluates to true" for a value?

[rgommers]

Neither, grammar change meaning "unknown may occur, if it does then emit an installer-too-old warning and treat the artifact or release in question as invalid and skip it". You can't guess true/false with this, because guessing wrong may lead to a broken install.

[konstin]

@konstin would you be able to refresh the pip statistics data? It's time to poke at this more seriously, and fresh data would help. Hopefully the tail has compressed a little (not holding my breath though).

Up-to-date data is in https://blog.schuetze.link/python-ecosystem-statistics/, let me know if you want something more specific.

[rgommers]

Thanks, that is super helpful. The pip stats are worse than I hoped, it's pretty 😢 that >30% of downloads are done with a version that is over 1,000 days old.

[konstin]

Wheel format: newer minor Wheel-Versions should remain consumable; unsupported major versions fail.

A main problem here is the lack of wheel version in the API that does not allow tools to skip too new wheels. But that's a different PEP.

I tend towards that this is overkill, but there's another framing: We're talking about the grammar of the Requires-Dist field. If we wanted, we could introduce a Requires-Dist2 with the new grammar, that is used of Requires-Dist by tools that support it, which lets us put both metadata into the field.

We may also be overengineering this. We could say that if you want to install the latest version of that ML library, please update your pip and/or uv version, which arguably is a fair requirement, the only problem is the bad error message (it doesn't tell you that you need to upgrade).

(I'm not married to any of these)

[rgommers]

Ah I wasn't aware of the wheel version not in index server API issue, thanks.

Re Requires-Dist2: yeah I think we touched on that before. It's possible but it feels like it won't help with anything that the grammar change doesn't already help with, for similar (maybe more) effort. And it's actually worse: you can't start using Requires-Dist2 without also population (or poisoning) Requires-Dist, and there's no good way of doing that. At least, if you assume that incorrect answers to "what dependencies are needed?" are not allowed.

We may also be overengineering this. [...] please update your pip and/or uv version

Yes, I think that's a valid alternative. My gut feeling is that that by itself isn't going to be accepted, but either way this is an option. Even if we do a grammar change as well, we're not going to wait for 1,500 days for that grammar change to propagate - at some point we'll just want to roll out new environment markers, and there will be breakage for old installers. I think it's more of a "how do we do better and avoid that breakage next time".

Also, a sanity check on engineering effort: I am thinking that the grammar change is relatively easy to implement - probably less work than shepherding a PEP about it all the way to acceptance. If I'm wrong about that though, please tell me.

[mgorny]

Yeah, I think that grammar change is a good step forward. And most likely the best we can hope for, at least short term.