Rust November 2023

rust@lists.fedoraproject.org

2 participants
3 discussions

Re: Rust bindings for Python (pyo3 versions <0.19, cpython) broken with Python 3.12
by Fabio Valentini 29 Nov '23

29 Nov '23

On Fri, Nov 17, 2023 at 1:34 AM Kevin Fenzi <kevin(a)scrye.com> wrote: > > On Mon, Nov 13, 2023 at 07:25:10PM +0100, Fabio Valentini wrote: > > > > Yup, I've mentioned that in the bug I filed for python-bcrypt - > > It might be as simple as bumping the dependency on pyo3 from v0.15 to v0.19. > > I've made an attempt here: > https://src.fedoraproject.org/rpms/python-bcrypt/pull-request/9 All dependent packages have now moved off of pyo3 v0.15 to at least pyo3 v0.19.2+ or the latest v0.20, both of which should be fine with Python 3.12. Thank you all for your help! I will retire the packages for pyo3 v0.15 from rawhide now. Fabio

1 0

Rust crates with optional namespacing and how to map them to package names
by Fabio Valentini 17 Nov '23

17 Nov '23

Hi all, I'm sorry in advance for the wall of text below, but I think it adds some necessary context to this issue. The cargo RFC for "Packages as (optional) namespaces" was recently approved, and will likely be implemented in a future version of cargo, and on the official package registry, crates.io: c.f. https://github.com/rust-lang/cargo/issues/12901 c.f. https://github.com/rust-lang/rfcs/pull/3243 With this being an *optional* namespacing, this raises a few problems for packaging. For example, the 1:1 mapping of crate name <-> package name is no longer possible, since there will be zero or one namespace associated with crates. I have found no other language-specific package registries that use "zero or one" namespaces other than NPM, so I don't think there's good "prior art" for this problem. Other language-specific registries I've looked at either use *no* namespacing at all, or always use *exactly one* level of namespacing - but not "zero or one", as will likely be implemented in the Rust crate ecosystem. After having been made aware of this RFC, I commented on the upstream tickets, warning them that this is a rather unusual setup, and will likely cause issues with how Fedora packages Rust crates. One of the Rust package maintainers from Debian added a similar comment. It looks like these decisions were made without considering the broader ecosystem and / or downstream consumers of Rust crates, and I only became aware of this "pending approval" RFC very late in the process, since there doesn't seem to have been any outreach to downstream consumers of Rust crates (neither any Fedora, nor any Debian Rust packagers were aware). With the RFC having been approved, it now raises the issue of how to represent the new optional namespace name in package names in Fedora. There was some discussion in the upstream ticket(s) about this, and I don't much like my conclusions - but here's a summary: - The only characters allowed in names of crates that are published on crates.io are a-z, A-Z, 0-9, hyphen, and underscore. These are all acceptable for RPM package names, which has allowed us to map "project published as $crate" -> "crate packaged as rust-$crate" without collisions (disregarding any possible collisions due to compat version suffices, which luckily has not happened yet). - The proposed separator between crate namespace and crate name is `::`(two colons, just like the import path separators in "use" statements in Rust code). This is not a valid character in RPM package names. - The intersection of the three sets "ascii characters", "characters that are valid in RPM package names", and "characters that cannot be part of a crate name" is vanishingly small. To the best of my knowledge, the intersection only contains two characters - the plus sign `+` and the dot `.`. - Rust crates as they are packaged in Fedora already use the `+` character as a separator for a different purpose in "feature subpackages'' (i.e. rust-$crate+$feature-devel), similarly to how Python packages handle "extras" with additional subpackages. As best as I can tell, this leaves us with only two options: The first option would be to adopt the `.` character as the separator character between crate namespace and crate name, as that appears to be the only choice that does not lead to any ambiguities (hyphen and underscore characters can be part of both crate namespace and crate name, and the plus sign is already used for a different purpose) - i.e. source packages being named like `rust-$namespace.$crate`, and subpackages being named like `rust-$namespace.$crate+$feature-devel`. The second option would be to "overload" the meaning of the `+` sign, and use it both as a separator between "crate namespace" and "crate name" *and* as the separator between "crate name" and "feature name". In this case, both the source package name and the binary package names will contain the `+` character. Using the `.` as the new separator, the source package names would not contain any `+` characters. However, this would not result in unambiguous package names in all cases (in particular, for "binary" packages). A `+` sign in the source package name unambiguously means that the project consists of both namespace *and* name, but in built "-devel" packages, only the occurrence of two `+` characters is unambiguous. However, for "-devel" packages that contain only one `+` character, it is possible for both `rust-$namespace+$crate-devel` (i.e. the "main" binary package, which encodes all mandatory dependencies for project "$namespace::$crate") and `rust-$crate+$feature-devel` (i.e. the binary package that encodes dependencies for the feature "$feature" for project "$crate") to coincide, with two source packages for different projects resulting in binary packages of the same name. I also don't consider this an "unlikely" thing to happen - rather, I think it will be *very likely*. For example, a future `serde::derive` crate and the subpackage for the "derive" feature of `serde` crate would both result in binary packages named `rust-serde+derive-devel`. Using the `.` as the separator character, these two cases would result in unique package names instead (i.e. `rust-serde.derive-devel` and `rust-serde+derive-devel`, respectively). So, the only choice that does not lead to ambiguities and name clashes appears to be to use the `.` character as the new separator between namespace name and crate name. The Packaging Guidelines are not really helpful in this case - they seem to only stipulate that the plus sign `+`, the dot `.`, and the underscore `_` MUST NOT be used as separators when packaging a project, but this only seems to apply to the source package name (?) if the upstream project's name consists of multiple parts, and already doesn't fit with "extra" feature packages like the ones that are used in Python or Rust packaging already. If we really need to adopt the `.` character as a new separator for namespaced Rust crates, I will ask the FPC to amend this section of the Packaging Guidelines to allow uses case like this, where there really is no other option: https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_separato… I'm open to discuss other options, but I don't see any. Please let me know what you think. Fabio

2 1

Rust bindings for Python (pyo3 versions <0.19, cpython) broken with Python 3.12
by Fabio Valentini 13 Nov '23

13 Nov '23

Hello Pythonistas and Rustaceans, TL;DR: Only PyO3 v0.19.2 (and later) will ever properly support Python 3.12. Port your Python projects to v0.19 **NOW**. Older versions of PyO3 (especially pyo3 v0.15, v0.16, v0.17, and v0.18) are *not* compatible with Python 3.12 due to some ABI changes in unicode strings and behavioural changes wrt/ "immortal" objects. This also affects all current versions of the "cpython" Rust bindings, with no timeline for Python 3.12 support. As far as I can tell, extensions that use pyo3 < v0.19 or the "cpython" bindings can (and likely will) not work as expected on Python 3.12 if they use the affected APIs (either by producing garbage data in strings that are passed over the FFI boundary, or by crashing). There are applications in Fedora that still rely on *ancient* versions of PyO3, potentially affected by this: - cpython: mercurial - pyo3 v0.15: fapolicy-analyzer, python-bcrypt, python-cryptography - pyo3 v0.16: python-y-py - pyo3 v0.17: unused compat packages, will be retired - pyo3 v0.18: matrix-synapse I *stongly* recommend to move all of these packages to pyo3 v0.19 in Rawhide as soon as possible. I will try to submit pull requests with the required changes for affected packages (except mercurial, since there's no version of the "cpython" crate that supports Python 3.12 in sight). There's already a few packages that depend on pyo3 v0.19, which I will rebuild in rawhide for pyo3 v0.19.2, which has much better support for Python 3.12 than v0.19.0 and v0.19.1 (breezy, python-rpds-py, orjson) unless there are any objections. As soon as no packages depend on the compat packages for old versions of pyo3 any longer, I will retire them from Rawhide (and F39, depending on the timing), since they will never work with Python 3.12 and nothing should use them. I've added <package>-maintainers(a)fedoraproject.org for all these packages to the CC of this message. Fabio Rust SIG / PyO3 maintainer in Fedora

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

Rust November 2023