Redesigning the %python_provide macro from scratch
by Miro Hrončok
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
I have realized several things I don't like about %python_provide:
1) It must be used conditionally (it is not defined in python-srpm-macros).
That means you always wrap it in %{?python_provide:...} in order to have a
"valid" specfile even when the macro is not yet defined (e.g. during SRPM
creation in Koji or on a packager's machine without python-rpm-macros installed).
2) You cannot use it with arbitrary versions. Suppose you package python3-foo
1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very
common, but it happens. %python_provide only takes the "name" as an argument,
always using %{?epoch} %{version} and %{release}.
3) You need to both add a virtual provide + use the macro. Suppose you want to
provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release}
%{?python_provide:%python_provide python3-pkg_resources}
4) When used with (sub)package name, it generates a duplicate dependency on
Fedora 33+ (and an rpmlint error).
5) It produces Obsoletes, but that might no longer be necessary nor desired.
6) Broken expectations about %_isa. It used to add %_isa provides based on wrong
data, it no longer does that (except on old releases and EPELs), can be used
manually with name%{?_isa}, but not on the old releases.
7) Undocuemnted error handling (e.g. the macro expands to nothing when used with
pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new
mechanism to do this and preserve the old macro as is, deprecating it in Fedora
36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is
basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested)
ideas are:
1) Define the macro in python-srpm-macros. No need to use it conditionally. We
can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we
start using the macro only after Fedora 30 goes EOL, we can make the macro
behave consistently across all Fedora versions.)
2) Accept version identifier as an optional argument, use %{?epoch} %{version}
and %{release} as default. (See for example %{pypi_source} on how this can be done.)
3) Make the macro also produce the provide for the given name and document that.
E.g. when you call it with python3-pkg_resources, it also provides
python3-pkg_resources (not only python-pkg_resources etc.).
4) Make it so that for given arguments, the macro will only expand to something
once per build. Hence when you use it with package name, the automatic provides
won't re-add the same provide again. This also means you cannot have 2 different
(sub)packages provide the same name-version-release, but that shall be very very
very uncommon need and can always be workarounded somehow if needed.
5) No obsoletes with the new macro. Packagers use manual obsoletes when desired.
6) Document clearly that there is no %{?_isa} (and there is no "backwards
compatibility" load to carry). When absolutely desired, packagers can call the
macro with %{name}%{?_isa}.
7) Support arbitrary names. Only provide the given name and nothing else if not
"recognized".
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with
lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use
%python_provides (note the s). The only problem I see is that it is likely to be
mistaken for the old one, but IMHO it shouldn't really hurt that much.
Usage example:
%package -n python3-setuptools
%python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33
python-setuptools = 46.6.6-6.fc33
python39-pkg_resources = 46.6.6-6.fc33
python3-pkg_resources = 46.6.6-6.fc33
python-pkg_resources = 46.6.6-6.fc33
python39-pkg_resources = 46.6.6-6.fc33
Another example:
%package -n python3-pillow
%python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33
python-pillow = 7.1.1-1.fc33
python39-pillow = 7.1.1-1.fc33
python3-PIL = 1.1.6-100
python-PIL = 1.1.6-100
python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip
%python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33
python-pip = 20.0.2-1.fc33
python39-pip = 20.0.2-1.fc33
(all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 4 months
Macronize %pytest
by Miro Hrončok
Hello Python packagers,
since there is no upstream supported universal test invocation for Python
(`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't
always do what we want in RPM's %check and/or is not always used by upstreams)
and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for
standardizing the way pytest is run.
Usage:
%check
%pytest
Or with options passed to pyets:
%check
%pytest -v -m "not network" tests/
(Here ends the tl;dr version.)
Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`?
We want to *test the code we ship*. In the general case this involves:
1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch)
2. Prepending $PATH with %{buildroot}%{_bindir}
3. Ensuring $PWD is not in sys.path
So the idea is (untested):
%pytest_cmd /usr/bin/pytest
%pytest %{expand:
PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"
\
PATH="%{buildroot}%{_bindir}:$PATH" \
%pytest_cmd
}
Where PYTHONPATH solves (1), PATH solves (2) and using `/usr/bin/pytest` over
`%{python3} -m pytest` kinda solves (3).
I say "kinda" because using `python -m pytest` explicitly adds $PWD to sys.path,
but using `pytest` doesn't always prevent it.
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-pytho...
I've been trying to completely solve (3) in my head in a general way, but it
always involves `cd`ing to a temporary directory before running the tests, but
plenty of test suites cannot handle that gracefully.
Unfortunately Python interpreter doesn't have a flag that says "don't add
current directory to sys.path, but respect $PYTHONPATH". In the long term, we
might propose to add it.
Where to have the macro defined? (Skip this part if not interested.)
I was thinking:
a) pytest package
b) pytest subpackage (required if rpm-build)
c) python3-rpm-macros
d) python-rpm-macros
e) pyproject-rpm-macros
And here is the pros and cons:
a+) self contained, can change with upstream if needed
a+) always present with pytest
a-) pytest must co-own macro dir or depend on rpm-build
b+) same as a+, but not a-
b-) additional layer of complexity
c+) easy
c+) the macro doesn't need to require pytest, can be used with arbitrary command
c+) can be generalized to non-pytest later if desired
c-) when pytest CLI changes significantly, we need to change it in different
package (but we are not using pytets CLI now at all, just Python/Shell
environment variables)
c~) technically not always present with pytest, but always present with
python3-devel
d+-) same as (c)
d-) the macro uses python3 specific things
e+) the new fancy macros are there
e-) the other macros from there (actually pyproject related) are not required
for this
e-) not always present with pytest nor python3-devel
Hence, I'd go with (c) python3-rpm-macros.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 5 months
Rawhide python-rpm-generators news: small speedup + %python_provide not needed in most cases
by Miro Hrončok
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in
Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to
older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by
executing a Shell script. This is a bit faster especially for packages with a
lot of files. For 10 000 files in one package, the speedup is roughly from ~80
to ~2 seconds (time of the entire package build incl. creating the files from a
Python script). That is a lot, but most of the real packages have much less
files, so I am afraid you won't feel it.
More importantly, %python_provide is not needed for package names.
Until now, we needed to this:
%package -n python3-foo
...
%{?python_provide:%python_provide python3-foo}
This adds automatic provides "python-foo" (and since recently, also
"python38-foo" for forwards/backwards compatibility with EL). This now happens
automagically. Any package called "python3-..." will have those provides added
implicitly by the Python generators when they are in the buildroot
(python3-devel brings those in, so for Python packages this means always).
This automation was possible due to the speedup mentioned above, doing it the
pre-RPM.16 way would be very slow, because the generator is called for every
packaged file (which is still the case now, but it can now be a Lua macro). The
~2 seconds from above include this new feature.
There are 2 cases where manual %python_provide calls are still needed:
1. Metapackages without files
No files => no dependency generator. (I recommend to %ghost the
dist-info/egg-info directory in such packages to workaround this -- we plan to
add more good stuff regarding Python [extras] that will require this anyway.)
2. Virtual provides
The dependency generator can see the (sub)package's name, but not any other
virtual provides. When you do:
%package -n python3-%{pypi_name}
Provides: python3-%{legacy_name} = %{version}-%{release}
You are still supposed to add:
%{?python_provide:%python_provide python3-%{legacy_name}}
(Also note that %python_provide adds obsoletes for older python-foo versions,
that was useful when we renamed everything from python-foo to python2-foo and
when we changed the "default" from python2- to python3-. We are keeping the
Obsoletes in the macro (for now), but I have decided to not automatically
generate the Obsoletes based on the package name. A) I don't consider them
really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T
sound very intriguing to me. This is a decision that may be revisited later if
it's bringing unforeseen trouble.)
You don't need to to actively remove the macro from your spec files, it does no
harm. If you prefer to maintain a single spec, keep it there until Fedora 32
goes EOL (and EPEL 8 if that's your target). The guidelines always recommended
using it the safe way (%{?python_provide:...}), so even if it ceases to exist in
the future, it can remain in specs. There is no plan to remove it in any near or
distant future, as it is still needed for the virtual provides. The generators
also use it internally (for DRY and consistent results).
If you'll get into trouble because of this, let us know and we'll fix it.
I'll make a note for myself to update the Python packaging guidelines.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 5 months
"fedora" namespace in redhat-rpm-config Lua files?
by Miro Hrončok
Hello, I've just opened this bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1829430
There are "fedora" namespaced files in redhat-rpm-config:
$ repoquery --repo=rawhide -l redhat-rpm-config | grep fedora
/usr/lib/rpm/lua/fedora
/usr/lib/rpm/lua/fedora/common.lua
/usr/lib/rpm/lua/fedora/rpm
/usr/lib/rpm/lua/fedora/srpm
/usr/lib/rpm/lua/fedora/srpm/forge.lua
/usr/lib/rpm/macros.d/macros.fedora-misc
/usr/lib/rpm/macros.d/macros.fedora-misc-srpm
I find it very confuisng, since the package is called "redhat" and other files
in /usr/lib/rpm are namespaced as such:
$ repoquery --repo=rawhide -l redhat-rpm-config | grep redhat
/usr/lib/rpm/redhat
/usr/lib/rpm/redhat/brp-ldconfig
/usr/lib/rpm/redhat/brp-mangle-shebangs
/usr/lib/rpm/redhat/brp-python-bytecompile
/usr/lib/rpm/redhat/brp-strip-lto
/usr/lib/rpm/redhat/config.guess
/usr/lib/rpm/redhat/config.sub
/usr/lib/rpm/redhat/dist.sh
/usr/lib/rpm/redhat/find-provides
/usr/lib/rpm/redhat/find-requires
/usr/lib/rpm/redhat/gpgverify
/usr/lib/rpm/redhat/macros
/usr/lib/rpm/redhat/redhat-annobin-cc1
/usr/lib/rpm/redhat/redhat-hardened-cc1
/usr/lib/rpm/redhat/redhat-hardened-ld
/usr/lib/rpm/redhat/rpmrc
Should we aim for a common namespace? The redhat-rpm-package will eventually be
merged into RHEL and "fedora" namespace in it will be even weirder there.
We found this when trying to add our own Lua files and trying to figure out
where to put them.
------
Sharing here for further discussion and awareness.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 5 months
Using macro definitions inside %changelog results in "error: no description in %changelog"
by Pooya Daravi
Hi I am trying to dynamically create my RPM files. So I am passing defines to rpmbuild:
```
rpmbuild --ba --define "_VERSION `<command_to_generate_version>`" --define "_CHANGELOG `cat <path_to_changelog>`" <path_to_spec_file>
```
I am able to dynamically use ${_VERSION} in the .spec file, but for changelog:
```
%changelog
%{_CHANGELOG}
```
Is not working. When running the above `rpmbuild` command I am getting this error: "error: no description in %changelog"
Is it not possible to use macro defines inside the %changelog section? If so is there any other way I could dynamically set the changelog and not have to generate a new spec file for each release?
Thank you,
Puya
3 years, 5 months
%optflags macro for clang?
by Jun Aruga
I am trying to run the unit tests for an application (that is only
header files) in the%check section of the RPM spec file, running cmake
on both gcc and clang with %optflags macro on Fedora rawhide.
According to the guideline
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_compiler_flags
, it seems %optflags macro works with both gcc and clang without the
restriction for a specific compiler.
This cmake command for gcc works.
```
CC=gcc CXX=g++ cmake .. \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_C_FLAGS="%{optflags}" \
-DCMAKE_CXX_FLAGS="%{optflags}"
%make_build
```
But this cmake command for clang does not work with the error
"clang-10: error: unknown argument: '-fstack-clash-protection'". The
used clang version is 10.0.0.
Should %{optflags} work for both gcc and clang? Is it a bug of the
%optflags macro?
```
CC=clang CXX=clang++ cmake .. \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_C_FLAGS="%{optflags}" \
-DCMAKE_CXX_FLAGS="%{optflags}"
%make_build
```
The log is here.
```
+ CC=clang
+ CXX=clang++
+ cmake .. -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON '-DCMAKE_C_FLAGS=-O2 -g
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
'-DCMAKE_CXX_FLAGS=-O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
-- The C compiler identification is Clang 10.0.0
-- The CXX compiler identification is Clang 10.0.0
-- Check for working C compiler: /usr/bin/clang
-- Check for working C compiler: /usr/bin/clang - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:60 (message):
The C compiler
"/usr/bin/clang"
is not able to compile a simple test program.
It fails with the following output:
Change Dir:
/builddir/build/BUILD/simde-0d632e1e0bd0a258ce3d3e240fbec4dd71234819/test/build-clang/CMakeFiles/CMakeTmp
----
Run Build Command(s):/usr/bin/gmake cmTC_686bc/fast &&
/usr/bin/gmake -f CMakeFiles/cmTC_686bc.dir/build.make
CMakeFiles/cmTC_686bc.dir/build
gmake[1]: Entering directory
'/builddir/build/BUILD/simde-0d632e1e0bd0a258ce3d3e240fbec4dd71234819/test/build-clang/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_686bc.dir/testCCompiler.c.o
/usr/bin/clang -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-o CMakeFiles/cmTC_686bc.dir/testCCompiler.c.o -c
/builddir/build/BUILD/simde-0d632e1e0bd0a258ce3d3e240fbec4dd71234819/test/build-clang/CMakeFiles/CMakeTmp/testCCompiler.c
clang-10: error: unknown argument: '-fstack-clash-protection'
gmake[1]: *** [CMakeFiles/cmTC_686bc.dir/build.make:83:
CMakeFiles/cmTC_686bc.dir/testCCompiler.c.o] Error 1
gmake[1]: Leaving directory
'/builddir/build/BUILD/simde-0d632e1e0bd0a258ce3d3e240fbec4dd71234819/test/build-clang/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:138: cmTC_686bc/fast] Error 2
```
Thanks.
--
Jun | He - His - Him
3 years, 5 months
Summary/Minutes from today's FPC Meeting (2020-04-09 16:00 - 17:10 UTC)
by James Antill
======================
#fedora-meeting-1: fpc
======================
Meeting started by geppetto at 16:00:34 UTC. The full logs are available
at
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-04-09/fpc.2020-04...
.
Meeting summary
---------------
* Roll Call (geppetto, 16:00:35)
* Schedule (geppetto, 16:09:29)
* LINK:
https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje...
(geppetto, 16:09:32)
* LINK:
https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje...
(geppetto, 16:10:36)
* #958 Update of the Haskell Packaging Guidelines (geppetto, 16:10:59)
* ACTION: Update of the Haskell Packaging Guidelines (+1:5, 0:0, -1:0)
(geppetto, 16:39:56)
* #959 Is it ok to have different version based on %{fedora}?
(geppetto, 16:40:34)
* Nobody likes it, and if it is allowed nobody wants it to be.
(geppetto, 17:00:08)
* Open Floor (geppetto, 17:00:14)
* ACTION: 966 request to use /usr/libexec/swift seems fine to everyone
(geppetto, 17:05:37)
Meeting ended at 17:10:04 UTC.
Action Items
------------
* Update of the Haskell Packaging Guidelines (+1:5, 0:0, -1:0)
* 966 request to use /usr/libexec/swift seems fine to everyone
Action Items, by person
-----------------------
* **UNASSIGNED**
* Update of the Haskell Packaging Guidelines (+1:5, 0:0, -1:0)
* 966 request to use /usr/libexec/swift seems fine to everyone
People Present (lines said)
---------------------------
* mhroncok (62)
* geppetto (50)
* tibbs (46)
* decathorpe (38)
* zodbot (13)
* limburgher (13)
* nim (10)
* tachoknight (6)
Generated by `MeetBot`_ 0.1.4
.. _`MeetBot`: http://wiki.debian.org/MeetBot
3 years, 5 months