Fedora 32 Self-Contained Change proposal: Track Changes in Taiga
by Ben Cotton
https://fedoraproject.org/wiki/Changes/fedora-change-wrangler
= Track Changes in Taiga =
The Motivation for this proposal is to propose using the Taiga
instance at teams.fedoraproject.org for Change processing.
[[User:Bcotton/UsePagureForChanges|Using Pagure]] was previously
proposed for this. The new Change Processing workflow aims to simplify
the process to improve visibility and ease of management.
== Summary ==
The current process allows contributors to propose changes in upcoming
Fedora releases. However, the management of those feature proposals is
cumbersome and requires several manual steps. This introduces more
opportunity for error and reduces the time available to the Change
Wrangler for more valuable contributions to Fedora. In particular, the
current process includes artifacts and discussion in the Fedora Wiki,
on mailing lists, FESCo tickets, Bugzilla tickets, and Docs tickets.
The current process is cumbersome with several manual steps,this
introduces more opportunity for error and reduces the time available
for the Change Wrangler.
A Cli Tool Written in Python that interacts with Taiga,Pagure and
Bugzilla.Functionality includes
Promoting,announcing,submitting,accepting and updating the changes
across the three platforms and over mailing list.
The new process will consolidate much of the information in
Taiga.Proposed Changes will be submitted as an issue in Taiga. The
description of the Issue will include the content with few exceptions:
* System-Wide or Self-Contained change will be indicated by a binary
in the issue metadata
* Fedora release will be indicated by a milestone in the issue metadata
* Change status will be indicated by a list selection in the issue metadata
== Owner ==
* Name: [[User:Pac23| Manas Mangaonkar]]
* Email: manasmangaonkar(a)gmail.com || pac23(a)fedoraproject.org
* Name: [[User:Bcotton | Ben Cotton]]
* Email: bcotton(a)redhat.com
== Detailed Description ==
As Part of GSoC 2019 the fedora-change-wrangler tool will be developed
to smooth out the workflow,reduce Manual Work involved for both the
Contributors and the Change-Wrangler(FPgm). The tool makes it easy by
covering all the functionality required for the process of moving
forward proposals.
The tool will be developed using Python 3.6+ and will interact with
Taiga/Pagure/Bugzilla via their API and the Mailing List via SMTP.
The General Workflow would be :
1. Change Owner opens an issue and fills in the fields. When they are
ready to submit the Change proposal, they set the status to "Ready
for Wrangler".
2. The Change Wrangler (FPgM) reviews the proposal. If it is
incomplete, they set the status back to "New" and inform the Change
Owner of what's needed. If it is ready to process, then...
Functionality covered
1. Promote
* The issue will be promoted to a user story in taiga,copies the
contents of the custom fields from the issue to the user story. Closes
the issue.
2. Announce
* The tool would have the functionality to enable announcing the
change proposal to devel and devel-announce lists using smtp. It will
also automatically change the story status to announced.
3. fesco-submit
* Allowing creation of a new issue in the FESco Pagure repo directly
from the command line.
4. Accept
* Once the changes are accepted the change-wrangler can create a
tracking bug in Bugzilla along with release notes on Pagure.THe status
of the user story is updated to accepted aswell.
5. Update
* Status can be changed to testable if BZ is "Modified" and to
Complete is BZ is "ON_QA"
6. Creation of Report
* The user can create a report to provide quick view of changes and
their status. It can be in wiki or Html form.
Techstack:
* Python3.6+
* SMTP
* Taiga/Pagure/Bugzilla API's
* Nano/Gedit/Vi ( Inbuilt support to edit text)
* Kerberos(For authentication)
The current sample arg created looks like this [ change-tool convert
--taiga <issue no> <issue no> ] . The advantage of this the Manager
would be able to specify unlimited no of issues to change at once in a
single command using the issue no in taiga to convert to user story.
== Benefit to Fedora ==
The proposed change will make the change-process easier for
everyone.Everyone would be able to see them in one place with
status,id filters. The current wiki process can be bit difficult for
formatting,having defined fields would mean easy access without the
cumbersome wiki edits.
Since changes would be submitted in Issue format on
Taiga,pre-submission discussions would be available thus getting
suggestions/feedback at the first stage itself would be easy for
everyone involved.
== Scope ==
* Proposal owners:
I will be working with the Change-Wrangler(Ben Cotton) throughout the
summer to create this tool from scratch.
* Other developers: N/A (not a System Wide Change)
* Release engineering:(no release engineering impact is expected)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
N/A (not a System Wide Change)
== How To Test ==
N/A (not a System Wide Change)
== User Experience ==
No visible Impact is expected.
== Dependencies ==
No other packages depend on this.
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) N/A (not a
System Wide Change)
Enough buffer time has been allocated to complete this during the GSoC period.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
4 years, 7 months
Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
= Switch RPMs to zstd compression =
== Summary ==
Binary RPMs are currently compressed with xz level 2.
Switching to zstd would increase decompression speed significantly.
== Owner ==
* Name: [[User:dmach| Daniel Mach]]
* Email: dmach(a)redhat.com
== Detailed Description ==
* The change requires setting a new compression algorithm in rpm
macros. Then a mass rebuild of all packages is required.
* The macro for setting the compression is: %define _binary_payload w19.zstdio
* The recommended compression level is 19. The builds will take
longer, but the additional compression time is negligible in the total
build time and it pays off in better compression ratio than xz lvl2
has.
* SRPM payload compression should stay at gzip (there's almost no
benefit in changing the compression, because SRPM's contents is
compressed already)
=== Use case: Firefox installation ===
I rebuilt firefox-66.0.5-1.fc30 with zstd level19.
Then I compared installation times with the original (xz compressed) package:
{| class="wikitable"
|-
! Compression !! Target File System !! Time
|-
| xz level 2 || tmpfs || 8s
|-
| xz level 2 || ext4 on nvme || 11s
|-
| zstd level 19 || tmpfs || 2s
|-
| zstd level 19 || ext4 on nvme || 4s
|-
|}
=== Comparison of compression algorithms and levels ===
Following table shows '''cpio''' and '''compressed cpio''' extraction
times into a tmpfs. Actual times in decompressing RPMs will differ due
to extracting on an actual disk and also some overhead in the RPM tool
(checks, scriptlets).
{| class="wikitable"
|-
! Compression !! Level !! Size B !! Size GiB
!! Compression time !! Compression time, 4 threads !!
Decompression time !! Comment
|-
| CPIO || - || 5016785692 || 4,7
|| - || - || -
||
|-
| xz || 2 || 1615017616 || 1,6
|| 9m55s || - || 1m36s
|| slow decompression
|-
| pxz || 2 || 1631869880 || 1,6
|| - || 6m11s || 1m38s
|| slow decompression
|-
| gzip || 9 || 2086354992 || 2,0
|| 10m23s || - || 31s
|| insufficient compression ratio
|-
| bzip2 || 9 || 1889161565 || 1,8
|| 8m || - || 2m50s
|| very slow decompression; compression ratio could be
better
|-
| zstd || 3 || 1913536587 || 1,8
|| 31s || 29s || 6,5s
||
|-
| zstd || 10 || 1737928978 || 1,7
|| 3m27s || 2m34s || 6,3s
||
|-
| zstd || 15 || 1717303256 || 1,7
|| 9m37s || 6m34s || 6,3s
|| identical compression speed to xz; fast decompression;
slightly worse compression ratio than xz
|-
| zstd || 17 || 1635525492 || 1,6
|| 16m16s || 11m20s || 6,7s
||
|-
| zstd || 19 || 1575843696 || 1,5
|| 24m2s || 18m55s || 7,7s
||
|-
|}
== Benefit to Fedora ==
* Faster installations/upgrades of user systems
* Faster koji builds (installations in build roots)
* Faster container builds
* Lower bandwidth on mirrors if we choose the highest compression level
== Scope ==
* Proposal owners: submit a patch to redhat-rpm-config
* Other developers: redhat-rpm-config maintainer: include the patch
and make a new build
* Release engineering: [https://pagure.io/releng/issue/8345 #8345]
mass rebuild is needed
== Upgrade/compatibility impact ==
* RPM in Fedora supports zstd compression already (from Fedora 28,
rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected.
* Fedora <= 27 and some other distros will not be able to decompress
zstd-compressed RPMs.
== How To Test ==
* dnf install <package>
* rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
* expected output: zstd 19
Also the overall system installation time should decrease significantly.
== User Experience ==
See '''Benefit to Fedora'''
== Dependencies ==
N/A
== Contingency Plan ==
* Contingency mechanism: Not needed, Fedora will stay at current compression.
* Contingency deadline: N/A
* Blocks release? No
* Blocks product? N/A
== Documentation ==
N/A
== Release Notes ==
RPMs have switched to zstd compression level 19.
Users will benefit from faster package decompression.
Users that build their packages will experience slightly longer build times.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
4 years, 7 months
Updating Rawhide vs GPG keys
by Vít Ondruch
Hi,
Can somebody please enlighten me, how to update Rawhide after branching
and not using --nogpgcheck?
It seems that Rawhide keys were added in fedora-repos-30-0.4. So this is
the package which is still "rawhide" package and has "f31" keys. But
this package was not probably signed, because this directory is empty:
https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.4/data/signed/
Installing anything from Rawhide fails, because of missing GPG keys.
So is there a way to get the GPG keys via DNF? Would it be possible to
sign fedora-repos and fedora-release packages by older key to allow
smooth updates?
Vít
4 years, 8 months
bootctl: no entry could be determined as default (Was: Upgrade to F30
gone wrong)
by Dridi Boukelmoune
On Sun, May 5, 2019 at 1:45 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
<snip>
> This makes the assumption, which was also made earlier in the thread,
> that it's somehow impossible to check what bootloader is installed.
> Why? My bootloader is happy to tell me its version:
> $ bootctl
> ...
> Current Boot Loader:
> Product: systemd-boot 241-565-g43d51bb
> Features: ✓ Boot counting
> ✓ Menu timeout control
> ✓ One-shot menu timeout control
> ✓ Default entry control
> ✓ One-shot entry control
> File: /EFI/systemd/systemd-bootx64.efi
> ...
> Nowadays it's gives the exact git commit it's built from, in the past
> it was just the release version, but either is enough. Therefore
> 'bootctl update' can fairly reliably *update*, i.e. do the installation
> if the thing we have is newer than the version already installed.
That's news to me, and unfortunately it doesn't look as nifty on my system:
...
Current Boot Loader:
Product: n/a
Features: ✗ Boot counting
✗ Menu timeout control
✗ One-shot menu timeout control
✗ Default entry control
✗ One-shot entry control
ESP: n/a
File: └─n/a
Available Boot Loaders on ESP:
ESP: /boot/efi (/dev/disk/by-partuuid/$UUID)
File: └─/EFI/BOOT/BOOTIA32.EFI
File: └─/EFI/BOOT/BOOTX64.EFI
Boot Loaders Listed in EFI Variables:
Title: Fedora
ID: 0x0001
Status: active, boot-order
Partition: /dev/disk/by-partuuid/$UUID
File: └─/EFI/fedora/shimx64.efi
Title: Linux Firmware Updater
ID: 0x0000
Status: active, boot-order
Partition: /dev/disk/by-partuuid/$UUID
File: └─/EFI/fedora/shimx64.efi
...
Where $UUID is the same for all three occurrences.
Dridi
4 years, 8 months
Where are armv7hl, i686 and ppc64le Container tar.xz files?
by Jun Aruga
I am looking for arch container archive files like
"Fedora-Container-Base-30-1.2.aarch64.tar.xz
" for Fedora 30.
I found the x86_64, aarch64 and s390x's archive files in below directory.
But where is the archive file of armv7hl, i686 and ppc64le?
I assume the multi archs except x86_64, aarch64 and s390x existed in
the age of Fedora 24, 25 in below releases directory.
And why below s390x does not have Fedora-Container-Base-*.tar.xz?
https://dl.fedoraproject.org/pub/fedora/linux/releases/30/Container/
x86_64/
images/
Fedora-Container-Base-30-1.2.x86_64.tar.xz
Fedora-Container-Minimal-Base-30-1.2.x86_64.tar.xz
aarch64/
images/
Fedora-Container-Base-30-1.2.aarch64.tar.xz
Fedora-Container-Minimal-Base-30-1.2.aarch64.tar.xz
https://dl.fedoraproject.org/pub/fedora-secondary/releases/30/Container/
s390x/
images/
Fedora-Container-Minimal-Base-30-1.2.s390x.tar.xz
=> No Fedora-Container-Base-*.tar.xz
Thanks.
--
Jun Aruga / He - His - Him
jaruga(a)redhat.com / IRC: jaruga
4 years, 8 months
F31 Self-Contained Change proposal: Include several modules in the
EFI build of Grub2 for security use-cases
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Include_security_modules_in_efi_Grub2
== Summary ==
Include Grub's "verify," "cryptodisk" and "luks" modules (and if
necessary, relevant "gcry" modules) in grubx64.efi of the
'grub2-efi-x64' package.
== Owner ==
* Name: [[User:pjones| Peter Jones]]
* Email: pjones(a)redhat.com
* Name: [[User:javierm| Javier Martinez Canillas]]
* Email: fmartine(a)redhat.com
== Detailed Description ==
Users utilising secure boot functionality on the UEFI platform cannot
insert modules that aren't in grubx64.efi. Paradoxically, this means
that security-conscious users cannot use grub's verify module, or
employ (near) full disk encryption using cryptodisk and luks.
The security benefits of signature verification would reach more users
if Fedora automated it by incorporating the process into
grub2-mkconfig.
For the long-term, to grant flexibility with grub2 modules on secure
boot instances, it may be advisable to sign the .mod files in the
'grub2-efi-x64-modules' package, modify grub2-mkconfig (or -install)
to copy the necessary modules into the EFI partition when required by
the user's configuration and then allow inserting of signed modules in
secure boot instances.
== Benefit to Fedora ==
This change will allow users to gain trust in the integrity of
early-launch code either through verification of signatures
(particularly useful for initramfs, which is particularly vulnerable
to possible offline modification) or encryption of the boot partition.
== Scope ==
* Proposal owners: Modify grub.macros file to include the
above-mentioned modules in the GRUB_MODULES variable.
* Other developers: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
Change only adds modules, so existing users should have no problems.
== How To Test ==
<b>For "verify":</b>
1. Generate a signing key with "gpg --gen-key" and copy it to the EFI partition
2. Add "trust <gpg key>" (but grub may inherit this from shim's MOK)
and "set check_signatures=enforce" to /etc/default/40_custom
3. Run grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
4. Create a file, /tmp/pass, with the key's passphrase, then execute:
for x in $(find /boot -name "*.cfg" -or -name "*.mod" -or -name
"vmlinuz*" -or -name "initramfs*" -or -name "grubenv"); do gpg --batch
--detach-sign --passphrase-fd 0 $x < /tmp/pass; done. Then, shred
/tmp/pass
5. Reboot. If system starts, change is successful
<b>For cryptography modules:</b>
1. Backup boot partition
2. Run cryptsetup luksFormat <boot partition's block device, for
example, /dev/sda2> --type luks1
3. Open luks container and restore backup
4. Add GRUB_ENABLE_CRYPTODISK=y to /etc/default/grub
5. Confirm that /etc/fstab has the correct UUID for /boot
6. Run grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
7. Reboot. Grub should ask for the password created in step 2. If
system then starts, change is successful
(If filesystem root is also encrypted, user may be asked for a
password twice. This can be mitigated with a keyfile for filesystem
root, or use of the clevis package, and likely, a tpm.)
== User Experience ==
Users may optionally elect to verify the integrity of boot code either
through verification of digital signatures or encryption of the boot
partition.
== Dependencies ==
Grub2-efi-x64-modules and grub2-tools-* depend on this package, but
require no change.
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) Revert the
shipped configuration
* Contingency deadline: Beta freeze
* Blocks release? N/A (not a System Wide Change)
* Blocks product? No
== Documentation ==
https://www.gnu.org/software/grub/manual/grub/html_node/Using-digital-sig...
https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system...
== Release Notes ==
Fedora now supports Grub's detached verify and cryptodisk
functionality natively, even on secure boot systems.
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
Pronouns: he/him
4 years, 9 months
rpmlint: library-not-linked-against-libc (parts of Python stdlib,
after gcc 9)
by Miro Hrončok
I've just spotted these when working on Python 3.8.0a1. This happens on 3.7 as
well since GCC 9:
python3-debug.x86_64: E: library-not-linked-against-libc
/usr/lib64/python3.7/lib-dynload/_contextvars.cpython-37dm-x86_64-linux-gnu.so
python3-debug.x86_64: E: library-not-linked-against-libc
/usr/lib64/python3.7/lib-dynload/_testimportmultiple.cpython-37dm-x86_64-linux-gnu.so
python3-libs.x86_64: E: library-not-linked-against-libc
/usr/lib64/python3.7/lib-dynload/_contextvars.cpython-37m-x86_64-linux-gnu.so
python3-test.x86_64: E: library-not-linked-against-libc
/usr/lib64/python3.7/lib-dynload/_testimportmultiple.cpython-37m-x86_64-linux-gnu.so
(Note that there are plenty of other extension modules that do not raise this
error.)
This doesn't happen with latest python3 built prior to the gcc update to 9.
$ rpmlint -I library-not-linked-against-libc
library-not-linked-against-libc:
That isn't helpful either.
I found a similar Debian thing [1] that says:
> It is theoretically possible to have a library which doesn't use any symbols
> from libc...
Do I care? Should I fix something? I honestly have no idea.
[1] https://lintian.debian.org/tags/library-not-linked-against-libc.html
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 9 months
Self Introduction: Gordon Messmer
by Gordon Messmer
Hello,
My name is Gordon Messmer. I've been a long-time user of Red Hat
systems, starting with Red Hat Linux 4.2 in 1997. I've been packaging
software in rpm format for use in business environments for very nearly
as long. I've also been a semi-active member of the Red Hat and Fedora
mailing lists most of those years.
My first contribution will be ansible-bender: a tool that builds
container images using ansible playbooks.
https://bugzilla.redhat.com/show_bug.cgi?id=1714377
4 years, 9 months