LTO and F33
by Jeff Law
So we're at a point where the F33 FTBFS issues related to LTO that I'm aware of
have been resolved (by opting the package out of LTO). I still expect some LTO
issues will pop up as packages fix things like missing dependencies, cmake
macros, etc. I continue to be available to investigate potential LTO issues, but
package maintainers will need to contact me as I'm not actively looking for new
LTO issues.
My focus is now turning to the packages with LTO opt-outs. I'll be extracting
bug reports for upstream (primarily GCC), trying simple workarounds for old style
symbol versioning, identifying backports from upstream GCC that allow us to
remove LTO opt-outs and the like. So there should be a trickle of opt-outs
removed, but otherwise should largely be invisible to the F33 release process.
I'd like to thank everyone involved for being patient while we worked through
various issues and I look forward to continuing to make incremental improvements
now that the bulk of the LTO work has landed.
jeff
3 years, 6 months
F34 Change: Reduce installation media size by improving the
compression ratio of SquashFS filesystem (Self-Contained Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/OptimizeSquashFS
== Summary ==
Improve compression ratio of SquashFS filesystem on the installation media.
== Owner ==
* Name: [[User:bkhomuts|Bohdan Khomutskyi]]
* Email: bkhomuts(a)redhat.com
== Detailed Description ==
As of Fedora 31, the LiveOS/squashfs.img file on the installation
image, is compressed with default block size of mksquashfs. The
default block size is 128k. Additionaly, lorax sets BCJ filter
depending on the architecture. Those parameters can be adjusted which
will lead to a better compression ratio and/or reduction of the CPU
usage at build time.
This is simple to achieve. Recently, Lorax has gotten support for
adjusting the compression options for mksquashfs via the configuration
file. The file should be altered as following:
<pre>
[compression]
bcj = no
args = -b 1M -Xdict-size 1M -no-recovery
</pre>
Where -b 1M and -Xdict-size 1M are block and dictionary sizes
respectively; bcj -- branch-call-jump filter.
Based on the results above, I'd suggest selecting the following
''optimal configuration'': XZ algorithm, with block size of 1MiB and
without BCJ filter (plain xz -b 1M, without -Xbcj x86).
On the right, you can see the impact of the compression algorithms on
installation time.
As can be seen from the picture on the right hand side, selecting
'plain xz -b 1M configuration' has minimal impact on the installation
time and CPU usage during the installation. The compression will
result in +6.51% or, in real terms, +24.94s additional installation
time, compared to the savings of 142 MiB on the installation media,
refer to the documentation section to download the ISO images. This
increase in installation time will be compensated by the change in the
installer: https://github.com/rhinstaller/anaconda/pull/2292
I noticed, that even with maximum compression, CPU is not fully
utilized during installation.
== Benefit to Fedora ==
* Reduction of the installation media size and the cost of storing and
distributing Fedora.
* Reduction of the CPU usage at build time. Depending on which
compression parameters chosen.
== Scope ==
* Proposal owners:
The build environment should have support for adjusting the Lorax
configuration file and -squashfs-only parameter. Lorax is a program
that produces the LiveOS/squashfs.img file on the installation media.
One of the ways to enable such customization is to introduce support
in Pungi to pass -c option to Lorax.
* Other developers:
The pungi utility should support passing the custom configuration file
location to the Lorax utility. This option should apply during
buildInstall phase of pungi.
* Release engineering: [https://pagure.io/releng/issue/9127]
<!-- Does this feature require coordination with release engineering
(e.g. changes to installer image generation or update package
delivery)? Is a mass rebuild required? include a link to the releng
issue.
The issue is required to be filed prior to feature submission, to
ensure that someone is on board to do any process development work and
testing, and that all changes make it into the pipeline; a bullet
point in a change is not sufficient communication -->
* Policies and guidelines: Not required.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
N/A (not a System Wide Change)
== How To Test ==
<pre>
mkdir -p /mnt/new /mnt/old
sudo mount -o loop,ro FedoraInstallationOld.iso
sudo mount -o loop,ro FedoraInstallationNew.iso
ls -l /mnt/{new,old}/LiveOS/squashfs.img
</pre>
And then calculate the size difference.
== User Experience ==
* Decreasing the installation image size will reduce cost of mirroring
and storing Fedora installation images.
* Decreasing the installation image size will reduce the download time.
* Increasing the block size on the current configuration with EXT4
file system, should increase latency while accessing the EXT4
filesystem. The exact impact is to be evaluated.
* The impact of latency will be reduced, if the plain SquashFS option
is be choosen.
== Dependencies ==
Pungi, a utility that builds the compose, should include new
functionality mentioned above.
Alternatively, the /etc/lorax/lorax.conf should be altered in the
environment where Lorax is running.
== Contingency Plan ==
N/A
== Documentation ==
https://pagure.io/releng/issue/9127.<br/>
mksquashfs(1)<br/>
lorax(1)<br/>
https://docs.pagure.org/pungi<br/>
Select ISOs can be downloaded at https://khomutsky.com/fedora-dvd/<br/>
Spreadsheet pictured above [[File:Comparison Table SquashFS.ods]]
[https://github.com/rhinstaller/anaconda/pull/2292 Multi-core
decompression of SquashFS merge request]
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 6 months
splitting out systemd-networkd,
systemd-standalone-{sysusers,tmpfiles} subpackages in F33+
by Zbigniew Jędrzejewski-Szmek
Hi,
the systemd package is getting a systemd-networkd subpackage split out
that will contain systemd-networkd, networkctl, and the associated data files.
This was requested by coreos maintainers: NetworkManager is used and skipping
systemd-networkd allows the installation footprint and potential user confusion
to be reduced a bit. (By 1.6 MB and an unknown amount, respectively.)
Appropriate Obsoletes are added on both the main package and the new
systemd-networkd subpackage, so the systemd-networkd subpackage should
be installed on upgrades. In addition, the new subpackage has Recommends from
the main package, so it will be installed in normal installations. The split
affects installations with --setopt=install_weak_deps=False. Please make sure
to pull in systemd-networkd.rpm independently if needed. Also note that
systemd-networkd.service was preset as *disabled* in Fedora, which means that
unless it was enabled by the user, the removal of systemd-networkd wouldn't
have an effect.
In addition, two more new subpackages are created: systemd-standalone-sysusers
and systemd-standalone-tmpfiles, with custom-linked systemd-sysusers and
systemd-tmpfiles binaries. They packages are 170kB and 260kB and pull in much
less dependencies compared to the normal systemd package (only glibc, libselinux,
and libacl). The goal here is to be able to use those packages in limited
environments where systemd itself is not necessary.
The main systemd systemd package Obsoletes the -standalone- packages, so it
should smoothly replace them whenever it is pulled in.
This change was done in systemd-246.6-2.fc34 in rawhide right now. (There are
some cleanups to move more files to the -networkd subpackage in the works for
246.6-3.fc34). Please give this a spin and report any issues.
The plan is to also do this split for F33 if no issues are noted in rawhide.
Zbyszek
3 years, 6 months
Packages in rawhide not showing in src.fedoraproject.org
by José Abílio Matos
Hi,
I am sorry if this has been discussed before but I do not remember it.
The culprit to this message was the review of R-pak
https://bugzilla.redhat.com/show_bug.cgi?id=1883047
While running fedora-review that uses mock I found that I had an issue
installing the corresponding rpm in rawhide. And surely enough running the
same command it failed:
# /usr/bin/dnf --installroot /var/lib/mock/fedora-rawhide-x86_64/root/ --
releasever 34 --setopt=deltarpm=False --allowerasing --disableplugin=local --
disableplugin=spacewalk install /home/jamatos/tmp/1883047-R-pak/results/R-
pak-0.1.2-1.fc34.noarch.rpm --setopt=tsflags=nocontexts
No matches found for the following disable plugin patterns: local, spacewalk
fedora
4.4 MB/s | 74 MB 00:16
Last metadata expiration check: 0:00:17 ago on Wed 30 Sep 2020 10:21:53 AM
WEST.
Error:
Problem: conflicting requests
- nothing provides R(rprojroot) >= 1.3.2 needed by R-pak-0.1.2-1.fc34.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to
use not only best candidate packages)
Consulting the page for R-rprojroot
https://src.fedoraproject.org/rpms/R-rprojroot
we also do not see any build for Fedora 34. I already saw this also for other
packages where it only shows after a rebuild.
On the other hand asking directly to rawhide we get:
# dnf -q list R-rprojroot --disablerepo=\* --enablerepo=rawhide
Available Packages
R-rprojroot.noarch 1.3.2-9.fc33 rawhide
And so in fact there is a match and so the packages is available at rawhide.
My doubt and the reason to post this is to ask if this is the same issue or if
it corresponds to different issues, one for src.fedoraproject.org (I always
forget the backend name) and another for mock.
Best regards,
PS: no linkers or debug formats were involved in the compose of this message.
:-)
--
José Abílio
3 years, 6 months
F34 Change proposal: Debug Info LLDB Index (System-Wide change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/DebugInfoLldbIndex
== Summary ==
Provide .debug_names debug info index for LLDB for clang-built
binaries using: clang -gdwarf-5 -gpubnames
Debuginfo index significantly accelerates loading of *.debug files by
debugger. Fedora currently provides ELF section .gdb_index for
[https://www.gnu.org/software/gdb/ GDB debugger].
[https://lldb.llvm.org/ LLDB debugger] cannot use .gdb_index (as it is
missing DIE offsets for more effective processing by LLDB) but LLDB
can use .debug_names index.
== Owner ==
* Name: [[User:jankratochvil| Jan Kratochvil ]]
* Email: jan.kratochvil(a)redhat.com
== Detailed Description ==
There are currently 3 formats of debug info index:
* .gdb_index: It is currently produced in Fedora by GDB
(/usr/bin/gdb-add-index), it is a part of rpmbuild process. It is
compatible with GDB but incompatible with LLDB as it is missing
essential DIE offsets needed by LLDB due to more effective (faster)
reading of DWARF by LLDB.
* .debug_names from GDB (augmentation "GDB\x00"): It can be produced
by GDB (/usr/bin/gdb-add-index -dwarf-5) but its format is
non-conforming to [http://www.dwarfstd.org/doc/DWARF5.pdf DWARF-5
standard]. LLDB expects DWARF-5 standard compliant .debug_names and
therefore it is incompatible with this format. It can be expected GDB
will fix the conformance in the future. Currently GDB .debug_names
format has no advantage over GDB .gdb_index format.
* .debug_names from clang (augmentation "LLVM0700"): It can be
produced by clang (clang -gdwarf-5 -gpubnames) for LLDB. It is
conforming to [http://www.dwarfstd.org/doc/DWARF5.pdf DWARF-5
standard], one can expect GDB will be able to read it in the future.
It would be good to produce index from GCC by GDB and to produce index
from clang by clang as the compatibility inside the same toolchain is
best tested and supported. Using index across toolchains (index from
GDB by LLDB or index from clang by GDB) should theoretically work but
in practice there exist subtle differences in interpretation of more
complicated DWARF constructs. It would be best to fix those but that
will be always an afterthought.
== Benefit to Fedora ==
* Faster startup of LLDB debugger using Fedora system *.debug files.
== Scope ==
* Proposal owners: It affects all clang-built packages generating
*-debuginfo.rpm.
* Other developers: none
* Policies and guidelines: All the needed changes should be done in
[https://src.fedoraproject.org/rpms/redhat-rpm-config
redhat-rpm-config]. The [https://src.fedoraproject.org/rpms/dwz dwz
package] can be then retired.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: The size differences are only for
*-debuginfo.rpm which is outside of scope of the listed objectives.
Currently the change will affect only packages using:
%global toolchain clang
Those are currently only these packages being built by clang and using
this %toolchain framework: dotnet3.1 libcxxabi mtxclient nheko simde
wine
FIXME: Which other Fedora packages are being built by clang?
== Upgrade/compatibility impact ==
Existing tools not supporting .debug_names will just ignore the
additional ELF section. The only issue is current GDB would get
confused by the clang .debug_names as it expects .debug_names to be in
its incompatible GDB format - FIXME: Provide a GDB bugfix patch.
Also each *-debuginfo.rpm has to exactly match NVRA of its binary
package the Fedora change compatibility is not applicable.
== How To Test ==
GDB should not get affected by the new .debug_names index from clang.
LLDB should load Fedora system *.debug files faster. LLDB
functionality should not be affected by the index from clang (that is
a part of LLVM development/testsuite).
"llvm-dwarfdump -debug-names *.debug" should show: Augmentation: 'LLVM0700'
== User Experience ==
No user visible change. This affects what tools can developers use.
== Dependencies ==
This Change is dependent on how is decided [[Changes/DebugInfoStandardization]].
This Change is dependent on RHEL-5 F-34 feature expected to be filed
by Mark Wielaard.
Mass rebuild is not required. Packages inherited from F-33 will just
miss the LLDB index and LLDB will load them more slower.
* .debug_names would need to be updated by DWZ but DWZ does not plan
to support .debug_names (according to Mark Wielaard).
** If DWZ is dropped ([[Changes/DebugInfoStandardization]] gets
approved) then clang can normally produce .debug_names for LLDB.
** If DWZ stays in use ([[Changes/DebugInfoStandardization]] gets
rejected) then there are multiple options. LLDB currently cannot
produce .debug_names (only clang can). GDB currently produces
incompatible .debug_names format.
*** [[Changes/DebugInfoStandardization]] should be applied at least
for clang-built packages, preferred by this proposal.
*** There was an idea DWZ would remove .debug_names. That would
effectively reject this Change and make LLVM Toolchain slower due to a
deficiency of the DWZ tool.
*** There isn't anyone willing to implement updating of .debug_names
into DWZ, moreover DWARF-5 standard does not specify a needed format
for combination of both DWZ and .debug_names. Also LLDB currently does
not support the DWZ format anyway.
*** When GDB produces .debug_names compatible with DWARF-5 in the
future it could theoretically produce .debug_names for LLDB. But that
would mean the index for LLDB (and therefore interpretation of DWARF
by LLDB) would be affected by interpretation of DWARF by GDB, that is
a toolchain compatibility nightmare.
* clang produces more effective .debug_names (one per executable; not
one per *.o compilation unit) when using -flto which is
[[LTOByDefault|default since F-33]] now.
== Contingency Plan ==
* Contingency mechanism: Revert the change in
[https://src.fedoraproject.org/rpms/redhat-rpm-config
redhat-rpm-config]. LLDB can continue loading Fedora system *.debug
files slightly slower.
* Contingency deadline: beta freeze
* Blocks release? No
* Blocks product? N/A
== Documentation ==
* [http://www.dwarfstd.org/doc/DWARF5.pdf DWARF-5] 6.1.1 Lookup by Name
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 6 months
Schedule for Wednesday's FESCo Meeting (2020-09-30)
by Miro Hrončok
Following is the list of topics that will be discussed in the
FESCo meeting Wednesday at 14:00UTC in #fedora-meeting-2 on
irc.freenode.net.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/UTCHowto
or run:
date -d '2020-09-30 14:00 UTC'
Links to all issues to be discussed can be found at:
https://pagure.io/fesco/report/meeting_agenda
= Discussed and Voted in the Ticket =
Formalize updated policies for what spins can change without asking (and what
can be changed with FESCo clearance)
https://pagure.io/fesco/issue/2418
APPROVED (+6, 0, -0)
F34 System-Wide Change: Remove support for SELinux runtime disable
https://pagure.io/fesco/issue/2471
APPROVED (+7, 1, -0)
F34 System-Wide Change: Wayland by Default for KDE Plasma Desktop
https://pagure.io/fesco/issue/2472
APPROVED (+4, 1, -0)
= New business =
#topic #2474 F34 System-Wide Change: Rust Crate Packages For Release Branches
https://pagure.io/fesco/issue/2474
#topic #2476 systemd-resolved by default should be deferred to a future release
https://pagure.io/fesco/issue/2476
#topic #2477 Fedora 33 schedule: Is it correct there is only one week between
Beta and Final Freeze?
https://pagure.io/fesco/issue/2477
= Open Floor =
For more complete details, please visit each individual
issue. The report of the agenda items can be found at
https://pagure.io/fesco/report/meeting_agenda
If you would like to add something to this agenda, you can
reply to this e-mail, file a new issue at
https://pagure.io/fesco, e-mail me directly, or bring it
up at the end of the meeting, during the open floor topic. Note
that added topics may be deferred until the following meeting.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 6 months