F34 Change proposal: Debug Info Standardization (from DWZ to
-fdebug-types-section) (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/DebugInfoStandardization
== Summary ==
Fedora 18 implemented [[Features/DwarfCompressor]]. As the format did
not get widespread and the tool is not much maintained it became
burden to make existing debugging tools compatible with Fedora debug
info.
== Owner ==
* Name: [[User:jankratochvil| Jan Kratochvil ]]
* Email: jan.kratochvil(a)redhat.com
== Detailed Description ==
Debug info files *.debug contained in *-debuginfo.rpm are very big in
general (x86_64 Fedora 32 distribution has debug/ directory of 82GB
while all its other files are only 75GB). There exist several methods
how to make the *-debuginfo.rpms at least a bit smaller. Fedora 18
started using DWZ tool (from [[Features/DwarfCompressor]]) while
[https://gcc.gnu.org/pipermail/gcc-patches/2008-August/246281.html
Google implemented] the same goal in a different way called
-fdebug-types-section.
Almost nobody uses existing Fedora DWZ (only Fedora/CentOS/RHEL and
SuSE OSes) and so its support is missing in tools like
[https://lldb.llvm.org/ LLDB],
[[llvm-dwarfdumphttps://llvm.org/docs/CommandGuide/llvm-dwarfdump.html|llvm-dwarfdump]]
or binutils readelf. -fdebug-types-section is used internally by
Google (produced by clang). Debian does not store any debug info
archives. Ubuntu uses neither -fdebug-types-section nor DWZ.
* DWZ advantage: On the whole Fedora distro it saves 3.3% (5GB of the
157GB distribution size)
** If the 3.3% size increase is a concern I can implement a different
optimization ([https://whova.com/embedded/session/llvm_202010/1193947/
talk (2)]) as a GCC post-processing phase which would require no
changes in any DWARF consumers.
* DWZ disadvantage: DWZ has currently less support across consumers
(LLDB, llvm-dwarfdump, binutils readelf)
* DWZ disadvantage: DWZ requires 8x times more complicated (LoC count)
support in consumers than -fdebug-types-section.
* DWZ disadvantage: DWZ cannot update LLVM .debug_names index which
can be generated only by clang (it cannot be regenerated later for
DWZ-compressed file)
* DWZ disadvantage: DWZ DWARF-5 support is a work-in-progress. DWZ has
been blocking DWARF-5 for Fedora for 3.5 years and only after I have
now proposed to drop DWZ Mark Wielaard has started porting DWZ to
DWARF-5. It can be expected next DWARF extensions will remain
unsupported again. Even currently there is no plan to support DWARF-5
features used by clang which may need -fdebug-types-section for
clang-built binaries or no size optimization of clang-built debug info
at all.
* DWZ disadvantage: Compilation (linking) requires for C++ up to 2x as
big disk space (as DWZ is processing files after linker and DWZ is
incompatible with -fdebug-types-section)
* DWZ disadvantage: Compilation (linking) is slower
This proposed DWARF format was originally submitted already for Fedora
18 as [[Features/DebugTypesSections]].
== Benefit to Fedora ==
* Better compatibility with existing debugging and tracing tools,
primarily [https://lldb.llvm.org/ LLDB].
* Less resource-intensive rebuilds of C++ packages (in disk space,
memory requirements and compilation time).
== Scope ==
* Proposal owners: It affects all packages generating *-debuginfo.rpm,
that is compiled (not scripted) languages.
* Other developers: Report any possible debuginfo incompatibility (unexpected).
* Release engineering: [https://pagure.io/releng/issues #Releng issue
number] (a check of an impact with Release Engineering is needed)
* Policies and guidelines: All the needed changes should be done in
[https://src.fedoraproject.org/rpms/redhat-rpm-config
redhat-rpm-config]. The [https://src.fedoraproject.org/rpms/dwz dwz
package] can be then retired.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: The size differences are only for
*-debuginfo.rpm which is outside of scope of the listed objectives.
== Upgrade/compatibility impact ==
As *-debuginfo.rpm have to exactly match NVRA of its binary package
the compatibility is not relevant. Existing tools supporting DWZ will
still support the DWZ file format in packages which have not been
rebuilt.
== How To Test ==
The change will update
[https://src.fedoraproject.org/rpms/redhat-rpm-config
redhat-rpm-config] by
[https://people.redhat.com/jkratoch/redhat-rpm-config-fdebug-types-section...
an -fdebug-types-section patch].
Then one can use rpmbuild to rebuild a package. For mock use
-a|--addrepo with modified redhat-rpm-config.rpm (with increased
NVRA). For packages already rebuilt in Koji nothing is needed.
Test programs like lldb and gdb if they still can print source code,
function parameters, variables etc.
One should also verify integrated testsuites of tools like clang,
lldb, gcc, binutils, gdb, elfutils or rpm are not regressing with the
-fdebug-types-section option.
One can also compare *.debug files built with/without DWZ and/or
-fdebug-types-section using
[https://src.fedoraproject.org/rpms/libabigail libabigail] utility
dwdiff but that will be rather done by the change owner.
== User Experience ==
No user visible change. This affects what tools can developers use.
== Dependencies ==
none
== Contingency Plan ==
* Contingency mechanism: Revert the change in
[https://src.fedoraproject.org/rpms/redhat-rpm-config
redhat-rpm-config]. Fedora can continue using DWZ, just some
debugging/tracing tools will stay incompatible.
* Contingency deadline: beta freeze
* Blocks release? No
* Blocks product? N/A
== Documentation ==
* [http://www.dwarfstd.org/doc/DWARF5.pdf DWARF-5] E.2 Using Type Units
* [https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fdebug-ty...
GCC -fdebug-types-section]
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
6 months, 2 weeks
readelf seems broken in F33
by Steve Grubb
Hello,
I was doing some binary analysis of files in F33 and have run across something odd.
readelf -s /usr/sbin/auditd | grep GLIBC
produces a lot of output like:
182: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
184: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
185: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
186: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
187: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.3.2 (2)
191: 0000000000000000 0 FUNC GLOBAL DEFAULT UND alarm(a)GLIBC_2.2.5 (3)
194: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
195: 0000000000000000 0 FUNC GLOBAL DEFAULT UND close(a)GLIBC_2.2.5 (3)
197: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...](a)GLIBC_2.2.5 (3)
It's missing a lot of symbols. Is this something with readelf or an odd
effect of the LTO changes?
-Steve
6 months, 2 weeks
Orphaning openbabel
by Dominik 'Rathann' Mierzejewski
Hi everyone,
I've finally admitted to myself that I have no time to take good care
of openbabel:
https://src.fedoraproject.org/rpms/openbabel
It has one FTBFS bug in rawhide (related to cmake macro changes)
and two requests to update. One in Fedora to move to the recent 3.x
series and one for EPEL to update from 2.3 to 2.4.
It has 9 consumers in Fedora:
avogadro
avogadro2
chemtool
ghemical
gnome-chemistry-utils
IQmol
kalzium
molsketch
xdrawchem
Hopefully, one of their maintainers (Bcc'd) or someone from SciTech SIG
(Cc'd) can pick this up.
I'll orphan it in a few days if nobody responds here with their
FAS ID.
Regards,
--
Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
6 months, 2 weeks
Orphaning vdr-skinsoppalusikka and the status of my packages (looking
for co-maintainers for Finnish spell checking)
by Ville-Pekka Vainio
Hi all,
As some of you may have noticed, I have been away from Fedora
packaging for quite a while. I've been hoping to find the time for
packaging work, but it hasn't happened. In hindsight, I should have
let the project know earlier.
I have now orphaned vdr-skinsoppalusikka, because I don't have a
working VDR installation anymore. If someone has the ability, my user
(vpv) can be dropped from the packages vdr, vdr-epgsearch, vdr-femon
and vdr-osdteletext.
My main interest in Fedora packaging used to be Finnish spell
checking. The spell checking stack, consisting of libreoffice-voikko,
libvoikko, voikko-fi and foma (as a dependency) went through quite
major changes in 2017-2019. The vocabulary package voikko-fi requires
foma to build. I managed to get the foma package reviewed and into
Fedora in 2018. It was automatically orphaned and then retired due to
FTBFS (https://src.fedoraproject.org/rpms/foma). The code has now been
fixed in Github (https://github.com/mhulden/foma) and it should build,
but I haven't tried it yet. If I get it to build, I might ask for it
to be added back. It seems like I'll be able to work on packaging for
a few days a year, so don't expect any miracles.
Once foma is built, voikko-fi can be built. Even though it shares some
history with the previous vocabulary package malaga-suomi-voikko, it
probably needs a new review. Then libvoikko could be updated to use
the new vocabulary. If anyone is interested in working on these, I
would be happy to get co-maintainers.
I'm willing to stay as a co-maintainer of gpodder and
python-mygpoclient. I code Python at $DAYJOB so I might be able to
help.
Best regards,
Ville-Pekka Vainio (vpv)
6 months, 2 weeks
paranoid md raid1 -> Btrfs migration tools?
by Daniel Pocock
Are there any tools in Fedora to help people migrate an md RAID1 array
to btrfs, with data verification checks?
For example
1) check for differences between source file sectors on each source drive
2) for each file, if the file format has a checksum, check it somehow
and alert on failures
The new emphasis on Btrfs is interesting but it is not much help if
people copy in files that are already spoiled. If people can still
recover a good copy from their md array then it is a good idea to do so.
I could imagine using kpartx to script a solution to (1) above, skipping
over the md headers. Some kind of shim may be needed to fool the kernel
to see a different UUID for each source volume so they can be mounted
simultaneously without md.
Regards,
Daniel
6 months, 3 weeks
LTO and F33
by Jeff Law
So we're at a point where the F33 FTBFS issues related to LTO that I'm aware of
have been resolved (by opting the package out of LTO). I still expect some LTO
issues will pop up as packages fix things like missing dependencies, cmake
macros, etc. I continue to be available to investigate potential LTO issues, but
package maintainers will need to contact me as I'm not actively looking for new
LTO issues.
My focus is now turning to the packages with LTO opt-outs. I'll be extracting
bug reports for upstream (primarily GCC), trying simple workarounds for old style
symbol versioning, identifying backports from upstream GCC that allow us to
remove LTO opt-outs and the like. So there should be a trickle of opt-outs
removed, but otherwise should largely be invisible to the F33 release process.
I'd like to thank everyone involved for being patient while we worked through
various issues and I look forward to continuing to make incremental improvements
now that the bulk of the LTO work has landed.
jeff
6 months, 3 weeks
F34 Change: Reduce installation media size by improving the
compression ratio of SquashFS filesystem (Self-Contained Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/OptimizeSquashFS
== Summary ==
Improve compression ratio of SquashFS filesystem on the installation media.
== Owner ==
* Name: [[User:bkhomuts|Bohdan Khomutskyi]]
* Email: bkhomuts(a)redhat.com
== Detailed Description ==
As of Fedora 31, the LiveOS/squashfs.img file on the installation
image, is compressed with default block size of mksquashfs. The
default block size is 128k. Additionaly, lorax sets BCJ filter
depending on the architecture. Those parameters can be adjusted which
will lead to a better compression ratio and/or reduction of the CPU
usage at build time.
This is simple to achieve. Recently, Lorax has gotten support for
adjusting the compression options for mksquashfs via the configuration
file. The file should be altered as following:
<pre>
[compression]
bcj = no
args = -b 1M -Xdict-size 1M -no-recovery
</pre>
Where -b 1M and -Xdict-size 1M are block and dictionary sizes
respectively; bcj -- branch-call-jump filter.
Based on the results above, I'd suggest selecting the following
''optimal configuration'': XZ algorithm, with block size of 1MiB and
without BCJ filter (plain xz -b 1M, without -Xbcj x86).
On the right, you can see the impact of the compression algorithms on
installation time.
As can be seen from the picture on the right hand side, selecting
'plain xz -b 1M configuration' has minimal impact on the installation
time and CPU usage during the installation. The compression will
result in +6.51% or, in real terms, +24.94s additional installation
time, compared to the savings of 142 MiB on the installation media,
refer to the documentation section to download the ISO images. This
increase in installation time will be compensated by the change in the
installer: https://github.com/rhinstaller/anaconda/pull/2292
I noticed, that even with maximum compression, CPU is not fully
utilized during installation.
== Benefit to Fedora ==
* Reduction of the installation media size and the cost of storing and
distributing Fedora.
* Reduction of the CPU usage at build time. Depending on which
compression parameters chosen.
== Scope ==
* Proposal owners:
The build environment should have support for adjusting the Lorax
configuration file and -squashfs-only parameter. Lorax is a program
that produces the LiveOS/squashfs.img file on the installation media.
One of the ways to enable such customization is to introduce support
in Pungi to pass -c option to Lorax.
* Other developers:
The pungi utility should support passing the custom configuration file
location to the Lorax utility. This option should apply during
buildInstall phase of pungi.
* Release engineering: [https://pagure.io/releng/issue/9127]
<!-- Does this feature require coordination with release engineering
(e.g. changes to installer image generation or update package
delivery)? Is a mass rebuild required? include a link to the releng
issue.
The issue is required to be filed prior to feature submission, to
ensure that someone is on board to do any process development work and
testing, and that all changes make it into the pipeline; a bullet
point in a change is not sufficient communication -->
* Policies and guidelines: Not required.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
N/A (not a System Wide Change)
== How To Test ==
<pre>
mkdir -p /mnt/new /mnt/old
sudo mount -o loop,ro FedoraInstallationOld.iso
sudo mount -o loop,ro FedoraInstallationNew.iso
ls -l /mnt/{new,old}/LiveOS/squashfs.img
</pre>
And then calculate the size difference.
== User Experience ==
* Decreasing the installation image size will reduce cost of mirroring
and storing Fedora installation images.
* Decreasing the installation image size will reduce the download time.
* Increasing the block size on the current configuration with EXT4
file system, should increase latency while accessing the EXT4
filesystem. The exact impact is to be evaluated.
* The impact of latency will be reduced, if the plain SquashFS option
is be choosen.
== Dependencies ==
Pungi, a utility that builds the compose, should include new
functionality mentioned above.
Alternatively, the /etc/lorax/lorax.conf should be altered in the
environment where Lorax is running.
== Contingency Plan ==
N/A
== Documentation ==
https://pagure.io/releng/issue/9127.<br/>
mksquashfs(1)<br/>
lorax(1)<br/>
https://docs.pagure.org/pungi<br/>
Select ISOs can be downloaded at https://khomutsky.com/fedora-dvd/<br/>
Spreadsheet pictured above [[File:Comparison Table SquashFS.ods]]
[https://github.com/rhinstaller/anaconda/pull/2292 Multi-core
decompression of SquashFS merge request]
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
6 months, 3 weeks