From: Clark Williams on gitlab.com Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
Change merge.py to handle input config files in similar manner to how merge.pl did it.
Signed-off-by: Clark Williams williams@redhat.com
--- redhat/configs/merge.pl | 72 ----------------------------- redhat/configs/merge.py | 109 +++++++++++++++++++++++++++---------------- redhat/Makefile | 2 +- redhat/kernel.spec.template | 10 ++-- 4 files changed, 73 insertions(+), 120 deletions(-)
From: Clark Williams williams@redhat.com
redhat: update merge.py to handle merge.pl corner cases
Change merge.py to handle input config files in similar manner to how merge.pl did it.
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py index blahblah..blahblah 100755 --- a/redhat/configs/merge.py +++ b/redhat/configs/merge.py @@ -5,54 +5,77 @@ # # merge.py - a direct replacement for merge.pl in the redhat/configs directory # -# invocation: python merge.py overrides baseconfig +# invocation: python merge.py overrides baseconfig [arch] # -# Both input files are kernel config files, where overides is config overides -# to the baseconfig file. Both are read into python dictionaries with the -# keys being the config name and the values being the config file text - -# The script iterates through the overrides keys adding/replacing the contents -# of the baseconfig values and then outputs the new baseconfig to stdout. +# This script merges two kernel configuration files, an override file and a +# base config file and writes the results to stdout. # +# The script reads the overrides into a dictionary, then reads the baseconfig +# file, looking for overrides and replacing any found, then printing the result +# to stdout. Finally any remaining (new) configs in the override are appended to the +# end of the output
import sys +import re import os.path
def usage(msg): - print(msg) - print("usage: merge.py overrides baseconfig") + sys.stderr.write(msg + "\n") + sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n") sys.exit(1)
-# read a config file and return a dictionary of the contents -def read_config_file(cfgfile): - configs = {} - with open(cfgfile) as f: - for l in [n.strip() for n in f.readlines()]: - if not l: continue - if l.startswith("# CONFIG_"): - configs[l.split()[1]] = l - continue - if l.startswith("CONFIG_"): - configs[l.split('=')[0]] = l - continue - return configs - - -if len(sys.argv) < 3: usage("must have two input files") - -# read in the overides file -if not os.path.exists(sys.argv[1]): - usage("overrides config file %s does not exist!" % sys.argv[1]) -overrides = read_config_file(sys.argv[1]) - -# read in the base config file -if not os.path.exists(sys.argv[2]): - usage("base config file %s does not exist!" % sys.argv[2]) -baseconfigs = read_config_file(sys.argv[2]) - -# update baseconfigs with the overrides values -baseconfigs.update(overrides) - -# print the new config to stdout -for v in baseconfigs.values(): - print(v) +isset = re.compile(r'^(CONFIG_\w+)=') +notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set') + +# search an input line for a config (set or notset) pattern +# if we get a match return the config that is being changed +def find_config(line): + if m := isset .match(line): + return m.group(1) + if m := notset.match(line): + return m.group(1) + return None + +######################################################### + +if len(sys.argv) < 3: + usage("must have two input files") + +override_file = sys.argv[1] +baseconfig_file = sys.argv[2] + +if not os.path.exists(override_file): + usage("overrides config file %s does not exist!" % override_file) + +if not os.path.exists(baseconfig_file): + usage("base configs file %s does not exist" % baseconfig_file) + +if len(sys.argv) == 4: + print("# %s" % sys.argv[3]) + +# read each line of the override file and store any configuration values +# in the overrides dictionary, keyed by the configuration name. +overrides = {} +with open(override_file, "rt") as f: + for l in [n.strip() for n in f.readlines()]: + if c := find_config(l): + overrides[c] = l + +# now read and print the base config, checking each line +# that defines a config value and printing the override if +# it exists +with open(baseconfig_file, "rt") as f: + for line in [ l.strip() for l in f.readlines() ]: + c = find_config(line) + if c and c in overrides: + print(overrides[c]) + del overrides[c] + else: + print(line) + +# print out the remaining configs (new values) +# from the overrides file +for c in overrides: + print (overrides[c]) + +sys.exit(0)
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Clark Williams on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1254465...
Updated to address review suggestions and pylint complaints.
From: Clark Williams williams@redhat.com
redhat: remove merge.pl and references to it
get rid of redhat/configs/merge.pl and any references to it in redhat/kernel.spec.template and redhat/Makefile
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/Makefile b/redhat/Makefile index blahblah..blahblah 100644 --- a/redhat/Makefile +++ b/redhat/Makefile @@ -634,7 +634,7 @@ sources-rh: $(TARBALL) generate-testpatch-tmp setup-source dist-configs-check scripts/mod/mod-sign.sh \ configs/flavors \ configs/generate_all_configs.sh \ - configs/merge.pl \ + configs/merge.py \ configs/process_configs.sh \ ../Makefile.rhelver \ README.rst \ diff --git a/redhat/configs/merge.pl b/redhat/configs/merge.pl deleted file mode 100755 index blahblah..blahblah 0 --- a/redhat/configs/merge.pl +++ /dev/null @@ -1,72 +0,0 @@ -#! /usr/bin/perl - -my @args=@ARGV; -my %configvalues; -my @configoptions; -my $configcounter = 0; - -# optionally print out the architecture as the first line of our output -my $arch = $args[2]; -if (defined $arch) { - print "# $arch\n"; -} - -# first, read the override file - -open (FILE,"$args[0]") || die "Could not open $args[0]"; -while (<FILE>) { - my $str = $_; - my $configname; - - if (/# ([\w]+) is not set/) { - $configname = $1; - } elsif (/^#/) { - # fall through on comments like 'avoid CONFIG_FOO=y' - ; - } elsif (/([\w]+)=/) { - $configname = $1; - } - - if (defined($configname) && !exists($configvalues{$configname})) { - $configvalues{$configname} = $str; - $configoptions[$configcounter] = $configname; - $configcounter ++; - } -}; - -# now, read and output the entire configfile, except for the overridden -# parts... for those the new value is printed. - -open (FILE2,"$args[1]") || die "Could not open $args[1]"; -while (<FILE2>) { - my $configname; - - if (/# ([\w]+) is not set/) { - $configname = $1; - } elsif (/^#/) { - # fall through on comments like 'avoid CONFIG_FOO=y' - ; - } elsif (/([\w]+)=/) { - $configname = $1; - } - - if (defined($configname) && exists($configvalues{$configname})) { - print "$configvalues{$configname}"; - delete($configvalues{$configname}); - } else { - print "$_"; - } -} - -# now print the new values from the overridden configfile -my $counter = 0; - -while ($counter < $configcounter) { - my $configname = $configoptions[$counter]; - if (exists($configvalues{$configname})) { - print "$configvalues{$configname}"; - } - $counter++; -} - -1; diff --git a/redhat/kernel.spec.template b/redhat/kernel.spec.template index blahblah..blahblah 100755 --- a/redhat/kernel.spec.template +++ b/redhat/kernel.spec.template @@ -850,7 +850,7 @@ Source2002: kvm_stat.logrotate # Some people enjoy building customized kernels from the dist-git in Fedora and # use this to override configuration options. One day they may all use the # source tree, but in the mean time we carry this to support the legacy workflow -Source3000: merge.pl +Source3000: merge.py Source3001: kernel-local %if %{patchlist_changelog} Source3002: Patchlist.changelog @@ -1042,7 +1042,7 @@ This package provides debug information for package kernel-tools. %{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} -p '.*%%{_bindir}/centrino-decode(.debug)?|.*%%{_bindir}/powernow-k8-decode(.debug)?|.*%%{_bindir}/cpupower(.debug)?|.*%%{_libdir}/libcpupower.*|.*%%{_bindir}/turbostat(.debug)?|.*%%{_bindir}/x86_energy_perf_policy(.debug)?|.*%%{_bindir}/tmon(.debug)?|.*%%{_bindir}/lsgpio(.debug)?|.*%%{_bindir}/gpio-hammer(.debug)?|.*%%{_bindir}/gpio-event-mon(.debug)?|.*%%{_bindir}/gpio-watch(.debug)?|.*%%{_bindir}/iio_event_monitor(.debug)?|.*%%{_bindir}/iio_generic_buffer(.debug)?|.*%%{_bindir}/lsiio(.debug)?|.*%%{_bindir}/intel-speed-select(.debug)?|.*%%{_bindir}/page_owner_sort(.debug)?|.*%%{_bindir}/slabinfo(.debug)?|.*%%{_sbindir}/intel_sdsi(.debug)?|XXX' -o kernel-tools-debuginfo.list}
%package -n rtla -Summary: RTLA: Real-Time Linux Analysis tools +Summary: RTLA: Real-Time Linux Analysis tools %description -n rtla The rtla tool is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But, instead of @@ -1512,7 +1512,7 @@ cd configs # Drop some necessary files from the source dir into the buildroot cp $RPM_SOURCE_DIR/kernel-*.config . cp %{SOURCE80} . -# merge.pl +# merge.py cp %{SOURCE3000} . # kernel-local cp %{SOURCE3001} . @@ -1523,12 +1523,12 @@ FLAVOR=%{primary_target} SPECVERSION=%{version} ./generate_all_configs.sh %{debu for i in %{all_arch_configs} do mv $i $i.tmp - ./merge.pl %{SOURCE3001} $i.tmp > $i + ./merge.py %{SOURCE3001} $i.tmp > $i %if %{with_gcov} echo "Merging with gcov options" cat %{SOURCE75} mv $i $i.tmp - ./merge.pl %{SOURCE75} $i.tmp > $i + ./merge.py %{SOURCE75} $i.tmp > $i %endif rm $i.tmp done
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Clark Williams williams@redhat.com
redhat: fixup pylint complaints
Added function doc strings Use f-strings for output Exlicitly use utf-8 encoding when processing config files
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py index blahblah..blahblah 100755 --- a/redhat/configs/merge.py +++ b/redhat/configs/merge.py @@ -20,6 +20,7 @@ import re import os.path
def usage(msg): + '''print a usage message and exit''' sys.stderr.write(msg + "\n") sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n") sys.exit(1) @@ -30,7 +31,8 @@ notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set') # search an input line for a config (set or notset) pattern # if we get a match return the config that is being changed def find_config(line): - if m := isset .match(line): + '''find a configuration line in the input and return the config name''' + if m := isset.match(line): return m.group(1) if m := notset.match(line): return m.group(1) @@ -45,26 +47,26 @@ override_file = sys.argv[1] baseconfig_file = sys.argv[2]
if not os.path.exists(override_file): - usage("overrides config file %s does not exist!" % override_file) + usage(f"overrides config file {override_file: s} does not exist!")
if not os.path.exists(baseconfig_file): - usage("base configs file %s does not exist" % baseconfig_file) + usage(f"base configs file {baseconfig_file: s} does not exist")
if len(sys.argv) == 4: - print("# %s" % sys.argv[3]) + print(f"# {sys.argv[3]:s}")
# read each line of the override file and store any configuration values # in the overrides dictionary, keyed by the configuration name. overrides = {} -with open(override_file, "rt") as f: - for l in [n.strip() for n in f.readlines()]: - if c := find_config(l): - overrides[c] = l +with open(override_file, "rt", encoding="utf-8") as f: + for line in [l.strip() for l in f.readlines()]: + if c := find_config(line): + overrides[c] = line
# now read and print the base config, checking each line # that defines a config value and printing the override if # it exists -with open(baseconfig_file, "rt") as f: +with open(baseconfig_file, "rt", encoding="utf-8") as f: for line in [ l.strip() for l in f.readlines() ]: c = find_config(line) if c and c in overrides: @@ -75,7 +77,7 @@ with open(baseconfig_file, "rt") as f:
# print out the remaining configs (new values) # from the overrides file -for c in overrides: - print (overrides[c]) +for v in overrides.values(): + print (v)
sys.exit(0)
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Herton R. Krzesinski on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1258074...
Looks like this part doesn't translate exactly what the perl part implemented:
``` if (defined($configname) && !exists($configvalues{$configname})) { $configvalues{$configname} = $str; $configoptions[$configcounter] = $configname; $configcounter ++; } ```
If the overrides file has the same config option listed twice, the perl seemed to ignore it. Eg.
CONFIG_FOO = y CONFIG_FOO = n
configvalues gets CONFIG_FOO = y next iteration configvalues is not assigned because CONFIG_FOO is already assigned previously
The python script is not checking this, always assign what it reads (```overrides[c] = line```)
Not sure if this is a real problem or not, but just pointing out. We can resolve this thread if it's not an issue.
From: Justin M. Forbes on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1258211...
This actually seems like the more desirable behavior, but in either case, having the same config item listed multiple times in the file is a bug. It does appear that this patch changes the config output, however it seems like logical changes:
``` --- a/kernel-x86_64-rhel.config +++ b/kernel-x86_64-rhel.config @@ -868,7 +868,6 @@ CONFIG_CPU_LITTLE_ENDIAN=y CONFIG_CPUMASK_KUNIT_TEST=m CONFIG_CPUMASK_OFFSTACK=y CONFIG_CPUSETS=y -# CONFIG_CPU_SUP_CENTAUR ignored for predicate CONFIG_EXPERT # CONFIG_CPU_THERMAL is not set CONFIG_CPU_UNRET_ENTRY=y # CONFIG_CRAMFS is not set @@ -5864,44 +5863,31 @@ CONFIG_SND_SOC_RT715_SDW=m # CONFIG_SND_SOC_SMDK_WM8994_PCM is not set # CONFIG_SND_SOC_SNOW is not set CONFIG_SND_SOC_SOF_ACPI=m -# CONFIG_SND_SOC_SOF_ALDERLAKE: CONFIG_SND_SOC_SOF_ALDERLAKE=m # CONFIG_SND_SOC_SOF_AMD_TOPLEVEL is not set -# CONFIG_SND_SOC_SOF_APOLLOLAKE: CONFIG_SND_SOC_SOF_APOLLOLAKE=m -# CONFIG_SND_SOC_SOF_BAYTRAIL: CONFIG_SND_SOC_SOF_BAYTRAIL=m -# CONFIG_SND_SOC_SOF_BROADWELL: CONFIG_SND_SOC_SOF_BROADWELL=m -# CONFIG_SND_SOC_SOF_CANNONLAKE: CONFIG_SND_SOC_SOF_CANNONLAKE=m -# CONFIG_SND_SOC_SOF_COFFEELAKE: CONFIG_SND_SOC_SOF_COFFEELAKE=m -# CONFIG_SND_SOC_SOF_COMETLAKE: CONFIG_SND_SOC_SOF_COMETLAKE=m -# CONFIG_SND_SOC_SOF_ELKHARTLAKE: CONFIG_SND_SOC_SOF_ELKHARTLAKE=m CONFIG_SND_SOC_SOF_GEMINILAKE=m CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC=y CONFIG_SND_SOC_SOF_HDA_LINK=y -# CONFIG_SND_SOC_SOF_ICELAKE: CONFIG_SND_SOC_SOF_ICELAKE=m # CONFIG_SND_SOC_SOF_IMX8M_SUPPORT is not set # CONFIG_SND_SOC_SOF_IMX8_SUPPORT is not set # CONFIG_SND_SOC_SOF_IMX_TOPLEVEL is not set -# CONFIG_SND_SOC_SOF_INTEL_SOUNDWIRE: CONFIG_SND_SOC_SOF_INTEL_SOUNDWIRE=m CONFIG_SND_SOC_SOF_INTEL_TOPLEVEL=y -# CONFIG_SND_SOC_SOF_JASPERLAKE: CONFIG_SND_SOC_SOF_JASPERLAKE=m CONFIG_SND_SOC_SOF_KABYLAKE=m -# CONFIG_SND_SOC_SOF_MERRIFIELD: CONFIG_SND_SOC_SOF_MERRIFIELD=m CONFIG_SND_SOC_SOF_METEORLAKE=m # CONFIG_SND_SOC_SOF_OF is not set CONFIG_SND_SOC_SOF_PCI=m CONFIG_SND_SOC_SOF_SKYLAKE=m -# CONFIG_SND_SOC_SOF_TIGERLAKE: CONFIG_SND_SOC_SOF_TIGERLAKE=m CONFIG_SND_SOC_SOF_TOPLEVEL=y # CONFIG_SND_SOC_SPDIF is not set @@ -7280,12 +7266,8 @@ CONFIG_XILINX_GMII2RGMII=m # CONFIG_XILLYBUS is not set # CONFIG_XILLYUSB is not set CONFIG_XMON_DEFAULT_RO_MODE=y -# CONFIG_XZ_DEC_ARM ignored for predicate CONFIG_EXPERT -# CONFIG_XZ_DEC_ARMTHUMB ignored for predicate CONFIG_EXPERT -# CONFIG_XZ_DEC_IA64 ignored for predicate CONFIG_EXPERT # CONFIG_XZ_DEC_MICROLZMA is not set CONFIG_XZ_DEC_POWERPC=y -# CONFIG_XZ_DEC_SPARC ignored for predicate CONFIG_EXPERT # CONFIG_XZ_DEC_TEST is not set CONFIG_XZ_DEC_X86=y CONFIG_XZ_DEC=y ```
Other arches included the same or just a subset as the SND_SOC bits were lacking deps. If you look closely, we are losing comments that shouldn't be in the file, and do not have an impact on the built kernel. Behavior of the built kernels should be identical. For future changes, after merge, things would work just as before the merge in that anyone making a config change should verify that the config change they are making has the desired effect.
kernel@lists.fedoraproject.org