From: Clark Williams on gitlab.com Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
Change merge.py to handle input config files in similar manner to how merge.pl did it.
Signed-off-by: Clark Williams williams@redhat.com
--- redhat/configs/merge.pl | 72 ---------------------------- redhat/configs/merge.py | 110 +++++++++++++++++++++++++++---------------- redhat/Makefile | 2 +- redhat/kernel.spec.template | 10 ++-- 4 files changed, 74 insertions(+), 120 deletions(-)
From: Clark Williams williams@redhat.com
redhat: update merge.py to handle merge.pl corner cases
Change merge.py to handle input config files in similar manner to how merge.pl did it.
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py index blahblah..blahblah 100755 --- a/redhat/configs/merge.py +++ b/redhat/configs/merge.py @@ -5,54 +5,77 @@ # # merge.py - a direct replacement for merge.pl in the redhat/configs directory # -# invocation: python merge.py overrides baseconfig +# invocation: python merge.py overrides baseconfig [arch] # -# Both input files are kernel config files, where overides is config overides -# to the baseconfig file. Both are read into python dictionaries with the -# keys being the config name and the values being the config file text - -# The script iterates through the overrides keys adding/replacing the contents -# of the baseconfig values and then outputs the new baseconfig to stdout. +# This script merges two kernel configuration files, an override file and a +# base config file and writes the results to stdout. # +# The script reads the overrides into a dictionary, then reads the baseconfig +# file, looking for overrides and replacing any found, then printing the result +# to stdout. Finally any remaining (new) configs in the override are appended to the +# end of the output
import sys +import re import os.path
def usage(msg): - print(msg) - print("usage: merge.py overrides baseconfig") + sys.stderr.write(msg + "\n") + sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n") sys.exit(1)
-# read a config file and return a dictionary of the contents -def read_config_file(cfgfile): - configs = {} - with open(cfgfile) as f: - for l in [n.strip() for n in f.readlines()]: - if not l: continue - if l.startswith("# CONFIG_"): - configs[l.split()[1]] = l - continue - if l.startswith("CONFIG_"): - configs[l.split('=')[0]] = l - continue - return configs - - -if len(sys.argv) < 3: usage("must have two input files") - -# read in the overides file -if not os.path.exists(sys.argv[1]): - usage("overrides config file %s does not exist!" % sys.argv[1]) -overrides = read_config_file(sys.argv[1]) - -# read in the base config file -if not os.path.exists(sys.argv[2]): - usage("base config file %s does not exist!" % sys.argv[2]) -baseconfigs = read_config_file(sys.argv[2]) - -# update baseconfigs with the overrides values -baseconfigs.update(overrides) - -# print the new config to stdout -for v in baseconfigs.values(): - print(v) +isset = re.compile(r'^(CONFIG_\w+)=') +notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set') + +# search an input line for a config (set or notset) pattern +# if we get a match return the config that is being changed +def find_config(line): + if m := isset .match(line): + return m.group(1) + if m := notset.match(line): + return m.group(1) + return None + +######################################################### + +if len(sys.argv) < 3: + usage("must have two input files") + +override_file = sys.argv[1] +baseconfig_file = sys.argv[2] + +if not os.path.exists(override_file): + usage("overrides config file %s does not exist!" % override_file) + +if not os.path.exists(baseconfig_file): + usage("base configs file %s does not exist" % baseconfig_file) + +if len(sys.argv) == 4: + print("# %s" % sys.argv[3]) + +# read each line of the override file and store any configuration values +# in the overrides dictionary, keyed by the configuration name. +overrides = {} +with open(override_file, "rt") as f: + for l in [n.strip() for n in f.readlines()]: + if c := find_config(l): + overrides[c] = l + +# now read and print the base config, checking each line +# that defines a config value and printing the override if +# it exists +with open(baseconfig_file, "rt") as f: + for line in [ l.strip() for l in f.readlines() ]: + c = find_config(line) + if c and c in overrides: + print(overrides[c]) + del overrides[c] + else: + print(line) + +# print out the remaining configs (new values) +# from the overrides file +for c in overrides: + print (overrides[c]) + +sys.exit(0)
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Clark Williams williams@redhat.com
redhat: remove merge.pl and references to it
get rid of redhat/configs/merge.pl and any references to it in redhat/kernel.spec.template and redhat/Makefile
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/Makefile b/redhat/Makefile index blahblah..blahblah 100644 --- a/redhat/Makefile +++ b/redhat/Makefile @@ -634,7 +634,7 @@ sources-rh: $(TARBALL) generate-testpatch-tmp setup-source dist-configs-check scripts/mod/mod-sign.sh \ configs/flavors \ configs/generate_all_configs.sh \ - configs/merge.pl \ + configs/merge.py \ configs/process_configs.sh \ ../Makefile.rhelver \ README.rst \ diff --git a/redhat/configs/merge.pl b/redhat/configs/merge.pl deleted file mode 100755 index blahblah..blahblah 0 --- a/redhat/configs/merge.pl +++ /dev/null @@ -1,72 +0,0 @@ -#! /usr/bin/perl - -my @args=@ARGV; -my %configvalues; -my @configoptions; -my $configcounter = 0; - -# optionally print out the architecture as the first line of our output -my $arch = $args[2]; -if (defined $arch) { - print "# $arch\n"; -} - -# first, read the override file - -open (FILE,"$args[0]") || die "Could not open $args[0]"; -while (<FILE>) { - my $str = $_; - my $configname; - - if (/# ([\w]+) is not set/) { - $configname = $1; - } elsif (/^#/) { - # fall through on comments like 'avoid CONFIG_FOO=y' - ; - } elsif (/([\w]+)=/) { - $configname = $1; - } - - if (defined($configname) && !exists($configvalues{$configname})) { - $configvalues{$configname} = $str; - $configoptions[$configcounter] = $configname; - $configcounter ++; - } -}; - -# now, read and output the entire configfile, except for the overridden -# parts... for those the new value is printed. - -open (FILE2,"$args[1]") || die "Could not open $args[1]"; -while (<FILE2>) { - my $configname; - - if (/# ([\w]+) is not set/) { - $configname = $1; - } elsif (/^#/) { - # fall through on comments like 'avoid CONFIG_FOO=y' - ; - } elsif (/([\w]+)=/) { - $configname = $1; - } - - if (defined($configname) && exists($configvalues{$configname})) { - print "$configvalues{$configname}"; - delete($configvalues{$configname}); - } else { - print "$_"; - } -} - -# now print the new values from the overridden configfile -my $counter = 0; - -while ($counter < $configcounter) { - my $configname = $configoptions[$counter]; - if (exists($configvalues{$configname})) { - print "$configvalues{$configname}"; - } - $counter++; -} - -1; diff --git a/redhat/kernel.spec.template b/redhat/kernel.spec.template index blahblah..blahblah 100755 --- a/redhat/kernel.spec.template +++ b/redhat/kernel.spec.template @@ -850,7 +850,7 @@ Source2002: kvm_stat.logrotate # Some people enjoy building customized kernels from the dist-git in Fedora and # use this to override configuration options. One day they may all use the # source tree, but in the mean time we carry this to support the legacy workflow -Source3000: merge.pl +Source3000: merge.py Source3001: kernel-local %if %{patchlist_changelog} Source3002: Patchlist.changelog @@ -1042,7 +1042,7 @@ This package provides debug information for package kernel-tools. %{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} -p '.*%%{_bindir}/centrino-decode(.debug)?|.*%%{_bindir}/powernow-k8-decode(.debug)?|.*%%{_bindir}/cpupower(.debug)?|.*%%{_libdir}/libcpupower.*|.*%%{_bindir}/turbostat(.debug)?|.*%%{_bindir}/x86_energy_perf_policy(.debug)?|.*%%{_bindir}/tmon(.debug)?|.*%%{_bindir}/lsgpio(.debug)?|.*%%{_bindir}/gpio-hammer(.debug)?|.*%%{_bindir}/gpio-event-mon(.debug)?|.*%%{_bindir}/gpio-watch(.debug)?|.*%%{_bindir}/iio_event_monitor(.debug)?|.*%%{_bindir}/iio_generic_buffer(.debug)?|.*%%{_bindir}/lsiio(.debug)?|.*%%{_bindir}/intel-speed-select(.debug)?|.*%%{_bindir}/page_owner_sort(.debug)?|.*%%{_bindir}/slabinfo(.debug)?|.*%%{_sbindir}/intel_sdsi(.debug)?|XXX' -o kernel-tools-debuginfo.list}
%package -n rtla -Summary: RTLA: Real-Time Linux Analysis tools +Summary: RTLA: Real-Time Linux Analysis tools %description -n rtla The rtla tool is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But, instead of @@ -1512,7 +1512,7 @@ cd configs # Drop some necessary files from the source dir into the buildroot cp $RPM_SOURCE_DIR/kernel-*.config . cp %{SOURCE80} . -# merge.pl +# merge.py cp %{SOURCE3000} . # kernel-local cp %{SOURCE3001} . @@ -1523,12 +1523,12 @@ FLAVOR=%{primary_target} SPECVERSION=%{version} ./generate_all_configs.sh %{debu for i in %{all_arch_configs} do mv $i $i.tmp - ./merge.pl %{SOURCE3001} $i.tmp > $i + ./merge.py %{SOURCE3001} $i.tmp > $i %if %{with_gcov} echo "Merging with gcov options" cat %{SOURCE75} mv $i $i.tmp - ./merge.pl %{SOURCE75} $i.tmp > $i + ./merge.py %{SOURCE75} $i.tmp > $i %endif rm $i.tmp done
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Clark Williams williams@redhat.com
redhat: fixup pylint complaints
Added function doc strings Use f-strings for output Exlicitly use utf-8 encoding when processing config files
Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py index blahblah..blahblah 100755 --- a/redhat/configs/merge.py +++ b/redhat/configs/merge.py @@ -20,6 +20,7 @@ import re import os.path
def usage(msg): + '''print a usage message and exit''' sys.stderr.write(msg + "\n") sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n") sys.exit(1) @@ -30,7 +31,8 @@ notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set') # search an input line for a config (set or notset) pattern # if we get a match return the config that is being changed def find_config(line): - if m := isset .match(line): + '''find a configuration line in the input and return the config name''' + if m := isset.match(line): return m.group(1) if m := notset.match(line): return m.group(1) @@ -45,26 +47,26 @@ override_file = sys.argv[1] baseconfig_file = sys.argv[2]
if not os.path.exists(override_file): - usage("overrides config file %s does not exist!" % override_file) + usage(f"overrides config file {override_file: s} does not exist!")
if not os.path.exists(baseconfig_file): - usage("base configs file %s does not exist" % baseconfig_file) + usage(f"base configs file {baseconfig_file: s} does not exist")
if len(sys.argv) == 4: - print("# %s" % sys.argv[3]) + print(f"# {sys.argv[3]:s}")
# read each line of the override file and store any configuration values # in the overrides dictionary, keyed by the configuration name. overrides = {} -with open(override_file, "rt") as f: - for l in [n.strip() for n in f.readlines()]: - if c := find_config(l): - overrides[c] = l +with open(override_file, "rt", encoding="utf-8") as f: + for line in [l.strip() for l in f.readlines()]: + if c := find_config(line): + overrides[c] = line
# now read and print the base config, checking each line # that defines a config value and printing the override if # it exists -with open(baseconfig_file, "rt") as f: +with open(baseconfig_file, "rt", encoding="utf-8") as f: for line in [ l.strip() for l in f.readlines() ]: c = find_config(line) if c and c in overrides: @@ -75,7 +77,7 @@ with open(baseconfig_file, "rt") as f:
# print out the remaining configs (new values) # from the overrides file -for c in overrides: - print (overrides[c]) +for v in overrides.values(): + print (v)
sys.exit(0)
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Clark Williams williams@redhat.com
redhat/configs: modify merge.py to match old overrides input
Change overrides file input to only use the first instance of a config that is specified and ignore any duplicates encountered after that.
Suggested-by: Herton R. Krzesinski herton@redhat.com Signed-off-by: Clark Williams williams@redhat.com
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py index blahblah..blahblah 100755 --- a/redhat/configs/merge.py +++ b/redhat/configs/merge.py @@ -60,7 +60,8 @@ if len(sys.argv) == 4: overrides = {} with open(override_file, "rt", encoding="utf-8") as f: for line in [l.strip() for l in f.readlines()]: - if c := find_config(line): + c = find_config(line) + if c and c not in overrides: overrides[c] = line
# now read and print the base config, checking each line
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
From: Justin M. Forbes on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1259574...
So this version does make the output match the output from before this MR, but that means we have our additional comments in the config files. I do not consider it a part of the scope of this MR, but it has pointed out that those comments exist, and should not. If we are going to accept this version of merge.py, someone should do another MR to cleanup all of those unnecessary comments.
From: Herton R. Krzesinski on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1259579...
Yes indeed. I think we can keep the logic the same as .pl right now, and may be we can improve the .py script later. Since original concern is solved, resolving this thread.
From: Herton R. Krzesinski on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1259584...
Yes I agree we could do improvements/changes to the script or cleanup the comments in another MR, since it's out of scope for this one, so approving this.
kernel@lists.fedoraproject.org