A little follow-on to the "Longing for git-bisect" thread, in a newly-created/subjected thread...
So, I've been poking around in the kernel spec file extensively the past few days already... Any objections to my attempting to implement the building of a kernel-vanilla package as well?
Roland, not sure if you were already planning to do this in conjunction with digging up your old patches, or if you were just throwing the patches out there, but either way, I'm happy to take on the task, but I'll defer to anyone else that really wants to do it.
I already got it working and posted the spec and Makefile patch here. (I've also got my stuff for building from git branches working now.) Nobody answered about whether they wanted it committed.
I didn't do it in a way intended to produce different variant rpms called kernel-vanilla-V-R, if that's what you had in mind. That requires basically duplicating all the innards of the spec file. I don't think that should be attempted without a complete cleanup of the spec file to become sane.
What I've done is sufficient for making a vanilla build like this one: http://porkchop.devel.redhat.com/brewroot/scratch/roland/task_700227/
Thanks, Roland
Roland McGrath wrote:
I already got it working and posted the spec and Makefile patch here.
Gah, sorry, I saw the patch, but missed that the spec mods were included in it too.
(I've also got my stuff for building from git branches working now.) Nobody answered about whether they wanted it committed.
I didn't do it in a way intended to produce different variant rpms called kernel-vanilla-V-R, if that's what you had in mind.
It is. And I was under the impression that's what Dave was thinking too, but I'll let him speak for himself.
That requires basically duplicating all the innards of the spec file. I don't think that should be attempted without a complete cleanup of the spec file to become sane.
More specifically, what I had in mind was to add just one more variant, so you'd have kernel, kernel-xen, kernel-kdump, kernel-PAE, etc., plus now a kernel-vanilla, which would be analogous to kernel minus patches -- i.e., don't bother with kernel-vanilla-xen, kernel-vanilla-kudmp, etc.
Not to say that a spec file sanitization/sane-itization wouldn't be good to do as well (which I'm also perfectly willing to take on -- a few steps in this direction were taken earlier today).
What I've done is sufficient for making a vanilla build like this one: http://porkchop.devel.redhat.com/brewroot/scratch/roland/task_700227/
Cool, I'll read your patch more carefully now. :)
It is. And I was under the impression that's what Dave was thinking too, but I'll let him speak for himself.
Well, do we want it in the package set? I figured we were talking about "informal" builds that might be put up for ftp, but not be an integrated part of the Fedora world. If one rpmbuild from the spec kicks out -vanilla packages along with other variants, and we turn that on only for rawhide, still it winds up "in the system", people install it as "normal" and wind up with upgrade issues and all that. Eventually we have clueless Fedora users saying "you use -PAE? well -vanilla is the Fedora kernel I like best!" and reporting in bugs "why this fix isn't in -vanilla too?"
It's easy to tweak my changes to put "vanilla" into %name instead of %release. I went with "kernel" and a long, funky %release because that's the tradition I've experienced for one-off and test builds people distribute. From my own experience juggling a dozen installations on two or three test boxes with many hack kernels installed each in an utterly disorganized fashion, it seems more likely to get resolved usefully when doing upgrades and such than an rpm named "kernel-foobar". That is, yum and rpm will get in your face all the time in the way that people are used to for juggling kernel rpms. (Also, I always shudder in superstitious fear that yum or who knows what will assume things about the kernel rpm being the way it always has been.)
With whatever name, separate vanilla builds could easily be made usable as their own little yum repository.
What is more work, and I think is more dubious, is having it be the same srpm that produces both fedora and vanilla builds. What will -bp do, make two source trees patched differently?
That requires basically duplicating all the innards of the spec file. I don't think that should be attempted without a complete cleanup of the spec file to become sane.
More specifically, what I had in mind was to add just one more variant, so you'd have kernel, kernel-xen, kernel-kdump, kernel-PAE, etc., plus now a kernel-vanilla, which would be analogous to kernel minus patches -- i.e., don't bother with kernel-vanilla-xen, kernel-vanilla-kudmp, etc.
That's no fun! Seriously, I think we need the full suite of variants (modulo xen until that's uptsream, etc.) for it really to be useful. Plenty of people are using -PAE and the point is to compare the foo you're using to foo-vanilla on a one-to-one basis, i.e. different source, same .config settings. And you want -debuginfo for all of those too.
Not to say that a spec file sanitization/sane-itization wouldn't be good to do as well (which I'm also perfectly willing to take on -- a few steps in this direction were taken earlier today).
I despise how much duplication there is in there. The debuginfo stuff I originally did with rpm macros got hand-expanded because older rpmbuild couldn't cope with it. Maybe nowadays there is nothing with an rpmbuild older than FC-5 we are trying to work with, so we could use more macros for new stuff. (I think brew got fixed to use the buildroot's rpmbuild, so anything that's fine for the target systems sharing the kernel spec source should be fine.) rpm macros are a hairy mess, but cleaning things up with clean new macros is a better bet than anything else, IMHO.
Cool, I'll read your patch more carefully now. :)
Here's the current version of it. You still need the nonintconfig patch variant I posted last time to go with this. I also attached below the shell script used by the GIT-using hacks that are in this version. http://porkchop.devel.redhat.com/brewroot/scratch/roland/task_703101/ shows what "make git/roland/scratch-build" does (the build magic works for "make git/roland-fedora/..." too, but the fedora patches don't apply to current upstream so it didn't build). This (the non-fedora flavor) is similar to the vanilla build, but automagically prepends:
%changelog * Thu Mar 29 2007 Roland McGrath roland@redhat.com - Experimental build from git sources (no Fedora patches) - git tag: v2.6.21-rc5 6fb04ccf5c5e054c4107090bed6e866489f1089f - git branch: upstream 28defbea64622f69d65a6079bf800cedb9915a5f - git branch: roland 0ca00c859e7ac4d9d053abd06376233cb4f3bf84
and includes:
Patch1: patch-2.6.21-rc5.bz2 Patch2: linux-2.6.21-rc5-upstream.patch Patch3: linux-2.6.21-rc5-roland.patch
Thanks, Roland
Index: Makefile =================================================================== RCS file: /cvs/dist/rpms/kernel/devel/Makefile,v retrieving revision 1.45 diff -u -B -p -r1.45 Makefile --- Makefile 19 Mar 2007 21:32:30 -0000 1.45 +++ Makefile 30 Mar 2007 21:45:57 -0000 @@ -68,3 +68,119 @@ reconfig:
# since i386 isn't a target... compile compile-short: DIST_DEFINES += --target $(shell uname -m) + +# +# Hacks for building vanilla (unpatched) kernel rpms. +# Use "make vanilla-TARGET" like "make TARGET" (make vanilla-scratch-build). +# +vanilla-%: $(SPECFILE:.spec=-vanilla.spec) + @$(MAKE) $* SPECFILE=$< + +$(SPECFILE:.spec=-vanilla.spec): $(SPECFILE) + @rm -f $@ + (echo %define nopatches 1; cat $<) > $@ + +scratch-build: test-srpm + $(BUILD_CLIENT) $(BUILD_FLAGS) --scratch $(COLLECTION) \ + $(SRCRPMDIR)/$(NAME)-$(VERSION)-$(RELEASE).src.rpm + +# Dismal kludge for building via brew from cvs after "make vanilla-tag". +ifdef BEEHIVE_SRPM_BUILD +export CHECKOUT_TAG ?= $(shell sed s/^.// CVS/Tag) +tag-pattern = $(TAG_NAME)-$(TAG_VERSION)-0_%_$(TAG_RELEASE) +ifeq (,$(filter-out $(tag-pattern),$(CHECKOUT_TAG))) +variant := $(patsubst $(tag-pattern),%,$(CHECKOUT_TAG)) +srpm: SPECFILE := $(wildcard $(SPECFILE:.spec=-$(variant).spec) \ + $(SPECFILE:.spec=-t.$(variant).spec)) +srpm beehive-sprm: RELEASE := 0.$(variant).$(RELEASE) +endif +endif + + +# +# Hacks for building kernel rpms from upstream code plus local GIT branches. +# Use "make git/BRANCH/TARGET" like "make TARGET". +# Use "make git/BRANCH-fedora/TARGET" to include Fedora patches on top. +# +ifndef GIT_SPEC +git/%: + @$(MAKE) GIT_SPEC=$(subst /,-,$(*D)) git-$(*F) +else +git-%: $(SPECFILE:.spec=-t.$(GIT_SPEC).spec) + @$(MAKE) GIT_SPEC= $* SPECFILE=$< +endif + +# +# Your git-branches.mk file can define GIT_DIR, e.g.: +# GIT_DIR = ${HOME}/kernel/.git +# Make sure GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL are also set +# or your rpm changelogs will look like crap. +# +# For each branch it can define a variable branch-BRANCH or tag-BRANCH +# giving the parent of BRANCH to diff against in a separate patch. If +# the parent is unknown, it will use $(branch-upstream) defaulting to +# "upstream". +# +# Defining tag-BRANCH means the tag corresponds to an upstream patch in +# the sources file, so that is used instead of generating a patch with +# git. If there is no tag-upstream defined, it will figure out a vNNN +# tag or vNNN-gitN pseudo-tag from the last patch in the sources file. +# For example: +# tag-some-hacks = v2.6.21-rc5 +# branch-more-hacks = first-hacks +# Leads to patches: +# git diff v2.6.21-rc5..more-hacks > linux-2.6.21-rc5-some-hacks.patch +# git diff some-hacks..more-hacks > linux-2.6.21-rc5-more-hacks.patch +# Whereas having no git-branches.mk at all but doing +# "make GIT_DIR=... git/mybranch/test-srpm" does: +# id=`cat patch-2.6.21-rc5-git4.id` # auto-fetched via upstream file +# git diff $id..upstream > linux-2.6.21-rc5-git4-upstream.patch +# git diff upstream..mybranch > linux-2.6.21-rc5-git4-mybranch.patch +# If the upstream patch (or any branch patch) is empty it's left out. +# +git-branches.mk:; +-include git-branches.mk + +branch-upstream ?= upstream + +ifdef GIT_DIR +export GIT_DIR +export GIT_AUTHOR_NAME +export GIT_AUTHOR_EMAIL +gen-patches ?= gen-patches + +ifndef havespec +$(SPECFILE:.spec=-t.%-fedora.spec): $(SPECFILE) $(gen-patches) FORCE + ./$(gen-patches) --fedora < $< > $@ $(gen-patches-args) +$(SPECFILE:.spec=-t.%.spec): $(SPECFILE) $(gen-patches) FORCE + ./$(gen-patches) < $< > $@ $(gen-patches-args) +.PRECIOUS: $(SPECFILE:.spec=-t.%.spec) $(SPECFILE:.spec=-t.%-fedora.spec) +endif + +spec-%: $(SPECFILE:.spec=-t.%.spec) ; +$(SPECFILE):; +FORCE:; + +gen-patches-args = ${srcdir} v$(VERSION) $(call heads,$*) +define heads +$(if $(tag-$1),$(filter-out v$(VERSION),$(tag-$1)),\ + $(call heads,$(firstword $(branch-$1) $(branch-upstream)))) $1 +endef + +files-%-fedora: + @echo $(SPECFILE:.spec=-t.$*-fedora.spec) + @$(call list-patches,$*) +files-%: + @echo $(SPECFILE:.spec=-t.$*.spec) + @$(call list-patches,$*) +define list-patches +$(if $(tag-$1),version=$(patsubst v%,%,$(tag-$1)),\ + $(call list-patches,$(firstword $(branch-$1) $(branch-upstream)))); \ +echo linux-$${version}-$1.patch +endef + +ifndef tag-$(branch-upstream) +tag-$(branch-upstream) := $(shell \ + sed -n 's/^.* *//;s/.bz2$$//;s/patch-/v/p' sources) +endif +endif Index: kernel-2.6.spec =================================================================== RCS file: /cvs/dist/rpms/kernel/devel/kernel-2.6.spec,v retrieving revision 1.3025 diff -u -B -p -r1.3025 kernel-2.6.spec --- kernel-2.6.spec 29 Mar 2007 00:00:55 -0000 1.3025 +++ kernel-2.6.spec 30 Mar 2007 21:45:57 -0000 @@ -32,7 +32,8 @@ Summary: The Linux kernel (the core of t %define sublevel 20 %define kversion 2.6.%{sublevel} %define rpmversion 2.6.%{sublevel} -%define release %(R="$Revision: 1.3025 $"; RR="${R##: }"; echo ${RR%%?})%{?dist} +%define specrelease %(R="$Revision: 1.3025 $"; RR="${R##: }"; echo ${RR%%?})%{?dist} +%define release %{specrelease}
%define make_target bzImage %define kernel_image x86 @@ -45,6 +46,28 @@ Summary: The Linux kernel (the core of t %define KVERREL %{PACKAGE_VERSION}-%{PACKAGE_RELEASE} %define hdrarch %_target_cpu
+%if 0%{!?nopatches:1} +%define nopatches 0 +%endif + +%if %{nopatches} +%define includexen 0 +%else +%define relsuffix .fedora +%endif + +%define using_upstream_branch 0 +%if 0%{?upstream_branch:1} +%define using_upstream_branch 1 +%define release 0.%{upstream_branch}%{?relsuffix}.%{specrelease} +%define buildxen 0 +%define buildxenPAE 0 +%else +%if %{nopatches} +%define release 0.vanilla.%{specrelease} +%endif +%endif + # groups of related archs #OLPC stuff %if 0%{?olpc} @@ -169,6 +192,15 @@ Summary: The Linux kernel (the core of t %define xen_image vmlinux.gz %endif
+%if %{nopatches} +%define signmodules 0 +# Ignore unknown options in our config-* files. +# Some options go with patches we're not applying. +%define oldconfig_target loose_nonint_oldconfig +%else +%define oldconfig_target nonint_oldconfig +%endif + # To temporarily exclude an architecture from being built, add it to # %nobuildarches. Do _NOT_ use the ExclusiveArch: line, because if we # don't build kernel-headers then the new build system will no longer let @@ -304,7 +336,15 @@ Source82: config-olpc-generic # # Patches 0 through 100 are meant for core subsystem upgrades # + +%if %{using_upstream_branch} +### BRANCH PATCH ### +%else +# Here should be only the patches up to the upstream canonical Linus tree. Patch1: patch-2.6.21-rc5.bz2 +%endif + +%if !%{nopatches} Patch3: git-geode.patch
# Patches 10 through 99 are for things that are going upstream really soon. @@ -360,7 +400,9 @@ Patch341: linux-2.6-mpc52xx-fec.patch # Patches 800 through 899 are reserved for bugfixes to the core system # and patches related to how RPMs are build # -Patch800: linux-2.6-build-nonintconfig.patch +%endif +Patch800: linux-2.6.17-nonintconfig.patch +%if !%{nopatches}
# Exec-shield. Patch810: linux-2.6-execshield.patch @@ -495,6 +537,7 @@ Patch20001: xen-11668-hvm_disable_fix.pa Patch20002: xen-dom0-reboot.patch
# END OF PATCH DEFINITIONS +%endif
BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root-%{_target_cpu}
@@ -867,9 +910,16 @@ cp -rl vanilla linux-%{kversion}.%{_targ
cd linux-%{kversion}.%{_target_cpu}
+%if %{using_upstream_branch} +### BRANCH APPLY ### +%else + # Update to latest upstream. %patch1 -p1
+%endif +%if !%{nopatches} + # Patches 10 through 100 are meant for core subsystem upgrades
# Roland's utrace ptrace replacement. @@ -945,13 +995,14 @@ cd linux-%{kversion}.%{_target_cpu} # Patches 800 through 899 are reserved for bugfixes to the core system # and patches related to how RPMs are build # - +%endif
# This patch adds a "make nonint_oldconfig" which is non-interactive and # also gives a list of missing options at the end. Useful for automated # builds (as used in the buildsystem). %patch800 -p1
+%if !%{nopatches} # Exec shield %patch810 -p1
@@ -1171,6 +1222,7 @@ mv drivers/xen/blktap/blktap.c drivers/x %patch10001 -p1
# END OF PATCH APPLICATIONS +%endif
cp %{SOURCE10} Documentation/
@@ -1201,7 +1253,7 @@ for i in *.config do mv $i .config Arch=`head -1 .config | cut -b 3-` - make ARCH=$Arch nonint_oldconfig > /dev/null + make ARCH=$Arch %{oldconfig_target} > /dev/null echo "# $Arch" > configs/$i cat .config >> configs/$i done @@ -1295,7 +1347,7 @@ BuildKernel() { KernelImage=arch/$Arch/boot/bzImage fi
- make -s ARCH=$Arch nonint_oldconfig > /dev/null + make -s ARCH=$Arch %{oldconfig_target} > /dev/null make -s ARCH=$Arch %{?_smp_mflags} $MakeTarget %{?sparse_mflags} make -s ARCH=$Arch %{?_smp_mflags} modules %{?sparse_mflags} || exit 1
#!/bin/sh
nopatches=1 if [ "x$1" = "x--fedora" ]; then nopatches=0 shift patchcomment="plus Fedora patches" else patchcomment="no Fedora patches" fi
if [ $# -lt 2 ]; then echo >&2 "Usage: GIT_DIR=REPO $0 TARBALL-TAG [PATCH-TAG...] BRANCH..." exit 2 fi
base=$1 shift
nextpatch=1 usepatch() { patches[$nextpatch]=$1 nextpatch=$(($nextpatch + 1)) }
loglines="- Experimental build from git sources ($patchcomment)\ " log() { logrev=$1 logtext="$(printf %-12s "git $2:")$3" loglines="${loglines}- $(printf %-35s "$logtext") ${logrev}\ " }
patch_headers() { p=1 while [ $p -lt $nextpatch ]; do echo "Patch$p: ${patches[$p]}\" p=$(($p + 1)) done }
patch_apply() { p=1 while [ $p -lt $nextpatch ]; do echo "%patch$p -p1\" p=$(($p + 1)) done }
base_rev() { local base=$1 if [ -r "$GIT_DIR/refs/tags/$base" ]; then tag_rev=`git-rev-parse $base` || { echo >&2 "Cannot find tag $base" exit 2 } return 0 fi case "$1" in v*-git*) local id=patch-${1#v}.id if [ ! -r $id ]; then make download UPSTREAM_FILES=$id UPSTREAM_CHECKS=-- 2> /dev/null 2>&1 fi [ -r $id ] && tag_rev=`cat $id` && return 0 ;; esac return 1 }
while base_rev $1; do base=$1 base_rev=$tag_rev shift usepatch patch-${base#v}.bz2 log $tag_rev tag $base done version=${base#v}
now="`date +'%Y-%m-%d %H:%M %Z'`" for branch; do
merge_base=`git-merge-base $base_rev $branch` || { echo >&2 "No common ancestor for $base and $branch" exit 2 } branch_rev=`git-rev-parse $branch`
file=linux-${version}-${branch}.patch git diff -p -r "${merge_base}" -r "${branch_rev}" > $file || exit if [ ! -s $file ]; then rm -f $file continue fi
usepatch $file log $branch_rev branch $branch
base="$branch" base_rev="$base" done
upstream_branch=`date -u -d "$now" +${branch}.%Y%m%dT%H%M | sed s/-/_/g` logdate=`date -d "$now" +'%a %b %d %Y'`
sed "/%define nopatches/c\ %define nopatches ${nopatches}\ %define upstream_branch ${upstream_branch} /^### BRANCH PATCH/a\ `patch_headers` ### /^### BRANCH APPLY/a\ `patch_apply` ### /^%changelog/a\ * ${logdate} ${GIT_AUTHOR_NAME} <${GIT_AUTHOR_EMAIL}>\ $loglines
"
Roland McGrath wrote:
It is. And I was under the impression that's what Dave was thinking too, but I'll let him speak for himself.
Well, do we want it in the package set? I figured we were talking about "informal" builds that might be put up for ftp, but not be an integrated part of the Fedora world.
This would appear to be question #1 we need to get answered. :)
If one rpmbuild from the spec kicks out -vanilla packages along with other variants, and we turn that on only for rawhide, still it winds up "in the system", people install it as "normal" and wind up with upgrade issues and all that. Eventually we have clueless Fedora users saying "you use -PAE? well -vanilla is the Fedora kernel I like best!" and reporting in bugs "why this fix isn't in -vanilla too?"
I don't think that's too big a concern, we could be 100% firm that we don't apply any patches whatsoever (outside of nonintconfig), its pure upstream.
The main benefit I see is that it would be really nice to spin them off the exact same rpmbuild, since that would mean the only possible variance is the set of patches applied.
However, yeah, it being "in the system" could indeed be annoying. And it could be a bit of unnecessary overhead, resulting in an identical kernel if from one kernel build to another, all that changes is the patches we add.
So perhaps, building "unofficial" builds, of all varieties, does make more sense, and then we just point folks at them when they have issues with our normal kernels and want to compare w/an upstream build.
It's easy to tweak my changes to put "vanilla" into %name instead of %release. I went with "kernel" and a long, funky %release because that's the tradition I've experienced for one-off and test builds people distribute. From my own experience juggling a dozen installations on two or three test boxes with many hack kernels installed each in an utterly disorganized fashion, it seems more likely to get resolved usefully when doing upgrades and such than an rpm named "kernel-foobar". That is, yum and rpm will get in your face all the time in the way that people are used to for juggling kernel rpms. (Also, I always shudder in superstitious fear that yum or who knows what will assume things about the kernel rpm being the way it always has been.)
I kinda like it in %name whichever route we go here. Along the same lines as Ingo's kernel-rt packages, it makes it easier to install them in parallel with normal kernels for testing. Otherwise, a vanilla build we only wanted someone to install for testing would trigger one of the user's two other installed kernels being un-installed with our current default yum setup.
With whatever name, separate vanilla builds could easily be made usable as their own little yum repository.
Okay, I think that's what I'm leaning toward now. :)
What is more work, and I think is more dubious, is having it be the same srpm that produces both fedora and vanilla builds. What will -bp do, make two source trees patched differently?
Good question...
That requires basically duplicating all the innards of the spec file. I don't think that should be attempted without a complete cleanup of the spec file to become sane.
More specifically, what I had in mind was to add just one more variant, so you'd have kernel, kernel-xen, kernel-kdump, kernel-PAE, etc., plus now a kernel-vanilla, which would be analogous to kernel minus patches -- i.e., don't bother with kernel-vanilla-xen, kernel-vanilla-kudmp, etc.
That's no fun! Seriously, I think we need the full suite of variants (modulo xen until that's uptsream, etc.) for it really to be useful. Plenty of people are using -PAE and the point is to compare the foo you're using to foo-vanilla on a one-to-one basis, i.e. different source, same .config settings. And you want -debuginfo for all of those too.
I'd have had vanilla-debuginfo built off the main spec too (plus vanilla-devel and vanilla-headers). But I'm persuaded to go all-out.
Not to say that a spec file sanitization/sane-itization wouldn't be good to do as well (which I'm also perfectly willing to take on -- a few steps in this direction were taken earlier today).
I despise how much duplication there is in there. The debuginfo stuff I originally did with rpm macros got hand-expanded because older rpmbuild couldn't cope with it. Maybe nowadays there is nothing with an rpmbuild older than FC-5 we are trying to work with
And we only have to care about FC-5 for another month or so, in which case I wouldn't bother worrying about anything but FC-6 and later.
so we could use more macros for new stuff. (I think brew got fixed to use the buildroot's rpmbuild, so anything that's fine for the target systems sharing the kernel spec source should be fine.)
Correct, the buildroot should be using the target system's rpmbuild.
rpm macros are a hairy mess, but cleaning things up with clean new macros is a better bet than anything else, IMHO.
Absolutely.
Cool, I'll read your patch more carefully now. :)
Here's the current version of it. You still need the nonintconfig patch variant I posted last time to go with this. I also attached below the shell script used by the GIT-using hacks that are in this version. http://porkchop.devel.redhat.com/brewroot/scratch/roland/task_703101/ shows what "make git/roland/scratch-build" does (the build magic works for "make git/roland-fedora/..." too, but the fedora patches don't apply to current upstream so it didn't build). This (the non-fedora flavor) is similar to the vanilla build, but automagically prepends:
%changelog
- Thu Mar 29 2007 Roland McGrath roland@redhat.com
- Experimental build from git sources (no Fedora patches)
- git tag: v2.6.21-rc5 6fb04ccf5c5e054c4107090bed6e866489f1089f
- git branch: upstream 28defbea64622f69d65a6079bf800cedb9915a5f
- git branch: roland 0ca00c859e7ac4d9d053abd06376233cb4f3bf84
and includes:
Patch1: patch-2.6.21-rc5.bz2 Patch2: linux-2.6.21-rc5-upstream.patch Patch3: linux-2.6.21-rc5-roland.patch
Thanks much, I'll try to digest all this over the weekend, and/or early Monday...
Still interested to hear what approach others think we should take to providing some "vanilla" kernels, but it would seem I'm now on board for your idea, but with "vanilla" (or "upstream" or "whatever") moved from %release to %name.
I kinda like it in %name whichever route we go here. Along the same lines as Ingo's kernel-rt packages, it makes it easier to install them in parallel with normal kernels for testing.
I was just remembering about Ingo's -rt builds. I hadn't looked. What he uses is nearly identical to what I wind up with. It should be easy to use a unified spec file for his -rt patch version too.
I've tweaked my hacks slightly to use name: kernel-vanilla and kernel-branch for the git stuff. For git builds, it now produces e.g. kernel-upstream-2.6.20-2.6.21.rc5.97.1.3025 or kernel-roland-fedora-PAE-2.6.20-2.6.18.rc6.256.1.3025 using the tag and "commit number" from git (1.5) describe, so upgrades for the same branch package should work right without the ginormous date string release numbers I had before. (Those changes are small but I don't have intermediate diffs on hand since I didn't commit the old version. I can send you the new files if you want.)
Thanks, Roland
Roland McGrath wrote:
I kinda like it in %name whichever route we go here. Along the same lines as Ingo's kernel-rt packages, it makes it easier to install them in parallel with normal kernels for testing.
I was just remembering about Ingo's -rt builds. I hadn't looked. What he uses is nearly identical to what I wind up with. It should be easy to use a unified spec file for his -rt patch version too.
Definitely doable. Probably lots more %if'age in the spec, but hey, there's already a ton...
I've tweaked my hacks slightly to use name: kernel-vanilla and kernel-branch for the git stuff. For git builds, it now produces e.g. kernel-upstream-2.6.20-2.6.21.rc5.97.1.3025 or kernel-roland-fedora-PAE-2.6.20-2.6.18.rc6.256.1.3025 using the tag and "commit number" from git (1.5) describe, so upgrades for the same branch package should work right without the ginormous date string release numbers I had before.
If I'm understanding correctly, my current implementation isn't quite as flexible as yours, as it only adds the ability to build kernel-vanilla (and all sub-variants/bits), but it does do so for both released and in-development -rc and -rc-git vanilla kernels. I've been toying around with a few different variations, shuffling %name, %version and %release around a bit, and I think I've come upon one that I'm liking.
My current version outputs a kernel-vanilla-2.6.21-0.rc5.git9.1.fc7 from a 1.3040-based spec, which is (hopefully) obviously 2.6.21-rc5-git9. A 2.6.20.4-based build should wind up being kernel-vanilla-2.6.20-4.1.fc6 using similar spec tweaks, and a forthcoming F7 2.6.21 kernel ought to wind up being kernel-vanilla-2.6.21-0.1.fc7.
(Those changes are small but I don't have intermediate diffs on hand since I didn't commit the old version. I can send you the new files if you want.)
Absolutely, shoot me whatcha got. I still have your prior changes in patch form (as well as a good portion merged into a working copy), extracting the diffs is no prob.
Renaming to kernel-vanilla while still using the same spec does introduce a few problems I hadn't considered...
1) it breaks some build setups, where spec and sources are both somewhere like %_topdir/sources/%name. The build gets confused if %name is now kernel-vanilla. I can work around it locally with a symlink in my build root, but something programmatic would be good. Haven't looked into this much just yet.
2) kernel-xyz-headers is going to conflict with kernel-headers. Perhaps solved by simply 'if kernel-xyz, kernel-xyz-headers Provides: kernel-headers'. I've got this in my latest edition, will see how it behaves...
Despite those issues, I still vote for the %name change.
Definitely doable. Probably lots more %if'age in the spec, but hey, there's already a ton...
Nope, pretty much just the same ones I added.
Absolutely, shoot me whatcha got. I still have your prior changes in patch form (as well as a good portion merged into a working copy), extracting the diffs is no prob.
Index: Makefile =================================================================== RCS file: /cvs/dist/rpms/kernel/devel/Makefile,v retrieving revision 1.45 diff -B -p -u -r1.45 Makefile --- Makefile 19 Mar 2007 21:32:30 -0000 1.45 +++ Makefile 3 Apr 2007 21:02:40 -0000 @@ -68,3 +68,120 @@ reconfig:
# since i386 isn't a target... compile compile-short: DIST_DEFINES += --target $(shell uname -m) + +# +# Hacks for building vanilla (unpatched) kernel rpms. +# Use "make vanilla-TARGET" like "make TARGET" (make vanilla-scratch-build). +# +vanilla-%: $(SPECFILE:.spec=-vanilla.spec) + @$(MAKE) $* SPECFILE=$< + +$(SPECFILE:.spec=-vanilla.spec): $(SPECFILE) + @rm -f $@ + (echo %define nopatches 1; cat $<) > $@ + +scratch-build: test-srpm + $(BUILD_CLIENT) $(BUILD_FLAGS) --scratch $(COLLECTION) \ + $(SRCRPMDIR)/$(NAME)-$(VERSION)-$(RELEASE).src.rpm + +# Dismal kludge for building via brew from cvs after "make vanilla-tag". +ifdef BEEHIVE_SRPM_BUILD +export CHECKOUT_TAG ?= $(shell sed s/^.// CVS/Tag) +tag-pattern = $(TAG_NAME)-$(TAG_VERSION)-0_%_$(TAG_RELEASE) +ifeq (,$(filter-out $(tag-pattern),$(CHECKOUT_TAG))) +variant := $(patsubst $(tag-pattern),%,$(CHECKOUT_TAG)) +srpm: SPECFILE := $(wildcard $(SPECFILE:.spec=-$(variant).spec) \ + $(SPECFILE:.spec=-t.$(variant).spec)) +srpm beehive-sprm: RELEASE := 0.$(variant).$(RELEASE) +endif +endif + + +# +# Hacks for building kernel rpms from upstream code plus local GIT branches. +# Use "make git/BRANCH/TARGET" like "make TARGET". +# Use "make git/BRANCH-fedora/TARGET" to include Fedora patches on top. +# +ifndef GIT_SPEC +git/%: + @$(MAKE) GIT_SPEC=$(subst /,-,$(*D)) git-$(*F) +else +git-%: $(SPECFILE:.spec=-t.$(GIT_SPEC).spec) + @$(MAKE) GIT_SPEC= $* SPECFILE=$< +endif + +# +# Your git-branches.mk file can define GIT_DIR, e.g.: +# GIT_DIR = ${HOME}/kernel/.git +# Make sure GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL are also set +# or your rpm changelogs will look like crap. +# +# For each branch it can define a variable branch-BRANCH or tag-BRANCH +# giving the parent of BRANCH to diff against in a separate patch. If +# the parent is unknown, it will use $(branch-upstream) defaulting to +# "upstream". +# +# Defining tag-BRANCH means the tag corresponds to an upstream patch in +# the sources file, so that is used instead of generating a patch with +# git. If there is no tag-upstream defined, it will figure out a vNNN +# tag or vNNN-gitN pseudo-tag from the last patch in the sources file. +# For example: +# tag-some-hacks = v2.6.21-rc5 +# branch-more-hacks = first-hacks +# Leads to patches: +# git diff v2.6.21-rc5..more-hacks > linux-2.6.21-rc5-some-hacks.patch +# git diff some-hacks..more-hacks > linux-2.6.21-rc5-more-hacks.patch +# Whereas having no git-branches.mk at all but doing +# "make GIT_DIR=... git/mybranch/test-srpm" does: +# id=`cat patch-2.6.21-rc5-git4.id` # auto-fetched via upstream file +# git diff $id..upstream > linux-2.6.21-rc5-git4-upstream.patch +# git diff upstream..mybranch > linux-2.6.21-rc5-git4-mybranch.patch +# If the upstream patch (or any branch patch) is empty it's left out. +# +git-branches.mk:; +-include git-branches.mk + +branch-upstream ?= upstream + +ifdef GIT_DIR +export GIT_DIR +export GIT_AUTHOR_NAME +export GIT_AUTHOR_EMAIL +gen-patches ?= gen-patches + +ifndef havespec +$(SPECFILE:.spec=-t.%-fedora.spec): $(SPECFILE) $(gen-patches) FORCE + ./$(gen-patches) --fedora < $< > $@ $(gen-patches-args) +$(SPECFILE:.spec=-t.%.spec): $(SPECFILE) $(gen-patches) FORCE + ./$(gen-patches) < $< > $@ $(gen-patches-args) +.PRECIOUS: $(SPECFILE:.spec=-t.%.spec) $(SPECFILE:.spec=-t.%-fedora.spec) +endif + +spec-%: $(SPECFILE:.spec=-t.%.spec) ; +$(SPECFILE):; +FORCE:; + +branch-of-* = $(firstword $(head-$*) $*) +gen-patches-args = --name $* v$(VERSION) $(call heads,$(branch-of-*)) +define heads +$(if $(tag-$1),$(filter-out v$(VERSION),$(tag-$1)),\ + $(call heads,$(firstword $(branch-$1) $(branch-upstream)))) $1 +endef + +files-%-fedora: + @echo $(SPECFILE:.spec=-t.$*-fedora.spec) + @$(call list-patches,$(branch-of-*)) +files-%: + @echo $(SPECFILE:.spec=-t.$*.spec) + @$(call list-patches,$(branch-of-*)) +define list-patches +$(if $(tag-$1),version=$(patsubst v%,%,$(tag-$1)),\ + $(call list-patches,$(firstword $(branch-$1) $(branch-upstream)))); \ +echo linux-$${version}-$1.patch +endef + +ifndef tag-$(branch-upstream) +tag-$(branch-upstream) := $(shell \ + sed -n 's/^.* *//;s/.bz2$$//;s/patch-/v/p' sources) +endif +endif Index: kernel-2.6.spec =================================================================== RCS file: /cvs/dist/rpms/kernel/devel/kernel-2.6.spec,v retrieving revision 1.3040 diff -B -p -u -r1.3040 kernel-2.6.spec --- kernel-2.6.spec 2 Apr 2007 19:39:38 -0000 1.3040 +++ kernel-2.6.spec 3 Apr 2007 21:02:40 -0000 @@ -60,7 +60,8 @@ Summary: The Linux kernel (the core of t %define sublevel 20 %define kversion 2.6.%{sublevel} %define rpmversion 2.6.%{sublevel} -%define release %(R="$Revision: 1.3040 $"; RR="${R##: }"; echo ${RR%%?})%{?dist}%{?buildid} +%define specrelease %(R="$Revision: 1.3040 $"; RR="${R##: }"; echo ${RR%%?})%{?dist}%{?buildid} +%define release %{specrelease}
%define make_target bzImage %define kernel_image x86 @@ -73,6 +74,24 @@ Summary: The Linux kernel (the core of t %define KVERREL %{PACKAGE_VERSION}-%{PACKAGE_RELEASE} %define hdrarch %_target_cpu
+%if 0%{!?nopatches:1} +%define nopatches 0 +%endif + +%if %{nopatches} +%define includexen 0 +%define variant -vanilla +%else +%define variant_fedora -fedora +%endif + +%define using_upstream_branch 0 +%if 0%{?upstream_branch:1} +%define using_upstream_branch 1 +%define variant -%{upstream_branch}%{?variant_fedora} +%define release %{upstream_branch_release}.%{specrelease} +%endif + # if requested, only build base kernel %if %{with_baseonly} %define with_smp 0 @@ -237,6 +256,15 @@ Summary: The Linux kernel (the core of t %define xen_image vmlinux.gz %endif
+%if %{nopatches} +%define signmodules 0 +# Ignore unknown options in our config-* files. +# Some options go with patches we're not applying. +%define oldconfig_target loose_nonint_oldconfig +%else +%define oldconfig_target nonint_oldconfig +%endif + # To temporarily exclude an architecture from being built, add it to # %nobuildarches. Do _NOT_ use the ExclusiveArch: line, because if we # don't build kernel-headers then the new build system will no longer let @@ -285,7 +313,7 @@ Summary: The Linux kernel (the core of t # %define kernel_prereq fileutils, module-init-tools, initscripts >= 8.11.1-1, mkinitrd >= 6.0.4-1
-Name: kernel +Name: kernel%{?variant} Group: System Environment/Kernel License: GPLv2 Version: %{rpmversion} @@ -373,8 +401,16 @@ Source82: config-olpc-generic # # Patches 0 through 100 are meant for core subsystem upgrades # + +%if %{using_upstream_branch} +### BRANCH PATCH ### +%else +# Here should be only the patches up to the upstream canonical Linus tree. Patch1: patch-2.6.21-rc5.bz2 Patch2: patch-2.6.21-rc5-git9.bz2 +%endif + +%if !%{nopatches} Patch3: git-geode.patch
# Patches 10 through 99 are for things that are going upstream really soon. @@ -448,7 +484,9 @@ Patch366: linux-2.6-ps3av-export-header. # Patches 800 through 899 are reserved for bugfixes to the core system # and patches related to how RPMs are build # -Patch800: linux-2.6-build-nonintconfig.patch +%endif +Patch800: linux-2.6.17-nonintconfig.patch +%if !%{nopatches}
# Exec-shield. Patch810: linux-2.6-execshield.patch @@ -582,6 +620,7 @@ Patch20001: xen-11668-hvm_disable_fix.pa Patch20002: xen-dom0-reboot.patch
# END OF PATCH DEFINITIONS +%endif
BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root-%{_target_cpu}
@@ -959,10 +998,17 @@ cp -rl vanilla linux-%{kversion}.%{_targ
cd linux-%{kversion}.%{_target_cpu}
+%if %{using_upstream_branch} +### BRANCH APPLY ### +%else + # Update to latest upstream. %patch1 -p1 %patch2 -p1
+%endif +%if !%{nopatches} + # Patches 10 through 100 are meant for core subsystem upgrades
# Roland's utrace ptrace replacement. @@ -1056,13 +1102,14 @@ cd linux-%{kversion}.%{_target_cpu} # Patches 800 through 899 are reserved for bugfixes to the core system # and patches related to how RPMs are build # - +%endif
# This patch adds a "make nonint_oldconfig" which is non-interactive and # also gives a list of missing options at the end. Useful for automated # builds (as used in the buildsystem). %patch800 -p1
+%if !%{nopatches} # Exec shield %patch810 -p1
@@ -1280,6 +1327,7 @@ mv drivers/xen/blktap/blktap.c drivers/x %patch10001 -p1
# END OF PATCH APPLICATIONS +%endif
cp %{SOURCE10} Documentation/
@@ -1310,7 +1358,7 @@ for i in *.config do mv $i .config Arch=`head -1 .config | cut -b 3-` - make ARCH=$Arch nonint_oldconfig > /dev/null + make ARCH=$Arch %{oldconfig_target} > /dev/null echo "# $Arch" > configs/$i cat .config >> configs/$i done @@ -1404,7 +1452,7 @@ BuildKernel() { KernelImage=arch/$Arch/boot/bzImage fi
- make -s ARCH=$Arch nonint_oldconfig > /dev/null + make -s ARCH=$Arch %{oldconfig_target} > /dev/null make -s ARCH=$Arch %{?_smp_mflags} $MakeTarget %{?sparse_mflags} make -s ARCH=$Arch %{?_smp_mflags} modules %{?sparse_mflags} || exit 1
#!/bin/sh
nopatches=1 if [ "x$1" = "x--fedora" ]; then nopatches=0 shift patchcomment="plus Fedora patches" else patchcomment="no Fedora patches" fi
name= if [ "x$1" = "x--name" ]; then shift name="$1" shift fi
if [ $# -lt 2 ]; then echo >&2 "Usage: GIT_DIR=REPO $0 [--fedora] [--name NAME]\ TARBALL-TAG [PATCH-TAG...] BRANCH..." exit 2 fi
base=$1 shift
nextpatch=1 usepatch() { patches[$nextpatch]=$1 nextpatch=$(($nextpatch + 1)) }
loglines="- Experimental build from git sources ($patchcomment)\ " log() { logrev=$1 logtext="$(printf %-12s "git $2:")$3" loglines="${loglines}- $(printf %-35s "$logtext") ${logrev}\ " }
patch_headers() { p=1 while [ $p -lt $nextpatch ]; do echo "Patch$p: ${patches[$p]}\" p=$(($p + 1)) done }
patch_apply() { p=1 while [ $p -lt $nextpatch ]; do echo "%patch$p -p1\" p=$(($p + 1)) done }
base_rev() { local base=$1 if [ -r "$GIT_DIR/refs/tags/$base" ]; then tag_rev=`git-rev-parse $base` || { echo >&2 "Cannot find tag $base" exit 2 } return 0 fi case "$1" in v*-git*) local id=patch-${1#v}.id if [ ! -r $id ]; then make download UPSTREAM_FILES=$id UPSTREAM_CHECKS=-- 2> /dev/null 2>&1 fi [ -r $id ] && tag_rev=`cat $id` && return 0 ;; esac return 1 }
while base_rev $1; do base=$1 base_rev=$tag_rev shift usepatch patch-${base#v}.bz2 log $tag_rev tag $base done version=${base#v}
now="`date +'%Y-%m-%d %H:%M %Z'`"
for branch; do
merge_base=`git-merge-base $base_rev $branch` || { echo >&2 "No common ancestor for $base and $branch" exit 2 } branch_rev=`git-rev-parse $branch`
file=linux-${version}-${branch}.patch git diff -p -r "${merge_base}" -r "${branch_rev}" > $file || exit if [ ! -s $file ]; then rm -f $file continue fi
usepatch $file log $branch_rev branch $branch
base="$branch" base_rev="$base" done name=`echo ${name:-${branch}} | sed s/-/_/g`
#upstream_branch=`date -u -d "$now" +${branch}.%Y%m%dT%H%M | sed s/-/_/g` upstream_branch=$name branch_rev=`git describe $base_rev | sed 's/-g[0-9a-f]*$//;s/-/./g;s/^v//'`
logdate=`date -d "$now" +'%a %b %d %Y'`
sed "/%define nopatches/c\ %define nopatches ${nopatches}\ %define upstream_branch ${name}\ %define upstream_branch_release ${branch_rev} /^### BRANCH PATCH/a\ `patch_headers` ### /^### BRANCH APPLY/a\ `patch_apply` ### /^%changelog/a\ * ${logdate} ${GIT_AUTHOR_NAME} <${GIT_AUTHOR_EMAIL}>\ $loglines
"
Roland McGrath wrote:
Definitely doable. Probably lots more %if'age in the spec, but hey, there's already a ton...
Nope, pretty much just the same ones I added.
Ah, sorry, I see where the disconnect is here... I've not taken some of the build system side of things into account, and was trying to do things solely via spec mods. I think I now understand the necessity of Makefile hacks too. :\
kernel@lists.fedoraproject.org