From: Don Zickus on gitlab.com Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
This change uses the update git push changes for ark-create-release.sh. The behavior moves the git push for gitlab-ci.yml to the script itself. No expected behavior fallout, but there are some odd legacy cases that may cause a subtle change.
Dependent on part2.
Signed-off-by: Don Zickus dzickus@redhat.com
--- redhat/scripts/ci/ark-ci-env.sh | 30 +++++++++++ redhat/scripts/ci/ark-create-release.sh | 74 +++++++++++----------------- redhat/scripts/ci/ark-update-configs.sh | 85 ++++++++++++++++++-------------- .gitlab-ci.yml | 12 +---- 4 files changed, 108 insertions(+), 93 deletions(-)
From: Don Zickus dzickus@redhat.com
CI: Introduce simple environment script
Simple code re-use across the ci scripts. The same variables and setup functions are used. Update the ci scripts to source the new script.
This changes the API to ark-create-release.sh, so modify the call to the script to accept the defaults, which mimics current behaviour.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index blahblah..blahblah 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -264,7 +264,7 @@ rawhide_release: script: - git checkout --track origin/ark-latest && git describe - git checkout --track origin/ark-infra && git describe - - redhat/scripts/ci/ark-create-release.sh "master" "$CI_PROJECT_ID" || exit_code=$? + - redhat/scripts/ci/ark-create-release.sh || exit_code=$? - if [ $exit_code -eq 3 ]; then echo "Tag exists, halting" && exit 0; fi; - if [ $exit_code -ne 0 ]; then echo "Unable to create release tag" && exit 1; fi; - git push gitlab os-build diff --git a/redhat/scripts/ci/ark-ci-env.sh b/redhat/scripts/ci/ark-ci-env.sh new file mode 100644 index blahblah..blahblah 100644 --- /dev/null +++ b/redhat/scripts/ci/ark-ci-env.sh @@ -0,0 +1,30 @@ +#! /bin/sh + +die() +{ + echo "$1" + exit 1 +} + +ci_pre_check() +{ + if test -n "${TO_PUSH}" && test -z "${GITLAB_URL}"; then + die "Please run 'git remote add gitlab <url>' to enable git-push." + fi + git diff-index --quiet HEAD || die "Dirty tree, please clean before merging." +} + +# Common variables for all CI scripts +UPSTREAM_REF=${1:-"master"} +BRANCH=${2:-"os-build"} +PROJECT_ID=${PROJECT_ID:-"13604247"} +TO_PUSH=${DIST_PUSH:-""} +GITLAB_URL="$(git remote get-url gitlab 2>/dev/null)" + +ci_pre_check + +export UPSTREAM_REF +export BRANCH +export PROJECT_ID +export TO_PUSH +export GITLAB_URL diff --git a/redhat/scripts/ci/ark-create-release.sh b/redhat/scripts/ci/ark-create-release.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-create-release.sh +++ b/redhat/scripts/ci/ark-create-release.sh @@ -7,8 +7,9 @@
set -e
-UPSTREAM_REF=${1-master} -PROJECT_ID=${2:-13604247} +# source common CI functions and variables +# shellcheck source=./redhat/scripts/ci/ark-ci-env.sh +. "$(dirname "$0")"/ark-ci-env.sh
# Detect if there's one or more prior releases for this upstream ref. if git describe "$UPSTREAM_REF" | grep -q -c '-g'; then diff --git a/redhat/scripts/ci/ark-update-configs.sh b/redhat/scripts/ci/ark-update-configs.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-update-configs.sh +++ b/redhat/scripts/ci/ark-update-configs.sh @@ -18,8 +18,9 @@
set -e
-UPSTREAM_REF=${1:-master} -test -n "$PROJECT_ID" || PROJECT_ID="${2:-13604247}" +# source common CI functions and variables +# shellcheck source=./redhat/scripts/ci/ark-ci-env.sh +. "$(dirname "$0")"/ark-ci-env.sh
ISSUE_DESCRIPTION="A merge conflict has occurred and must be resolved manually.
@@ -31,8 +32,7 @@ To resolve this, do the following: 4. git push "
-git checkout os-build -BRANCH="$(git branch --show-current)" +git checkout "${BRANCH}" if ! git merge -m "Merge '$UPSTREAM_REF' into '$BRANCH'" "$UPSTREAM_REF"; then git merge --abort printf "Merge conflict; halting!\n"
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: Remove legacy tag check cruft
Originally code was added to handle the script being repeatedly called by detecting an existing tag and aborting to prevent a mess.
Over the years we removed the ark-patches, did more git merges and strengthened dist-release. This simplified the code paths and remove a lot of fragile cases.
The original checks are unnecessary now that 'make dist-release' blocks any changes from going in. 'make dist-release-tag' will also fail on existing tags.
Remove the complicated check and rely on existing mechanisms.
In addition, ark-patches and its release structure died a couple of years ago, remove that cruft too. Instead just do the detached head for ark-latest.
Also removed hardcoded 'os-build' and replaced with a variable in prep for other branches to use this script (like rt and automotive).
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-create-release.sh b/redhat/scripts/ci/ark-create-release.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-create-release.sh +++ b/redhat/scripts/ci/ark-create-release.sh @@ -11,40 +11,13 @@ set -e # shellcheck source=./redhat/scripts/ci/ark-ci-env.sh . "$(dirname "$0")"/ark-ci-env.sh
-# Detect if there's one or more prior releases for this upstream ref. -if git describe "$UPSTREAM_REF" | grep -q -c '-g'; then - SHORT_COMMIT=$(git describe "$UPSTREAM_REF" | cut -d "g" -f 2) - BASE_RELEASE=$(git tag -l | grep "$SHORT_COMMIT" | tail -n 1) -else - if git describe "$UPSTREAM_REF" | grep -q -c "-"; then - RC_LEVEL="0.$(git describe "$UPSTREAM_REF" | cut -d "-" -f 2)" - VERSION=$(git describe "$UPSTREAM_REF" | cut -d "-" -f 1 | cut -c 2-) - else - RC_LEVEL="" - VERSION=$(git describe "$UPSTREAM_REF" | cut -c 2-) - fi - BASE_RELEASE=$(git tag -l | grep -E "kernel-$VERSION.0-$RC_LEVEL.[0-9]+" | tail -n 1) -fi -if [ -n "$BASE_RELEASE" ]; then - printf "There's already a release for %s (tagged as %s); if you're trying \ - to create a new release check out that tag, apply any commits you \ - want, and then run "touch localversion && make dist-release && make \ - dist-release-tag".\n" "$UPSTREAM_REF" "$BASE_RELEASE" - exit 3 -fi - -git checkout os-build +git checkout "${BRANCH}" touch localversion make dist-release
-if git tag -v "$UPSTREAM_REF" > /dev/null 2>&1; then - git checkout -b ark/"$UPSTREAM_REF" ark/patches/"$UPSTREAM_REF" - RELEASE_BRANCHES=" ark/$UPSTREAM_REF ark/patches/$UPSTREAM_REF" -else - # This is a snapshot release so we're only going to make a tag - git checkout --detach os-build && git describe - RELEASE_BRANCHES="" -fi +# prep ark-latest branch +git checkout --detach "${BRANCH}" && git describe + MR_PATCHES=$(gitlab project-merge-request list --project-id="$PROJECT_ID" \ --labels="Include in Releases" --state=opened | grep -v "^$" | sort | \ awk '{ print "https://gitlab.com/cki-project/kernel-ark/-/merge_requests/" $2 ".patch" }')
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: ark-create-release: Add a robust check if nothing changed
There are rare cases where upstream hasn't updated and no new 'include in release' patches were created. The nightly cron job may stumble upon no new changes. This is blocked by the failure of 'make dist-release-tag'.
Make this a little cleaner by gracefully detecting and exiting.
We do this by saving the original sha head and comparing it to ark-latest's sha head. If it hasn't changed then neither os-build nor ark-latest added new code. This mean ark-infra won't change either and there is nothing to tag. Just exit early.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-create-release.sh b/redhat/scripts/ci/ark-create-release.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-create-release.sh +++ b/redhat/scripts/ci/ark-create-release.sh @@ -13,6 +13,7 @@ set -e
git checkout "${BRANCH}" touch localversion +old_head="$(git rev-parse HEAD)" make dist-release
# prep ark-latest branch @@ -25,7 +26,16 @@ for patch_url in $MR_PATCHES; do curl -sL "$patch_url" | git am done
+# if dist-release doesn't update anything, then there is a good chance the +# tag already exists and infra changes have already been applied. Let's +# skip those conditions and exit gracefully. make dist-release +new_head="$(git rev-parse HEAD)" +if test "$old_head" == "$new_head"; then + echo "Nothing changed, skipping updates" + exit 0 +fi + make dist-release-tag RELEASE=$(git describe) git checkout ark-latest
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: sync ark-create-release push commands with output
A user can run the ci scripts and either auto push the changes or not. If the changes are not pushed, a message is printed explaining the steps needed to properly push the changes.
Over time the real push and the output got out of sync. Just wrap them in variables and use eval to execute them. This should keep things in sync for a good while.
Also, notice some cruft was modified. The output did not match the reality inside the .gitlab-ci.yml file. And the RELEASE stuff was never used.
The only things being pushed are: BRANCH(os-build), release tag, ark-latest, and ark-infra.
No behaviour changes.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-create-release.sh b/redhat/scripts/ci/ark-create-release.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-create-release.sh +++ b/redhat/scripts/ci/ark-create-release.sh @@ -37,7 +37,7 @@ if test "$old_head" == "$new_head"; then fi
make dist-release-tag -RELEASE=$(git describe) +RELEASE=$(git describe) #grab the tag git checkout ark-latest git reset --hard "$RELEASE"
@@ -59,14 +59,16 @@ git add makefile Makefile.rhelver Makefile redhat
git commit -m "bulk merge ark-infra as of $(date)"
-printf "All done! +echo +test "$TO_PUSH" && PUSH_VERB="Pushing" || PUSH_VERB="To push" +PUSH_STR="all release artifacts" +PUSH_CMD="git push gitlab ${BRANCH} && \ + git push gitlab "${RELEASE}" && \ + git push -f gitlab ark-latest && \ + git push -f gitlab ark-infra +"
-To push all the release artifacts, run: - -git push os-build -for branch in $(git branch | grep configs/"$(date +%%F)"); do -\tgit push -o merge_request.create -o merge_request.target=os-build\ - -o merge_request.remove_source_branch upstream "$branch" -done -git push upstream %s%s -git push -f upstream ark-latest\n" "$RELEASE" "$RELEASE_BRANCHES" +#Push branch +echo "# $PUSH_VERB $PUSH_STR" +echo "$PUSH_CMD" +test "$TO_PUSH" && eval "$PUSH_CMD"
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: ark-update-configs: misc changes
Simple changes in preparation to help with readability with later changes.
No behavior changes.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-update-configs.sh b/redhat/scripts/ci/ark-update-configs.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-update-configs.sh +++ b/redhat/scripts/ci/ark-update-configs.sh @@ -22,6 +22,15 @@ set -e # shellcheck source=./redhat/scripts/ci/ark-ci-env.sh . "$(dirname "$0")"/ark-ci-env.sh
+finish() +{ + rm "$TMPFILE" +} +trap finish EXIT + +TMPFILE=".push-warnings" +touch $TMPFILE + ISSUE_DESCRIPTION="A merge conflict has occurred and must be resolved manually.
To resolve this, do the following: @@ -45,7 +54,7 @@ if ! git merge -m "Merge '$UPSTREAM_REF' into '$BRANCH'" "$UPSTREAM_REF"; then --description "$ISSUE_DESCRIPTION" fi fi - exit 1 + die "Merge conflicts" fi
# Generates and commits all the pending configs @@ -60,7 +69,9 @@ new_head="$(git rev-parse HEAD)" # Converts each new pending config from above into its finalized git # configs/<date>/<config> branch. These commits are used for Merge # Requests. -if [ "$old_head" != "$new_head" ]; then +[ "$old_head" != "$new_head" ] && CONFIGS_ADDED="1" || CONFIGS_ADDED="" + +if test "$CONFIGS_ADDED"; then ./redhat/scripts/genspec/gen_config_patches.sh else printf "No new configuration values exposed from merging %s into $BRANCH\n" "$UPSTREAM_REF"
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: ark-update-configs: sync push command and output
A rather large change that just moves everything into variables. The end result is the same except the push command is also echo'd before being eval'd. This allows the script to keep the command and output in sync.
The script relies on the trap feature to cleanup the tempfile if anything goes wrong.
No functional changes.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/redhat/scripts/ci/ark-update-configs.sh b/redhat/scripts/ci/ark-update-configs.sh index blahblah..blahblah 100755 --- a/redhat/scripts/ci/ark-update-configs.sh +++ b/redhat/scripts/ci/ark-update-configs.sh @@ -73,42 +73,40 @@ new_head="$(git rev-parse HEAD)"
if test "$CONFIGS_ADDED"; then ./redhat/scripts/genspec/gen_config_patches.sh + PUSH_VERB="Pushing" else printf "No new configuration values exposed from merging %s into $BRANCH\n" "$UPSTREAM_REF" + PUSH_VERB="To push" fi
-if test -n "$DIST_PUSH"; then - tmpfile=".push-warnings" - touch $tmpfile - - echo "Pushing branch $(git branch --show-current) to $(git remote get-url gitlab)" - git push gitlab HEAD - - echo "Pushing config update branches" - for branch in $(git branch | grep configs/"$(date +%F)"); do - git push \ - -o merge_request.create \ - -o merge_request.target="$BRANCH" \ - -o merge_request.remove_source_branch \ - gitlab "$branch" 2>&1 | tee -a $tmpfile - done - - # GitLab server side warnings do not fail git-push but leave verbose - # WARNING messages. Grep for those and consider it a script - # failure. Make sure all branches are pushed first as follow up - # git-pushes may succeed. - grep -q "remote:[ ]* WARNINGS" $tmpfile && RC=1 || RC=0 - rm $tmpfile - test $RC && exit $RC -else - printf " -To push all the release artifacts, run: +echo +PUSH_STR="branch ${BRANCH} to ${GITLAB_URL}" +PUSH_CMD="git push gitlab ${BRANCH}" +PUSH_CONFIG_STR="config update branches" +PUSH_CONFIG_CMD="for branch in $(git branch | grep configs/"$(date +%F)"); do + git push \ + -o merge_request.create \ + -o merge_request.target="$BRANCH" \ + -o merge_request.remove_source_branch \ + gitlab "$branch" 2>&1 | tee -a $TMPFILE +done +"
-git push gitlab HEAD -for branch in $(git branch | grep configs/"$(date +%%F)"); do -\tgit push -o merge_request.create -o merge_request.target=$BRANCH\ - -o merge_request.remove_source_branch upstream gitlab "$branch" -done\n" +#Push branch +echo "# $PUSH_VERB $PUSH_STR" +echo "$PUSH_CMD" +test "$TO_PUSH" && eval "$PUSH_CMD"
+#Push config branches if created +if test "$CONFIGS_ADDED"; then + echo + echo "# $PUSH_VERB $PUSH_CONFIG_STR" + echo "$PUSH_CONFIG_CMD" + test "$TO_PUSH" && eval "$PUSH_CONFIG_CMD" fi
+# GitLab server side warnings do not fail git-push but leave verbose +# WARNING messages. Grep for those and consider it a script +# failure. Make sure all branches are pushed first as follow up +# git-pushes may succeed. +grep -q "remote:[ ]* WARNINGS" $TMPFILE && die "Server side warnings"
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Don Zickus dzickus@redhat.com
CI: rawhide_release: switch to using script to push
Currently the gitlab-ci.yml script pushes the gitlab changes. Switch to using the script to make testing and debugging easier.
No functional changes, but script pushing is minimally tested.
Signed-off-by: Don Zickus dzickus@redhat.com
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index blahblah..blahblah 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -264,17 +264,7 @@ rawhide_release: script: - git checkout --track origin/ark-latest && git describe - git checkout --track origin/ark-infra && git describe - - redhat/scripts/ci/ark-create-release.sh || exit_code=$? - - if [ $exit_code -eq 3 ]; then echo "Tag exists, halting" && exit 0; fi; - - if [ $exit_code -ne 0 ]; then echo "Unable to create release tag" && exit 1; fi; - - git push gitlab os-build - # ark-create-release.sh creates a tag for ark-latest to be pushed. - # Make sure the ark-latest branch is checked out. - - git checkout ark-latest - - git push gitlab "$(git describe)" - - git push gitlab ark/* - - git push -f gitlab ark-latest - - git push -f gitlab ark-infra + - DIST_PUSH=1 redhat/scripts/ci/ark-create-release.sh retry: 2 rules: - *only-os-build-schedule
-- https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519
From: Scott Weaver on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519#note_1499480...
Resolving this thread based on Michael's approval.
From: Scott Weaver on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2519#note_1499831...
@dzickusrh this needs rebased after the change made in !2624. One thing that might need adjusted in `ark-create-release.sh` is to move the `set -e` to after sourcing `ark-ci-env.sh`. Then we'll see any error messages generated by `ci_pre_check`.
Looking at `ark-create-release.sh`, it seems to me that we shouldn't need !2624 because the first step after sourcing is to switch to `os-build` but I guess there's no guarantee that the version of the script in the `ark-infra` branch is really the version we want to execute. Otherwise, LGTM.
kernel@lists.fedoraproject.org