[SSSD] [RFC] Add basic support for CI test execution

Lukas Slebodnik lslebodn at redhat.com
Wed Jun 11 07:52:28 UTC 2014


On (10/06/14 19:28), Nikolai Kondrashov wrote:
>Hi everyone,
>
>I would like to collect feedback about the attached patch adding support for
>basic CI test runs.
>
>Please ask any questions, give any suggestions and voice any objections you
>think are appropriate.
>
>Thank you.
>
>Nick

>From dbb98dfc935a46d7ba1fefaba5e599758d87f98f Mon Sep 17 00:00:00 2001
>From: Nikolai Kondrashov <Nikolai.Kondrashov at redhat.com>
>Date: Tue, 25 Mar 2014 12:01:00 +0200
>Subject: [PATCH 1/1] Add basic support for CI test execution
>
>Add basic support for executing continuous integration (CI) tests on
>RHEL7, Fedora 20 and Debian Testing.
>
>This adds two front-end scripts which can be executed either locally by
>developers, or on the CI server: contrib/ci/run and contrib/ci/clean.
>The first one would run the tests and the second would wipe out the
>artifacts. Note that contrib/ci/run would itself clean before running.
>
>The contrib/ci/run script accepts options to choose either full or just
>essential set of tests (-f/--full/-e/--essential), with the essential
>set selected by default.
>
>A default local run would produce output like this one (for now):
>
>    install-deps:           success  00:00:01 ci-install-deps.log
>    autoreconf:             success  00:00:15 ci-autoreconf.log
>    DEBUG BUILD:                              ci-build-debug
>    configure:              success  00:00:17 ci-build-debug/ci-configure.log
>    make-tests:             success  00:00:41 ci-build-debug/ci-make-tests.log
>    make-check-valgrind:    failure  00:00:49 ci-build-debug/ci-make-check-valgrind.log
>    SUCCESS
>
>and would complete in just above two minutes on a modern machine.
>
>A full run (contrib/ci/run -f) would produce something like this:
>
>    install-deps:           success  00:00:01 ci-install-deps.log
>    autoreconf:             success  00:00:15 ci-autoreconf.log
>    DEBUG BUILD:                              ci-build-debug
>    configure:              success  00:00:17 ci-build-debug/ci-configure.log
>    make-distcheck:         success  00:04:26 ci-build-debug/ci-make-distcheck.log
>    make-tests:             success  00:00:43 ci-build-debug/ci-make-tests.log
>    make-check-valgrind:    failure  00:00:49 ci-build-debug/ci-make-check-valgrind.log
>    COVERAGE BUILD:                           ci-build-coverage
>    configure:              success  00:00:43 ci-build-coverage/ci-configure.log
>    scan-make-tests:        success  00:05:35 ci-build-coverage/ci-scan-make-tests.log
>    scan report:                              ci-build-coverage/ci-report-scan/index.html
>    scan-check:             failure  00:00:00 ci-build-coverage/ci-scan-check.log
>    lcov-pre:               success  00:00:07 ci-build-coverage/ci-lcov-pre.log
>    make-check:             success  00:00:27 ci-build-coverage/ci-make-check.log
>    lcov-post:              success  00:00:15 ci-build-coverage/ci-lcov-post.log
>    lcov-merge:             success  00:00:01 ci-build-coverage/ci-lcov-merge.log
>    genhtml:                success  00:00:03 ci-build-coverage/ci-genhtml.log
>    coverage report:                          ci-build-coverage/ci-report-coverage/index.html
>    SUCCESS
>
>and would complete in about 15 minutes.
>
>An example of a full run on Jenkins is available on our CI server [1].
>
>The first stage (install-deps) would invoke sudo to install dependencies
>using the appropriate package manager. A relatively strict rule can be
>added to sudoers to avoid password prompts.
>
>Valgrind (make-check-valgrind) and Clang analyzer (scan-check) stage
>failures are displayed, but ignored for now, until the underlying code
>issues are fixed.
>
>Future plans include considering using Epoxy [2] as the framework, to
>increase robustness and flexibility.
>
>[1] http://sssd-ci.idm.lab.eng.brq.redhat.com:8080/job/new_private_master_fedora20/214/console
The host is not accesible. It should not be in commit message.

$ ping sssd-ci.idm.lab.eng.brq.redhat.com
ping: unknown host sssd-ci.idm.lab.eng.brq.redhat.com

>[2] https://github.com/spbnick/epoxy
>---
> Makefile.am                  |   9 +-
> contrib/ci/clean             |  25 ++++
> contrib/ci/configure.sh      | 111 ++++++++++++++++
> contrib/ci/deps.sh           | 142 ++++++++++++++++++++
> contrib/ci/distro.sh         |  68 ++++++++++
> contrib/ci/misc.sh           |  31 +++++
> contrib/ci/run               | 310 +++++++++++++++++++++++++++++++++++++++++++
> contrib/ci/valgrind-condense |  89 +++++++++++++
> 8 files changed, 784 insertions(+), 1 deletion(-)
> create mode 100755 contrib/ci/clean
> create mode 100644 contrib/ci/configure.sh
> create mode 100644 contrib/ci/deps.sh
> create mode 100644 contrib/ci/distro.sh
> create mode 100644 contrib/ci/misc.sh
> create mode 100755 contrib/ci/run
> create mode 100755 contrib/ci/valgrind-condense
>
>diff --git a/Makefile.am b/Makefile.am
>index f64c77a..98c0d13 100644
>--- a/Makefile.am
>+++ b/Makefile.am
>@@ -300,6 +300,9 @@ dist_noinst_SCRIPTS = \
>     contrib/rhel/update_debug_levels.py \
>     contrib/fedora/bashrc_sssd \
>     contrib/fedora/make_srpm.sh \
>+    contrib/ci/clean \
>+    contrib/ci/run \
>+    contrib/ci/valgrind-condense \
>     src/tests/pyhbac-test.py \
>     src/tests/pysss_murmur-test.py
> 
>@@ -310,7 +313,11 @@ dist_noinst_DATA = \
>     src/config/testconfigs/sssd-badversion.conf \
>     src/config/testconfigs/sssd-invalid.conf \
>     src/config/testconfigs/sssd-invalid-badbool.conf \
>-    src/config/etc/sssd.api.d/crash_test_dummy
>+    src/config/etc/sssd.api.d/crash_test_dummy \
>+    contrib/ci/configure.sh \
>+    contrib/ci/deps.sh \
>+    contrib/ci/distro.sh \
>+    contrib/ci/misc.sh
> 
> ###############################
> # Global compilation settings #
>diff --git a/contrib/ci/clean b/contrib/ci/clean
>new file mode 100755
>index 0000000..ee18c10
>--- /dev/null
>+++ b/contrib/ci/clean
>@@ -0,0 +1,25 @@
>+#!/bin/bash
>+#
>+# Clean source tree after a run of integration tests.
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+set -o nounset -o pipefail -o errexit
>+export PATH=`dirname "\`readlink -f \"\$0\"\`"`:$PATH
>+
>+. misc.sh
>+
>+rm_rf_ro ci-*
>diff --git a/contrib/ci/configure.sh b/contrib/ci/configure.sh
>new file mode 100644
>index 0000000..a731604
>--- /dev/null
>+++ b/contrib/ci/configure.sh
>@@ -0,0 +1,111 @@
>+#
>+# Configure option management.
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+if [ -z ${_CONFIGURE_SH+set} ]; then
>+declare -r _CONFIGURE_SH=
>+
>+. distro.sh
>+
>+declare -r _CONFIGURE_ARCH=`uname -m`
>+
>+# List of "configure" arguments.
>+declare -a CONFIGURE_ARG_LIST=(
>+    "--build=$_CONFIGURE_ARCH-linux-gnu"
>+    "--datadir=/usr/share"
>+    "--disable-dependency-tracking"
>+    "--disable-rpath"
>+    "--disable-static"
>+    "--host=$_CONFIGURE_ARCH-linux-gnu"
>+    "--includedir=/usr/include"
>+    "--infodir=/usr/share/info"
>+    "--localstatedir=/var"
>+    "--mandir=/usr/share/man"
>+    "--prefix=/usr"
>+    "--sysconfdir=/etc"
>+    "--with-initscript=systemd"
>+    "--with-syslog=journald"
>+)
>+
>+
>+if [[ "$DISTRO_BRANCH" == -redhat-* ]]; then
>+    declare -r _CONFIGURE_LIB=`rpm --eval '/%{_lib}'`
>+    declare -r _CONFIGURE_USR_LIB=`rpm --eval '%{_libdir}'`
>+    declare -r _CONFIGURE_USR_LIBEXEC=`rpm --eval '%{_libexecdir}'`
>+    CONFIGURE_ARG_LIST+=(
>+        "--bindir=/usr/bin"
>+        "--enable-ldb-version-check"
>+        "--enable-nsslibdir=/$_CONFIGURE_LIB"
>+        "--enable-pammoddir=/$_CONFIGURE_LIB/security"
>+        "--exec-prefix=/usr"
>+        "--libdir=$_CONFIGURE_USR_LIB"
>+        "--libexecdir=$_CONFIGURE_USR_LIBEXEC"
>+        "--program-prefix="
>+        "--sbindir=/usr/sbin"
>+        "--sharedstatedir=/var/lib"
>+        "--with-db-path=/var/lib/sss/db"
>+        "--with-init-dir=/etc/rc.d/init.d"
>+        "--with-krb5-rcache-dir=/var/cache/krb5rcache"
>+        "--with-mcache-path=/var/lib/sss/mc"
>+        "--with-pipe-path=/var/lib/sss/pipes"
>+        "--with-pubconf-path=/var/lib/sss/pubconf"
>+    )
>+fi
>+
>+if [[ "$DISTRO_BRANCH" == -debian-* ]]; then
>+    declare -r _CONFIGURE_MULTIARCH=`dpkg-architecture -qDEB_HOST_MULTIARCH`
>+    declare -r _CONFIGURE_LIB="/lib/$_CONFIGURE_MULTIARCH"
>+    declare -r _CONFIGURE_USR_LIB="/usr/lib/$_CONFIGURE_MULTIARCH"
>+    CONFIGURE_ARG_LIST+=(
>+        "--enable-krb5-locator-plugin"
>+        "--enable-nsslibdir=$_CONFIGURE_LIB"
>+        "--enable-pammoddir=$_CONFIGURE_LIB/security"
>+        "--libdir=$_CONFIGURE_USR_LIB"
>+        "--libexecdir=$_CONFIGURE_USR_LIB"
>+        "--with-autofs"
>+        "--with-krb5-plugin-path=$_CONFIGURE_USR_LIB/krb5/plugins/krb5"
>+        "--with-ldb-lib-dir=$_CONFIGURE_USR_LIB/ldb/modules/ldb"
>+        "--with-ssh"
>+        "--with-sudo"
>+        "--with-systemdunitdir=/lib/systemd/system"
>+    )
>+fi
>+
>+declare -r -a CONFIGURE_ARG_LIST
>+
>+# Format the configure command.
>+# Args: [cmd [arg...]]
>+function configure_format()
>+{
>+    if [ $# == 0 ]; then
>+        printf './configure'
>+    else
>+        printf '%q' "$1"
>+        shift
>+    fi
>+    printf ' %q' "${CONFIGURE_ARG_LIST[@]}" "$@"
>+    printf '\n'
>+}
>+
>+# Run the configure command.
>+# Args: [cmd [arg...]]
>+function configure_run()
>+{
>+    `configure_format "$@"`
>+}
>+
>+fi # _CONFIGURE_SH
>diff --git a/contrib/ci/deps.sh b/contrib/ci/deps.sh
>new file mode 100644
>index 0000000..4517ef5
>--- /dev/null
>+++ b/contrib/ci/deps.sh
>@@ -0,0 +1,142 @@
>+#
>+# Dependency management.
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+if [ -z ${_DEPS_SH+set} ]; then
>+declare -r _DEPS_SH=
>+
>+. distro.sh
>+
>+# Dependency list
>+declare -a DEPS_LIST=(
>+    autoconf
>+    automake
>+    lcov
>+    libtool
>+    valgrind
>+    xqilla
>+)
>+
>+if [[ "$DISTRO_BRANCH" == -redhat-* ]]; then
>+    DEPS_LIST+=(
>+        augeas-devel
>+        bind-utils
>+        c-ares-devel
>+        check-devel
>+        cifs-utils-devel
>+        clang-analyzer
>+        dbus-devel
>+        dbus-libs
>+        docbook-style-xsl
>+        doxygen
>+        findutils
>+        gettext-devel
>+        glib2-devel
>+        keyutils-libs-devel
>+        krb5-devel
>+        libcmocka-devel
>+        libcollection-devel
>+        libdhash-devel
>+        libini_config-devel
>+        libldb-devel
>+        libnl3-devel
>+        libselinux-devel
>+        libsemanage-devel
>+        libsmbclient-devel
>+        libtalloc-devel
>+        libtdb-devel
>+        libtevent-devel
>+        libxml2
>+        libxslt
>+        nspr-devel
>+        nss-devel
>+        openldap-devel
>+        pam-devel
>+        pcre-devel
>+        pkgconfig
>+        popt-devel
>+        python-devel
>+        samba4-devel
>+        selinux-policy-targeted
>+        systemd-devel
>+    )
I would like to avoind maintain two different lists with build dependency for
fedora/rhel based OSes. We already have listed all build dependencies in
sssd spec file. It acn be used with rhel6, rhel7 and recent versions of fedora.
You can install build dependencies with command:
    a) yum-builddep path/to/package.src.rpm
    b) yum-builddep path/to/package.spec

sssd.spec file can be created from contrib/sssd.spec.in with few replacements.
    @see contrib/fedora/make_srpm.sh

I am fine with maintaning development packages for other distributions.

>+fi
>+
>+if [[ "$DISTRO_BRANCH" == -debian-* ]]; then
>+    DEPS_LIST+=(
>+        autopoint
>+        check
>+        cifs-utils
>+        clang
>+        dh-apparmor
>+        dnsutils
>+        docbook-xml
>+        docbook-xsl
>+        gettext
>+        krb5-config
>+        libaugeas-dev
>+        libc-ares-dev
>+        libcmocka-dev
>+        libcollection-dev
>+        libdbus-1-dev
>+        libdhash-dev
>+        libglib2.0-dev
>+        libini-config-dev
>+        libkeyutils-dev
>+        libkrb5-dev
>+        libldap2-dev
>+        libldb-dev
>+        libltdl-dev
>+        libnl-3-dev
>+        libnl-route-3-dev
>+        libnspr4-dev
>+        libnss3-dev
>+        libpam0g-dev
>+        libpcre3-dev
>+        libpopt-dev
>+        libsasl2-dev
>+        libselinux1-dev
>+        libsemanage1-dev
>+        libsmbclient-dev
>+        libsystemd-journal-dev
>+        libtalloc-dev
>+        libtdb-dev
>+        libtevent-dev
>+        libxml2-utils
>+        python-dev
>+        samba-dev
>+        systemd
>+        xml-core
>+        xsltproc
>+    )
>+fi
>+
>+declare -a -r DEPS_LIST
>+
>+# Install dependencies.
>+function deps_install()
>+{
>+    distro_pkg_install "${DEPS_LIST[@]}"
>+}
>+
>+# Remove dependencies.
>+function deps_remove()
>+{
>+    distro_pkg_remove "${DEPS_LIST[@]}"
>+}
>+
>+fi # _DEPS_SH
>diff --git a/contrib/ci/distro.sh b/contrib/ci/distro.sh
>new file mode 100644
>index 0000000..945ab2a
>--- /dev/null
>+++ b/contrib/ci/distro.sh
>@@ -0,0 +1,68 @@
>+#
>+# Distribution version discovery
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+if [ -z ${_DISTRO_SH+set} ]; then
>+declare -r _DISTRO_SH=
>+
>+# Distribution branch (lowercase)
>+declare DISTRO_BRANCH=
>+
>+if [ -e /etc/redhat-release ]; then
>+    DISTRO_BRANCH+=-redhat
>+elif [ -e /etc/debian_version ]; then
>+    DISTRO_BRANCH+=-debian
>+else
>+    DISTRO_BRANCH+=-unknown
>+fi
>+
>+DISTRO_BRANCH+=-`lsb_release --id | sed -e 's/^[^:]*:\s*\(.*\)$/\L\1\E/'`
>+DISTRO_BRANCH+=-`lsb_release --release | sed -e 's/^[^:]*:\s*\(.*\)$/\L\1\E/'`
>+DISTRO_BRANCH+=-
The executable file lsb_release needn't be installed and it will cause
warnings.
sh-4.2$ DISTRO_BRANCH+=-`lsb_release --id | sed -e 's/^[^:]*:\s*\(.*\)$/\L\1\E/'`
sh: lsb_release: command not found

>+
>+declare -r DISTRO_BRANCH
>+
>+
>+# Install packages.
>+# Args: [pkg_name...]
>+function distro_pkg_install()
>+{
>+    if [[ "$DISTRO_BRANCH" == -redhat-* ]]; then
>+        [ $# != 0 ] && sudo yum --assumeyes install -- "$@"
>+    elif [[ "$DISTRO_BRANCH" == -debian-* ]]; then
>+        [ $# != 0 ] && sudo apt-get --yes install -- "$@"
>+    else
>+        echo "Cannot install packages on $DISTRO_BRANCH" >&2
>+        exit 1
>+    fi
>+}
>+
>+# Remove packages.
>+# Args: [pkg_name...]
>+function distro_pkg_remove()
>+{
>+    if [[ "$DISTRO_BRANCH" == -redhat-* ]]; then
>+        [ $# != 0 ] && sudo yum --assumeyes remove -- "$@"
>+    elif [[ "$DISTRO_BRANCH" == -debian-* ]]; then
>+        [ $# != 0 ] && sudo apt-get --yes remove -- "$@"
>+    else
>+        echo "Cannot remove packages on $DISTRO_BRANCH" >&2
>+        exit 1
>+    fi
>+}
>+
>+fi # _DISTRO_SH
>diff --git a/contrib/ci/misc.sh b/contrib/ci/misc.sh
>new file mode 100644
>index 0000000..2c205cd
>--- /dev/null
>+++ b/contrib/ci/misc.sh
>@@ -0,0 +1,31 @@
>+#
>+# Miscellaneous routines.
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+if [ -z ${_MISC_SH+set} ]; then
>+declare -r _MISC_SH=
>+
>+# Remove files and directories recursively, forcing write permissions on
>+# directories.
>+# Args: path...
>+function rm_rf_ro()
>+{
>+    chmod -Rf u+w -- "$@" || true
>+    rm -Rf -- "$@"
If rm is called in recursive mode from script it will be good to print command
to the standard output before execution or run rm in verbose mode.
    echo  rm -Rf -- "$@"
    rm -Rf -- "$@"

>+}
>+
>+fi # _MISC_SH
>diff --git a/contrib/ci/run b/contrib/ci/run
>new file mode 100755
>index 0000000..a66f5a8
>--- /dev/null
>+++ b/contrib/ci/run
>@@ -0,0 +1,310 @@
>+#!/bin/bash
>+#
>+# Run continuous integration tests.
>+#
>+# Copyright (C) 2014 Red Hat
>+#
>+# This program is free software; you can redistribute it and/or modify
>+# it under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# This program is distributed in the hope that it will be useful,
>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>+
>+set -o nounset -o pipefail -o errexit
>+export PATH=`dirname "\`readlink -f \"\$0\"\`"`:$PATH
>+
>+. deps.sh
>+. configure.sh
>+. misc.sh
>+
>+declare -r DEBUG_CFLAGS="-O0 -g3"
In my opinion, sssd should be built without optimisation only for code
coverage. Some warnings are not visible without optimisation.

>+declare -r COVERAGE_CFLAGS="$DEBUG_CFLAGS --coverage"
>+declare -r CPU_NUM=`getconf _NPROCESSORS_ONLN`
>+declare -r TITLE_WIDTH=24
>+declare -r RESULT_WIDTH=18
>+declare BASE_PFX=""
>+declare BASE_DIR=`pwd`
>+declare FULL=false
>+
>+# Output program usage information.
>+function usage()
>+{
>+    cat <<EOF
>+Usage: `basename "$0"` [OPTION...]
>+Run continuous integration tests.
>+
>+Options:
>+    -h, --help          Output this help message and exit.
>+    -p, --prefix=STRING Use STRING as the prefix to prepend to file and
>+                        directory paths in output.
>+    -e, --essential     Run an essential subset of tests, completing faster.
>+    -f, --full          Run all tests.
>+
>+Default options: --essential
>+EOF
>+}
>+
>+# Output a file display path: a path relocated from base directory (BASE_DIR)
>+# to base prefix (BASE_PFX).
>+# Args: path
>+function disppath()
>+{
>+    declare -r path=`readlink -f "$1"`
>+    printf "%s" "$BASE_PFX${path:${#BASE_DIR}+1}"
>+}
>+
>+# Run a stage.
>+# Args: id cmd [arg...]
>+function stage()
>+{
>+    declare -r id="$1";     shift
>+    declare -r log="ci-$id.log"
>+    declare status
>+    declare start
>+    declare end
>+    declare duration
>+
>+    printf "%-${TITLE_WIDTH}s" "$id:"
>+
>+    {
>+        printf "Start: "
>+        start=`date +%s`
>+        date --date="@$start"
>+        set +o errexit
>+        (
>+            set -o errexit -o xtrace
>+            "$@"
>+        )
>+        status=$?
>+        set -o errexit
>+        printf "End: "
>+        end=`date +%s`
>+        date --date="@$end"
>+    } &> "$log"
>+
>+    duration=$((end - start))
>+
>+    if [ "$status" == 0 ]; then
>+        printf 'success  '
>+    else
>+        printf 'failure  '
>+    fi
>+    printf "%02u:%02u:%02u " \
>+           $((duration / (60 * 60))) \
>+           $((duration / 60 % 60)) \
>+           $((duration % 60))
>+    disppath "$log"
>+    printf "\n"
>+
>+    return "$status"
>+}
>+
>+# Run "scan-build" with output to a single specified directory, instead of
>+# a subdirectory.
>+# Args: dir [scan_build_arg...]
>+function scan_build_single()
>+{
>+    declare -r dir="$1";    shift
>+    declare entry
>+    declare subdir
>+    declare status
>+
>+    set +o errexit
>+    scan-build -o "$dir" "$@"
>+    status="$?"
>+    set -o errexit
>+
>+    for entry in "$dir/"*; do
>+        if [ -n "${subdir+set}" ] || ! [ -d "$entry" ]; then
>+            echo 'Unexpected entries in scan-build output directory' >&2
>+            exit 1
>+        fi
>+        subdir="$entry"
>+    done
>+
>+    mv "$subdir/"* "$dir"
>+    rmdir "$subdir"
>+    return "$status"
>+}
>+
>+# Check if a scan-build result directory has any non-empty .plist files.
>+# Args: dir
>+function scan_check()
>+{
>+    declare -r dir="$1"
>+    declare f
>+    for f in "$dir"/*.plist; do
>+        if [ "`xqilla -i \"\$f\" /dev/stdin \
            Could you use xmllint instead of xqilla.
We need xmllint for building manual pages. I didn't try it, but here is the 2nd
link from quick google search.
http://stackoverflow.com/questions/15461737/how-to-execute-xpath-one-liners-from-shell

>+                     <<<'count(/plist/dict/array[count(*) > 0])'`" != 0 ]; then
>+            return 1
>+        fi
>+    done
>+    return 0
>+}
>+
>+# Run debug build checks.
>+function build_debug()
>+{
>+    export CFLAGS="$DEBUG_CFLAGS"
>+    declare test_dir
>+    declare test_dir_distcheck
>+    declare status
>+
>+    test_dir=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
>+    stage configure         `configure_format \
>+                                "$BASE_DIR/configure" \
>+                                    --with-test-dir="$test_dir"`
>+
>+    if "$FULL"; then
>+        test_dir_distcheck=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
>+        # Single thread due to https://fedorahosted.org/sssd/ticket/2354
>+        status=0
>+        stage make-distcheck    make distcheck \
>+                                    DISTCHECK_CONFIGURE_FLAGS=" \
>+                                        --with-test-dir=$test_dir" ||
>+                status=$?
>+        mv "$test_dir_distcheck" ci-test-dir-distcheck
>+        ((status == 0))
>+    fi
>+
>+    # Not building "tests" due to https://fedorahosted.org/sssd/ticket/2350
>+    stage make-tests        make -j $CPU_NUM check LOG_COMPILER=true
>+
>+    # Ignored until issues found by Valgrind are fixed
We have already ticket for this, but it will not be so easy.
We need to find some compromise if we do not want to ignore valgrind issues
forever.
    https://fedorahosted.org/sssd/ticket/1052

>+    status=0
>+    CK_FORK=no \
>+        stage make-check-valgrind \
>+                make -j $CPU_NUM check \
>+                        LOG_COMPILER=libtool \
>+                        LOG_FLAGS='--mode=execute \
>+                            valgrind-condense 99 \
>+                                --vgdb=no \
>+                                --trace-children=yes \
>+                                --trace-children-skip=*/bin/*,*/sbin/* \
>+                                --leak-check=full' ||
>+            status=$?
>+    mv "$test_dir" ci-test-dir
>+
>+    unset CFLAGS
>+}
>+
>+# Run coverage build checks.
>+function build_coverage()
>+{
>+    declare -r scan_report_dir="ci-report-scan"
>+    declare -r coverage_report_dir="ci-report-coverage"
>+    declare test_dir
>+
>+    export CFLAGS="$COVERAGE_CFLAGS"
>+
>+    test_dir=`mktemp --directory /dev/shm/ci-test-dir.XXXXXXXX`
>+    stage configure         scan-build `configure_format \
>+                                       "$BASE_DIR/configure" \
>+                                           --with-test-dir="$test_dir"`
>+
>+    # Build everything, including tests
>+    # Not building "tests" due to https://fedorahosted.org/sssd/ticket/2350
>+    stage scan-make-tests   scan_build_single \
>+                                "$scan_report_dir" \
>+                                -plist-html \
>+                                --html-title="sssd - scan-build report" \
>+                                make -j $CPU_NUM check LOG_COMPILER=true
>+    printf "%-$((TITLE_WIDTH + RESULT_WIDTH))s%s\n" \
>+            "scan report:" \
>+            "`disppath \"\$scan_report_dir/index.html\"`"
>+    # Ignored until issues found by the scanner are fixed
clang static analyzer has some false positives. I do not think we will fix all
of them.

>+    stage scan-check        scan_check "$scan_report_dir" ||
>+                            true
>+
I would prefer to split this function into two parts.
    * running static analyser
    * code coverage.
The compilation with scan-build is slow and it is not necessary for code
coverage.

>+    stage lcov-pre          lcov --capture --initial --directory . \
>+                                 --output-file lcov_base.info
>+    # Run tests
>+    stage make-check        scan-build make -j $CPU_NUM check || true
>+    mv "$test_dir" ci-test-dir
>+
>+    stage lcov-post         lcov --capture --directory . \
>+                                 --output-file lcov_check.info
>+    stage lcov-merge        lcov --add-tracefile lcov_base.info \
>+                                 --add-tracefile lcov_check.info \
>+                                 --output-file lcov.info
Could you also remove output for external libraries and tests.
    lcov --remove libbash_test.info "/usr*" ...

LS



More information about the sssd-devel mailing list